126 changed files with 6682 additions and 18146 deletions
--- a/README.rst
+++ b/README.rst
@ -1,161 +1,162 @@
 piker
 -----
-trading gear for hackers
+trading gear for hackers.
 |gh_actions|
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
    :target: https://actions-badge.atrox.dev/piker/pikers/goto
-``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
+``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
-real-time computational trading targeted at `hardcore Linux users
+computational trading targeted at `hardcore Linux users <comp_trader>`_ .
 <comp_trader>`_ .
-we use much bleeding edge tech including (but not limited to):
+we use as much bleeding edge tech as possible including (but not limited to):
 - latest python for glue_
- uv_ for packaging and distribution
+- trio_ & tractor_ for our distributed, multi-core, real-time streaming
- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
+  `structured concurrency`_ runtime B)
- Qt_ for pristine low latency UIs
+- Qt_ for pristine high performance UIs
- pyqtgraph_ (which we've extended) for real-time charting and graphics
+- pyqtgraph_ for real-time charting
- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
+- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
- `apache arrow and parquet`_ for time-series storage
+- `apache arrow and parquet`_ for time series history management
  persistence and sharing
 - (prototyped) techtonicdb_ for L2 book storage
-potential projects we might integrate with soon,
+.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
-
+    :target: https://travis-ci.org/pikers/piker
 - (already prototyped in ) techtonicdb_ for L2 book storage
 .. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
 .. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _uv: https://docs.astral.sh/uv/
 .. _trio: https://github.com/python-trio/trio
 .. _tractor: https://github.com/goodboy/tractor
 .. _structured concurrency: https://trio.discourse.group/
 .. _marketstore: https://github.com/alpacahq/marketstore
 .. _techtonicdb: https://github.com/0b01/tectonicdb
 .. _Qt: https://www.qt.io/
 .. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
 .. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _apache arrow and parquet: https://arrow.apache.org/faq/
 .. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
-.. _techtonicdb: https://github.com/0b01/tectonicdb
+.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
-focus and feats:
+focus and features:
-****************
+*******************
-fitting with these tenets, we're always open to new
+- 100% federated: your code, your hardware, your data feeds, your broker fills.
-framework/lib/service interop suggestions and ideas!
+- zero web: low latency, native software that doesn't try to re-invent the OS
 - maximal **privacy**: prevent brokers and mms from knowing your
  planz; smack their spreads with dark volume.
 - zero clutter: modal, context oriented UIs that echew minimalism, reduce
  thought noise and encourage un-emotion.
 - first class parallelism: built from the ground up on next-gen structured concurrency
  primitives.
 - traders first: broker/exchange/asset-class agnostic
 - systems grounded: real-time financial signal processing that will
  make any queuing or DSP eng juice their shorts.
 - non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
 - data collaboration: every process and protocol is multi-host scalable.
 - fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
- **100% federated**:
+fitting with these tenets, we're always open to new framework suggestions and ideas.
  your code, your hardware, your data feeds, your broker fills.
- **zero web**:
+building the best looking, most reliable, keyboard friendly trading
-  low latency as a prime objective, native UIs and modern IPC
+platform is the dream; join the cause.
  protocols without trying to re-invent the "OS-as-an-app"..
 - **maximal privacy**:
  prevent brokers and mms from knowing your planz; smack their
  spreads with dark volume from a VPN tunnel.
 - **zero clutter**:
  modal, context oriented UIs that echew minimalism, reduce thought
  noise and encourage un-emotion.
 - **first class parallelism**:
  built from the ground up on a next-gen structured concurrency
  supervision sys.
 - **traders first**:
  broker/exchange/venue/asset-class/money-sys agnostic
 - **systems grounded**:
  real-time financial signal processing (fsp) that will make any
  queuing or DSP eng juice their shorts.
 - **non-tina UX**:
  sleek, powerful keyboard driven interaction with expected use in
  tiling wms (or maybe even a DDE).
 - **data collab at scale**:
  every actor-process and protocol is multi-host aware.
 - **fight club ready**:
  zero interest in adoption by suits; no corporate friendly license,
  ever.
 building the hottest looking, fastest, most reliable, keyboard
 friendly FOSS trading platform is the dream; join the cause.
-a sane install with `uv`
+sane install with `poetry`
-************************
+**************************
-bc why install with `python` when you can faster with `rust` ::
+TODO!
-    uv lock
+
 rigorous install on ``nixos`` using ``poetry2nix``
 **************************************************
 TODO!
 hacky install on nixos
 **********************
-``NixOS`` is our core devs' distro of choice for which we offer
+`NixOS` is our core devs' distro of choice for which we offer
 a stringently defined development shell envoirment that can be loaded with::
-    nix-shell default.nix
+    nix-shell develop.nix
 this will setup the required python environment to run piker, make sure to
 run::
    pip install -r requirements.txt -e .
 once after loading the shell
-start a chart
+install wild-west style via `pip`
-*************
+*********************************
-run a realtime OHLCV chart stand-alone::
+``piker`` is currently under heavy pre-alpha development and as such
 should be cloned from this repo and hacked on directly.
-    piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
+for a development install::
-this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
+    git clone git@github.com:pikers/piker.git
-overlayed on the same graph. Use of `piker` without first starting
+    cd piker
-a daemon (`pikerd` - see below) means there is an implicit spawning of the
+    virtualenv env
-multi-actor-runtime (implemented as a `tractor` app).
+    source ./env/bin/activate
-
+    pip install -r requirements.txt -e .
 For additional subsystem feats available through our chart UI see the
 various sub-readmes:
 - order control using a mouse-n-keyboard UX B)
 - cross venue market-pair (what most call "symbol") search, select, overlay Bo
 - financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
 - src-asset derivatives scan for anal, like the infamous "max pain" XO
-spawn a daemon standalone
+check out our charts
-*************************
+********************
-we call the root actor-process the ``pikerd``. it can be (and is
+bet you weren't expecting this from the foss::
-recommended normally to be) started separately from the ``piker
+
-chart`` program::
+    piker -l info -b kraken -b binance chart btcusdt.binance --pdb
 this runs the main chart (currently with 1m sampled OHLC) in in debug
 mode and you can practice paper trading using the following
 micro-manual:
 ``order_mode`` (
    edge triggered activation by any of the following keys,
    ``mouse-click`` on y-level to submit at that price
    ):
    - ``f``/ ``ctl-f`` to stage buy
    - ``d``/ ``ctl-d`` to stage sell
    - ``a`` to stage alert
 ``search_mode`` (
    ``ctl-l`` or ``ctl-space`` to open,
    ``ctl-c`` or ``ctl-space`` to close
    ) :
    - begin typing to have symbol search automatically lookup
      symbols from all loaded backend (broker) providers
    - arrow keys and mouse click to navigate selection
    - vi-like ``ctl-[hjkl]`` for navigation
 you can also configure your position allocation limits from the
 sidepane.
 run in distributed mode
 ***********************
 start the service manager and data feed daemon in the background and
 connect to it::
    pikerd -l info --pdb
 the daemon does nothing until a ``piker``-client (like ``piker
 chart``) connects and requests some particular sub-system. for
 a connecting chart ``pikerd`` will spawn and manage at least,
- a data-feed daemon: ``datad`` which does all the work of comms with
+connect your chart::
  the backend provider (in this case the ``binance`` cex).
 - a paper-trading engine instance, ``paperboi.binance``, (if no live
  account has been configured) which allows for auto/manual order
  control against the live quote stream.
-*using* an actor-service (aka micro-daemon) manager which dynamically
+    piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
 supervises various sub-subsystems-as-services throughout the ``piker``
 runtime-stack.
 now you can (implicitly) connect your chart::
-    piker chart btcusdt.spot.binance
+enjoy persistent real-time data feeds tied to daemon lifetime. the next
-
+time you spawn a chart it will load much faster since the data feed has
-since ``pikerd`` was started separately you can now enjoy a persistent
+been cached and is now always running live in the background until you
-real-time data stream tied to the daemon-tree's lifetime. i.e. the next
+kill ``pikerd``.
 time you spawn a chart it will obviously not only load much faster
 (since the underlying ``datad.binance`` is left running with its
 in-memory IPC data structures) but also the data-feed and any order
 mgmt states should be persistent until you finally cancel ``pikerd``.
 if anyone asks you what this project is about
 *********************************************
-you don't talk about it; just use it.
+you don't talk about it.
 how do i get involved?
@ -165,15 +166,6 @@ enter the matrix.
 how come there ain't that many docs
 ***********************************
-i mean we want/need them but building the core right has been higher
+suck it up, learn the code; no one is trying to sell you on anything.
-prio then marketting (and likely will stay that way Bp).
+also, we need lotsa help so if you want to start somewhere and can't
-
+necessarily write serious code, this might be the place for you!
 soo, suck it up bc,
 - no one is trying to sell you on anything
 - learning the code base is prolly way more valuable
 - the UI/UXs are intended to be "intuitive" for any hacker..
 we obviously need tonz help so if you want to start somewhere and
 can't necessarily write "advanced" concurrent python/rust code, this
 helping document literally anything might be the place for you!
--- a/config/brokers.toml
+++ b/config/brokers.toml
@ -2,15 +2,13 @@
 # ---- CEXY ----
 ################
 [binance]
 accounts.paper = 'paper'
 accounts.usdtm = 'futes'
-futes.use_testnet = false
+futes.use_testnet = true
 futes.api_key = ''
 futes.api_secret = ''
 accounts.spot = 'spot'
-spot.use_testnet = false
+spot.use_testnet = true
 spot.api_key = ''
 spot.api_secret = ''
--- a/default.nix
+++ b/default.nix
@ -1,134 +0,0 @@
 with (import <nixpkgs> {});
 let
  glibStorePath = lib.getLib glib;
  zlibStorePath = lib.getLib zlib;
  zstdStorePath = lib.getLib zstd;
  dbusStorePath = lib.getLib dbus;
  libGLStorePath = lib.getLib libGL;
  freetypeStorePath = lib.getLib freetype;
  qt6baseStorePath = lib.getLib qt6.qtbase;
  fontconfigStorePath = lib.getLib fontconfig;
  libxkbcommonStorePath = lib.getLib libxkbcommon;
  xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
  qtpyStorePath = lib.getLib python312Packages.qtpy;
  pyqt6StorePath = lib.getLib python312Packages.pyqt6;
  pyqt6SipStorePath = lib.getLib python312Packages.pyqt6-sip;
  rapidfuzzStorePath = lib.getLib python312Packages.rapidfuzz;
  qdarkstyleStorePath = lib.getLib python312Packages.qdarkstyle;
  xorgLibX11StorePath = lib.getLib xorg.libX11;
  xorgLibxcbStorePath = lib.getLib xorg.libxcb;
  xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
  xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
  xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
  xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
  xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
 in
 stdenv.mkDerivation {
  name = "piker-qt6-uv";
  buildInputs = [
    # System requirements.
    glib
    zlib
    dbus
    zstd
    libGL
    freetype
    qt6.qtbase
    libgcc.lib
    fontconfig
    libxkbcommon
    # Xorg requirements
    xcb-util-cursor
    xorg.libxcb
    xorg.libX11
    xorg.xcbutilwm
    xorg.xcbutilimage
    xorg.xcbutilerrors
    xorg.xcbutilkeysyms
    xorg.xcbutilrenderutil
    # Python requirements.
    python312Full
    python312Packages.uv
    python312Packages.qdarkstyle
    python312Packages.rapidfuzz
    python312Packages.pyqt6
    python312Packages.qtpy
  ];
  src = null;
  shellHook = ''
    set -e
    # Set the Qt plugin path
    # export QT_DEBUG_PLUGINS=1
    QTBASE_PATH="${qt6baseStorePath}/lib"
    QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
    QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
    LIB_GCC_PATH="${libgcc.lib}/lib"
    GLIB_PATH="${glibStorePath}/lib"
    ZSTD_PATH="${zstdStorePath}/lib"
    ZLIB_PATH="${zlibStorePath}/lib"
    DBUS_PATH="${dbusStorePath}/lib"
    LIBGL_PATH="${libGLStorePath}/lib"
    FREETYPE_PATH="${freetypeStorePath}/lib"
    FONTCONFIG_PATH="${fontconfigStorePath}/lib"
    LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
    XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
    XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
    XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
    XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
    XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
    XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
    XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
    XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
    export LD_LIBRARY_PATH
    RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.12/site-packages"
    QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.12/site-packages"
    QTPY_PATH="${qtpyStorePath}/lib/python3.12/site-packages"
    PYQT6_PATH="${pyqt6StorePath}/lib/python3.12/site-packages"
    PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.12/site-packages"
    PATCH="$PATCH:$RPDFUZZ_PATH"
    PATCH="$PATCH:$QDRKSTYLE_PATH"
    PATCH="$PATCH:$QTPY_PATH"
    PATCH="$PATCH:$PYQT6_PATH"
    PATCH="$PATCH:$PYQT6_SIP_PATH"
    export PATCH
    # Install deps
    uv lock
  '';
 }
--- a/develop.nix
+++ b/develop.nix
@ -1,34 +1,28 @@
 with (import <nixpkgs> {});
-
+with python310Packages;
 stdenv.mkDerivation {
-  name = "poetry-env";
+  name = "pip-env";
  buildInputs = [
    # System requirements.
    readline
    # TODO: hacky non-poetry install stuff we need to get rid of!!
-    poetry
+    virtualenv
-    # virtualenv
+    setuptools
-    # setuptools
+    pip
    # pip
    # Python requirements (enough to get a virtualenv going).
    python311Full
    # obviously, and see below for hacked linking
-    python311Packages.pyqt5
+    pyqt5
-    python311Packages.pyqt5_sip
+
-    # python311Packages.qtpy
+    # Python requirements (enough to get a virtualenv going).
    python310Full
    # numerics deps
-    python311Packages.levenshtein
+    python310Packages.python-Levenshtein
-    python311Packages.fastparquet
+    python310Packages.fastparquet
-    python311Packages.polars
+    python310Packages.polars
  ];
  # environment.sessionVariables = {
  #   LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
  # };
  src = null;
  shellHook = ''
    # Allow the use of wheels.
@ -36,12 +30,13 @@ stdenv.mkDerivation {
    # Augment the dynamic linker path
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
    export QT_QPA_PLATFORM_PLUGIN_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.version}/plugins";
-    if [ ! -d ".venv" ]; then
+    if [ ! -d "venv" ]; then
-        poetry install --with uis
+        virtualenv venv
    fi
-    poetry shell
+    source venv/bin/activate
  '';
 }
--- a/dockering/ib/docker-compose.yml
+++ b/dockering/ib/docker-compose.yml
@ -19,9 +19,8 @@ services:
    # other image tags available:
    # https://github.com/waytrade/ib-gateway-docker#supported-tags
-    # image: waytrade/ib-gateway:1012.2i
+    # image: waytrade/ib-gateway:981.3j
-    image: ghcr.io/gnzsnz/ib-gateway:latest
+    image: waytrade/ib-gateway:1012.2i
    restart: 'no'  # restart on boot whenev there's a crash or user clicsk
    network_mode: 'host'
--- a/dockering/ib/ibc.ini
+++ b/dockering/ib/ibc.ini
@ -117,57 +117,9 @@ SecondFactorDevice=
 # If you use the IBKR Mobile app for second factor authentication,
 # and you fail to complete the process before the time limit imposed
-# by IBKR, this setting tells IBC whether to automatically restart
+# by IBKR, you can use this setting to tell IBC to exit: arrangements
-# the login sequence, giving you another opportunity to complete
+# can then be made to automatically restart IBC in order to initiate
-# second factor authentication. 
+# the login sequence afresh. Otherwise, manual intervention at TWS's
 #
 # Permitted values are 'yes' and 'no'.
 #
 # If this setting is not present or has no value, then the value
 # of the deprecated ExitAfterSecondFactorAuthenticationTimeout is
 # used instead. If this also has no value, then this setting defaults
 # to 'no'.
 #
 # NB: you must be using IBC v3.14.0 or later to use this setting:
 # earlier versions ignore it.
 ReloginAfterSecondFactorAuthenticationTimeout=
 # This setting is only relevant if
 # ReloginAfterSecondFactorAuthenticationTimeout is set to 'yes',
 # or if ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
 #
 # It controls how long (in seconds) IBC waits for login to complete
 # after the user acknowledges the second factor authentication
 # alert at the IBKR Mobile app. If login has not completed after
 # this time, IBC terminates.
 # The default value is 60.
 SecondFactorAuthenticationExitInterval=
 # This setting specifies the timeout for second factor authentication
 # imposed by IB. The value is in seconds. You should not change this
 # setting unless you have reason to believe that IB has changed the
 # timeout. The default value is 180.
 SecondFactorAuthenticationTimeout=180
 # DEPRECATED SETTING
 # ------------------
 #
 # ExitAfterSecondFactorAuthenticationTimeout - THIS SETTING WILL BE
 # REMOVED IN A FUTURE RELEASE. For IBC version 3.14.0 and later, see
 # the notes for ReloginAfterSecondFactorAuthenticationTimeout above.
 #
 # For IBC versions earlier than 3.14.0: If you use the IBKR Mobile
 # app for second factor authentication, and you fail to complete the
 # process before the time limit imposed by IBKR, you can use this
 # setting to tell IBC to exit: arrangements can then be made to
 # automatically restart IBC in order to initiate the login sequence
 # afresh. Otherwise, manual intervention at TWS's
 # Second Factor Authentication dialog is needed to complete the
 # login.
 #
@ -180,18 +132,29 @@ SecondFactorAuthenticationTimeout=180
 ExitAfterSecondFactorAuthenticationTimeout=no
 # This setting is only relevant if
 # ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
 #
 # It controls how long (in seconds) IBC waits for login to complete
 # after the user acknowledges the second factor authentication
 # alert at the IBKR Mobile app. If login has not completed after
 # this time, IBC terminates.
 # The default value is 40.
 SecondFactorAuthenticationExitInterval=
 # Trading Mode
 # ------------
 #
-# This indicates whether the live account or the paper trading
+# TWS 955 introduced a new Trading Mode combo box on its login
-# account corresponding to the supplied credentials is to be used.
+# dialog. This indicates whether the live account or the paper
-# The allowed values are 'live' (the default) and 'paper'.
+# trading account corresponding to the supplied credentials is
-#
+# to be used. The allowed values are 'live' (the default) and
-# If this is set to 'live', then the credentials for the live
+# 'paper'. For earlier versions of TWS this setting has no
-# account must be supplied. If it is set to 'paper', then either
+# effect.
 # the live or the paper-trading credentials may be supplied.
-TradingMode=paper
+TradingMode=
 # Paper-trading Account Warning
@ -225,7 +188,7 @@ AcceptNonBrokerageAccountWarning=yes
 #
 # The default value is 60.
-LoginDialogDisplayTimeout=60
+LoginDialogDisplayTimeout=20
@ -254,15 +217,7 @@ LoginDialogDisplayTimeout=60
 # but they are acceptable.
 #
 # The default is the current working directory when IBC is
-# started, unless the TWS_SETTINGS_PATH setting in the relevant
+# started.
 # start script is set.
 #
 # If both this setting and TWS_SETTINGS_PATH are set, then this
 # setting takes priority. Note that if they have different values,
 # auto-restart will not work.
 #
 # NB: this setting is now DEPRECATED. You should use the
 # TWS_SETTINGS_PATH setting in the relevant start script.
 IbDir=/root/Jts
@ -329,32 +284,15 @@ ExistingSessionDetectedAction=primary
 # Override TWS API Port Number
 # ----------------------------
 #
-# If OverrideTwsApiPort is set to an integer, IBC changes the 
+# If OverrideTwsApiPort is set to an integer, IBC changes the
-# 'Socket port' in TWS's API configuration to that number shortly 
+# 'Socket port' in TWS's API configuration to that number shortly
-# after startup (but note that for the FIX Gateway, this setting is
+# after startup. Leaving the setting blank will make no change to
-# actually stored in jts.ini rather than the Gateway's settings
+# the current setting. This setting is only intended for use in
-# file). Leaving the setting blank will make no change to 
+# certain specialized situations where the port number needs to
 # the current setting. This setting is only intended for use in 
 # certain specialized situations where the port number needs to 
 # be set dynamically at run-time, and for the FIX Gateway: most
 # non-FIX users will never need it, so don't use it unless you know
 # you need it.
 OverrideTwsApiPort=4000
 # Override TWS Master Client ID
 # -----------------------------
 #
 # If OverrideTwsMasterClientID is set to an integer, IBC changes the
 # 'Master Client ID' value in TWS's API configuration to that 
 # value shortly after startup. Leaving the setting blank will make
 # no change to the current setting. This setting is only intended 
 # for use in certain specialized situations where the value needs to
 # be set dynamically at run-time: most users will never need it,
 # so don't use it unless you know you need it.
-OverrideTwsMasterClientID=
+; OverrideTwsApiPort=4002
 # Read-only Login
@ -364,13 +302,11 @@ OverrideTwsMasterClientID=
 # account security programme, the user will not be asked to perform
 # the second factor authentication action, and login to TWS will
 # occur automatically in read-only mode: in this mode, placing or
-# managing orders is not allowed. 
+# managing orders is not allowed. If set to 'no', and the user is
-#
+# enrolled in IB's account security programme, the user must perform
-# If set to 'no', and the user is enrolled in IB's account security
+# the relevant second factor authentication action to complete the
-# programme, the second factor authentication process is handled
+# login. 
-# according to the Second Factor Authentication Settings described
+
 # elsewhere in this file.
 #
 # If the user is not enrolled in IB's account security programme,
 # this setting is ignored. The default is 'no'.
@ -390,44 +326,7 @@ ReadOnlyLogin=no
 # set the relevant checkbox (this only needs to be done once) and
 # not provide a value for this setting.
-ReadOnlyApi=
+ReadOnlyApi=no
 # API Precautions
 # ---------------
 # 
 # These settings relate to the corresponding 'Precautions' checkboxes in the
 # API section of the Global Configuration dialog.
 #
 # For all of these, the accepted values are:
 # - 'yes' sets the checkbox
 # - 'no' clears the checkbox
 # - if not set, the existing TWS/Gateway configuration is unchanged
 #
 # NB: thess settings are really only supplied for the benefit of new TWS
 # or Gateway instances that are being automatically installed and
 # started without user intervention, or where user settings are not preserved
 # between sessions (eg some Docker containers). Where a user is involved, they
 # should use the Global Configuration to set the relevant checkboxes and not
 # provide values for these settings.
 BypassOrderPrecautions=
 BypassBondWarning=
 BypassNegativeYieldToWorstConfirmation=
 BypassCalledBondWarning=
 BypassSameActionPairTradeWarning=
 BypassPriceBasedVolatilityRiskWarning=
 BypassUSStocksMarketDataInSharesWarning=
 BypassRedirectOrderWarning=
 BypassNoOverfillProtectionPrecaution=
 # Market data size for US stocks - lots or shares
@ -482,145 +381,54 @@ AcceptBidAskLastSizeDisplayUpdateNotification=accept
 SendMarketDataInLotsForUSstocks=
 # Trusted API Client IPs
 # ----------------------
 #
 # NB: THIS SETTING IS ONLY RELEVANT FOR THE GATEWAY, AND ONLY WHEN FIX=yes.
 # In all other cases it is ignored.
 #
 # This is a list of IP addresses separated by commas. API clients with IP
 # addresses in this list are able to connect to the API without Gateway
 # generating the 'Incoming connection' popup.
 #
 # Note that 127.0.0.1 is always permitted to connect, so do not include it
 # in this setting.
 TrustedTwsApiClientIPs=
 # Reset Order ID Sequence
 # -----------------------
 #
 # The setting resets the order id sequence for orders submitted via the API, so
 # that the next invocation of the `NextValidId` API callback will return the
 # value 1. The reset occurs when TWS starts.
 #
 # Note that order ids are reset for all API clients, except those that have
 # outstanding (ie incomplete) orders: their order id sequence carries on as
 # before.
 #
 # Valid values are 'yes', 'true', 'false' and 'no'. The default is 'no'.
 ResetOrderIdsAtStart=
 # This setting specifies IBC's action when TWS displays the dialog asking for
 # confirmation of a request to reset the API order id sequence.
 #
 # Note that the Gateway never displays this dialog, so this setting is ignored
 # for a Gateway session.
 #
 # Valid values consist of two strings separated by a solidus '/'. The first
 # value specifies the action to take when the order id reset request resulted
 # from setting ResetOrderIdsAtStart=yes. The second specifies the action to
 # take when the order id reset request is a result of the user clicking the
 # 'Reset API order ID sequence' button in the API configuration. Each value
 # must be one of the following:
 #
 #    'confirm' 
 #        order ids will be reset
 #
 #    'reject' 
 #        order ids will not be reset
 #
 #    'ignore' 
 #        IBC will ignore the dialog. The user must take action.
 #
 #    The default setting is ignore/ignore
 # Examples:
 #
 #    'confirm/reject' - confirm order id reset only if ResetOrderIdsAtStart=yes
 #                       and reject any user-initiated requests
 #
 #    'ignore/confirm' - user must decide what to do if ResetOrderIdsAtStart=yes
 #                       and confirm user-initiated requests
 #
 #    'reject/ignore'  - reject order id reset if  ResetOrderIdsAtStart=yes but
 #                       allow user to handle user-initiated requests 
 ConfirmOrderIdReset=
 # =============================================================================
-# 4.   TWS Auto-Logoff and Auto-Restart
+# 4.   TWS Auto-Closedown
 # =============================================================================
 #
-# TWS and Gateway insist on being restarted every day. Two alternative
+# IMPORTANT NOTE: Starting with TWS 974, this setting no longer
-# automatic options are offered: 
+# works properly, because IB have changed the way TWS handles its
 # autologoff mechanism. 
 #
-#    - Auto-Logoff: at a specified time, TWS shuts down tidily, without
+# You should now configure the TWS autologoff time to something
-#      restarting.
+# convenient for you, and restart IBC each day.
 #
-#    - Auto-Restart: at a specified time, TWS shuts down and then restarts
+# Alternatively, discontinue use of IBC and use the auto-relogin
-#      without the user having to re-autheticate.
+# mechanism within TWS 974 and later versions (note that the 
-#
+# auto-relogin mechanism provided by IB is not available if you
-# The normal way to configure the time at which this happens is via the Lock
+# use IBC).
 # and Exit section of the Configuration dialog. Once this time has been
 # configured in this way, the setting persists until the user changes it again.
 #
 # However, there are situations where there is no user available to do this
 # configuration, or where there is no persistent storage (for example some
 # Docker images). In such cases, the auto-restart or auto-logoff time can be
 # set whenever IBC starts with the settings below.
 #
 # The value, if specified, must be a time in HH:MM AM/PM format, for example
 # 08:00 AM or 10:00 PM. Note that there must be a single space between the
 # two parts of this value; also that midnight is "12:00 AM" and midday is
 # "12:00 PM".
 #
 # If no value is specified for either setting, the currently configured
 # settings will apply. If a value is supplied for one setting, the other
 # setting is cleared. If values are supplied for both settings, only the
 # auto-restart time is set, and the auto-logoff time is cleared.
 #
 # Note that for a normal TWS/Gateway installation with persistent storage
 # (for example on a desktop computer) the value will be persisted as if the
 # user had set it via the configuration dialog.
 #
 # If you choose to auto-restart, you should take note of the considerations
 # described at the link below. Note that where this information mentions
 # 'manual authentication', restarting IBC will do the job (IBKR does not
 # recognise the existence of IBC in its docuemntation).
 #
 #  https://www.interactivebrokers.com/en/software/tws/twsguide.htm#usersguidebook/configuretws/auto_restart_info.htm
 #
 # If you use the "RESTART" command via the IBC command server, and IBC is
 # running any version of the Gateway (or a version of TWS earlier than 1018),
 # note that this will set the Auto-Restart time in Gateway/TWS's configuration
 # dialog to the time at which the restart actually happens (which may be up to
 # a minute after the RESTART command is issued). To prevent future auto-
 # restarts at this time, you must make sure you have set AutoLogoffTime or
 # AutoRestartTime to your desired value before running IBC. NB: this does not
 # apply to TWS from version 1018 onwards.
-AutoLogoffTime=
+# Set to yes or no (lower case).
 #
 #   yes   means allow TWS to shut down automatically at its
 # 	  specified shutdown time, which is set via the TWS
 #	  configuration menu.
 #
 #   no    means TWS never shuts down automatically.
 #
 # NB: IB recommends that you do not keep TWS running
 # continuously. If you set this setting to 'no', you may
 # experience incorrect TWS operation.
 #
 # NB: the default for this setting is 'no'. Since this will
 # only work properly with TWS versions earlier than 974, you
 # should explicitly set this to 'yes' for version 974 and later.
 IbAutoClosedown=yes
 AutoRestartTime=
 # =============================================================================
 # 5.   TWS Tidy Closedown Time
 # =============================================================================
 #
-# Specifies a time at which TWS will close down tidily, with no restart.
+# NB: starting with TWS 974 this is no longer a useful option
 # because both TWS and Gateway now have the same auto-logoff
 # mechanism, and IBC can no longer avoid this.
 #
-# There is little reason to use this setting. It is similar to AutoLogoffTime,
+# Note that giving this setting a value does not change TWS's
-# but can include a day-of-the-week, whereas AutoLogoffTime and AutoRestartTime
+# auto-logoff in any way: any setting will be additional to the
-# apply every day. So for example you could use ClosedownAt in conjunction with
+# TWS auto-logoff.
 # AutoRestartTime to shut down TWS on Friday evenings after the markets
 # close, without it running on Saturday as well.
 #
 # To tell IBC to tidily close TWS at a specified time every
 # day, set this value to <hh:mm>, for example:
@ -679,7 +487,7 @@ AcceptIncomingConnectionAction=reject
 #   no    means the dialog remains on display and must be
 #         handled by the user.
-AllowBlindTrading=no
+AllowBlindTrading=yes
 # Save Settings on a Schedule
@ -722,26 +530,6 @@ AllowBlindTrading=no
 SaveTwsSettingsAt=
 # Confirm Crypto Currency Orders Automatically
 # --------------------------------------------
 #
 # When you place an order for a cryptocurrency contract, a dialog is displayed
 # asking you to confirm that you want to place the order, and notifying you
 # that you are placing an order to trade cryptocurrency with Paxos, a New York
 # limited trust company, and not at Interactive Brokers.
 #
 #   transmit    means that the order will be placed automatically, and the
 #               dialog will then be closed
 #
 #   cancel      means that the order will not be placed, and the dialog will
 #               then be closed
 #
 #   manual      means that IBC will take no action and the user must deal
 #               with the dialog
 ConfirmCryptoCurrencyOrders=transmit
 # =============================================================================
 # 7.   Settings Specific to Indian Versions of TWS
@ -778,17 +566,13 @@ DismissNSEComplianceNotice=yes
 #
 # The port number that IBC listens on for commands
 # such as "STOP". DO NOT set this to the port number
-# used for TWS API connections.
+# used for TWS API connections. There is no good reason
-#
+# to change this setting unless the port is used by
-# The convention is to use 7462 for this port,
+# some other application (typically another instance of
-# but it must be set to a different value from any other
+# IBC). The default value is 0, which tells IBC not to
-# IBC instance that might run at the same time.
+# start the command server
 #
 # The default value is 0, which tells IBC not to start
 # the command server
 #CommandServerPort=7462
 CommandServerPort=0
 # Permitted Command Sources
@ -799,19 +583,19 @@ CommandServerPort=0
 # IBC.  Commands can always be sent from the
 # same host as IBC is running on.
-ControlFrom=
+ControlFrom=127.0.0.1
 # Address for Receiving Commands
 # ------------------------------
 #
 # Specifies the IP address on which the Command Server
-# is to listen. For a multi-homed host, this can be used
+# is so listen. For a multi-homed host, this can be used
 # to specify that connection requests are only to be
 # accepted on the specified address. The default is to
 # accept connection requests on all local addresses.
-BindAddress=
+BindAddress=127.0.0.1
 # Command Prompt
@ -837,7 +621,7 @@ CommandPrompt=
 # information is sent. The default is that such information
 # is not sent.
-SuppressInfoMessages=yes
+SuppressInfoMessages=no
@ -867,10 +651,10 @@ SuppressInfoMessages=yes
 # The LogStructureScope setting indicates which windows are
 # eligible for structure logging:
 #
-#    - (default value) if set to 'known', only windows that
+#    - if set to 'known', only windows that IBC recognizes
-#      IBC recognizes are eligible - these are windows that
+#      are eligible - these are windows that IBC has some
-#      IBC has some interest in monitoring, usually to take
+#      interest in monitoring, usually to take some action
-#      some action on the user's behalf;
+#      on the user's behalf;
 #
 #    - if set to 'unknown', only windows that IBC does not
 #      recognize are eligible. Most windows displayed by
@ -883,8 +667,9 @@ SuppressInfoMessages=yes
 #    - if set to 'all', then every window displayed by TWS
 #      is eligible.
 #
 # The default value is 'known'.
-LogStructureScope=known
+LogStructureScope=all
 # When to Log Window Structure
@ -897,15 +682,13 @@ LogStructureScope=known
 #       structure of an eligible window the first time it
 #       is encountered;
 #
 #     - if set to 'openclose', the structure is logged every
 #       time an eligible window is opened or closed;
 #
 #    - if set to 'activate', the structure is logged every
 #      time an eligible window is made active;
 #
-#    - (default value) if set to 'never' or 'no' or 'false',
+#    - if set to 'never' or 'no' or 'false', structure
-#      structure information is never logged.
+#      information is never logged.
 #
 # The default value is 'never'.
 LogStructureWhen=never
@ -925,3 +708,4 @@ LogStructureWhen=never
 #LogComponents=
--- a/flake.lock
+++ b/flake.lock
@ -1,138 +0,0 @@
 {
  "nodes": {
    "flake-utils": {
      "inputs": {
        "systems": "systems"
      },
      "locked": {
        "lastModified": 1689068808,
        "narHash": "sha256-6ixXo3wt24N/melDWjq70UuHQLxGV8jZvooRanIHXw0=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "919d646de7be200f3bf08cb76ae1f09402b6f9b4",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "flake-utils_2": {
      "inputs": {
        "systems": "systems_2"
      },
      "locked": {
        "lastModified": 1689068808,
        "narHash": "sha256-6ixXo3wt24N/melDWjq70UuHQLxGV8jZvooRanIHXw0=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "919d646de7be200f3bf08cb76ae1f09402b6f9b4",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nix-github-actions": {
      "inputs": {
        "nixpkgs": [
          "poetry2nix",
          "nixpkgs"
        ]
      },
      "locked": {
        "lastModified": 1688870561,
        "narHash": "sha256-4UYkifnPEw1nAzqqPOTL2MvWtm3sNGw1UTYTalkTcGY=",
        "owner": "nix-community",
        "repo": "nix-github-actions",
        "rev": "165b1650b753316aa7f1787f3005a8d2da0f5301",
        "type": "github"
      },
      "original": {
        "owner": "nix-community",
        "repo": "nix-github-actions",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1692174805,
        "narHash": "sha256-xmNPFDi/AUMIxwgOH/IVom55Dks34u1g7sFKKebxUm0=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "caac0eb6bdcad0b32cb2522e03e4002c8975c62e",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
        "ref": "nixos-unstable",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "poetry2nix": {
      "inputs": {
        "flake-utils": "flake-utils_2",
        "nix-github-actions": "nix-github-actions",
        "nixpkgs": [
          "nixpkgs"
        ]
      },
      "locked": {
        "lastModified": 1692048894,
        "narHash": "sha256-cDw03rso2V4CDc3Mll0cHN+ztzysAvdI8pJ7ybbz714=",
        "ref": "refs/heads/pyqt6",
        "rev": "b059ad4c3051f45d6c912e17747aae37a9ec1544",
        "revCount": 2276,
        "type": "git",
        "url": "file:///home/lord_fomo/repos/poetry2nix"
      },
      "original": {
        "type": "git",
        "url": "file:///home/lord_fomo/repos/poetry2nix"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs",
        "poetry2nix": "poetry2nix"
      }
    },
    "systems": {
      "locked": {
        "lastModified": 1681028828,
        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
        "owner": "nix-systems",
        "repo": "default",
        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
        "type": "github"
      },
      "original": {
        "owner": "nix-systems",
        "repo": "default",
        "type": "github"
      }
    },
    "systems_2": {
      "locked": {
        "lastModified": 1681028828,
        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
        "owner": "nix-systems",
        "repo": "default",
        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
        "type": "github"
      },
      "original": {
        "owner": "nix-systems",
        "repo": "default",
        "type": "github"
      }
    }
  },
  "root": "root",
  "version": 7
 }
--- a/flake.nix
+++ b/flake.nix
@ -1,180 +0,0 @@
 # NOTE: to convert to a poetry2nix env like this here are the
 # steps:
 # - install poetry in your system nix config
 # - convert the repo to use poetry using `poetry init`:
 #   https://python-poetry.org/docs/basic-usage/#initialising-a-pre-existing-project
 # - then manually ensuring all deps are converted over:
 # - add this file to the repo and commit it
 # - 
 # GROKin tips:
 # - CLI eps are (ostensibly) added via an `entry_points.txt`:
 #   - https://packaging.python.org/en/latest/specifications/entry-points/#file-format
 #   - https://github.com/nix-community/poetry2nix/blob/master/editable.nix#L49
 {
  description = "piker: trading gear for hackers (pkged with poetry2nix)";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  # see https://github.com/nix-community/poetry2nix/tree/master#api
  inputs.poetry2nix = {
    # url = "github:nix-community/poetry2nix";
    # url = "github:K900/poetry2nix/qt5-explicit-deps";
    url = "/home/lord_fomo/repos/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };
  outputs = {
    self,
    nixpkgs,
    flake-utils,
    poetry2nix,
  }:
    # TODO: build cross-OS and use the `${system}` var thingy..
    flake-utils.lib.eachDefaultSystem (system:
      let
        # use PWD as sources
        projectDir = ./.;
        pyproject = ./pyproject.toml;
        poetrylock = ./poetry.lock;
        # TODO: port to 3.11 and support both versions?
        python = "python3.10";
        # for more functions and examples.
        # inherit
        # (poetry2nix.legacyPackages.${system})
        # mkPoetryApplication;
        # pkgs = nixpkgs.legacyPackages.${system};
        pkgs = nixpkgs.legacyPackages.x86_64-linux;
        lib = pkgs.lib;
        p2npkgs = poetry2nix.legacyPackages.x86_64-linux;
        # define all pkg overrides per dep, see edgecases.md:
        # https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md
        # TODO: add these into the json file:
        # https://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json
        pypkgs-build-requirements = {
          asyncvnc = [ "setuptools" ];
          eventkit = [ "setuptools" ];
          ib-insync = [ "setuptools" "flake8" ];
          msgspec = [ "setuptools"];
          pdbp = [ "setuptools" ];
          pyqt6-sip = [ "setuptools" ];
          tabcompleter = [ "setuptools" ];
          tractor = [ "setuptools" ];
          tricycle = [ "setuptools" ];
          trio-typing = [ "setuptools" ];
          trio-util = [ "setuptools" ];
          xonsh = [ "setuptools" ];
        };
        # auto-generate override entries
        p2n-overrides = p2npkgs.defaultPoetryOverrides.extend (self: super:
          builtins.mapAttrs (package: build-requirements:
            (builtins.getAttr package super).overridePythonAttrs (old: {
              buildInputs = (
                old.buildInputs or [ ]
              ) ++ (
                builtins.map (
                  pkg: if builtins.isString pkg then builtins.getAttr pkg super else pkg
                  ) build-requirements
              );
            })
          ) pypkgs-build-requirements
        );
        # override some ahead-of-time compiled extensions
        # to be built with their wheels.
        ahot_overrides = p2n-overrides.extend(
          final: prev: {
            # llvmlite = prev.llvmlite.override {
            #   preferWheel = false;
            # };
            # TODO: get this workin with p2n and nixpkgs..
            # pyqt6 = prev.pyqt6.override {
            #   preferWheel = true;
            # };
            # NOTE: this DOESN'T work atm but after a fix
            # to poetry2nix, it will and actually this line
            # won't be needed - thanks @k900:
            # https://github.com/nix-community/poetry2nix/pull/1257
            pyqt5 = prev.pyqt5.override {
              # withWebkit = false;
              preferWheel = true;
            };
            # see PR from @k900:
            # https://github.com/nix-community/poetry2nix/pull/1257
            # pyqt5-qt5 = prev.pyqt5-qt5.override {
            #   withWebkit = false;
            #   preferWheel = true;
            # };
            # TODO: patch in an override for polars to build
            # from src! See the details likely needed from
            # the cryptography entry:
            # https://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix#L426-L435
            polars = prev.polars.override {
              preferWheel = true;
            };
          }
      );
      # WHY!? -> output-attrs that `nix develop` scans for:
      # https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html#flake-output-attributes
      in
      rec {
        packages = {
          # piker = poetry2nix.legacyPackages.x86_64-linux.mkPoetryEditablePackage {
          #   editablePackageSources = { piker = ./piker; };
          piker = p2npkgs.mkPoetryApplication {
            projectDir = projectDir;
            # SEE ABOVE for auto-genned input set, override
            # buncha deps with extras.. like `setuptools` mostly.
            # TODO: maybe propose a patch to p2n to show that you
            # can even do this in the edgecases docs?
            overrides = ahot_overrides;
            # XXX: won't work on llvmlite..
            # preferWheels = true;
          };
        };
        # devShells.default = pkgs.mkShell {
        #   projectDir = projectDir;
        #   python = "python3.10";
        #   overrides = ahot_overrides;
        #   inputsFrom = [ self.packages.x86_64-linux.piker ];
        #   packages = packages;
        #   # packages = [ poetry2nix.packages.${system}.poetry ];
        # };
        # TODO: grok the difference here..
        # - avoid re-cloning git repos on every develop entry..
        # - ideally allow hacking on the src code of some deps
        #   (tractor, pyqtgraph, tomlkit, etc.) WITHOUT having to
        #   re-install them every time a change is made.
        # - boot a usable xonsh inside the poetry virtualenv when
        #   defined via a custom entry point?
        devShells.default = p2npkgs.mkPoetryEnv {
        # env = p2npkgs.mkPoetryEnv {
            projectDir = projectDir;
            python = pkgs.python310;
            overrides = ahot_overrides;
            editablePackageSources = packages;
              # piker = "./";
              # tractor = "../tractor/";
            # };  # wut?
        };
      }
    );  # end of .outputs scope
 }
--- a/piker/toolz/profile.py
+++ b/piker/toolz/profile.py
--- a/piker/accounting/README.rst
+++ b/piker/accounting/README.rst
@ -1,16 +0,0 @@
 .accounting
 -----------
 A subsystem for transaction processing, storage and historical
 measurement.
 .pnl
 ----
 BEP, the break even price: the price at which liquidating
 a remaining position results in a zero PnL since the position was
 "opened" in the destination asset.
 PPU: price-per-unit: the "average cost" (in cumulative mean terms)
 of the "entry" transactions which "make a position larger"; taking
 a profit relative to this price means that you will "make more
 profit then made prior" since the position was opened.
--- a/piker/accounting/init.py
+++ b/piker/accounting/init.py
@ -21,21 +21,17 @@ for tendiez.
 '''
 from ..log import get_logger
 from .calc import (
    iter_by_dt,
 )
 from ._ledger import (
    iter_by_dt,
    Transaction,
    TransactionLedger,
    open_trade_ledger,
 )
 from ._pos import (
-    Account,
+    load_pps_from_ledger,
    load_account,
    load_account_from_ledger,
    open_pps,
    open_account,
    Position,
    PpTable,
 )
 from ._mktinfo import (
    Asset,
@ -51,25 +47,22 @@ from ._allocate import (
    Allocator,
 )
 log = get_logger(__name__)
 __all__ = [
    'Account',
    'Allocator',
    'Asset',
    'MktPair',
    'Position',
    'PpTable',
    'Symbol',
    'Transaction',
    'TransactionLedger',
    'dec_digits',
    'digits_to_dec',
    'iter_by_dt',
-    'load_account',
+    'load_pps_from_ledger',
    'load_account_from_ledger',
    'mk_allocator',
    'open_account',
    'open_pps',
    'open_trade_ledger',
    'unpack_fqme',
@ -89,7 +82,7 @@ def get_likely_pair(
    '''
    try:
-        src_name_start: str = bs_mktid.rindex(src)
+        src_name_start = bs_mktid.rindex(src)
    except (
        ValueError,   # substr not found
    ):
@ -100,8 +93,25 @@ def get_likely_pair(
        # log.warning(
        #     f'No src fiat {src} found in {bs_mktid}?'
        # )
-        return None
+        return
-    likely_dst: str = bs_mktid[:src_name_start]
+    likely_dst = bs_mktid[:src_name_start]
    if likely_dst == dst:
        return bs_mktid
 if __name__ == '__main__':
    import sys
    from pprint import pformat
    args = sys.argv
    assert len(args) > 1, 'Specifiy account(s) from `brokers.toml`'
    args = args[1:]
    for acctid in args:
        broker, name = acctid.split('.')
        trans, updated_pps = load_pps_from_ledger(broker, name)
        print(
            f'Processing transactions into pps for {broker}:{acctid}\n'
            f'{pformat(trans)}\n\n'
            f'{pformat(updated_pps)}'
        )
--- a/piker/accounting/_allocate.py
+++ b/piker/accounting/_allocate.py
@ -25,7 +25,7 @@ from bidict import bidict
 from ._pos import Position
 from . import MktPair
-from piker.types import Struct
+from ..data.types import Struct
 _size_units = bidict({
@ -118,9 +118,9 @@ class Allocator(Struct):
        ld: int = mkt.size_tick_digits
        size_unit = self.size_unit
-        live_size = live_pp.cumsize
+        live_size = live_pp.size
        abs_live_size = abs(live_size)
-        abs_startup_size = abs(startup_pp.cumsize)
+        abs_startup_size = abs(startup_pp.size)
        u_per_slot, currency_per_slot = self.step_sizes()
@ -213,6 +213,8 @@ class Allocator(Struct):
            slots_used = self.slots_used(
                Position(
                    mkt=mkt,
                    size=order_size,
                    ppu=price,
                    bs_mktid=mkt.bs_mktid,
                )
            )
@ -239,7 +241,7 @@ class Allocator(Struct):
        Calc and return the number of slots used by this ``Position``.
        '''
-        abs_pp_size = abs(pp.cumsize)
+        abs_pp_size = abs(pp.size)
        if self.size_unit == 'currency':
            # live_currency_size = size or (abs_pp_size * pp.ppu)
--- a/piker/accounting/_ledger.py
+++ b/piker/accounting/_ledger.py
@ -21,77 +21,65 @@ Trade and transaction ledger processing.
 from __future__ import annotations
 from collections import UserDict
 from contextlib import contextmanager as cm
 from functools import partial
 from pathlib import Path
 from pprint import pformat
 from types import ModuleType
 from typing import (
    Any,
    Callable,
-    Generator,
+    Iterator,
-    Literal,
+    Union,
-    TYPE_CHECKING,
+    Generator
 )
 from pendulum import (
    datetime,
    DateTime,
    from_timestamp,
    parse,
 )
 import tomli_w  # for fast ledger writing
-from piker.types import Struct
+from .. import config
-from piker import config
+from ..data.types import Struct
 from ..log import get_logger
-from .calc import (
+from ._mktinfo import (
-    iter_by_dt,
+    Symbol,  # legacy
    MktPair,
    Asset,
 )
 if TYPE_CHECKING:
    from ..data._symcache import (
        SymbologyCache,
    )
 log = get_logger(__name__)
 TxnType = Literal[
    'clear',
    'transfer',
    # TODO: see https://github.com/pikers/piker/issues/510
    # 'split',
    # 'rename',
    # 'resize',
    # 'removal',
 ]
 class Transaction(Struct, frozen=True):
-    # NOTE: this is a unified acronym also used in our `MktPair`
+    # TODO: unify this with the `MktPair`,
-    # and can stand for any of a
+    # once we have that as a required field,
-    # "fully qualified <blank> endpoint":
+    # we don't really need the fqme any more..
    # - "market" in the case of financial trades
    #   (btcusdt.spot.binance).
    # - "merkel (tree)" aka a blockchain system "wallet tranfers"
    #   (btc.blockchain)
    # - "money" for tradtitional (digital databases)
    #   *bank accounts* (usd.swift, eur.sepa)
    fqme: str
-    tid: str | int  # unique transaction id
+    tid: Union[str, int]  # unique transaction id
    size: float
    price: float
    cost: float  # commisions or other additional costs
-    dt: DateTime
+    dt: datetime
    # the "event type" in terms of "market events" see above and
    # https://github.com/pikers/piker/issues/510
    etype: TxnType = 'clear'
    # TODO: we can drop this right since we
    # can instead expect the backend to provide this
    # via the `MktPair`?
-    expiry: DateTime | None = None
+    expiry: datetime | None = None
    # TODO: drop the Symbol type, construct using
    # t.sys (the transaction system)
    # the underlying "transaction system", normally one of a ``MktPair``
    # (a description of a tradable double auction) or a ledger-recorded
    # ("ledger" in any sense as long as you can record transfers) of any
    # sort) ``Asset``.
    sym: MktPair | Asset | Symbol | None = None
    @property
    def sys(self) -> Symbol:
        return self.sym
    # (optional) key-id defined by the broker-service backend which
    # ensures the instrument-symbol market key for this record is unique
@ -100,16 +88,15 @@ class Transaction(Struct, frozen=True):
    # service.
    bs_mktid: str | int | None = None
-    def to_dict(
+    def to_dict(self) -> dict:
-        self,
+        dct = super().to_dict()
-        **kwargs,
+
-    ) -> dict:
+        # TODO: switch to sys!
-        dct: dict[str, Any] = super().to_dict(**kwargs)
+        dct.pop('sym')
        # ensure we use a pendulum formatted
        # ISO style str here!@
        dct['dt'] = str(self.dt)
        return dct
@ -121,45 +108,17 @@ class TransactionLedger(UserDict):
    outside.
    '''
    # NOTE: see `open_trade_ledger()` for defaults, this should
    # never be constructed manually!
    def __init__(
        self,
        ledger_dict: dict,
        file_path: Path,
        account: str,
        mod: ModuleType,  # broker mod
        tx_sort: Callable,
        symcache: SymbologyCache,
    ) -> None:
-        self.account: str = account
+        self.file_path = file_path
-        self.file_path: Path = file_path
+        self.tx_sort = tx_sort
        self.mod: ModuleType = mod
        self.tx_sort: Callable = tx_sort
        self._symcache: SymbologyCache = symcache
        # any added txns we keep in that form for meta-data
        # gathering purposes
        self._txns: dict[str, Transaction] = {}
        super().__init__(ledger_dict)
    def __repr__(self) -> str:
        return (
            f'TransactionLedger: {len(self)}\n'
            f'{pformat(list(self.data))}'
        )
    @property
    def symcache(self) -> SymbologyCache:
        '''
        Read-only ref to backend's ``SymbologyCache``.
        '''
        return self._symcache
    def update_from_t(
        self,
        t: Transaction,
@ -170,14 +129,14 @@ class TransactionLedger(UserDict):
        '''
        self.data[t.tid] = t.to_dict()
        self._txns[t.tid] = t
-    def iter_txns(
+    def iter_trans(
        self,
-        symcache: SymbologyCache | None = None,
+        mkt_by_fqme: dict[str, MktPair],
        broker: str = 'paper',
    ) -> Generator[
-        Transaction,
+        tuple[str, Transaction],
        None,
        None,
    ]:
@ -186,125 +145,129 @@ class TransactionLedger(UserDict):
        form via generator.
        '''
-        symcache = symcache or self._symcache
+        if broker != 'paper':
            raise NotImplementedError('Per broker support not dun yet!')
-        if self.account == 'paper':
+            # TODO: lookup some standard normalizer
-            from piker.clearing import _paper_engine
+            # func in the backend?
-            norm_trade: Callable = partial(
+            # from ..brokers import get_brokermod
-                _paper_engine.norm_trade,
+            # mod = get_brokermod(broker)
-                brokermod=self.mod,
+            # trans_dict = mod.norm_trade_records(self.data)
            # NOTE: instead i propose the normalizer is
            # a one shot routine (that can be lru cached)
            # and instead call it for each entry incrementally:
            # normer = mod.norm_trade_record(txdict)
        # TODO: use tx_sort here yah?
        for txdict in self.tx_sort(self.data.values()):
        # for tid, txdict in self.data.items():
            # special field handling for datetimes
            # to ensure pendulum is used!
            tid: str = txdict['tid']
            fqme: str = txdict.get('fqme') or txdict['fqsn']
            dt: DateTime = parse(txdict['dt'])
            expiry: str | None = txdict.get('expiry')
            if not (mkt := mkt_by_fqme.get(fqme)):
                # we can't build a trans if we don't have
                # the ``.sys: MktPair`` info, so skip.
                continue
            tx = Transaction(
                fqme=fqme,
                tid=txdict['tid'],
                dt=dt,
                price=txdict['price'],
                size=txdict['size'],
                cost=txdict.get('cost', 0),
                bs_mktid=txdict['bs_mktid'],
                # TODO: change to .sys!
                sym=mkt,
                expiry=parse(expiry) if expiry else None,
            )
            yield tid, tx
-        else:
+    def to_trans(
            norm_trade: Callable = self.mod.norm_trade
        # datetime-sort and pack into txs
        for tid, txdict in self.tx_sort(self.data.items()):
            txn: Transaction = norm_trade(
                tid,
                txdict,
                pairs=symcache.pairs,
                symcache=symcache,
            )
            yield txn
    def to_txns(
        self,
-        symcache: SymbologyCache | None = None,
+        **kwargs,
    ) -> dict[str, Transaction]:
        '''
-        Return entire output from ``.iter_txns()`` in a ``dict``.
+        Return entire output from ``.iter_trans()`` in a ``dict``.
        '''
-        txns: dict[str, Transaction] = {}
+        return dict(self.iter_trans(**kwargs))
        for t in self.iter_txns(symcache=symcache):
-            if not t:
+    def write_config(
-                log.warning(f'{self.mod.name}:{self.account} TXN is -> {t}')
+        self,
                continue
-            txns[t.tid] = t
+    ) -> None:
        return txns
    def write_config(self) -> None:
        '''
-        Render the self.data ledger dict to its TOML file form.
+        Render the self.data ledger dict to it's TOML file form.
        ALWAYS order datetime sorted!
        '''
-        is_paper: bool = self.account == 'paper'
+        cpy = self.data.copy()
        symcache: SymbologyCache = self._symcache
        towrite: dict[str, Any] = {}
-        for tid, txdict in self.tx_sort(self.data.copy()):
+        for tid, trans in cpy.items():
-            # write blank-str expiry for non-expiring assets
+
            # drop key for non-expiring assets
            txdict = towrite[tid] = self.data[tid]
            if (
                'expiry' in txdict
                and txdict['expiry'] is None
            ):
-                txdict['expiry'] = ''
+                txdict.pop('expiry')
-            # (maybe) re-write old acro-key
+            # re-write old acro-key
-            if (
+            fqme = txdict.get('fqsn')
-                is_paper
+            if fqme:
                # if symcache is empty/not supported (yet), don't
                # bother xD
                and symcache.mktmaps
            ):
                fqme: str = txdict.pop('fqsn', None) or txdict['fqme']
                bs_mktid: str | None = txdict.get('bs_mktid')
                if (
                    fqme not in symcache.mktmaps
                    or (
                        # also try to see if this is maybe a paper
                        # engine ledger in which case the bs_mktid
                        # should be the fqme as well!
                        bs_mktid
                        and fqme != bs_mktid
                    )
                ):
                    # always take any (paper) bs_mktid if defined and
                    # in the backend's cache key set.
                    if bs_mktid in symcache.mktmaps:
                        fqme: str = bs_mktid
                    else:
                        best_fqme: str = list(symcache.search(fqme))[0]
                        log.warning(
                            f'Could not find FQME: {fqme} in qualified set?\n'
                            f'Qualifying and expanding {fqme} -> {best_fqme}'
                        )
                        fqme = best_fqme
                if (
                    bs_mktid
                    and bs_mktid != fqme
                ):
                    # in paper account case always make sure both the
                    # fqme and bs_mktid are fully qualified..
                    txdict['bs_mktid'] = fqme
                # in paper ledgers always write the latest
                # symbology key field: an FQME.
                txdict['fqme'] = fqme
            towrite[tid] = txdict
        with self.file_path.open(mode='wb') as fp:
            tomli_w.dump(towrite, fp)
 def iter_by_dt(
    records: dict[str, dict[str, Any]] | list[dict],
    # NOTE: parsers are looked up in the insert order
    # so if you know that the record stats show some field
    # is more common then others, stick it at the top B)
    parsers: dict[tuple[str], Callable] = {
        'dt': None,  # parity case
        'datetime': parse,  # datetime-str
        'time': from_timestamp,  # float epoch
    },
    key: Callable | None = None,
 ) -> Iterator[tuple[str, dict]]:
    '''
    Iterate entries of a ``records: dict`` table sorted by entry recorded
    datetime presumably set at the ``'dt'`` field in each entry.
    '''
    def dyn_parse_to_dt(txdict: dict[str, Any]) -> DateTime:
        k, v, parser = next(
            (k, txdict[k], parsers[k]) for k in parsers if k in txdict
        )
        return parser(v) if parser else v
    if isinstance(records, dict):
        records = records.values()
    for entry in sorted(
        records,
        key=key or dyn_parse_to_dt,
    ):
        yield entry
 def load_ledger(
    brokername: str,
    acctid: str,
    # for testing or manual load from file
    dirpath: Path | None = None,
 ) -> tuple[dict, Path]:
    '''
    Load a ledger (TOML) file from user's config directory:
@ -319,11 +282,7 @@ def load_ledger(
    except ModuleNotFoundError:
        import tomli as tomllib
-    ldir: Path = (
+    ldir: Path = config._config_dir / 'accounting' / 'ledgers'
        dirpath
        or
        config._config_dir / 'accounting' / 'ledgers'
    )
    if not ldir.is_dir():
        ldir.mkdir()
@ -349,15 +308,8 @@ def open_trade_ledger(
    broker: str,
    account: str,
    allow_from_sync_code: bool = False,
    symcache: SymbologyCache | None = None,
    # default is to sort by detected datetime-ish field
    tx_sort: Callable = iter_by_dt,
    rewrite: bool = False,
    # for testing or manual load from file
    _fp: Path | None = None,
 ) -> Generator[TransactionLedger, None, None]:
    '''
@ -369,52 +321,18 @@ def open_trade_ledger(
    name as defined in the user's ``brokers.toml`` config.
    '''
-    from ..brokers import get_brokermod
+    ledger_dict, fpath = load_ledger(broker, account)
    mod: ModuleType = get_brokermod(broker)
    ledger_dict, fpath = load_ledger(
        broker,
        account,
        dirpath=_fp,
    )
    cpy = ledger_dict.copy()
    # XXX NOTE: if not provided presume we are being called from
    # sync code and need to maybe run `trio` to generate..
    if symcache is None:
        # XXX: be mega pendantic and ensure the caller knows what
        # they're doing!
        if not allow_from_sync_code:
            raise RuntimeError(
                'You MUST set `allow_from_sync_code=True` when '
                'calling `open_trade_ledger()` from sync code! '
                'If you are calling from async code you MUST '
                'instead pass a `symcache: SymbologyCache`!'
            )
        from ..data._symcache import (
            get_symcache,
        )
        symcache: SymbologyCache = get_symcache(broker)
    assert symcache
    ledger = TransactionLedger(
        ledger_dict=cpy,
        file_path=fpath,
-        account=account,
+        tx_sort=tx_sort,
        mod=mod,
        symcache=symcache,
        tx_sort=getattr(mod, 'tx_sort', tx_sort),
    )
    try:
        yield ledger
    finally:
-        if (
+        if ledger.data != ledger_dict:
-            ledger.data != ledger_dict
+
            or rewrite
        ):
            # TODO: show diff output?
            # https://stackoverflow.com/questions/12956957/print-diff-of-python-dictionaries
            log.info(f'Updating ledger for {fpath}:\n')
--- a/piker/accounting/_mktinfo.py
+++ b/piker/accounting/_mktinfo.py
@ -36,7 +36,7 @@ from typing import (
    Literal,
 )
-from piker.types import Struct
+from ..data.types import Struct
 # TODO: make these literals..
@ -130,29 +130,8 @@ class Asset(Struct, frozen=True):
    # should not be explicitly required in our generic API.
    info: dict | None = None
-    # `None` is not toml-compat so drop info
+    # TODO?
-    # if no extra data added..
+    # _to_dict_skip = {'info'}
    def to_dict(
        self,
        **kwargs,
    ) -> dict:
        dct = super().to_dict(**kwargs)
        if (info := dct.pop('info', None)):
            dct['info'] = info
        assert dct['tx_tick']
        return dct
    @classmethod
    def from_msg(
        cls,
        msg: dict[str, Any],
    ) -> Asset:
        return cls(
            tx_tick=Decimal(str(msg.pop('tx_tick'))),
            info=msg.pop('info', None),
            **msg,
        )
    def __str__(self) -> str:
        return self.name
@ -309,40 +288,12 @@ class MktPair(Struct, frozen=True):
    # strike price, call or put, swap type, exercise model, etc.
    contract_info: list[str] | None = None
    # TODO: rename to sectype since all of these can
    # be considered "securities"?
    _atype: str = ''
    # allow explicit disable of the src part of the market
    # pair name -> useful for legacy markets like qqq.nasdaq.ib
    _fqme_without_src: bool = False
    # NOTE: when cast to `str` return fqme
    def __str__(self) -> str:
        return self.fqme
    def to_dict(
        self,
        **kwargs,
    ) -> dict:
        d = super().to_dict(**kwargs)
        d['src'] = self.src.to_dict(**kwargs)
        if not isinstance(self.dst, str):
            d['dst'] = self.dst.to_dict(**kwargs)
        else:
            d['dst'] = str(self.dst)
        d['price_tick'] = str(self.price_tick)
        d['size_tick'] = str(self.size_tick)
        if self.contract_info is None:
            d.pop('contract_info')
        # d.pop('_fqme_without_src')
        return d
    @classmethod
    def from_msg(
        cls,
@ -353,32 +304,36 @@ class MktPair(Struct, frozen=True):
        Constructor for a received msg-dict normally received over IPC.
        '''
-        if not isinstance(
+        dst_asset_msg = msg.pop('dst')
-            dst_asset_msg := msg.pop('dst'),
+        src_asset_msg = msg.pop('src')
-            str,
+
-        ):
+        if isinstance(dst_asset_msg, str):
-            dst: Asset = Asset.from_msg(dst_asset_msg)  # .copy()
+            src: str = str(src_asset_msg)
            assert isinstance(src, str)
            return cls.from_fqme(
                dst_asset_msg,
                src=src,
                **msg,
            )
        else:
-            dst: str = dst_asset_msg
+            # NOTE: we call `.copy()` here to ensure
            # type casting!
            dst = Asset(**dst_asset_msg).copy()
            if not isinstance(src_asset_msg, str):
                src = Asset(**src_asset_msg).copy()
            else:
                src = str(src_asset_msg)
        src_asset_msg: dict = msg.pop('src')
        src: Asset = Asset.from_msg(src_asset_msg)  # .copy()
        # XXX NOTE: ``msgspec`` can encode `Decimal` but it doesn't
        # decide to it by default since we aren't spec-cing these
        # msgs as structs proper to get them to decode implictily
        # (yet) as per,
        # - https://github.com/pikers/piker/pull/354
        # - https://github.com/goodboy/tractor/pull/311
        # SO we have to ensure we do a struct type
        # case (which `.copy()` does) to ensure we get the right
        # type!
        return cls(
            dst=dst,
            src=src,
            price_tick=Decimal(msg.pop('price_tick')),
            size_tick=Decimal(msg.pop('size_tick')),
            **msg,
        # XXX NOTE: ``msgspec`` can encode `Decimal`
        # but it doesn't decide to it by default since
        # we aren't spec-cing these msgs as structs, SO
        # we have to ensure we do a struct type case (which `.copy()`
        # does) to ensure we get the right type!
        ).copy()
    @property
@ -406,20 +361,7 @@ class MktPair(Struct, frozen=True):
        ):
            _fqme = f'{fqme}.{broker}'
-        broker, mkt_ep_key, venue, expiry = unpack_fqme(_fqme)
+        broker, mkt_ep_key, venue, suffix = unpack_fqme(_fqme)
        kven: str = kwargs.pop('venue', venue)
        if venue:
            assert venue == kven
        else:
            venue = kven
        exp: str = kwargs.pop('expiry', expiry)
        if expiry:
            assert exp == expiry
        else:
            expiry = exp
        dst: Asset = Asset.guess_from_mkt_ep_key(
            mkt_ep_key,
            atype=kwargs.get('_atype'),
@ -431,15 +373,14 @@ class MktPair(Struct, frozen=True):
        # which we expect to be filled in by some
        # backend client with access to that data-info.
        return cls(
            dst=dst,
            # XXX: not resolved to ``Asset`` :(
-            #src=src,
+            dst=dst,
            broker=broker,
            venue=venue,
            # XXX NOTE: we presume this token
            # if the expiry for now!
-            expiry=expiry,
+            expiry=suffix,
            price_tick=price_tick,
            size_tick=size_tick,
@ -545,7 +486,7 @@ class MktPair(Struct, frozen=True):
        '''
        key: str = (
            self.pair(delim_char=delim_char)
-            if not (without_src or self._fqme_without_src)
+            if not without_src
            else str(self.dst)
        )
@ -614,7 +555,7 @@ class MktPair(Struct, frozen=True):
        if isinstance(self.dst, Asset):
            return str(self.dst.atype)
-        return 'UNKNOWN'
+        return 'unknown'
    @property
    def price_tick_digits(self) -> int:
--- a/piker/accounting/_pos.py
+++ b/piker/accounting/_pos.py
--- a/piker/accounting/calc.py
+++ b/piker/accounting/calc.py
@ -1,698 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Calculation routines for balance and position tracking such that
 you know when you're losing money (if possible) XD
 '''
 from __future__ import annotations
 from collections.abc import ValuesView
 from contextlib import contextmanager as cm
 from math import copysign
 from typing import (
    Any,
    Callable,
    Iterator,
    TYPE_CHECKING,
 )
 import polars as pl
 from pendulum import (
    DateTime,
    from_timestamp,
    parse,
 )
 if TYPE_CHECKING:
    from ._ledger import (
        Transaction,
        TransactionLedger,
    )
 def ppu(
    clears: Iterator[Transaction],
    # include transaction cost in breakeven price
    # and presume the worst case of the same cost
    # to exit this transaction (even though in reality
    # it will be dynamic based on exit stratetgy).
    cost_scalar: float = 2,
    # return the ledger of clears as a (now dt sorted) dict with
    # new position fields inserted alongside each entry.
    as_ledger: bool = False,
 ) -> float | list[(str, dict)]:
    '''
    Compute the "price-per-unit" price for the given non-zero sized
    rolling position.
    The recurrence relation which computes this (exponential) mean
    per new clear which **increases** the accumulative postiion size
    is:
    ppu[-1] = (
        ppu[-2] * accum_size[-2]
        +
        ppu[-1] * size
    ) / accum_size[-1]
    where `cost_basis` for the current step is simply the price
    * size of the most recent clearing transaction.
    -----
    TODO: get the BEP computed and working similarly!
    -----
    the equivalent "break even price" or bep at each new clear
    event step conversely only changes when an "position exiting
    clear" which **decreases** the cumulative dst asset size:
    bep[-1] = ppu[-1] - (cum_pnl[-1] / cumsize[-1])
    '''
    asize_h: list[float] = []  # historical accumulative size
    ppu_h: list[float] = []  # historical price-per-unit
    # ledger: dict[str, dict] = {}
    ledger: list[dict] = []
    t: Transaction
    for t in clears:
        clear_size: float = t.size
        clear_price: str | float = t.price
        is_clear: bool = not isinstance(clear_price, str)
        last_accum_size = asize_h[-1] if asize_h else 0
        accum_size: float = last_accum_size + clear_size
        accum_sign = copysign(1, accum_size)
        sign_change: bool = False
        # on transfers we normally write some non-valid
        # price since withdrawal to another account/wallet
        # has nothing to do with inter-asset-market prices.
        # TODO: this should be better handled via a `type: 'tx'`
        # field as per existing issue surrounding all this:
        # https://github.com/pikers/piker/issues/510
        if isinstance(clear_price, str):
            # TODO: we can't necessarily have this commit to
            # the overall pos size since we also need to
            # include other positions contributions to this
            # balance or we might end up with a -ve balance for
            # the position..
            continue
        # test if the pp somehow went "passed" a net zero size state
        # resulting in a change of the "sign" of the size (+ve for
        # long, -ve for short).
        sign_change = (
            copysign(1, last_accum_size) + accum_sign == 0
            and last_accum_size != 0
        )
        # since we passed the net-zero-size state the new size
        # after sum should be the remaining size the new
        # "direction" (aka, long vs. short) for this clear.
        if sign_change:
            clear_size: float = accum_size
            abs_diff: float = abs(accum_size)
            asize_h.append(0)
            ppu_h.append(0)
        else:
            # old size minus the new size gives us size diff with
            # +ve -> increase in pp size
            # -ve -> decrease in pp size
            abs_diff = abs(accum_size) - abs(last_accum_size)
        # XXX: LIFO breakeven price update. only an increaze in size
        # of the position contributes the breakeven price,
        # a decrease does not (i.e. the position is being made
        # smaller).
        # abs_clear_size = abs(clear_size)
        abs_new_size: float | int = abs(accum_size)
        if (
            abs_diff > 0
            and is_clear
        ):
            cost_basis = (
                # cost basis for this clear
                clear_price * abs(clear_size)
                +
                # transaction cost
                accum_sign * cost_scalar * t.cost
            )
            if asize_h:
                size_last: float = abs(asize_h[-1])
                cb_last: float = ppu_h[-1] * size_last
                ppu: float = (cost_basis + cb_last) / abs_new_size
            else:
                ppu: float = cost_basis / abs_new_size
        else:
            # TODO: for PPU we should probably handle txs out
            # (aka withdrawals) similarly by simply not having
            # them contrib to the running PPU calc and only
            # when the next entry clear comes in (which will
            # then have a higher weighting on the PPU).
            # on "exit" clears from a given direction,
            # only the size changes not the price-per-unit
            # need to be updated since the ppu remains constant
            # and gets weighted by the new size.
            ppu: float = ppu_h[-1] if ppu_h else 0  # set to previous value
        # extend with new rolling metric for this step
        ppu_h.append(ppu)
        asize_h.append(accum_size)
        # ledger[t.tid] = {
            # 'txn': t,
        # ledger[t.tid] = t.to_dict() | {
        ledger.append((
            t.tid,
            t.to_dict() | {
                'ppu': ppu,
                'cumsize': accum_size,
                'sign_change': sign_change,
                # TODO: cum_pnl, bep
            }
        ))
    final_ppu = ppu_h[-1] if ppu_h else 0
    # TODO: once we have etypes in all ledger entries..
    # handle any split info entered (for now) manually by user
    # if self.split_ratio is not None:
    #     final_ppu /= self.split_ratio
    if as_ledger:
        return ledger
    else:
        return final_ppu
 def iter_by_dt(
    records: (
        dict[str, dict[str, Any]]
        | ValuesView[dict]  # eg. `Position._events.values()`
        | list[dict]
        | list[Transaction]  # XXX preferred!
    ),
    # NOTE: parsers are looked up in the insert order
    # so if you know that the record stats show some field
    # is more common then others, stick it at the top B)
    parsers: dict[str, Callable | None] = {
        'dt': parse,  # parity case
        'datetime': parse,  # datetime-str
        'time': from_timestamp,  # float epoch
    },
    key: Callable | None = None,
 ) -> Iterator[tuple[str, dict]]:
    '''
    Iterate entries of a transaction table sorted by entry recorded
    datetime presumably set at the ``'dt'`` field in each entry.
    '''
    if isinstance(records, dict):
        records: list[tuple[str, dict]] = list(records.items())
    def dyn_parse_to_dt(
        tx: tuple[str, dict[str, Any]] | Transaction,
    ) -> DateTime:
        # handle `.items()` inputs
        if isinstance(tx, tuple):
            tx = tx[1]
        # dict or tx object?
        isdict: bool = isinstance(tx, dict)
        # get best parser for this record..
        for k in parsers:
            if (
                isdict and k in tx
                 or getattr(tx, k, None)
            ):
                v = tx[k] if isdict else tx.dt
                assert v is not None, f'No valid value for `{k}`!?'
                # only call parser on the value if not None from
                # the `parsers` table above (when NOT using
                # `.get()`), otherwise pass through the value and
                # sort on it directly
                if (
                    not isinstance(v, DateTime)
                    and (parser := parsers.get(k))
                ):
                    return parser(v)
                else:
                    return v
        else:
            # XXX: should never get here..
            breakpoint()
    entry: tuple[str, dict] | Transaction
    for entry in sorted(
        records,
        key=key or dyn_parse_to_dt,
    ):
        # NOTE the type sig above; either pairs or txns B)
        yield entry
 # TODO: probably just move this into the test suite or
 # keep it here for use from as such?
 # def ensure_state(self) -> None:
 #     '''
 #     Audit either the `.cumsize` and `.ppu` local instance vars against
 #     the clears table calculations and return the calc-ed values if
 #     they differ and log warnings to console.
 #     '''
 #     # clears: list[dict] = self._clears
 #     # self.first_clear_dt = min(clears, key=lambda e: e['dt'])['dt']
 #     last_clear: dict = clears[-1]
 #     csize: float = self.calc_size()
 #     accum: float = last_clear['accum_size']
 #     if not self.expired():
 #         if (
 #             csize != accum
 #             and csize != round(accum * (self.split_ratio or 1))
 #         ):
 #             raise ValueError(f'Size mismatch: {csize}')
 #     else:
 #         assert csize == 0, 'Contract is expired but non-zero size?'
 #     if self.cumsize != csize:
 #         log.warning(
 #             'Position state mismatch:\n'
 #             f'{self.cumsize} => {csize}'
 #         )
 #         self.cumsize = csize
 #     cppu: float = self.calc_ppu()
 #     ppu: float = last_clear['ppu']
 #     if (
 #         cppu != ppu
 #         and self.split_ratio is not None
 #         # handle any split info entered (for now) manually by user
 #         and cppu != (ppu / self.split_ratio)
 #     ):
 #         raise ValueError(f'PPU mismatch: {cppu}')
 #     if self.ppu != cppu:
 #         log.warning(
 #             'Position state mismatch:\n'
 #             f'{self.ppu} => {cppu}'
 #         )
 #         self.ppu = cppu
@cm
 def open_ledger_dfs(
    brokername: str,
    acctname: str,
    ledger: TransactionLedger | None = None,
    **kwargs,
 ) -> tuple[
    dict[str, pl.DataFrame],
    TransactionLedger,
 ]:
    '''
    Open a ledger of trade records (presumably from some broker
    backend), normalize the records into `Transactions` via the
    backend's declared endpoint, cast to a `polars.DataFrame` which
    can update the ledger on exit.
    '''
    from piker.toolz import open_crash_handler
    with open_crash_handler():
        if not ledger:
            import time
            from ._ledger import open_trade_ledger
            now = time.time()
            with open_trade_ledger(
                    brokername,
                    acctname,
                    rewrite=True,
                    allow_from_sync_code=True,
                    # proxied through from caller
                    **kwargs,
            ) as ledger:
                if not ledger:
                    raise ValueError(f'No ledger for {acctname}@{brokername} exists?')
                print(f'LEDGER LOAD TIME: {time.time() - now}')
        yield ledger_to_dfs(ledger), ledger
 def ledger_to_dfs(
    ledger: TransactionLedger,
 ) -> dict[str, pl.DataFrame]:
    txns: dict[str, Transaction] = ledger.to_txns()
    # ldf = pl.DataFrame(
    #     list(txn.to_dict() for txn in txns.values()),
    ldf = pl.from_dicts(
        list(txn.to_dict() for txn in txns.values()),
        # only for ordering the cols
        schema=[
            ('fqme', str),
            ('tid', str),
            ('bs_mktid', str),
            ('expiry', str),
            ('etype', str),
            ('dt', str),
            ('size', pl.Float64),
            ('price', pl.Float64),
            ('cost', pl.Float64),
        ],
    ).sort(  # chronological order
        'dt'
    ).with_columns([
        pl.col('dt').str.to_datetime(),
        # pl.col('expiry').str.to_datetime(),
        # pl.col('expiry').dt.date(),
    ])
    # filter out to the columns matching values filter passed
    # as input.
    # if filter_by_ids:
    #     for col, vals in filter_by_ids.items():
    #         str_vals = set(map(str, vals))
    #         pred: pl.Expr = pl.col(col).eq(str_vals.pop())
    #         for val in str_vals:
    #             pred |= pl.col(col).eq(val)
    #     fdf = df.filter(pred)
    # TODO: originally i had tried just using a plain ol' groupby
    # + agg here but the issue was re-inserting to the src frame.
    # however, learning more about `polars` seems like maybe we can
    # use `.over()`?
    # https://pola-rs.github.io/polars/py-polars/html/reference/expressions/api/polars.Expr.over.html#polars.Expr.over
    # => CURRENTLY we break up into a frame per mkt / fqme
    dfs: dict[str, pl.DataFrame] = ldf.partition_by(
        'bs_mktid',
        as_dict=True,
    )
    # TODO: not sure if this is even possible but..
    # - it'd be more ideal to use `ppt = df.groupby('fqme').agg([`
    # - ppu and bep calcs!
    for key in dfs:
        # covert to lazy form (since apparently we might need it
        # eventually ...)
        df: pl.DataFrame = dfs[key]
        ldf: pl.LazyFrame = df.lazy()
        df = dfs[key] = ldf.with_columns([
            pl.cumsum('size').alias('cumsize'),
            # amount of source asset "sent" (via buy txns in
            # the market) to acquire the dst asset, PER txn.
            # when this value is -ve (i.e. a sell operation) then
            # the amount sent is actually "returned".
            (
                (pl.col('price') * pl.col('size'))
                +
                (pl.col('cost')) # * pl.col('size').sign())
            ).alias('dst_bot'),
        ]).with_columns([
            # rolling balance in src asset units
            (pl.col('dst_bot').cumsum() * -1).alias('src_balance'),
            # "position operation type" in terms of increasing the
            # amount in the dst asset (entering) or decreasing the
            # amount in the dst asset (exiting).
            pl.when(
                pl.col('size').sign() == pl.col('cumsize').sign()
            ).then(
                pl.lit('enter')  # see above, but is just price * size per txn
            ).otherwise(
                pl.when(pl.col('cumsize') == 0)
                .then(pl.lit('exit_to_zero'))
                .otherwise(pl.lit('exit'))
            ).alias('descr'),
            (pl.col('cumsize').sign() == pl.col('size').sign())
            .alias('is_enter'),
        ]).with_columns([
            # pl.lit(0, dtype=pl.Utf8).alias('virt_cost'),
            pl.lit(0, dtype=pl.Float64).alias('applied_cost'),
            pl.lit(0, dtype=pl.Float64).alias('pos_ppu'),
            pl.lit(0, dtype=pl.Float64).alias('per_txn_pnl'),
            pl.lit(0, dtype=pl.Float64).alias('cum_pos_pnl'),
            pl.lit(0, dtype=pl.Float64).alias('pos_bep'),
            pl.lit(0, dtype=pl.Float64).alias('cum_ledger_pnl'),
            pl.lit(None, dtype=pl.Float64).alias('ledger_bep'),
            # TODO: instead of the iterative loop below i guess we
            # could try using embedded lists to track which txns
            # are part of which ppu / bep calcs? Not sure this will
            # look any better nor be any more performant though xD
            # pl.lit([[0]], dtype=pl.List(pl.Float64)).alias('list'),
        # choose fields to emit for accounting puposes
        ]).select([
            pl.exclude([
                'tid',
                # 'dt',
                'expiry',
                'bs_mktid',
                'etype',
                # 'is_enter',
            ]),
        ]).collect()
        # compute recurrence relations for ppu and bep
        last_ppu: float = 0
        last_cumsize: float = 0
        last_ledger_pnl: float = 0
        last_pos_pnl: float = 0
        virt_costs: list[float, float] = [0., 0.]
        # imperatively compute the PPU (price per unit) and BEP
        # (break even price) iteratively over the ledger, oriented
        # around each position state: a state of split balances in
        # > 1 asset.
        for i, row in enumerate(df.iter_rows(named=True)):
            cumsize: float = row['cumsize']
            is_enter: bool = row['is_enter']
            price: float = row['price']
            size: float = row['size']
            # the profit is ALWAYS decreased, aka made a "loss"
            # by the constant fee charged by the txn provider!
            # see below in final PnL calculation and row element
            # set.
            txn_cost: float = row['cost']
            pnl: float = 0
            # ALWAYS reset per-position cum PnL
            if last_cumsize == 0:
                last_pos_pnl: float = 0
            # a "position size INCREASING" or ENTER transaction
            # which "makes larger", in src asset unit terms, the
            # trade's side-size of the destination asset:
            # - "buying" (more) units of the dst asset
            # - "selling" (more short) units of the dst asset
            if is_enter:
                # Naively include transaction cost in breakeven
                # price and presume the worst case of the
                # exact-same-cost-to-exit this transaction's worth
                # of size even though in reality it will be dynamic
                # based on exit strategy, price, liquidity, etc..
                virt_cost: float = txn_cost
                # cpu: float = cost / size
                # cummean of the cost-per-unit used for modelling
                # a projected future exit cost which we immediately
                # include in the costs incorporated to BEP on enters
                last_cum_costs_size, last_cpu = virt_costs
                cum_costs_size: float = last_cum_costs_size + abs(size)
                cumcpu = (
                    (last_cpu * last_cum_costs_size)
                    +
                    txn_cost
                ) / cum_costs_size
                virt_costs = [cum_costs_size, cumcpu]
                txn_cost = txn_cost + virt_cost
                # df[i, 'virt_cost'] = f'{-virt_cost} FROM {cumcpu}@{cum_costs_size}'
                # a cumulative mean of the price-per-unit acquired
                # in the destination asset:
                # https://en.wikipedia.org/wiki/Moving_average#Cumulative_average
                # You could also think of this measure more
                # generally as an exponential mean with `alpha
                # = 1/N` where `N` is the current number of txns
                # included in the "position" defining set:
                # https://en.wikipedia.org/wiki/Exponential_smoothing
                ppu: float = (
                    (
                        (last_ppu * last_cumsize)
                        +
                        (price * size)
                    ) /
                    cumsize
                )
            # a "position size DECREASING" or EXIT transaction
            # which "makes smaller" the trade's side-size of the
            # destination asset:
            # - selling previously bought units of the dst asset
            #   (aka 'closing' a long position).
            # - buying previously borrowed and sold (short) units
            #   of the dst asset (aka 'covering'/'closing' a short
            #   position).
            else:
                # only changes on position size increasing txns
                ppu: float = last_ppu
                # UNWIND IMPLIED COSTS FROM ENTRIES
                # => Reverse the virtual/modelled (2x predicted) txn
                # cost that was included in the least-recently
                # entered txn that is still part of the current CSi
                # set.
                # => we look up the cost-per-unit cumsum and apply
                # if over the current txn size (by multiplication)
                # and then reverse that previusly applied cost on
                # the txn_cost for this record.
                #
                # NOTE: current "model" is just to previously assumed 2x
                # the txn cost for a matching enter-txn's
                # cost-per-unit; we then immediately reverse this
                # prediction and apply the real cost received here.
                last_cum_costs_size, last_cpu = virt_costs
                prev_virt_cost: float = last_cpu * abs(size)
                txn_cost: float = txn_cost - prev_virt_cost  # +ve thus a "reversal"
                cum_costs_size: float = last_cum_costs_size - abs(size)
                virt_costs = [cum_costs_size, last_cpu]
                # df[i, 'virt_cost'] = (
                #     f'{-prev_virt_cost} FROM {last_cpu}@{cum_costs_size}'
                # )
                # the per-txn profit or loss (PnL) given we are
                # (partially) "closing"/"exiting" the position via
                # this txn.
                pnl: float = (last_ppu - price) * size
            # always subtract txn cost from total txn pnl
            txn_pnl: float = pnl - txn_cost
            # cumulative PnLs per txn
            last_ledger_pnl = (
                last_ledger_pnl + txn_pnl
            )
            last_pos_pnl = df[i, 'cum_pos_pnl'] = (
                last_pos_pnl + txn_pnl
            )
            if cumsize == 0:
                last_ppu = ppu = 0
            # compute the BEP: "break even price", a value that
            # determines at what price the remaining cumsize can be
            # liquidated such that the net-PnL on the current
            # position will result in ZERO gain or loss from open
            # to close including all txn costs B)
            if (
                abs(cumsize) > 0  # non-exit-to-zero position txn
            ):
                cumsize_sign: float = copysign(1, cumsize)
                ledger_bep: float = (
                    (
                        (ppu * cumsize)
                        -
                        (last_ledger_pnl * cumsize_sign)
                    ) / cumsize
                )
                # NOTE: when we "enter more" dst asset units (aka
                # increase position state) AFTER having exited some
                # units (aka decreasing the pos size some) the bep
                # needs to be RECOMPUTED based on new ppu such that
                # liquidation of the cumsize at the bep price
                # results in a zero-pnl for the existing position
                # (since the last one).
                # for position lifetime BEP we never can have
                # a valid value once the position is "closed"
                # / full exitted Bo
                pos_bep: float = (
                    (
                        (ppu * cumsize)
                        -
                        (last_pos_pnl * cumsize_sign)
                    ) / cumsize
                )
            # inject DF row with all values
            df[i, 'pos_ppu'] = ppu
            df[i, 'per_txn_pnl'] = txn_pnl
            df[i, 'applied_cost'] = -txn_cost
            df[i, 'cum_pos_pnl'] = last_pos_pnl
            df[i, 'pos_bep'] = pos_bep
            df[i, 'cum_ledger_pnl'] = last_ledger_pnl
            df[i, 'ledger_bep'] = ledger_bep
            # keep backrefs to suffice reccurence relation
            last_ppu: float = ppu
            last_cumsize: float = cumsize
    # TODO?: pass back the current `Position` object loaded from
    # the account as well? Would provide incentive to do all
    # this ledger loading inside a new async open_account().
    # bs_mktid: str = df[0]['bs_mktid']
    # pos: Position = acnt.pps[bs_mktid]
    return dfs
--- a/piker/accounting/cli.py
+++ b/piker/accounting/cli.py
@ -19,7 +19,6 @@ CLI front end for trades ledger and position tracking management.
 '''
 from __future__ import annotations
 from pprint import pformat
 from rich.console import Console
@ -38,11 +37,13 @@ from ..calc import humanize
 from ..brokers._daemon import broker_init
 from ._ledger import (
    load_ledger,
    TransactionLedger,
    # open_trade_ledger,
    # TransactionLedger,
 )
-from .calc import (
+from ._pos import (
-    open_ledger_dfs,
+    PpTable,
    load_pps_from_ledger,
    # load_account,
 )
@ -239,73 +240,54 @@ def sync(
 def disect(
    # "fully_qualified_account_name"
    fqan: str,
-    fqme: str,  # for ib
+    bs_mktid: str,  # for ib
    # TODO: in tractor we should really have
    # a debug_mode ctx for wrapping any kind of code no?
    pdb: bool = False,
-    bs_mktid: str = typer.Option(
+
        None,
        "-bid",
    ),
    loglevel: str = typer.Option(
        'error',
        "-l",
    ),
 ):
    from piker.log import get_console_log
    from piker.toolz import open_crash_handler
    get_console_log(loglevel)
    pair: tuple[str, str]
    if not (pair := unpack_fqan(fqan)):
        raise ValueError('{fqan} malformed!?')
    brokername, account = pair
-    # ledger dfs groupby-partitioned by fqme
+    # ledger: TransactionLedger
-    dfs: dict[str, pl.DataFrame]
+    records: dict[str, dict]
-    # actual ledger instance
+    table: PpTable
-    ldgr: TransactionLedger
+    records, table = load_pps_from_ledger(
        brokername,
        account,
        filter_by_ids={bs_mktid},
    )
    df = pl.DataFrame(
        list(records.values()),
        # schema=[
        #     ('tid', str),
        #     ('fqme', str),
        #     ('dt', str),
        #     ('size', pl.Float64),
        #     ('price', pl.Float64),
        #     ('cost', pl.Float64),
        #     ('expiry', str),
        #     ('bs_mktid', str),
        # ],
    ).select([
        pl.col('fqme'),
        pl.col('dt').str.to_datetime(),
        # pl.col('expiry').dt.datetime(),
        pl.col('size'),
        pl.col('price'),
    ])
-    pl.Config.set_tbl_cols(-1)
+    assert not df.is_empty()
-    pl.Config.set_tbl_rows(-1)
+    breakpoint()
-    with (
+    # tractor.pause_from_sync()
-        open_crash_handler(),
+    # with open_trade_ledger(
-        open_ledger_dfs(
+    #     brokername,
-            brokername,
+    #     account,
-            account,
+    # ) as ledger:
-        ) as (dfs, ldgr),
+    #     for tid, rec in ledger.items():
-    ):
+    #         bs_mktid: str = rec['bs_mktid']
        # look up specific frame for fqme-selected asset
        if (df := dfs.get(fqme)) is None:
            mktids2fqmes: dict[str, list[str]] = {}
            for bs_mktid in dfs:
                df: pl.DataFrame = dfs[bs_mktid]
                fqmes: pl.Series[str] = df['fqme']
                uniques: list[str] = fqmes.unique()
                mktids2fqmes[bs_mktid] = set(uniques)
                if fqme in uniques:
                    break
            print(
                f'No specific ledger for fqme={fqme} could be found in\n'
                f'{pformat(mktids2fqmes)}?\n'
                f'Maybe the `{brokername}` backend uses something '
                'else for its `bs_mktid` then the `fqme`?\n'
                'Scanning for matches in unique fqmes per frame..\n'
            )
        # :pray:
        assert not df.is_empty()
        # muck around in pdbp REPL
        breakpoint()
        # TODO: we REALLY need a better console REPL for this
        # kinda thing..
        # - `xonsh` is an obvious option (and it looks amazin) but
        # we need to figure out how to embed it better then just:
        # from xonsh.main import main
        # main(argv=[])
        # which will not actually inject the `df` to globals?
--- a/piker/brokers/init.py
+++ b/piker/brokers/init.py
@ -50,7 +50,7 @@ __brokers__: list[str] = [
    'binance',
    'ib',
    'kraken',
-    'kucoin',
+    'kucoin'
    # broken but used to work
    # 'questrade',
@ -71,7 +71,7 @@ def get_brokermod(brokername: str) -> ModuleType:
    Return the imported broker module by name.
    '''
-    module: ModuleType = import_module('.' + brokername, 'piker.brokers')
+    module = import_module('.' + brokername, 'piker.brokers')
    # we only allow monkeying because it's for internal keying
    module.name = module.__name__.split('.')[-1]
    return module
@ -106,6 +106,6 @@ async def open_cached_client(
    ) as (cache_hit, client):
        if cache_hit:
-            log.runtime(f'Reusing existing {client}')
+            log.info(f'Reusing existing {client}')
        yield client
--- a/piker/brokers/_daemon.py
+++ b/piker/brokers/_daemon.py
@ -179,6 +179,9 @@ def broker_init(
        subpath: str = f'{modpath}.{submodname}'
        enabled.append(subpath)
        # TODO XXX: DO WE NEED THIS?
        # enabled.append('piker.data.feed')
    return (
        brokermod,
        start_actor_kwargs,  # to `ActorNursery.start_actor()`
--- a/piker/brokers/_util.py
+++ b/piker/brokers/_util.py
@ -18,11 +18,10 @@
 Handy cross-broker utils.
 """
 from __future__ import annotations
 from functools import partial
 import json
-import httpx
+import asks
 import logging
 from ..log import (
@ -51,7 +50,6 @@ class SymbolNotFound(BrokerError):
    "Symbol not found by broker search"
 # TODO: these should probably be moved to `.tsp/.data`?
 class NoData(BrokerError):
    '''
    Symbol data not permitted or no data
@ -61,15 +59,14 @@ class NoData(BrokerError):
    def __init__(
        self,
        *args,
-        info: dict|None = None,
+        frame_size: int = 1000,
    ) -> None:
        super().__init__(*args)
        self.info: dict|None = info
        # when raised, machinery can check if the backend
        # set a "frame size" for doing datetime calcs.
-        # self.frame_size: int = 1000
+        self.frame_size: int = 1000
 class DataUnavailable(BrokerError):
@ -91,18 +88,16 @@ class DataThrottle(BrokerError):
 def resproc(
-    resp: httpx.Response,
+    resp: asks.response_objects.Response,
    log: logging.Logger,
    return_json: bool = True,
    log_resp: bool = False,
-) -> httpx.Response:
+) -> asks.response_objects.Response:
-    '''
+    """Process response and return its json content.
    Process response and return its json content.
    Raise the appropriate error on non-200 OK responses.
-
+    """
    '''
    if not resp.status_code == 200:
        raise BrokerError(resp.body)
    try:
--- a/piker/brokers/binance/init.py
+++ b/piker/brokers/binance/init.py
@ -32,19 +32,12 @@ from .feed import (
 )
 from .broker import (
    open_trade_dialog,
    get_cost,
 )
 from .venues import (
    SpotPair,
    FutesPair,
 )
 __all__ = [
    'get_client',
    'get_mkt_info',
    'get_cost',
    'SpotPair',
    'FutesPair',
    'open_trade_dialog',
    'open_history_client',
    'open_symbol_search',
--- a/piker/brokers/binance/api.py
+++ b/piker/brokers/binance/api.py
@ -1,8 +1,8 @@
 # piker: trading gear for hackers
 # Copyright (C)
-#  Guillermo Rodriguez (aka ze jefe)
+#   Guillermo Rodriguez (aka ze jefe)
-#  Tyler Goodlet
+#   Tyler Goodlet
-#  (in stewardship for pikers)
+#   (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -25,7 +25,6 @@ from __future__ import annotations
 from collections import ChainMap
 from contextlib import (
    asynccontextmanager as acm,
    AsyncExitStack,
 )
 from datetime import datetime
 from pprint import pformat
@ -42,7 +41,8 @@ import trio
 from pendulum import (
    now,
 )
-import httpx
+import asks
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 from piker import config
@ -52,13 +52,9 @@ from piker.clearing._messages import (
 from piker.accounting import (
    Asset,
    digits_to_dec,
    MktPair,
 )
 from piker.types import Struct
 from piker.data import (
    def_iohlcv_fields,
    match_from_pairs,
 )
 from piker.data.types import Struct
 from piker.data import def_iohlcv_fields
 from piker.brokers import (
    resproc,
    SymbolNotFound,
@ -68,6 +64,7 @@ from .venues import (
    PAIRTYPES,
    Pair,
    MarketType,
    _spot_url,
    _futes_url,
    _testnet_futes_url,
@ -77,18 +74,16 @@ from .venues import (
 log = get_logger('piker.brokers.binance')
-def get_config() -> dict[str, Any]:
+def get_config() -> dict:
    conf: dict
    path: Path
-    conf, path = config.load(
+    conf, path = config.load()
-        conf_name='brokers',
+
-        touch_if_dne=True,
+    section = conf.get('binance')
-    )
+
    section: dict = conf.get('binance')
    if not section:
-        log.warning(
+        log.warning(f'No config section found for binance in {path}')
            f'No config section found for binance in {path}'
        )
        return {}
    return section
@ -144,7 +139,7 @@ def binance_timestamp(
 class Client:
    '''
-    Async ReST API client using `trio` + `httpx` B)
+    Async ReST API client using ``trio`` + ``asks`` B)
    Supports all of the spot, margin and futures endpoints depending
    on method.
@ -153,17 +148,10 @@ class Client:
    def __init__(
        self,
        venue_sessions: dict[
            str,  # venue key
            tuple[httpx.AsyncClient, str]  # session, eps path
        ],
        conf: dict[str, Any],
        # TODO: change this to `Client.[mkt_]venue: MarketType`?
        mkt_mode: MarketType = 'spot',
    ) -> None:
        self.conf = conf
        # build out pair info tables for each market type
        # and wrap in a chain-map view for search / query.
        self._spot_pairs: dict[str, Pair] = {}  # spot info table
@ -190,13 +178,44 @@ class Client:
        # market symbols for use by search. See `.exch_info()`.
        self._pairs: ChainMap[str, Pair] = ChainMap()
        # spot EPs sesh
        self._sesh = asks.Session(connections=4)
        self._sesh.base_location: str = _spot_url
        # spot testnet
        self._test_sesh: asks.Session = asks.Session(connections=4)
        self._test_sesh.base_location: str = _testnet_spot_url
        # margin and extended spot endpoints session.
        self._sapi_sesh = asks.Session(connections=4)
        self._sapi_sesh.base_location: str = _spot_url
        # futes EPs sesh
        self._fapi_sesh = asks.Session(connections=4)
        self._fapi_sesh.base_location: str = _futes_url
        # futes testnet
        self._test_fapi_sesh: asks.Session = asks.Session(connections=4)
        self._test_fapi_sesh.base_location: str = _testnet_futes_url
        # global client "venue selection" mode.
        # set this when you want to switch venues and not have to
        # specify the venue for the next request.
        self.mkt_mode: MarketType = mkt_mode
-        # per-mkt-venue API client table
+        # per 8
-        self.venue_sesh = venue_sessions
+        self.venue_sesh: dict[
            str,  # venue key
            tuple[asks.Session, str]  # session, eps path
        ] = {
            'spot': (self._sesh, '/api/v3/'),
            'spot_testnet': (self._test_sesh, '/fapi/v1/'),
            'margin': (self._sapi_sesh, '/sapi/v1/'),
            'usdtm_futes': (self._fapi_sesh, '/fapi/v1/'),
            'usdtm_futes_testnet': (self._test_fapi_sesh, '/fapi/v1/'),
            # 'futes_coin': self._dapi,  # TODO
        }
        # lookup for going from `.mkt_mode: str` to the config
        # subsection `key: str`
@ -211,6 +230,40 @@ class Client:
            'futes': ['usdtm_futes'],
        }
        # for creating API keys see,
        # https://www.binance.com/en/support/faq/how-to-create-api-keys-on-binance-360002502072
        self.conf: dict = get_config()
        for key, subconf in self.conf.items():
            if api_key := subconf.get('api_key', ''):
                venue_keys: list[str] = self.confkey2venuekeys[key]
                venue_key: str
                sesh: asks.Session
                for venue_key in venue_keys:
                    sesh, _ = self.venue_sesh[venue_key]
                    api_key_header: dict = {
                        # taken from official:
                        # https://github.com/binance/binance-futures-connector-python/blob/main/binance/api.py#L47
                        "Content-Type": "application/json;charset=utf-8",
                        # TODO: prolly should just always query and copy
                        # in the real latest ver?
                        "User-Agent": "binance-connector/6.1.6smbz6",
                        "X-MBX-APIKEY": api_key,
                    }
                    sesh.headers.update(api_key_header)
                    # if `.use_tesnet = true` in the config then
                    # also add headers for the testnet session which
                    # will be used for all order control
                    if subconf.get('use_testnet', False):
                        testnet_sesh, _ = self.venue_sesh[
                            venue_key + '_testnet'
                        ]
                        testnet_sesh.headers.update(api_key_header)
    def _mk_sig(
        self,
        data: dict,
@ -229,6 +282,7 @@ class Client:
                'to define the creds for auth-ed endpoints!?'
            )
        # XXX: Info on security and authentification
        # https://binance-docs.github.io/apidocs/#endpoint-security-type
        if not (api_secret := subconf.get('api_secret')):
@ -257,7 +311,7 @@ class Client:
        params: dict,
        method: str = 'get',
-        venue: str|None = None,  # if None use `.mkt_mode` state
+        venue: str | None = None,  # if None use `.mkt_mode` state
        signed: bool = False,
        allow_testnet: bool = False,
@ -268,9 +322,8 @@ class Client:
        - /fapi/v3/ USD-M FUTURES, or
        - /api/v3/ SPOT/MARGIN
-        account/market endpoint request depending on either passed in
+        account/market endpoint request depending on either passed in `venue: str`
-        `venue: str` or the current setting `.mkt_mode: str` setting,
+        or the current setting `.mkt_mode: str` setting, default `'spot'`.
        default `'spot'`.
        Docs per venue API:
@ -299,6 +352,9 @@ class Client:
                venue=venue_key,
            )
        sesh: asks.Session
        path: str
        # Check if we're configured to route order requests to the
        # venue equivalent's testnet.
        use_testnet: bool = False
@ -323,12 +379,11 @@ class Client:
            # ctl machinery B)
            venue_key += '_testnet'
-        client: httpx.AsyncClient
+        sesh, path = self.venue_sesh[venue_key]
-        path: str
+
-        client, path = self.venue_sesh[venue_key]
+        meth: Callable = getattr(sesh, method)
        meth: Callable = getattr(client, method)
        resp = await meth(
-            url=path + endpoint,
+            path=path + endpoint,
            params=params,
            timeout=float('inf'),
        )
@ -341,6 +396,7 @@ class Client:
    ) -> None:
        # lookup internal mkt-specific pair table to update
        pair_table: dict[str, Pair] = self._venue2pairs[venue]
        asset_table: dict[str, Asset] = self._venue2assets[venue]
        # make API request(s)
        resp = await self._api(
@ -352,7 +408,6 @@ class Client:
            venue=venue,
            allow_testnet=False,  # XXX: never use testnet for symbol lookups
        )
        mkt_pairs = resp['symbols']
        if not mkt_pairs:
            raise SymbolNotFound(f'No market pairs found!?:\n{resp}')
@ -370,60 +425,28 @@ class Client:
                item['filters'] = filters
            pair_type: Type = PAIRTYPES[venue]
-            try:
+            pair: Pair = pair_type(**item)
                pair: Pair = pair_type(**item)
            except Exception as e:
                e.add_note(
                    "\nDon't panic, prolly stupid binance changed their symbology schema again..\n"
                    'Check out their API docs here:\n\n'
                    'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
                )
                raise
            pair_table[pair.symbol.upper()] = pair
            # update an additional top-level-cross-venue-table
            # `._pairs: ChainMap` for search B0
            pairs_view_subtable[pair.bs_fqme] = pair
            # XXX WOW: TURNS OUT THIS ISN'T TRUE !?
            # > (populate `Asset` table for spot mkts only since it
            # > should be a superset of any other venues such as
            # > futes or margin)
            if venue == 'spot':
-                dst_sectype: str = 'crypto_currency'
+                if (name := pair.quoteAsset) not in asset_table:
                    asset_table[name] = Asset(
                        name=name,
                        atype='crypto_currency',
                        tx_tick=digits_to_dec(pair.quoteAssetPrecision),
                    )
-            elif venue in {'usdtm_futes'}:
+                if (name := pair.baseAsset) not in asset_table:
-                dst_sectype: str = 'future'
+                    asset_table[name] = Asset(
-                if pair.contractType == 'PERPETUAL':
+                        name=name,
-                    dst_sectype: str = 'perpetual_future'
+                        atype='crypto_currency',
                        tx_tick=digits_to_dec(pair.baseAssetPrecision),
                    )
            spot_asset_table: dict[str, Asset] = self._venue2assets['spot']
            ven_asset_table: dict[str, Asset] = self._venue2assets[venue]
            if (
                (name := pair.quoteAsset) not in spot_asset_table
            ):
                spot_asset_table[pair.bs_src_asset] = Asset(
                    name=name,
                    atype='crypto_currency',
                    tx_tick=digits_to_dec(pair.quoteAssetPrecision),
                )
            if (
                (name := pair.baseAsset) not in ven_asset_table
            ):
                if venue != 'spot':
                    assert dst_sectype != 'crypto_currency'
                ven_asset_table[pair.bs_dst_asset] = Asset(
                    name=name,
                    atype=dst_sectype,
                    tx_tick=digits_to_dec(pair.baseAssetPrecision),
                )
        # log.warning(
        #     f'Assets not YET found in spot set: `{pformat(dne)}`!?'
        # )
        # NOTE: make merged view of all market-type pairs but
        # use market specific `Pair.bs_fqme` for keys!
        # this allows searching for market pairs with different
@ -435,29 +458,16 @@ class Client:
        if venue == 'spot':
            return
-        # TODO: maybe use this assets response for non-spot venues?
+        assets: list[dict] = resp.get('assets', ())
-        # -> issue is we do the exch_info queries conc, so we can't
+        for entry in assets:
-        # guarantee order for inter-table lookups..
+            name: str = entry['asset']
-        # if venue ep delivers an explicit set of assets copy just
+            asset_table[name] = self._venue2assets['spot'].get(name)
        # ensure they are also already listed in the spot equivs.
        # assets: list[dict] = resp.get('assets', ())
        # for entry in assets:
        #     name: str = entry['asset']
        #     spot_asset_table: dict[str, Asset] = self._venue2assets['spot']
        #     if name not in spot_asset_table:
        #         log.warning(
        #             f'COULDNT FIND ASSET {name}\n{entry}\n'
        #             f'ADDING AS FUTES ONLY!?'
        #         )
        #     asset_table: dict[str, Asset] = self._venue2assets[venue]
        #     asset_table[name] = spot_asset_table.get(name)
    async def exch_info(
        self,
        sym: str | None = None,
        venue: MarketType | None = None,
        expiry: str | None = None,
    ) -> dict[str, Pair] | Pair:
        '''
@ -473,20 +483,9 @@ class Client:
        '''
        pair_table: dict[str, Pair] = self._venue2pairs[
-            venue
+            venue or self.mkt_mode
            or
            self.mkt_mode
        ]
-        if (
+        if cached_pair := pair_table.get(sym):
            expiry
            and 'perp' not in expiry.lower()
        ):
            sym: str = f'{sym}_{expiry}'
        if (
            sym
            and (cached_pair := pair_table.get(sym))
        ):
            return cached_pair
        venues: list[str] = ['spot', 'usdtm_futes']
@ -494,52 +493,14 @@ class Client:
            venues: list[str] = [venue]
        # batch per-venue download of all exchange infos
-        async with trio.open_nursery() as tn:
+        async with trio.open_nursery() as rn:
            for ven in venues:
-                tn.start_soon(
+                rn.start_soon(
                    self._cache_pairs,
                    ven,
                )
-        if sym:
+        return pair_table[sym] if sym else self._pairs
            return pair_table[sym]
        else:
            return self._pairs
    async def get_assets(
        self,
        venue: str | None = None,
    ) -> dict[str, Asset]:
        if (
            venue
            and venue != 'spot'
        ):
            venues = [venue]
        else:
            venues = ['usdtm_futes']
        ass_table: dict[str, Asset] = self._venue2assets['spot']
        # merge in futes contracts with a sectype suffix
        for venue in venues:
            ass_table |= self._venue2assets[venue]
        return ass_table
    async def get_mkt_pairs(self) -> dict[str, Pair]:
        '''
        Flatten the multi-venue (chain) map of market pairs
        to a fqme indexed table for data layer caching.
        '''
        flat: dict[str, Pair] = {}
        for venmap in self._pairs.maps:
            for bs_fqme, pair in venmap.items():
                flat[pair.bs_fqme] = pair
        return flat
    # TODO: unused except by `brokers.core.search_symbols()`?
    async def search_symbols(
@ -549,32 +510,20 @@ class Client:
    ) -> dict[str, Any]:
-        fq_pairs: dict[str, Pair] = await self.exch_info()
+        fq_pairs: dict = await self.exch_info()
-        # TODO: cache this list like we were in
+        matches = fuzzy.extractBests(
-        # `open_symbol_search()`?
+            pattern,
-        # keys: list[str] = list(fq_pairs)
+            fq_pairs,
        return match_from_pairs(
            pairs=fq_pairs,
            query=pattern.upper(),
            score_cutoff=50,
        )
-
+        # repack in dict form
-    def pair2venuekey(
+        return {item[0]['symbol']: item[0]
-        self,
+                for item in matches}
        pair: Pair,
    ) -> str:
        return {
            'USDTM': 'usdtm_futes',
            'SPOT': 'spot',
            # 'COINM': 'coin_futes',
            # ^-TODO-^ bc someone might want it..?
        }[pair.venue]
    async def bars(
        self,
-        mkt: MktPair,
+        symbol: str,
        start_dt: datetime | None = None,
        end_dt: datetime | None = None,
@ -604,20 +553,16 @@ class Client:
        start_time = binance_timestamp(start_dt)
        end_time = binance_timestamp(end_dt)
        bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
        # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
        bars = await self._api(
            'klines',
            params={
-                # NOTE: always query using their native symbology!
+                'symbol': symbol.upper(),
                'symbol': mkt.bs_mktid.upper(),
                'interval': '1m',
                'startTime': start_time,
                'endTime': end_time,
                'limit': limit
            },
            venue=self.pair2venuekey(bs_pair),
            allow_testnet=False,
        )
        new_bars: list[tuple] = []
@ -934,148 +879,17 @@ class Client:
        await self.close_listen_key(key)
 _venue_urls: dict[str, str] = {
    'spot': (
        _spot_url,
        '/api/v3/',
    ),
    'spot_testnet': (
        _testnet_spot_url,
        '/fapi/v1/'
    ),
    # margin and extended spot endpoints session.
    # TODO: did this ever get implemented fully?
    # 'margin': (
    #     _spot_url,
    #     '/sapi/v1/'
    # ),
    'usdtm_futes': (
        _futes_url,
        '/fapi/v1/',
    ),
    'usdtm_futes_testnet': (
        _testnet_futes_url,
        '/fapi/v1/',
    ),
    # TODO: for anyone who actually needs it ;P
    # 'coin_futes': ()
 }
 def init_api_keys(
    client: Client,
    conf: dict[str, Any],
 ) -> None:
    '''
    Set up per-venue API keys each http client according to the user's
    `brokers.conf`.
    For ex, to use spot-testnet and live usdt futures APIs:
    ```toml
        [binance]
        # spot test net
        spot.use_testnet = true
        spot.api_key = '<spot_api_key_from_binance_account>'
        spot.api_secret = '<spot_api_key_password>'
        # futes live
        futes.use_testnet = false
        accounts.usdtm = 'futes'
        futes.api_key = '<futes_api_key_from_binance>'
        futes.api_secret = '<futes_api_key_password>''
        # if uncommented will use the built-in paper engine and not
        # connect to `binance` API servers for order ctl.
        # accounts.paper = 'paper'
    ```
    '''
    for key, subconf in conf.items():
        if api_key := subconf.get('api_key', ''):
            venue_keys: list[str] = client.confkey2venuekeys[key]
            venue_key: str
            client: httpx.AsyncClient
            for venue_key in venue_keys:
                client, _ = client.venue_sesh[venue_key]
                api_key_header: dict = {
                    # taken from official:
                    # https://github.com/binance/binance-futures-connector-python/blob/main/binance/api.py#L47
                    "Content-Type": "application/json;charset=utf-8",
                    # TODO: prolly should just always query and copy
                    # in the real latest ver?
                    "User-Agent": "binance-connector/6.1.6smbz6",
                    "X-MBX-APIKEY": api_key,
                }
                client.headers.update(api_key_header)
                # if `.use_tesnet = true` in the config then
                # also add headers for the testnet session which
                # will be used for all order control
                if subconf.get('use_testnet', False):
                    testnet_sesh, _ = client.venue_sesh[
                        venue_key + '_testnet'
                    ]
                    testnet_sesh.headers.update(api_key_header)
@acm
-async def get_client(
+async def get_client() -> Client:
    mkt_mode: MarketType = 'spot',
 ) -> Client:
    '''
    Construct an single `piker` client which composes multiple underlying venue
    specific API clients both for live and test networks.
-    '''
+    client = Client()
-    venue_sessions: dict[
+    await client.exch_info()
-        str,  # venue key
+    log.info(
-        tuple[httpx.AsyncClient, str]  # session, eps path
+        f'{client} in {client.mkt_mode} mode: caching exchange infos..\n'
-    ] = {}
+        'Cached multi-market pairs:\n'
-    async with AsyncExitStack() as client_stack:
+        f'spot: {len(client._spot_pairs)}\n'
-        for name, (base_url, path) in _venue_urls.items():
+        f'usdtm_futes: {len(client._ufutes_pairs)}\n'
-            api: httpx.AsyncClient = await client_stack.enter_async_context(
+        f'Total: {len(client._pairs)}\n'
-                httpx.AsyncClient(
+    )
                    base_url=base_url,
                    # headers={},
-                    # TODO: is there a way to numerate this?
+    yield client
                    # https://www.python-httpx.org/advanced/clients/#why-use-a-client
                    # connections=4
                )
            )
            venue_sessions[name] = (
                api,
                path,
            )
        conf: dict[str, Any] = get_config()
        # for creating API keys see,
        # https://www.binance.com/en/support/faq/how-to-create-api-keys-on-binance-360002502072
        client = Client(
            venue_sessions=venue_sessions,
            conf=conf,
            mkt_mode=mkt_mode,
        )
        init_api_keys(
            client=client,
            conf=conf,
        )
        fq_pairs: dict[str, Pair] = await client.exch_info()
        assert fq_pairs
        log.info(
            f'Loaded multi-venue `Client` in mkt_mode={client.mkt_mode!r}\n\n'
            f'Symbology Summary:\n'
            f'------ - ------\n'
            f'spot: {len(client._spot_pairs)}\n'
            f'usdtm_futes: {len(client._ufutes_pairs)}\n'
            '------ - ------\n'
            f'total: {len(client._pairs)}\n'
        )
        yield client
--- a/piker/brokers/binance/broker.py
+++ b/piker/brokers/binance/broker.py
@ -36,6 +36,7 @@ import trio
 from piker.accounting import (
    Asset,
    # MktPair,
 )
 from piker.brokers._util import (
    get_logger,
@ -48,9 +49,7 @@ from piker.brokers import (
    open_cached_client,
    BrokerError,
 )
-from piker.clearing import (
+from piker.clearing import OrderDialogs
    OrderDialogs,
 )
 from piker.clearing._messages import (
    BrokerdOrder,
    BrokerdOrderAck,
@ -72,33 +71,6 @@ from .api import Client
 log = get_logger('piker.brokers.binance')
 # Fee schedule template, mostly for paper engine fees modelling.
 # https://www.binance.com/en/support/faq/what-are-market-makers-and-takers-360007720071
 def get_cost(
    price: float,
    size: float,
    is_taker: bool = False,
 ) -> float:
    # https://www.binance.com/en/fee/trading
    cb: float = price * size
    match is_taker:
        case True:
            return cb * 0.001000
        case False if cb < 1e6:
            return cb * 0.001000
        case False if 1e6 >= cb < 5e6:
            return cb * 0.000900
        # NOTE: there's more but are you really going
        # to have a cb bigger then this per trade?
        case False if cb >= 5e6:
            return cb * 0.000800
 async def handle_order_requests(
    ems_order_stream: tractor.MsgStream,
    client: Client,
@ -260,24 +232,16 @@ async def open_trade_dialog(
    account_name: str = 'usdtm'
    use_testnet: bool = False
    # TODO: if/when we add .accounting support we need to
    # do a open_symcache() call.. though maybe we can hide
    # this in a new async version of open_account()?
    async with open_cached_client('binance') as client:
-        subconf: dict|None = client.conf.get(venue_name)
+        subconf: dict = client.conf[venue_name]
        use_testnet = subconf.get('use_testnet', False)
        # XXX: if no futes.api_key or spot.api_key has been set we
        # always fall back to the paper engine!
-        if (
+        if not subconf.get('api_key'):
            not subconf
            or
            not subconf.get('api_key')
        ):
            await ctx.started('paper')
            return
        use_testnet: bool = subconf.get('use_testnet', False)
    async with (
        open_cached_client('binance') as client,
    ):
@ -357,7 +321,7 @@ async def open_trade_dialog(
                        if balance > 0:
                            balances[spot_asset] = (balance, last_update_t)
-                            # await tractor.pause()
+                            # await tractor.breakpoint()
                # @position response:
                # {'positions': [{'entryPrice': '0.0',
@ -436,7 +400,7 @@ async def open_trade_dialog(
            #   and comparison with binance's own position calcs.
            # - load pps and accounts using accounting apis, write
            #   the ledger and account files
-            #   - table: Account
+            #   - table: PpTable
            #   - ledger: TransactionLedger
            async with (
--- a/piker/brokers/binance/feed.py
+++ b/piker/brokers/binance/feed.py
@ -24,11 +24,8 @@ from contextlib import (
    aclosing,
 )
 from datetime import datetime
-from functools import (
+from functools import partial
    partial,
 )
 import itertools
 from pprint import pformat
 from typing import (
    Any,
    AsyncGenerator,
@ -42,12 +39,12 @@ from trio_typing import TaskStatus
 from pendulum import (
    from_timestamp,
 )
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 import tractor
 from piker.brokers import (
    open_cached_client,
    NoData,
 )
 from piker._cacheables import (
    async_lifo_cache,
@ -57,8 +54,9 @@ from piker.accounting import (
    DerivTypes,
    MktPair,
    unpack_fqme,
    digits_to_dec,
 )
-from piker.types import Struct
+from piker.data.types import Struct
 from piker.data.validate import FeedInit
 from piker.data._web_bs import (
    open_autorecon_ws,
@ -110,7 +108,6 @@ class AggTrade(Struct, frozen=True):
 async def stream_messages(
    ws: NoBsWs,
 ) -> AsyncGenerator[NoBsWs, dict]:
    # TODO: match syntax here!
@ -221,8 +218,6 @@ def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
    }
 # TODO, why aren't frame resp `log.info()`s showing in upstream
 # code?!
@acm
 async def open_history_client(
    mkt: MktPair,
@ -255,30 +250,24 @@ async def open_history_client(
            else:
                client.mkt_mode = 'spot'
-            array: np.ndarray = await client.bars(
+            # NOTE: always query using their native symbology!
-                mkt=mkt,
+            mktid: str = mkt.bs_mktid
            array = await client.bars(
                mktid,
                start_dt=start_dt,
                end_dt=end_dt,
            )
            if array.size == 0:
                raise NoData(
                    f'No frame for {start_dt} -> {end_dt}\n'
                )
            times = array['time']
-            if not times.any():
+            if (
-                raise ValueError(
+                end_dt is None
-                    'Bad frame with null-times?\n\n'
+            ):
-                    f'{times}'
+                inow = round(time.time())
                )
            if end_dt is None:
                inow: int = round(time.time())
                if (inow - times[-1]) > 60:
-                    await tractor.pause()
+                    await tractor.breakpoint()
            start_dt = from_timestamp(times[0])
            end_dt = from_timestamp(times[-1])
            return array, start_dt, end_dt
        yield get_ohlc, {'erlangs': 3, 'rate': 3}
@ -288,113 +277,69 @@ async def open_history_client(
 async def get_mkt_info(
    fqme: str,
-) -> tuple[MktPair, Pair] | None:
+) -> tuple[MktPair, Pair]:
    # uppercase since kraken bs_mktid is always upper
-    if 'binance' not in fqme.lower():
+    if 'binance' not in fqme:
        fqme += '.binance'
-    mkt_mode: str = ''
+    bs_fqme, _, broker = fqme.rpartition('.')
    broker, mkt_ep, venue, expiry = unpack_fqme(fqme)
-    # NOTE: we always upper case all tokens to be consistent with
+    # NOTE: see the `FutesPair.bs_fqme: str` implementation
-    # binance's symbology style for pairs, like `BTCUSDT`, but in
+    # to understand the reverse market info lookup below.
-    # theory we could also just keep things lower case; as long as
+    mkt_mode = venue = venue.lower() or 'spot'
-    # we're consistent and the symcache matches whatever this func
+    _atype: str = ''
    # returns, always!
    expiry: str = expiry.upper()
    venue: str = venue.upper()
    venue_lower: str = venue.lower()
    # XXX TODO: we should change the usdtm_futes name to just
    # usdm_futes (dropping the tether part) since it turns out that
    # there are indeed USD-tokens OTHER THEN tether being used as
    # the margin assets.. it's going to require a wholesale
    # (variable/key) rename as well as file name adjustments to any
    # existing tsdb set..
    if 'usd' in venue_lower:
        mkt_mode: str = 'usdtm_futes'
    # NO IDEA what these contracts (some kinda DEX-ish futes?) are
    # but we're masking them for now..
    elif (
        'defi' in venue_lower
        # TODO: handle coinm futes which have a margin asset that
        # is some crypto token!
        # https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
        or 'btc' in venue_lower
    ):
        return None
    else:
        # NOTE: see the `FutesPair.bs_fqme: str` implementation
        # to understand the reverse market info lookup below.
        mkt_mode = venue_lower or 'spot'
    if (
        venue
-        and 'spot' not in venue_lower
+        and 'spot' not in venue.lower()
        # XXX: catch all in case user doesn't know which
        # venue they want (usdtm vs. coinm) and we can choose
        # a default (via config?) once we support coin-m APIs.
-        or 'perp' in venue_lower
+        or 'perp' in bs_fqme.lower()
    ):
-        if not mkt_mode:
+        mkt_mode: str = f'{venue.lower()}_futes'
-            mkt_mode: str = f'{venue_lower}_futes'
+        if 'perp' in expiry:
            _atype = 'perpetual_future'
        else:
            _atype = 'future'
    async with open_cached_client(
        'binance',
    ) as client:
-        assets: dict[str, Asset] = await client.get_assets()
+        # switch mode depending on input pattern parsing
        pair_str: str = mkt_ep.upper()
        # switch venue-mode depending on input pattern parsing
        # since we want to use a particular endpoint (set) for
        # pair info lookup!
        client.mkt_mode = mkt_mode
-        pair: Pair = await client.exch_info(
+        pair_str: str = mkt_ep.upper()
-            pair_str,
+        pair: Pair = await client.exch_info(pair_str)
            venue=mkt_mode,  # explicit
            expiry=expiry,
        )
        if 'futes' in mkt_mode:
            assert isinstance(pair, FutesPair)
        dst: Asset | None = assets.get(pair.bs_dst_asset)
        if (
            not dst
            # TODO: a known asset DNE list?
            # and pair.baseAsset == 'DEFI'
        ):
            log.warning(
                f'UNKNOWN {venue} asset {pair.baseAsset} from,\n'
                f'{pformat(pair.to_dict())}'
            )
            # XXX UNKNOWN missing "asset", though no idea why?
            # maybe it's only avail in the margin venue(s): /dapi/ ?
            return None
        mkt = MktPair(
-            dst=dst,
+            dst=Asset(
-            src=assets[pair.bs_src_asset],
+                name=pair.baseAsset,
                atype='crypto',
                tx_tick=digits_to_dec(pair.baseAssetPrecision),
            ),
            src=Asset(
                name=pair.quoteAsset,
                atype='crypto',
                tx_tick=digits_to_dec(pair.quoteAssetPrecision),
            ),
            price_tick=pair.price_tick,
            size_tick=pair.size_tick,
            bs_mktid=pair.symbol,
            expiry=expiry,
            venue=venue,
            broker='binance',
-
+            _atype=_atype,
            # NOTE: sectype is always taken from dst, see
            # `MktPair.type_key` and `Client._cache_pairs()`
            # _atype=sectype,
        )
-        return mkt, pair
+        both = mkt, pair
        return both
@acm
@ -465,8 +410,6 @@ async def stream_quotes(
    ):
        init_msgs: list[FeedInit] = []
        for sym in symbols:
            mkt: MktPair
            pair: Pair
            mkt, pair = await get_mkt_info(sym)
            # build out init msgs according to latest spec
@ -515,6 +458,7 @@ async def stream_quotes(
            # start streaming
            async for typ, quote in msg_gen:
                # period = time.time() - last
                # hz = 1/period if period else float('inf')
                # if hz > 60:
@ -528,11 +472,10 @@ async def open_symbol_search(
    ctx: tractor.Context,
 ) -> Client:
    # NOTE: symbology tables are loaded as part of client
    # startup in ``.api.get_client()`` and in this case
    # are stored as `Client._pairs`.
    async with open_cached_client('binance') as client:
        # load all symbols locally for fast search
        fqpairs_cache = await client.exch_info()
        # TODO: maybe we should deliver the cache
        # so that client's can always do a local-lookup-first
        # style try and then update async as (new) match results
@ -543,15 +486,14 @@ async def open_symbol_search(
            pattern: str
            async for pattern in stream:
-                # NOTE: pattern fuzzy-matching is done within
+                matches = fuzzy.extractBests(
                # the methd impl.
                pairs: dict[str, Pair] = await client.search_symbols(
                    pattern,
                    fqpairs_cache,
                    score_cutoff=50,
                )
-                # repack in fqme-keyed table
+                # repack in dict form
-                byfqme: dict[str, Pair] = {}
+                await stream.send({
-                for pair in pairs.values():
+                    item[0].bs_fqme: item[0]
-                    byfqme[pair.bs_fqme] = pair
+                    for item in matches
-
+                })
                await stream.send(byfqme)
--- a/piker/brokers/binance/venues.py
+++ b/piker/brokers/binance/venues.py
@ -1,5 +1,8 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C)
 #   Guillermo Rodriguez (aka ze jefe)
 #   Tyler Goodlet
 #   (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -26,7 +29,7 @@ from decimal import Decimal
 from msgspec import field
-from piker.types import Struct
+from piker.data.types import Struct
 # API endpoint paths by venue / sub-API
@ -62,7 +65,7 @@ MarketType = Literal[
    'spot',
    # 'margin',
    'usdtm_futes',
-    # 'coinm_futes',
+    # 'coin_futes',
 ]
@ -84,7 +87,6 @@ def get_api_eps(venue: MarketType) -> tuple[str, str]:
 class Pair(Struct, frozen=True, kw_only=True):
    symbol: str
    status: str
    orderTypes: list[str]
@ -118,10 +120,6 @@ class Pair(Struct, frozen=True, kw_only=True):
    def bs_fqme(self) -> str:
        return self.symbol
    @property
    def bs_mktid(self) -> str:
        return f'{self.symbol}.{self.venue}'
 class SpotPair(Pair, frozen=True):
@ -137,31 +135,15 @@ class SpotPair(Pair, frozen=True):
    quoteOrderQtyMarketAllowed: bool
    isSpotTradingAllowed: bool
    isMarginTradingAllowed: bool
    otoAllowed: bool
    defaultSelfTradePreventionMode: str
    allowedSelfTradePreventionModes: list[str]
    permissions: list[str]
    permissionSets: list[list[str]]
    # NOTE: see `.data._symcache.SymbologyCache.load()` for why
    ns_path: str = 'piker.brokers.binance:SpotPair'
    @property
    def venue(self) -> str:
        return 'SPOT'
    @property
    def bs_fqme(self) -> str:
        return f'{self.symbol}.SPOT'
    @property
    def bs_src_asset(self) -> str:
        return f'{self.quoteAsset}'
    @property
    def bs_dst_asset(self) -> str:
        return f'{self.baseAsset}'
 class FutesPair(Pair):
@ -181,14 +163,12 @@ class FutesPair(Pair):
    quoteAsset: str  # 'USDT',
    quotePrecision: int  # 8,
    requiredMarginPercent: float  # '5.0000',
    settlePlan: int  # 0,
    timeInForce: list[str]  # ['GTC', 'IOC', 'FOK', 'GTX'],
    triggerProtect: float  # '0.0500',
    underlyingSubType: list[str]  # ['PoW'],
    underlyingType: str  # 'COIN'
    # NOTE: see `.data._symcache.SymbologyCache.load()` for why
    ns_path: str = 'piker.brokers.binance:FutesPair'
    # NOTE: for compat with spot pairs and `MktPair.src: Asset`
    # processing..
    @property
@ -196,101 +176,32 @@ class FutesPair(Pair):
        return self.quotePrecision
    @property
-    def expiry(self) -> str:
+    def bs_fqme(self) -> str:
        symbol: str = self.symbol
        contype: str = self.contractType
        match contype:
            case (
                'CURRENT_QUARTER'
                | 'CURRENT_QUARTER DELIVERING'
                | 'NEXT_QUARTER'  # su madre binance..
            ):
                pair, _, expiry = symbol.partition('_')
                assert pair == self.pair  # sanity
                return f'{expiry}'
            case 'PERPETUAL':
                return 'PERP'
            case '':
                subtype: list[str] = self.underlyingSubType
                if not subtype:
                    if self.status == 'PENDING_TRADING':
                        return 'PENDING'
                match subtype:
                    case ['DEFI']:
                        return 'PERP'
        # wow, just wow you binance guys suck..
        if self.status == 'PENDING_TRADING':
            return 'PENDING'
        # XXX: yeah no clue then..
        raise ValueError(
            f'Bad .expiry token match: {contype} for {symbol}'
        )
    @property
    def venue(self) -> str:
        symbol: str = self.symbol
        ctype: str = self.contractType
        margin: str = self.marginAsset
        match ctype:
            case 'PERPETUAL':
-                return f'{margin}M'
+                return f'{symbol}.{margin}M.PERP'
-            case (
+            case 'CURRENT_QUARTER':
-                'CURRENT_QUARTER'
+                pair, _, expiry = symbol.partition('_')
-                | 'CURRENT_QUARTER DELIVERING'
+                return f'{pair}.{margin}M.{expiry}'
                | 'NEXT_QUARTER'  # su madre binance..
            ):
                _, _, expiry = symbol.partition('_')
                return f'{margin}M'
            case '':
                subtype: list[str] = self.underlyingSubType
                if not subtype:
                    if self.status == 'PENDING_TRADING':
-                        return f'{margin}M'
+                        return f'{symbol}.{margin}M.PENDING'
-                match subtype:
+                match subtype[0]:
-                    case (
+                    case 'DEFI':
-                        ['DEFI']
+                        return f'{symbol}.{subtype}.PERP'
                        | ['USDC']
                    ):
                        return f'{subtype[0]}'
        # XXX: yeah no clue then..
-        raise ValueError(
+        return f'{symbol}.WTF.PWNED.BBQ'
            f'Bad .venue token match: {ctype}'
        )
    @property
    def bs_fqme(self) -> str:
        symbol: str = self.symbol
        ctype: str = self.contractType
        venue: str = self.venue
        pair: str = self.pair
        match ctype:
            case (
                'CURRENT_QUARTER'
                | 'NEXT_QUARTER'  # su madre binance..
            ):
                pair, _, expiry = symbol.partition('_')
                assert pair == self.pair
        return f'{pair}.{venue}.{self.expiry}'
    @property
    def bs_src_asset(self) -> str:
        return f'{self.quoteAsset}'
    @property
    def bs_dst_asset(self) -> str:
        return f'{self.baseAsset}.{self.venue}'
 PAIRTYPES: dict[MarketType, Pair] = {
--- a/piker/brokers/cli.py
+++ b/piker/brokers/cli.py
@ -454,18 +454,8 @@ def mkt_info(
@cli.command()
@click.argument('pattern', required=True)
 # TODO: move this to top level click/typer context for all subs
@click.option(
    '--pdb',
    is_flag=True,
    help='Enable tractor debug mode',
 )
@click.pass_obj
-def search(
+def search(config, pattern):
    config: dict,
    pattern: str,
    pdb: bool,
 ):
    '''
    Search for symbols from broker backend(s).
@ -478,26 +468,23 @@ def search(
        async with maybe_open_pikerd(
            loglevel=config['loglevel'],
            debug_mode=pdb,
        ):
            return await func()
-    from piker.toolz import open_crash_handler
+    quotes = trio.run(
-    with open_crash_handler():
+        main,
-        quotes = trio.run(
+        partial(
-            main,
+            core.symbol_search,
-            partial(
+            brokermods,
-                core.symbol_search,
+            pattern,
-                brokermods,
+        ),
-                pattern,
+    )
            ),
        )
-        if not quotes:
+    if not quotes:
-            log.error(f"No matches could be found for {pattern}?")
+        log.error(f"No matches could be found for {pattern}?")
-            return
+        return
-        click.echo(colorize_json(quotes))
+    click.echo(colorize_json(quotes))
@cli.command()
@ -506,11 +493,9 @@ def search(
@click.option('--delete', '-d', flag_value=True, help='Delete section')
@click.pass_obj
 def brokercfg(config, section, value, delete):
-    '''
+    """If invoked with no arguments, open an editor to edit broker configs file
-    If invoked with no arguments, open an editor to edit broker
+    or get / update an individual section.
-    configs file or get / update an individual section.
+    """
    '''
    from .. import config
    if section:
--- a/piker/brokers/core.py
+++ b/piker/brokers/core.py
@ -95,15 +95,15 @@ async def option_chain(
            return await client.option_chains(contracts)
-# async def contracts(
+async def contracts(
-#     brokermod: ModuleType,
+    brokermod: ModuleType,
-#     symbol: str,
+    symbol: str,
-# ) -> Dict[str, Dict[str, Dict[str, Any]]]:
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
-#     """Return option contracts (all expiries) for ``symbol``.
+    """Return option contracts (all expiries) for ``symbol``.
-#     """
+    """
-#     async with brokermod.get_client() as client:
+    async with brokermod.get_client() as client:
-#         # return await client.get_all_contracts([symbol])
+        # return await client.get_all_contracts([symbol])
-#         return await client.get_all_contracts([symbol])
+        return await client.get_all_contracts([symbol])
 async def bars(
@ -117,6 +117,21 @@ async def bars(
        return await client.bars(symbol, **kwargs)
 async def mkt_info(
    brokermod: ModuleType,
    fqme: str,
    **kwargs,
 ) -> MktPair:
    '''
    Return MktPair info from broker including src and dst assets.
    '''
    return await brokermod.get_mkt_info(
        fqme.replace(brokermod.name, '')
    )
 async def search_w_brokerd(name: str, pattern: str) -> dict:
    async with open_cached_client(name) as client:
@ -145,11 +160,7 @@ async def symbol_search(
        async with maybe_spawn_brokerd(
            mod.name,
-            infect_asyncio=getattr(
+            infect_asyncio=getattr(mod, '_infect_asyncio', False),
                mod,
                '_infect_asyncio',
                False,
            ),
        ) as portal:
            results.append((
@ -167,20 +178,3 @@ async def symbol_search(
            n.start_soon(search_backend, mod.name)
    return results
 async def mkt_info(
    brokermod: ModuleType,
    fqme: str,
    **kwargs,
 ) -> MktPair:
    '''
    Return MktPair info from broker including src and dst assets.
    '''
    async with open_cached_client(brokermod.name) as client:
        assert client
        return await brokermod.get_mkt_info(
            fqme.replace(brokermod.name, '')
        )
--- a/piker/brokers/deribit/api.py
+++ b/piker/brokers/deribit/api.py
@ -34,7 +34,7 @@ from typing import (
 import pendulum
 import trio
 from trio_typing import TaskStatus
-from rapidfuzz import process as fuzzy
+from fuzzywuzzy import process as fuzzy
 import numpy as np
 from tractor.trionics import (
    broadcast_receiver,
@ -52,11 +52,8 @@ from cryptofeed.defines import (
 )
 from cryptofeed.symbols import Symbol
-from piker.data import (
+from piker.data.types import Struct
-    def_iohlcv_fields,
+from piker.data import def_iohlcv_fields
    match_from_pairs,
    Struct,
 )
 from piker.data._web_bs import (
    open_jsonrpc_session
 )
@ -82,7 +79,7 @@ _testnet_ws_url = 'wss://test.deribit.com/ws/api/v2'
 class JSONRPCResult(Struct):
    jsonrpc: str = '2.0'
    id: int
-    result: Optional[list[dict]] = None
+    result: Optional[dict] = None
    error: Optional[dict] = None
    usIn: int
    usOut: int
@ -292,29 +289,24 @@ class Client:
        currency: str = 'btc',  # BTC, ETH, SOL, USDC
        kind: str = 'option',
        expired: bool = False
    ) -> dict[str, Any]:
        """Get symbol info for the exchange.
-    ) -> dict[str, dict]:
+        """
        '''
        Get symbol infos.
        '''
        if self._pairs:
            return self._pairs
        # will retrieve all symbols by default
-        params: dict[str, str] = {
+        params = {
            'currency': currency.upper(),
            'kind': kind,
            'expired': str(expired).lower()
        }
-        resp: JSONRPCResult = await self.json_rpc(
+        resp = await self.json_rpc('public/get_instruments', params)
-            'public/get_instruments',
+        results = resp.result
-            params,
+
-        )
+        instruments = {
        # convert to symbol-keyed table
        results: list[dict] | None = resp.result
        instruments: dict[str, dict] = {
            item['instrument_name'].lower(): item
            for item in results
        }
@ -327,7 +319,6 @@ class Client:
    async def cache_symbols(
        self,
    ) -> dict:
        if not self._pairs:
            self._pairs = await self.symbol_info()
@ -338,23 +329,17 @@ class Client:
        pattern: str,
        limit: int = 30,
    ) -> dict[str, Any]:
-        '''
+        data = await self.symbol_info()
        Fuzzy search symbology set for pairs matching `pattern`.
-        '''
+        matches = fuzzy.extractBests(
-        pairs: dict[str, Any] = await self.symbol_info()
+            pattern,
-        matches: dict[str, Pair] = match_from_pairs(
+            data,
            pairs=pairs,
            query=pattern.upper(),
            score_cutoff=35,
            limit=limit
        )
-
+        # repack in dict form
-       # repack in name-keyed table
+        return {item[0]['instrument_name'].lower(): item[0]
-        return {
+                for item in matches}
            pair['instrument_name'].lower(): pair
            for pair in matches.values()
        }
    async def bars(
        self,
--- a/piker/brokers/deribit/feed.py
+++ b/piker/brokers/deribit/feed.py
@ -26,7 +26,7 @@ import time
 import trio
 from trio_typing import TaskStatus
 import pendulum
-from rapidfuzz import process as fuzzy
+from fuzzywuzzy import process as fuzzy
 import numpy as np
 import tractor
--- a/piker/brokers/ib/init.py
+++ b/piker/brokers/ib/init.py
@ -30,33 +30,23 @@ from .api import (
 )
 from .feed import (
    open_history_client,
    open_symbol_search,
    stream_quotes,
 )
 from .broker import (
    open_trade_dialog,
 )
 from .ledger import (
    norm_trade,
    norm_trade_records,
    tx_sort,
 )
 from .symbols import (
    get_mkt_info,
    open_symbol_search,
    _search_conf,
 )
 __all__ = [
    'get_client',
    'get_mkt_info',
    'norm_trade',
    'norm_trade_records',
    'open_trade_dialog',
    'open_history_client',
    'open_symbol_search',
    'stream_quotes',
    '_search_conf',
    'tx_sort',
 ]
 _brokerd_mods: list[str] = [
@ -66,7 +56,6 @@ _brokerd_mods: list[str] = [
 _datad_mods: list[str] = [
    'feed',
    'symbols',
 ]
@ -86,8 +75,3 @@ _spawn_kwargs = {
 # know if ``brokerd`` should be spawned with
 # ``tractor``'s aio mode.
 _infect_asyncio: bool = True
 # XXX NOTE: for now we disable symcache with this backend since
 # there is no clearly simple nor practical way to download "all
 # symbology info" for all supported venues..
 _no_symcache: bool = True
--- a/piker/brokers/ib/_flex_reports.py
+++ b/piker/brokers/ib/_flex_reports.py
@ -159,11 +159,7 @@ def load_flex_trades(
    for acctid in trades_by_account:
        trades_by_id = trades_by_account[acctid]
-        with open_trade_ledger(
+        with open_trade_ledger('ib', acctid) as ledger_dict:
            'ib',
            acctid,
            allow_from_sync_code=True,
        ) as ledger_dict:
            tid_delta = set(trades_by_id) - set(ledger_dict)
            log.info(
                'New trades detected\n'
--- a/piker/brokers/ib/_util.py
+++ b/piker/brokers/ib/_util.py
@ -88,25 +88,16 @@ async def data_reset_hack(
    api_port: str = str(ib_client.client.port)
    vnc_host: str
    vnc_port: int
-    vnc_sockaddr: tuple[str] | None = client.conf.get('vnc_addrs')
+    vnc_host, vnc_port = client.conf['vnc_addrs'].get(
    no_setup_msg:str = (
        f'No data reset hack test setup for {vnc_sockaddr}!\n'
        'See config setup tips @\n'
        'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
    )
    if not vnc_sockaddr:
        log.warning(
            no_setup_msg
            +
            'REQUIRES A `vnc_addrs: array` ENTRY'
        )
    vnc_host, vnc_port = vnc_sockaddr.get(
        api_port,
        ('localhost', 3003)
    )
    no_setup_msg:str = (
        f'No data reset hack test setup for {vnc_host}!\n'
        'See setup @\n'
        'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
    )
    global _reset_tech
    match _reset_tech:
@ -259,7 +250,7 @@ def i3ipc_xdotool_manual_click_hack() -> None:
                        timeout=timeout,
                    )
-        # re-activate and focus original window
+    # re-activate and focus original window
        subprocess.call([
            'xdotool',
            'windowactivate', '--sync', str(orig_win_id),
--- a/piker/brokers/ib/api.py
+++ b/piker/brokers/ib/api.py
--- a/piker/brokers/ib/broker.py
+++ b/piker/brokers/ib/broker.py
--- a/piker/brokers/ib/feed.py
+++ b/piker/brokers/ib/feed.py
@ -21,40 +21,33 @@ from __future__ import annotations
 import asyncio
 from contextlib import (
    asynccontextmanager as acm,
    nullcontext,
 )
 from decimal import Decimal
 from dataclasses import asdict
 from datetime import datetime
 from functools import partial
 from pprint import pformat
 from math import isnan
 import time
 from typing import (
    Any,
    Callable,
-    TYPE_CHECKING,
+    Awaitable,
 )
 from async_generator import aclosing
 from fuzzywuzzy import process as fuzzy
 import ib_insync as ibis
 import numpy as np
-from pendulum import (
+import pendulum
    now,
    from_timestamp,
    # DateTime,
    Duration,
    duration as mk_duration,
 )
 import tractor
 import trio
 from trio_typing import TaskStatus
-from piker.accounting import (
+from .._util import (
    MktPair,
 )
 from piker.data.validate import FeedInit
 from piker.brokers._util import (
    NoData,
    DataUnavailable,
    SymbolNotFound,
 )
 from .api import (
    # _adhoc_futes_set,
@ -70,10 +63,14 @@ from .api import (
    RequestError,
 )
 from ._util import data_reset_hack
-from .symbols import get_mkt_info
+from piker._cacheables import (
-
+    async_lifo_cache,
-if TYPE_CHECKING:
+)
-    from trio._core._run import Task
+from piker.accounting import (
    Asset,
    MktPair,
 )
 from piker.data.validate import FeedInit
 # XXX NOTE: See available types table docs:
@ -143,21 +140,32 @@ async def open_history_client(
    #   memory.
    # IB's internal symbology does not expect the "source asset" in
-    # the "symbol name", what we call the "market pair name". This is
+    # the "symbol name", what we call the "market name". This is
    # common in most legacy market brokers since it's presumed that
    # given a certain stock exchange, listed assets are traded
-    # "from" a particular source fiat, normally something like USD
+    # "from" a particular source fiat, normally something like USD.
-    # on the given venue-provider, eg. nasdaq, nyse, etc.
+    if (
-    fqme_kwargs: dict[str, Any] = {}
+        mkt.src
-    if mkt.dst.atype != 'fiat':
+        and mkt.src.atype == 'fiat'
-        fqme_kwargs  = {
+    ):
-            'without_src': True,  # default is True
+        fqme_kwargs: dict[str, Any] = {}
            'delim_char': '',  # bc they would normally use a frickin `.` smh
        }
-    fqme: str = mkt.get_bs_fqme(**(fqme_kwargs))
+        if mkt.dst.atype == 'forex':
            # XXX: for now we do need the src token kept in since
            fqme_kwargs  = {
                'without_src': False,  # default is True
                'delim_char': '',  # bc they would normally use a frickin `.` smh
            }
        fqme: str = mkt.get_bs_fqme(**(fqme_kwargs))
    else:
        fqme = mkt.bs_fqme
    async with open_data_client() as proxy:
        max_timeout: float = 2.
        mean: float = 0
        count: int = 0
@ -165,13 +173,12 @@ async def open_history_client(
        head_dt: None | datetime = None
        if (
            # fx cons seem to not provide this endpoint?
            # TODO: guard against all contract types which don't
            # support it?
            'idealpro' not in fqme
        ):
-            head_dt: datetime | None = await proxy.maybe_get_head_time(
+            try:
-                fqme=fqme
+                head_dt = await proxy.get_head_time(fqme=fqme)
-            )
+            except RequestError:
                head_dt = None
        async def get_hist(
            timeframe: float,
@ -179,15 +186,8 @@ async def open_history_client(
            start_dt: datetime | None = None,
        ) -> tuple[np.ndarray, str]:
            nonlocal max_timeout, mean, count
            if (
                start_dt
                and start_dt.timestamp() == 0
            ):
                await tractor.pause()
            query_start = time.time()
            out, timedout = await get_bars(
                proxy,
@ -208,48 +208,24 @@ async def open_history_client(
                    f'mean: {mean}'
                )
-            # could be trying to retreive bars over weekend
+            if (
-            if out is None:
+                out is None
            ):
                # could be trying to retreive bars over weekend
                log.error(f"Can't grab bars starting at {end_dt}!?!?")
-                if (
+                raise NoData(
-                    end_dt
+                    f'{end_dt}',
-                    and head_dt
+                    # frame_size=2000,
-                    and end_dt <= head_dt
+                )
                ):
                    raise DataUnavailable(
                        f'First timestamp is {head_dt}\n'
                        f'But {end_dt} was requested..'
                    )
-                else:
+            if (
-                    raise NoData(
+                end_dt
-                        info={
+                and head_dt
-                            'fqme': fqme,
+                and end_dt <= head_dt
-                            'head_dt': head_dt,
+            ):
-                            'start_dt': start_dt,
+                raise DataUnavailable(f'First timestamp is {head_dt}')
                            'end_dt': end_dt,
                            'timedout': timedout,
                        },
                    )
-            # also see return type for `get_bars()`
+            bars, bars_array, first_dt, last_dt = out
            bars: ibis.objects.BarDataList
            bars_array: np.ndarray
            first_dt: datetime
            last_dt: datetime
            (
                bars,
                bars_array,
                first_dt,
                last_dt,
            ) = out
            # TODO: audit the sampling period here as well?
            # timestep should always be at least as large as the
            # period step.
            # tdiff: np.ndarray = np.diff(bars_array['time'])
            # if (tdiff < timeframe).any():
            #     await tractor.pause()
            # volume cleaning since there's -ve entries,
            # wood luv to know what crookery that is..
@ -263,18 +239,7 @@ async def open_history_client(
        # quite sure why.. needs some tinkering and probably
        # a lookthrough of the ``ib_insync`` machinery, for eg. maybe
        # we have to do the batch queries on the `asyncio` side?
-        yield (
+        yield get_hist, {'erlangs': 1, 'rate': 3}
            get_hist,
            {
                'erlangs': 1,  # max conc reqs
                'rate': 3,  # max req rate
                'frame_types': {  # expected frame sizes
                    1: mk_duration(seconds=2e3),
                    60: mk_duration(days=2),
                }
            },
        )
 _pacing: str = (
@ -360,7 +325,7 @@ async def wait_on_data_reset(
    return False
-_data_resetter_task: Task | None = None
+_data_resetter_task: trio.Task | None = None
 _failed_resets: int = 0
@ -386,15 +351,7 @@ async def get_bars(
    task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
-) -> tuple[
+) -> (dict, np.ndarray):
    tuple[  # result tuple
        ibis.objects.BarDataList,
        np.ndarray,
        datetime,
        datetime,
    ] | None,
    bool,  # timed out hint
 ]:
    '''
    Retrieve historical data from a ``trio``-side task using
    a ``MethoProxy``.
@ -419,11 +376,7 @@ async def get_bars(
        while _failed_resets < max_failed_resets:
            try:
-                (
+                out = await proxy.bars(
                    bars,
                    bars_array,
                    dt_duration,
                ) = await proxy.bars(
                    fqme=fqme,
                    end_dt=end_dt,
                    sample_period_s=timeframe,
@ -434,58 +387,44 @@ async def get_bars(
                    # current impl) to detect a cancel case.
                    # timeout=timeout,
                )
-                # usually either a request during a venue closure
+                if out is None:
-                # or into a large (weekend) closure gap.
+                    raise NoData(f'{end_dt}')
-                if not bars:
+
-                    # no data returned?
+                bars, bars_array, dt_duration = out
                    log.warning(
                        'History frame is blank?\n'
                        f'start_dt: {start_dt}\n'
                        f'end_dt: {end_dt}\n'
                        f'duration: {dt_duration}\n'
                    )
                    # NOTE: REQUIRED to pass back value..
                    result = None
                    return None
                # not enough bars signal, likely due to venue
                # operational gaps.
-                if end_dt:
+                too_little: bool = False
-                    dur_s: float = len(bars) * timeframe
+                if (
-                    bars_dur = Duration(seconds=dur_s)
+                    end_dt
-                    dt_dur_s: float = dt_duration.in_seconds()
+                    and (
-                    if dur_s < dt_dur_s:
+                        not bars
-                        log.warning(
+                        or (too_little :=
-                            'History frame is shorter then expected?\n'
+                            start_dt
-                            f'start_dt: {start_dt}\n'
+                            and (len(bars) * timeframe)
-                            f'end_dt: {end_dt}\n'
+                                < dt_duration.in_seconds()
                            f'duration: {dt_dur_s}\n'
                            f'frame duration seconds: {dur_s}\n'
                            f'dur diff: {dt_duration - bars_dur}\n'
                        )
-                        # NOTE: we used to try to get a minimal
+                    )
-                        # set of bars by recursing but this ran
+                ):
-                        # into possible infinite query loops
+                    if (
-                        # when logic in the `Client.bars()` dt
+                        end_dt
-                        # diffing went bad. So instead for now
+                        or too_little
-                        # we just return the
+                    ):
-                        # shorter-then-expected history with
+                        log.warning(
-                        # a warning.
+                            f'History is blank for {dt_duration} from {end_dt}'
-                        # TODO: in the future it prolly makes
+                        )
-                        # the most send to do venue operating
+                        end_dt -= dt_duration
-                        # hours lookup and
+                        continue
                        # timestamp-in-operating-range set
                        # checking to know for sure if we can
                        # safely and quickly ignore non-uniform history
                        # frame timestamp gaps..
                        # end_dt -= dt_duration
                        # continue
                        # await tractor.pause()
-                first_dt = from_timestamp(
+                    raise NoData(f'{end_dt}')
                if bars_array is None:
                    raise SymbolNotFound(fqme)
                first_dt = pendulum.from_timestamp(
                    bars[0].date.timestamp())
-                last_dt = from_timestamp(
+                last_dt = pendulum.from_timestamp(
                    bars[-1].date.timestamp())
                time = bars_array['time']
@ -498,22 +437,15 @@ async def get_bars(
                if data_cs:
                    data_cs.cancel()
-                # NOTE: setting this is critical!
+                result = (bars, bars_array, first_dt, last_dt)
                result = (
                    bars,  # ib native
                    bars_array,  # numpy
                    first_dt,
                    last_dt,
                )
                # signal data reset loop parent task
                result_ready.set()
                # NOTE: this isn't getting collected anywhere!
                return result
            except RequestError as err:
-                msg: str = err.message
+                msg = err.message
                if 'No market data permissions for' in msg:
                    # TODO: signalling for no permissions searches
@ -535,7 +467,7 @@ async def get_bars(
                    if end_dt is not None:
                        end_dt = end_dt.subtract(days=1)
                    elif end_dt is None:
-                        end_dt = now().subtract(days=1)
+                        end_dt = pendulum.now().subtract(days=1)
                    log.warning(
                        f'NO DATA found ending @ {end_dt}\n'
@ -551,29 +483,21 @@ async def get_bars(
                    nodatas_count += 1
                    continue
-                elif (
+                elif 'API historical data query cancelled' in err.message:
                    'API historical data query cancelled'
                    in
                    err.message
                ):
                    log.warning(
                        'Query cancelled by IB (:eyeroll:):\n'
                        f'{err.message}'
                    )
                    continue
                elif (
                    'Trading TWS session is connected from a different IP'
-                    in
+                        in err.message
                    err.message
                ):
                    log.warning("ignoring ip address warning")
                    continue
                # XXX: more or less same as above timeout case
-                elif (
+                elif _pacing in msg:
                    _pacing in msg
                ):
                    log.warning(
                        'History throttle rate reached!\n'
                        'Resetting farms with `ctrl-alt-f` hack\n'
@ -625,10 +549,9 @@ async def get_bars(
                # don't double invoke the reset hack if another
                # requester task already has it covered.
                continue
            else:
                _data_resetter_task = trio.lowlevel.current_task()
-                unset_resetter: bool = True
+                unset_resetter = True
            # spawn new data reset task
            data_cs, reset_done = await nurse.start(
@ -641,16 +564,31 @@ async def get_bars(
            # sync wait on reset to complete
            await reset_done.wait()
-    _data_resetter_task = (
+    _data_resetter_task = None if unset_resetter else _data_resetter_task
        None
        if unset_resetter
        else _data_resetter_task
    )
    assert result
-    return (
+    return result, data_cs is not None
-        result,
+
-        data_cs is not None,
+
-    )
+# re-mapping to piker asset type names
 # https://github.com/erdewit/ib_insync/blob/master/ib_insync/contract.py#L113
 _asset_type_map = {
    'STK': 'stock',
    'OPT': 'option',
    'FUT': 'future',
    'CONTFUT': 'continuous_future',
    'CASH': 'forex',
    'IND': 'index',
    'CFD': 'cfd',
    'BOND': 'bond',
    'CMDTY': 'commodity',
    'FOP': 'futures_option',
    'FUND': 'mutual_fund',
    'WAR': 'warrant',
    'IOPT': 'warran',
    'BAG': 'bag',
    'CRYPTO': 'crypto',  # bc it's diff then fiat?
    # 'NEWS': 'news',
 }
 _quote_streams: dict[str, trio.abc.ReceiveStream] = {}
@ -671,8 +609,8 @@ async def _setup_quote_stream(
        # making them mostly useless and explains why the scanner
        # is always slow XD
        # '293',  # Trade count for day
-        # '294',  # Trade rate / minute
+        '294',  # Trade rate / minute
-        # '295',  # Vlm rate / minute
+        '295',  # Vlm rate / minute
    ),
    contract: Contract | None = None,
@ -856,6 +794,96 @@ def normalize(
    return data
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
    proxy: MethodProxy | None = None,
 ) -> tuple[MktPair, ibis.ContractDetails]:
    # XXX: we don't need to split off any fqme broker part?
    # bs_fqme, _, broker = fqme.partition('.')
    proxy: MethodProxy
    if proxy is not None:
        client_ctx = nullcontext(proxy)
    else:
        client_ctx = open_data_client
    async with client_ctx as proxy:
        try:
            (
                con,  # Contract
                details,  # ContractDetails
            ) = await proxy.get_sym_details(symbol=fqme)
        except ConnectionError:
            log.exception(f'Proxy is ded {proxy._aio_ns}')
            raise
    # TODO: more consistent field translation
    atype = _asset_type_map[con.secType]
    if atype == 'commodity':
        venue: str = 'cmdty'
    else:
        venue = con.primaryExchange or con.exchange
    price_tick: Decimal = Decimal(str(details.minTick))
    if atype == 'stock':
        # XXX: GRRRR they don't support fractional share sizes for
        # stocks from the API?!
        # if con.secType == 'STK':
        size_tick = Decimal('1')
    else:
        size_tick: Decimal = Decimal(str(details.minSize).rstrip('0'))
        # |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
    # NOTE: this is duplicate from the .broker.norm_trade_records()
    # routine, we should factor all this parsing somewhere..
    expiry_str = str(con.lastTradeDateOrContractMonth)
    # if expiry:
    #     expiry_str: str = str(pendulum.parse(
    #         str(expiry).strip(' ')
    #     ))
    # TODO: currently we can't pass the fiat src asset because
    # then we'll get a `MNQUSD` request for history data..
    # we need to figure out how we're going to handle this (later?)
    # but likely we want all backends to eventually handle
    # ``dst/src.venue.`` style !?
    src: str | Asset = ''
    if atype == 'forex':
        src = Asset(
            name=str(con.currency),
            atype='fiat',
            tx_tick=Decimal('0.01'),  # right?
        )
    mkt = MktPair(
        dst=Asset(
            name=con.symbol.lower(),
            atype=atype,
            tx_tick=size_tick,
        ),
        src=src,
        price_tick=price_tick,
        size_tick=size_tick,
        bs_mktid=str(con.conId),
        venue=str(venue),
        expiry=expiry_str,
        broker='ib',
        # TODO: options contract info as str?
        # contract_info=<optionsdetails>
    )
    return mkt, details
 async def stream_quotes(
    send_chan: trio.abc.SendChannel,
@ -883,10 +911,7 @@ async def stream_quotes(
    proxy: MethodProxy
    mkt: MktPair
    details: ibis.ContractDetails
-    async with (
+    async with open_data_client() as proxy:
        open_data_client() as proxy,
        # trio.open_nursery() as tn,
    ):
        mkt, details = await get_mkt_info(
            sym,
            proxy=proxy,  # passed to avoid implicit client load
@ -895,7 +920,7 @@ async def stream_quotes(
        init_msg = FeedInit(mkt_info=mkt)
        if mkt.dst.atype in {
-            'fiat',
+            'forex',
            'index',
            'commodity',
        }:
@ -906,50 +931,27 @@ async def stream_quotes(
        init_msgs.append(init_msg)
        con: Contract = details.contract
-        first_ticker: Ticker | None = None
+        first_ticker: Ticker = await proxy.get_quote(contract=con)
        first_quote: dict = normalize(first_ticker)
        log.warning(f'FIRST QUOTE: {first_quote}')
        # TODO: we should instead spawn a task that waits on a feed to start
        # and let it wait indefinitely..instead of this hard coded stuff.
        with trio.move_on_after(1):
-            first_ticker: Ticker = await proxy.get_quote(
+            first_ticker = await proxy.get_quote(contract=con)
                contract=con,
                raise_on_timeout=False,
            )
-        if first_ticker:
+        # it might be outside regular trading hours so see if we can at
-            first_quote: dict = normalize(first_ticker)
+        # least grab history.
            # TODO: we need a stack-oriented log levels filters for
            # this!
            # log.info(message, filter={'stack': 'live_feed'}) ?
            log.runtime(
                'Rxed init quote:\n\n'
                f'{pformat(first_quote)}\n'
            )
        # NOTE: it might be outside regular trading hours for
        # assets with "standard venue operating hours" so we
        # only "pretend the feed is live" when the dst asset
        # type is NOT within the NON-NORMAL-venue set: aka not
        # commodities, forex or crypto currencies which CAN
        # always return a NaN on a snap quote request during
        # normal venue hours. In the case of a closed venue
        # (equitiies, futes, bonds etc.) we at least try to
        # grab the OHLC history.
        if (
-            first_ticker
+            isnan(first_ticker.last)  # last quote price value is nan
            and
            isnan(first_ticker.last)
            # SO, if the last quote price value is NaN we ONLY
            # "pretend to do" `feed_is_live.set()` if it's a known
            # dst asset venue with a lot of closed operating hours.
            and mkt.dst.atype not in {
                'commodity',
-                'fiat',
+                'forex',
                'crypto',
            }
        ):
-            task_status.started((
+            task_status.started((init_msgs, first_quote))
                init_msgs,
                first_quote,
            ))
            # it's not really live but this will unblock
            # the brokerd feed task to tell the ui to update?
@ -959,28 +961,6 @@ async def stream_quotes(
            await trio.sleep_forever()
            return  # we never expect feed to come up?
        # TODO: we should instead spawn a task that waits on a feed
        # to start and let it wait indefinitely..instead of this
        # hard coded stuff.
        # async def wait_for_first_quote():
        #     with trio.CancelScope() as cs:
        # XXX: MUST acquire a ticker + first quote before starting
        # the live quotes loop!
        # with trio.move_on_after(1):
        first_ticker = await proxy.get_quote(
            contract=con,
            raise_on_timeout=True,
        )
        first_quote: dict = normalize(first_ticker)
        # TODO: we need a stack-oriented log levels filters for
        # this!
        # log.info(message, filter={'stack': 'live_feed'}) ?
        log.runtime(
            'Rxed init quote:\n'
            f'{pformat(first_quote)}'
        )
        cs: trio.CancelScope | None = None
        startup: bool = True
        while (
@ -1001,11 +981,8 @@ async def stream_quotes(
                    # only on first entry at feed boot up
                    if startup:
-                        startup: bool = False
+                        startup = False
-                        task_status.started((
+                        task_status.started((init_msgs, first_quote))
                            init_msgs,
                            first_quote,
                        ))
                    # start a stream restarter task which monitors the
                    # data feed event.
@ -1029,7 +1006,7 @@ async def stream_quotes(
                            # generally speaking these feeds don't
                            # include vlm data.
-                            atype: str = mkt.dst.atype
+                            atype = mkt.dst.atype
                            log.info(
                                f'No-vlm {mkt.fqme}@{atype}, skipping quote poll'
                            )
@ -1065,8 +1042,7 @@ async def stream_quotes(
                            quote = normalize(ticker)
                            log.debug(f"First ticker received {quote}")
-                        # tell data-layer spawner-caller that live
+                        # tell caller quotes are now coming in live
                        # quotes are now streaming.
                        feed_is_live.set()
                        # last = time.time()
@ -1078,3 +1054,141 @@ async def stream_quotes(
                            # ugh, clear ticks since we've consumed them
                            ticker.ticks = []
                            # last = time.time()
@tractor.context
 async def open_symbol_search(
    ctx: tractor.Context,
 ) -> None:
    # TODO: load user defined symbol set locally for fast search?
    await ctx.started({})
    async with (
        open_client_proxies() as (proxies, _),
        open_data_client() as data_proxy,
    ):
        async with ctx.open_stream() as stream:
            # select a non-history client for symbol search to lighten
            # the load in the main data node.
            proxy = data_proxy
            for name, proxy in proxies.items():
                if proxy is data_proxy:
                    continue
                break
            ib_client = proxy._aio_ns.ib
            log.info(f'Using {ib_client} for symbol search')
            last = time.time()
            async for pattern in stream:
                log.info(f'received {pattern}')
                now = time.time()
                # this causes tractor hang...
                # assert 0
                assert pattern, 'IB can not accept blank search pattern'
                # throttle search requests to no faster then 1Hz
                diff = now - last
                if diff < 1.0:
                    log.debug('throttle sleeping')
                    await trio.sleep(diff)
                    try:
                        pattern = stream.receive_nowait()
                    except trio.WouldBlock:
                        pass
                if (
                    not pattern
                    or pattern.isspace()
                    # XXX: not sure if this is a bad assumption but it
                    # seems to make search snappier?
                    or len(pattern) < 1
                ):
                    log.warning('empty pattern received, skipping..')
                    # TODO: *BUG* if nothing is returned here the client
                    # side will cache a null set result and not showing
                    # anything to the use on re-searches when this query
                    # timed out. We probably need a special "timeout" msg
                    # or something...
                    # XXX: this unblocks the far end search task which may
                    # hold up a multi-search nursery block
                    await stream.send({})
                    continue
                log.info(f'searching for {pattern}')
                last = time.time()
                # async batch search using api stocks endpoint and module
                # defined adhoc symbol set.
                stock_results = []
                async def stash_results(target: Awaitable[list]):
                    try:
                        results = await target
                    except tractor.trionics.Lagged:
                        print("IB SYM-SEARCH OVERRUN?!?")
                        return
                    stock_results.extend(results)
                for i in range(10):
                    with trio.move_on_after(3) as cs:
                        async with trio.open_nursery() as sn:
                            sn.start_soon(
                                stash_results,
                                proxy.search_symbols(
                                    pattern=pattern,
                                    upto=5,
                                ),
                            )
                            # trigger async request
                            await trio.sleep(0)
                    if cs.cancelled_caught:
                        log.warning(
                            f'Search timeout? {proxy._aio_ns.ib.client}'
                        )
                        continue
                    else:
                        break
                    # # match against our ad-hoc set immediately
                    # adhoc_matches = fuzzy.extractBests(
                    #     pattern,
                    #     list(_adhoc_futes_set),
                    #     score_cutoff=90,
                    # )
                    # log.info(f'fuzzy matched adhocs: {adhoc_matches}')
                    # adhoc_match_results = {}
                    # if adhoc_matches:
                    #     # TODO: do we need to pull contract details?
                    #     adhoc_match_results = {i[0]: {} for i in
                    #     adhoc_matches}
                log.debug(f'fuzzy matching stocks {stock_results}')
                stock_matches = fuzzy.extractBests(
                    pattern,
                    stock_results,
                    score_cutoff=50,
                )
                # matches = adhoc_match_results | {
                matches = {
                    item[0]: {} for item in stock_matches
                }
                # TODO: we used to deliver contract details
                # {item[2]: item[0] for item in stock_matches}
                log.debug(f"sending matches: {matches.keys()}")
                await stream.send(matches)
--- a/piker/brokers/ib/ledger.py
+++ b/piker/brokers/ib/ledger.py
@ -18,382 +18,159 @@
 Trade transaction accounting and normalization.
 '''
 from __future__ import annotations
 from bisect import insort
 from dataclasses import asdict
 from decimal import Decimal
 from functools import partial
 from pprint import pformat
 from typing import (
    Any,
    Callable,
    TYPE_CHECKING,
 )
 from bidict import bidict
-from pendulum import (
+import pendulum
    DateTime,
    parse,
    from_timestamp,
 )
 from ib_insync import (
    Contract,
    Commodity,
    Fill,
    Execution,
    CommissionReport,
 )
 from piker.types import Struct
 from piker.data import (
    SymbologyCache,
 )
 from piker.accounting import (
    Asset,
    dec_digits,
    digits_to_dec,
    Transaction,
    MktPair,
    iter_by_dt,
 )
 from ._flex_reports import parse_flex_dt
 from ._util import log
 if TYPE_CHECKING:
    from .api import (
        Client,
        MethodProxy,
    )
 tx_sort: Callable = partial(
    iter_by_dt,
    parsers={
        'dateTime': parse_flex_dt,
        'datetime': parse,
        # XXX: for some some fucking 2022 and
        # back options records.. f@#$ me..
        'date': parse,
    }
 )
 def norm_trade(
    tid: str,
    record: dict[str, Any],
    # this is the dict that was returned from
    # `Client.get_mkt_pairs()` and when running offline ledger
    # processing from `.accounting`, this will be the table loaded
    # into `SymbologyCache.pairs`.
    pairs: dict[str, Struct],
    symcache: SymbologyCache | None = None,
 ) -> Transaction | None:
    conid: int = str(record.get('conId') or record['conid'])
    bs_mktid: str = str(conid)
    # NOTE: sometimes weird records (like BTTX?)
    # have no field for this?
    comms: float = -1 * (
        record.get('commission')
        or record.get('ibCommission')
        or 0
    )
    if not comms:
        log.warning(
            'No commissions found for record?\n'
            f'{pformat(record)}\n'
        )
    price: float = (
        record.get('price')
        or record.get('tradePrice')
    )
    if price is None:
        log.warning(
            'No `price` field found in record?\n'
            'Skipping normalization..\n'
            f'{pformat(record)}\n'
        )
        return None
    # the api doesn't do the -/+ on the quantity for you but flex
    # records do.. are you fucking serious ib...!?
    size: float|int = (
        record.get('quantity')
        or record['shares']
    ) * {
        'BOT': 1,
        'SLD': -1,
    }[record['side']]
    symbol: str = record['symbol']
    exch: str = (
        record.get('listingExchange')
        or record.get('primaryExchange')
        or record['exchange']
    )
    # NOTE: remove null values since `tomlkit` can't serialize
    # them to file.
    if dnc := record.pop('deltaNeutralContract', None):
        record['deltaNeutralContract'] = dnc
    # likely an opts contract record from a flex report..
    # TODO: no idea how to parse ^ the strike part from flex..
    # (00010000 any, or 00007500 tsla, ..)
    # we probably must do the contract lookup for this?
    if (
        '   ' in symbol
        or '--' in exch
    ):
        underlying, _, tail = symbol.partition('   ')
        exch: str = 'opt'
        expiry: str = tail[:6]
        # otype = tail[6]
        # strike = tail[7:]
        log.warning(
            f'Skipping option contract -> NO SUPPORT YET!\n'
            f'{symbol}\n'
        )
        return None
    # timestamping is way different in API records
    dtstr: str = record.get('datetime')
    date: str = record.get('date')
    flex_dtstr: str = record.get('dateTime')
    if dtstr or date:
        dt: DateTime = parse(dtstr or date)
    elif flex_dtstr:
        # probably a flex record with a wonky non-std timestamp..
        dt: DateTime = parse_flex_dt(record['dateTime'])
    # special handling of symbol extraction from
    # flex records using some ad-hoc schema parsing.
    asset_type: str = (
        record.get('assetCategory')
        or record.get('secType')
        or 'STK'
    )
    if (expiry := (
            record.get('lastTradeDateOrContractMonth')
            or record.get('expiry')
        )
    ):
        expiry: str = str(expiry).strip(' ')
        # NOTE: we directly use the (simple and usually short)
        # date-string expiry token when packing the `MktPair`
        # since we want the fqme to contain *that* token.
        # It might make sense later to instead parse and then
        # render different output str format(s) for this same
        # purpose depending on asset-type-market down the road.
        # Eg. for derivs we use the short token only for fqme
        # but use the isoformat('T') for transactions and
        # account file position entries?
        # dt_str: str = pendulum.parse(expiry).isoformat('T')
    # XXX: pretty much all legacy market assets have a fiat
    # currency (denomination) determined by their venue.
    currency: str = record['currency']
    src = Asset(
        name=currency.lower(),
        atype='fiat',
        tx_tick=Decimal('0.01'),
    )
    match asset_type:
        case 'FUT':
            # XXX (flex) ledger entries don't necessarily have any
            # simple 3-char key.. sometimes the .symbol is some
            # weird internal key that we probably don't want in the
            # .fqme => we should probably just wrap `Contract` to
            # this like we do other crypto$ backends XD
            # NOTE: at least older FLEX records should have
            # this field.. no idea about API entries..
            local_symbol: str | None = record.get('localSymbol')
            underlying_key: str = record.get('underlyingSymbol')
            descr: str | None = record.get('description')
            if (
                not (
                    local_symbol
                    and symbol in local_symbol
                )
                and (
                        descr
                        and symbol not in descr
                    )
            ):
                con_key, exp_str = descr.split(' ')
                symbol: str = underlying_key or con_key
            dst = Asset(
                name=symbol.lower(),
                atype='future',
                tx_tick=Decimal('1'),
            )
        case 'STK':
            dst = Asset(
                name=symbol.lower(),
                atype='stock',
                tx_tick=Decimal('1'),
            )
        case 'CASH':
            if currency not in symbol:
                # likely a dict-casted `Forex` contract which
                # has .symbol as the dst and .currency as the
                # src.
                name: str = symbol.lower()
            else:
                # likely a flex-report record which puts
                # EUR.USD as the symbol field and just USD in
                # the currency field.
                name: str = symbol.lower().replace(f'.{src.name}', '')
            dst = Asset(
                name=name,
                atype='fiat',
                tx_tick=Decimal('0.01'),
            )
        case 'OPT':
            dst = Asset(
                name=symbol.lower(),
                atype='option',
                tx_tick=Decimal('1'),
                # TODO: we should probably always cast to the
                # `Contract` instance then dict-serialize that for
                # the `.info` field!
                # info=asdict(Option()),
            )
        case 'CMDTY':
            from .symbols import _adhoc_symbol_map
            con_kwargs, _ = _adhoc_symbol_map[symbol.upper()]
            dst = Asset(
                name=symbol.lower(),
                atype='commodity',
                tx_tick=Decimal('1'),
                info=asdict(Commodity(**con_kwargs)),
            )
    # try to build out piker fqme from record.
    # src: str = record['currency']
    price_tick: Decimal = digits_to_dec(dec_digits(price))
    # NOTE: can't serlialize `tomlkit.String` so cast to native
    atype: str = str(dst.atype)
    # if not (mkt := symcache.mktmaps.get(bs_mktid)):
    mkt = MktPair(
        bs_mktid=bs_mktid,
        dst=dst,
        price_tick=price_tick,
        # NOTE: for "legacy" assets, volume is normally discreet, not
        # a float, but we keep a digit in case the suitz decide
        # to get crazy and change it; we'll be kinda ready
        # schema-wise..
        size_tick=Decimal('1'),
        src=src,  # XXX: normally always a fiat
        _atype=atype,
        venue=exch,
        expiry=expiry,
        broker='ib',
        _fqme_without_src=(atype != 'fiat'),
    )
    fqme: str = mkt.fqme
    # XXX: if passed in, we fill out the symcache ad-hoc in order
    # to make downstream accounting work..
    if symcache is not None:
        orig_mkt: MktPair | None  = symcache.mktmaps.get(bs_mktid)
        if (
            orig_mkt
            and orig_mkt.fqme != mkt.fqme
        ):
            log.warning(
            # print(
                f'Contracts with common `conId`: {bs_mktid} mismatch..\n'
                f'{orig_mkt.fqme} -> {mkt.fqme}\n'
                # 'with DIFF:\n'
                # f'{mkt - orig_mkt}'
            )
        symcache.mktmaps[bs_mktid] = mkt
        symcache.mktmaps[fqme] = mkt
        symcache.assets[src.name] = src
        symcache.assets[dst.name] = dst
    # NOTE: for flex records the normal fields for defining an fqme
    # sometimes won't be available so we rely on two approaches for
    # the "reverse lookup" of piker style fqme keys:
    # - when dealing with API trade records received from
    #   `IB.trades()` we do a contract lookup at he time of processing
    # - when dealing with flex records, it is assumed the record
    #   is at least a day old and thus the TWS position reporting system
    #   should already have entries if the pps are still open, in
    #   which case, we can pull the fqme from that table (see
    #   `trades_dialogue()` above).
    return Transaction(
        fqme=fqme,
        tid=tid,
        size=size,
        price=price,
        cost=comms,
        dt=dt,
        expiry=expiry,
        bs_mktid=str(conid),
    )
 def norm_trade_records(
    ledger: dict[str, Any],
    symcache: SymbologyCache | None = None,
 ) -> dict[str, Transaction]:
    '''
-    Normalize (xml) flex-report or (recent) API trade records into
+    Normalize a flex report or API retrieved executions
-    our ledger format with parsing for `MktPair` and `Asset`
+    ledger into our standard record format.
    extraction to fill in the `Transaction.sys: MktPair` field.
    '''
    records: list[Transaction] = []
    for tid, record in ledger.items():
        conid = record.get('conId') or record['conid']
        comms = record.get('commission')
        if comms is None:
            comms = -1*record['ibCommission']
-        txn = norm_trade(
+        price = record.get('price') or record['tradePrice']
            tid,
            record,
-            # NOTE: currently no symcache support
+        # the api doesn't do the -/+ on the quantity for you but flex
-            pairs={},
+        # records do.. are you fucking serious ib...!?
-            symcache=symcache,
+        size = record.get('quantity') or record['shares'] * {
-        )
+            'BOT': 1,
            'SLD': -1,
        }[record['side']]
-        if txn is None:
+        exch = record['exchange']
        lexch = record.get('listingExchange')
        # NOTE: remove null values since `tomlkit` can't serialize
        # them to file.
        dnc = record.pop('deltaNeutralContract', False)
        if dnc is not None:
            record['deltaNeutralContract'] = dnc
        suffix = lexch or exch
        symbol = record['symbol']
        # likely an opts contract record from a flex report..
        # TODO: no idea how to parse ^ the strike part from flex..
        # (00010000 any, or 00007500 tsla, ..)
        # we probably must do the contract lookup for this?
        if '   ' in symbol or '--' in exch:
            underlying, _, tail = symbol.partition('   ')
            suffix = exch = 'opt'
            expiry = tail[:6]
            # otype = tail[6]
            # strike = tail[7:]
            print(f'skipping opts contract {symbol}')
            continue
-        # inject txns sorted by datetime
+        # timestamping is way different in API records
        dtstr = record.get('datetime')
        date = record.get('date')
        flex_dtstr = record.get('dateTime')
        if dtstr or date:
            dt = pendulum.parse(dtstr or date)
        elif flex_dtstr:
            # probably a flex record with a wonky non-std timestamp..
            dt = parse_flex_dt(record['dateTime'])
        # special handling of symbol extraction from
        # flex records using some ad-hoc schema parsing.
        asset_type: str = record.get(
            'assetCategory'
        ) or record.get('secType', 'STK')
        # TODO: XXX: WOA this is kinda hacky.. probably
        # should figure out the correct future pair key more
        # explicitly and consistently?
        if asset_type == 'FUT':
            # (flex) ledger entries don't have any simple 3-char key?
            symbol = record['symbol'][:3]
            asset_type: str = 'future'
        elif asset_type == 'STK':
            asset_type: str = 'stock'
        # try to build out piker fqme from record.
        expiry = (
            record.get('lastTradeDateOrContractMonth')
            or record.get('expiry')
        )
        if expiry:
            expiry = str(expiry).strip(' ')
            suffix = f'{exch}.{expiry}'
            expiry = pendulum.parse(expiry)
        # src: str = record['currency']
        price_tick: Decimal = digits_to_dec(dec_digits(price))
        pair = MktPair.from_fqme(
            fqme=f'{symbol}.{suffix}.ib',
            bs_mktid=str(conid),
            _atype=str(asset_type),  # XXX: can't serlialize `tomlkit.String`
            price_tick=price_tick,
            # NOTE: for "legacy" assets, volume is normally discreet, not
            # a float, but we keep a digit in case the suitz decide
            # to get crazy and change it; we'll be kinda ready
            # schema-wise..
            size_tick='1',
        )
        fqme = pair.fqme
        # NOTE: for flex records the normal fields for defining an fqme
        # sometimes won't be available so we rely on two approaches for
        # the "reverse lookup" of piker style fqme keys:
        # - when dealing with API trade records received from
        #   `IB.trades()` we do a contract lookup at he time of processing
        # - when dealing with flex records, it is assumed the record
        #   is at least a day old and thus the TWS position reporting system
        #   should already have entries if the pps are still open, in
        #   which case, we can pull the fqme from that table (see
        #   `trades_dialogue()` above).
        insort(
            records,
-            txn,
+            Transaction(
                fqme=fqme,
                sym=pair,
                tid=tid,
                size=size,
                price=price,
                cost=comms,
                dt=dt,
                expiry=expiry,
                bs_mktid=str(conid),
            ),
            key=lambda t: t.dt
        )
@ -402,49 +179,50 @@ def norm_trade_records(
 def api_trades_to_ledger_entries(
    accounts: bidict[str, str],
    fills: list[Fill],
-) -> dict[str, dict]:
+    # TODO: maybe we should just be passing through the
    # ``ib_insync.order.Trade`` instance directly here
    # instead of pre-casting to dicts?
    trade_entries: list[dict],
 ) -> dict:
    '''
-    Convert API execution objects entry objects into
+    Convert API execution objects entry objects into ``dict`` form,
-    flattened-``dict`` form, pretty much straight up without
+    pretty much straight up without modification except add
-    modification except add a `pydatetime` field from the parsed
+    a `pydatetime` field from the parsed timestamp.
    timestamp so that on write
    '''
-    trades_by_account: dict[str, dict] = {}
+    trades_by_account = {}
-    for fill in fills:
+    for t in trade_entries:
        # NOTE: example of schema we pull from the API client.
        # {
        #     'commissionReport': CommissionReport(...
        #     'contract': {...
        #     'execution': Execution(...
        #     'time': 1654801166.0
        # }
-        # NOTE: for the schema, see the defn for `Fill` which is
+        # flatten all sub-dicts and values into one top level entry.
-        # a `NamedTuple` subtype
+        entry = {}
-        fdict: dict = fill._asdict()
+        for section, val in t.items():
-
+            match section:
        # flatten all (sub-)objects and convert to dicts.
        # with values packed into one top level entry.
        val: CommissionReport | Execution | Contract
        txn_dict: dict[str, Any] = {}
        for attr_name, val in fdict.items():
            match attr_name:
                # value is a `@dataclass` subtype
                case 'contract' | 'execution' | 'commissionReport':
-                    txn_dict.update(asdict(val))
+                    # sub-dict cases
                    entry.update(val)
                case 'time':
                    # ib has wack ns timestamps, or is that us?
                    continue
                # TODO: we can remove this case right since there's
                # only 4 fields on a `Fill`?
                case _:
-                    txn_dict[attr_name] = val
+                    entry[section] = val
-        tid = str(txn_dict['execId'])
+        tid = str(entry['execId'])
-        dt = from_timestamp(txn_dict['time'])
+        dt = pendulum.from_timestamp(entry['time'])
-        txn_dict['datetime'] = str(dt)
+        # TODO: why isn't this showing seconds in the str?
-        acctid = accounts[txn_dict['acctNumber']]
+        entry['pydatetime'] = dt
-
+        entry['datetime'] = str(dt)
-        # NOTE: only inserted (then later popped) for sorting below!
+        acctid = accounts[entry['acctNumber']]
        txn_dict['pydatetime'] = dt
        if not tid:
            # this is likely some kind of internal adjustment
@ -455,18 +233,13 @@ def api_trades_to_ledger_entries(
            #   the user from the accounts window in TWS where they can
            #   manually set the avg price and size:
            #   https://api.ibkr.com/lib/cstools/faq/web1/index.html#/tag/DTWS_ADJ_AVG_COST
-            log.warning(
+            log.warning(f'Skipping ID-less ledger entry:\n{pformat(entry)}')
                'Skipping ID-less ledger txn_dict:\n'
                f'{pformat(txn_dict)}'
            )
            continue
        trades_by_account.setdefault(
            acctid, {}
-        )[tid] = txn_dict
+        )[tid] = entry
    # TODO: maybe we should just bisect.insort() into a list of
    # tuples and then return a dict of that?
    # sort entries in output by python based datetime
    for acctid in trades_by_account:
        trades_by_account[acctid] = dict(sorted(
@ -475,55 +248,3 @@ def api_trades_to_ledger_entries(
        ))
    return trades_by_account
 async def update_ledger_from_api_trades(
    fills: list[Fill],
    client: Client | MethodProxy,
    accounts_def_inv: bidict[str, str],
    # NOTE: provided for ad-hoc insertions "as transactions are
    # processed" -> see `norm_trade()` signature requirements.
    symcache: SymbologyCache | None = None,
 ) -> tuple[
    dict[str, Transaction],
    dict[str, dict],
 ]:
    # XXX; ERRGGG..
    # pack in the "primary/listing exchange" value from a
    # contract lookup since it seems this isn't available by
    # default from the `.fills()` method endpoint...
    fill: Fill
    for fill in fills:
        con: Contract = fill.contract
        conid: str = con.conId
        pexch: str | None = con.primaryExchange
        if not pexch:
            cons = await client.get_con(conid=conid)
            if cons:
                con = cons[0]
                pexch = con.primaryExchange or con.exchange
            else:
                # for futes it seems like the primary is always empty?
                pexch: str = con.exchange
        # pack in the ``Contract.secType``
        # entry['asset_type'] = condict['secType']
    entries: dict[str, dict] = api_trades_to_ledger_entries(
        accounts_def_inv,
        fills,
    )
    # normalize recent session's trades to the `Transaction` type
    trans_by_acct: dict[str, dict[str, Transaction]] = {}
    for acctid, trades_by_id in entries.items():
        # normalize to transaction form
        trans_by_acct[acctid] = norm_trade_records(
            trades_by_id,
            symcache=symcache,
        )
    return trans_by_acct, entries
--- a/piker/brokers/ib/symbols.py
+++ b/piker/brokers/ib/symbols.py
@ -1,615 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Symbology search and normalization.
 '''
 from __future__ import annotations
 from contextlib import (
    nullcontext,
 )
 from decimal import Decimal
 import time
 from typing import (
    Awaitable,
    TYPE_CHECKING,
 )
 from rapidfuzz import process as fuzzy
 import ib_insync as ibis
 import tractor
 import trio
 from piker.accounting import (
    Asset,
    MktPair,
    unpack_fqme,
 )
 from piker._cacheables import (
    async_lifo_cache,
 )
 from ._util import (
    log,
 )
 if TYPE_CHECKING:
    from .api import (
        MethodProxy,
        Client,
    )
 _futes_venues = (
    'GLOBEX',
    'NYMEX',
    'CME',
    'CMECRYPTO',
    'COMEX',
    # 'CMDTY',  # special name case..
    'CBOT',  # (treasury) yield futures
 )
 _adhoc_cmdty_set = {
    # metals
    # https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=69067924
    'xauusd.cmdty',  # london gold spot ^
    'xagusd.cmdty',  # silver spot
 }
 # NOTE: if you aren't seeing one of these symbol's futues contracts
 # show up, it's likely the `.<venue>` part is wrong!
 _adhoc_futes_set = {
    # equities
    'nq.cme',
    'mnq.cme',  # micro
    'es.cme',
    'mes.cme',  # micro
    # cypto$
    'brr.cme',
    'mbt.cme',  # micro
    'ethusdrr.cme',
    # agriculture
    'he.comex',  # lean hogs
    'le.comex',  # live cattle (geezers)
    'gf.comex',  # feeder cattle (younguns)
    # raw
    'lb.comex',  # random len lumber
    'gc.comex',
    'mgc.comex',  # micro
    # oil & gas
    'cl.nymex',
    'ni.comex',  # silver futes
    'qi.comex',  # mini-silver futes
    # treasury yields
    # etfs by duration:
    # SHY -> IEI -> IEF -> TLT
    'zt.cbot',  # 2y
    'z3n.cbot',  # 3y
    'zf.cbot',  # 5y
    'zn.cbot',  # 10y
    'zb.cbot',  # 30y
    # (micros of above)
    '2yy.cbot',
    '5yy.cbot',
    '10y.cbot',
    '30y.cbot',
 }
 # taken from list here:
 # https://www.interactivebrokers.com/en/trading/products-spot-currencies.php
 _adhoc_fiat_set = set((
    'USD, AED, AUD, CAD,'
    'CHF, CNH, CZK, DKK,'
    'EUR, GBP, HKD, HUF,'
    'ILS, JPY, MXN, NOK,'
    'NZD, PLN, RUB, SAR,'
    'SEK, SGD, TRY, ZAR'
    ).split(' ,')
 )
 # manually discovered tick discrepancies,
 # onl god knows how or why they'd cuck these up..
 _adhoc_mkt_infos: dict[int | str, dict] = {
    'vtgn.nasdaq': {'price_tick': Decimal('0.01')},
 }
 # map of symbols to contract ids
 _adhoc_symbol_map = {
    # https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=69067924
    # NOTE: some cmdtys/metals don't have trade data like gold/usd:
    # https://groups.io/g/twsapi/message/44174
    'XAUUSD': ({'conId': 69067924}, {'whatToShow': 'MIDPOINT'}),
 }
 for qsn in _adhoc_futes_set:
    sym, venue = qsn.split('.')
    assert venue.upper() in _futes_venues, f'{venue}'
    _adhoc_symbol_map[sym.upper()] = (
        {'exchange': venue},
        {},
    )
 # exchanges we don't support at the moment due to not knowing
 # how to do symbol-contract lookup correctly likely due
 # to not having the data feeds subscribed.
 _exch_skip_list = {
    'ASX',  # aussie stocks
    'MEXI',  # mexican stocks
    # no idea
    'NSE',
    'VALUE',
    'FUNDSERV',
    'SWB2',
    'PSE',
    'PHLX',
 }
 # optional search config the backend can register for
 # it's symbol search handling (in this case we avoid
 # accepting patterns before the kb has settled more then
 # a quarter second).
 _search_conf = {
    'pause_period': 6 / 16,
 }
@tractor.context
 async def open_symbol_search(ctx: tractor.Context) -> None:
    '''
    Symbology search brokerd-endpoint.
    '''
    from .api import open_client_proxies
    from .feed import open_data_client
    # TODO: load user defined symbol set locally for fast search?
    await ctx.started({})
    async with (
        open_client_proxies() as (proxies, _),
        open_data_client() as data_proxy,
    ):
        async with ctx.open_stream() as stream:
            # select a non-history client for symbol search to lighten
            # the load in the main data node.
            proxy = data_proxy
            for name, proxy in proxies.items():
                if proxy is data_proxy:
                    continue
                break
            ib_client = proxy._aio_ns.ib
            log.info(
                f'Using API client for symbol-search\n'
                f'{ib_client}\n'
            )
            last = time.time()
            async for pattern in stream:
                log.info(f'received {pattern}')
                now: float = time.time()
                # this causes tractor hang...
                # assert 0
                assert pattern, 'IB can not accept blank search pattern'
                # throttle search requests to no faster then 1Hz
                diff = now - last
                if diff < 1.0:
                    log.debug('throttle sleeping')
                    await trio.sleep(diff)
                    try:
                        pattern = stream.receive_nowait()
                    except trio.WouldBlock:
                        pass
                if (
                    not pattern
                    or pattern.isspace()
                    # XXX: not sure if this is a bad assumption but it
                    # seems to make search snappier?
                    or len(pattern) < 1
                ):
                    log.warning('empty pattern received, skipping..')
                    # TODO: *BUG* if nothing is returned here the client
                    # side will cache a null set result and not showing
                    # anything to the use on re-searches when this query
                    # timed out. We probably need a special "timeout" msg
                    # or something...
                    # XXX: this unblocks the far end search task which may
                    # hold up a multi-search nursery block
                    await stream.send({})
                    continue
                log.info(f'searching for {pattern}')
                last = time.time()
                # async batch search using api stocks endpoint and module
                # defined adhoc symbol set.
                stock_results = []
                async def extend_results(
                    target: Awaitable[list]
                ) -> None:
                    try:
                        results = await target
                    except tractor.trionics.Lagged:
                        print("IB SYM-SEARCH OVERRUN?!?")
                        return
                    stock_results.extend(results)
                for _ in range(10):
                    with trio.move_on_after(3) as cs:
                        async with trio.open_nursery() as sn:
                            sn.start_soon(
                                extend_results,
                                proxy.search_symbols(
                                    pattern=pattern,
                                    upto=5,
                                ),
                            )
                            # trigger async request
                            await trio.sleep(0)
                    if cs.cancelled_caught:
                        log.warning(
                            f'Search timeout? {proxy._aio_ns.ib.client}'
                        )
                        continue
                    elif stock_results:
                        break
                    # else:
                    # await tractor.pause()
                    # # match against our ad-hoc set immediately
                    # adhoc_matches = fuzzy.extract(
                    #     pattern,
                    #     list(_adhoc_futes_set),
                    #     score_cutoff=90,
                    # )
                    # log.info(f'fuzzy matched adhocs: {adhoc_matches}')
                    # adhoc_match_results = {}
                    # if adhoc_matches:
                    #     # TODO: do we need to pull contract details?
                    #     adhoc_match_results = {i[0]: {} for i in
                    #     adhoc_matches}
                log.debug(f'fuzzy matching stocks {stock_results}')
                stock_matches = fuzzy.extract(
                    pattern,
                    stock_results,
                    score_cutoff=50,
                )
                # matches = adhoc_match_results | {
                matches = {
                    item[0]: {} for item in stock_matches
                }
                # TODO: we used to deliver contract details
                # {item[2]: item[0] for item in stock_matches}
                log.debug(f"sending matches: {matches.keys()}")
                await stream.send(matches)
 # re-mapping to piker asset type names
 # https://github.com/erdewit/ib_insync/blob/master/ib_insync/contract.py#L113
 _asset_type_map = {
    'STK': 'stock',
    'OPT': 'option',
    'FUT': 'future',
    'CONTFUT': 'continuous_future',
    'CASH': 'fiat',
    'IND': 'index',
    'CFD': 'cfd',
    'BOND': 'bond',
    'CMDTY': 'commodity',
    'FOP': 'futures_option',
    'FUND': 'mutual_fund',
    'WAR': 'warrant',
    'IOPT': 'warran',
    'BAG': 'bag',
    'CRYPTO': 'crypto',  # bc it's diff then fiat?
    # 'NEWS': 'news',
 }
 def parse_patt2fqme(
    # client: Client,
    pattern: str,
 ) -> tuple[str, str, str, str]:
    # TODO: we can't use this currently because
    # ``wrapper.starTicker()`` currently cashes ticker instances
    # which means getting a singel quote will potentially look up
    # a quote for a ticker that it already streaming and thus run
    # into state clobbering (eg. list: Ticker.ticks). It probably
    # makes sense to try this once we get the pub-sub working on
    # individual symbols...
    # XXX UPDATE: we can probably do the tick/trades scraping
    # inside our eventkit handler instead to bypass this entirely?
    currency = ''
    # fqme parsing stage
    # ------------------
    if '.ib' in pattern:
        _, symbol, venue, expiry = unpack_fqme(pattern)
    else:
        symbol = pattern
        expiry = ''
        # # another hack for forex pairs lul.
        # if (
        #     '.idealpro' in symbol
        #     # or '/' in symbol
        # ):
        #     exch: str = 'IDEALPRO'
        #     symbol = symbol.removesuffix('.idealpro')
        #     if '/' in symbol:
        #         symbol, currency = symbol.split('/')
        # else:
        # TODO: yes, a cache..
        # try:
        #     # give the cache a go
        #     return client._contracts[symbol]
        # except KeyError:
        #     log.debug(f'Looking up contract for {symbol}')
        expiry: str = ''
        if symbol.count('.') > 1:
            symbol, _, expiry = symbol.rpartition('.')
        # use heuristics to figure out contract "type"
        symbol, venue = symbol.upper().rsplit('.', maxsplit=1)
    return symbol, currency, venue, expiry
 def con2fqme(
    con: ibis.Contract,
    _cache: dict[int, (str, bool)] = {}
 ) -> tuple[str, bool]:
    '''
    Convert contracts to fqme-style strings to be used both in
    symbol-search matching and as feed tokens passed to the front
    end data deed layer.
    Previously seen contracts are cached by id.
    '''
    # should be real volume for this contract by default
    calc_price: bool = False
    if con.conId:
        try:
            # TODO: LOL so apparently IB just changes the contract
            # ID (int) on a whim.. so we probably need to use an
            # FQME style key after all...
            return _cache[con.conId]
        except KeyError:
            pass
    suffix: str = con.primaryExchange or con.exchange
    symbol: str = con.symbol
    expiry: str = con.lastTradeDateOrContractMonth or ''
    match con:
        case ibis.Option():
            # TODO: option symbol parsing and sane display:
            symbol = con.localSymbol.replace(' ', '')
        case (
            ibis.Commodity()
            # search API endpoint returns std con box..
            | ibis.Contract(secType='CMDTY')
        ):
            # commodities and forex don't have an exchange name and
            # no real volume so we have to calculate the price
            suffix = con.secType
            # no real volume on this tract
            calc_price = True
        case ibis.Forex() | ibis.Contract(secType='CASH'):
            dst, src = con.localSymbol.split('.')
            symbol = ''.join([dst, src])
            suffix = con.exchange or 'idealpro'
            # no real volume on forex feeds..
            calc_price = True
    if not suffix:
        entry = _adhoc_symbol_map.get(
            con.symbol or con.localSymbol
        )
        if entry:
            meta, kwargs = entry
            cid = meta.get('conId')
            if cid:
                assert con.conId == meta['conId']
            suffix = meta['exchange']
    # append a `.<suffix>` to the returned symbol
    # key for derivatives that normally is the expiry
    # date key.
    if expiry:
        suffix += f'.{expiry}'
    fqme_key = symbol.lower()
    if suffix:
        fqme_key = '.'.join((fqme_key, suffix)).lower()
    _cache[con.conId] = fqme_key, calc_price
    return fqme_key, calc_price
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
    proxy: MethodProxy | None = None,
 ) -> tuple[MktPair, ibis.ContractDetails]:
    if '.ib' not in fqme:
        fqme += '.ib'
    broker, pair, venue, expiry = unpack_fqme(fqme)
    proxy: MethodProxy
    if proxy is not None:
        client_ctx = nullcontext(proxy)
    else:
        from .feed import (
            open_data_client,
        )
        client_ctx = open_data_client
    async with client_ctx as proxy:
        try:
            (
                con,  # Contract
                details,  # ContractDetails
            ) = await proxy.get_sym_details(fqme=fqme)
        except ConnectionError:
            log.exception(f'Proxy is ded {proxy._aio_ns}')
            raise
    # TODO: more consistent field translation
    atype = _asset_type_map[con.secType]
    if atype == 'commodity':
        venue: str = 'cmdty'
    else:
        venue = con.primaryExchange or con.exchange
    price_tick: Decimal = Decimal(str(details.minTick))
    ib_min_tick_gt_2: Decimal = Decimal('0.01')
    if (
        price_tick < ib_min_tick_gt_2
    ):
        # TODO: we need to add some kinda dynamic rounding sys
        # to our MktPair i guess?
        # not sure where the logic should sit, but likely inside
        # the `.clearing._ems` i suppose...
        log.warning(
            'IB seems to disallow a min price tick < 0.01 '
            'when the price is > 2.0..?\n'
            f'Decreasing min tick precision for {fqme} to 0.01'
        )
        # price_tick = ib_min_tick
        # await tractor.pause()
    if atype == 'stock':
        # XXX: GRRRR they don't support fractional share sizes for
        # stocks from the API?!
        # if con.secType == 'STK':
        size_tick = Decimal('1')
    else:
        size_tick: Decimal = Decimal(
            str(details.minSize).rstrip('0')
        )
        # |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
    # NOTE: this is duplicate from the .broker.norm_trade_records()
    # routine, we should factor all this parsing somewhere..
    expiry_str = str(con.lastTradeDateOrContractMonth)
    # if expiry:
    #     expiry_str: str = str(pendulum.parse(
    #         str(expiry).strip(' ')
    #     ))
    # TODO: currently we can't pass the fiat src asset because
    # then we'll get a `MNQUSD` request for history data..
    # we need to figure out how we're going to handle this (later?)
    # but likely we want all backends to eventually handle
    # ``dst/src.venue.`` style !?
    src = Asset(
        name=str(con.currency).lower(),
        atype='fiat',
        tx_tick=Decimal('0.01'),  # right?
    )
    dst = Asset(
        name=con.symbol.lower(),
        atype=atype,
        tx_tick=size_tick,
    )
    mkt = MktPair(
        src=src,
        dst=dst,
        price_tick=price_tick,
        size_tick=size_tick,
        bs_mktid=str(con.conId),
        venue=str(venue),
        expiry=expiry_str,
        broker='ib',
        # TODO: options contract info as str?
        # contract_info=<optionsdetails>
        _fqme_without_src=(atype != 'fiat'),
    )
    # just.. wow.
    if entry := _adhoc_mkt_infos.get(mkt.bs_fqme):
        log.warning(f'Frickin {mkt.fqme} has an adhoc {entry}..')
        new = mkt.to_dict()
        new['price_tick'] = entry['price_tick']
        new['src'] = src
        new['dst'] = dst
        mkt = MktPair(**new)
    # if possible register the bs_mktid to the just-built
    # mkt so that it can be retreived by order mode tasks later.
    # TODO NOTE: this is going to be problematic if/when we split
    # out the datatd vs. brokerd actors since the mktmap lookup
    # table will now be inaccessible..
    if proxy is not None:
        client: Client = proxy._aio_ns
        client._contracts[mkt.bs_fqme] = con
        client._cons2mkts[con] = mkt
    return mkt, details
--- a/piker/brokers/kraken/init.py
+++ b/piker/brokers/kraken/init.py
@ -19,36 +19,23 @@ Kraken backend.
 Sub-modules within break into the core functionalities:
- .api: for the core API machinery which generally
+- ``broker.py`` part for orders / trading endpoints
-        a ``asks``/``trio-websocket`` implemented ``Client``.
+- ``feed.py`` for real-time data feed endpoints
- .broker: part for orders / trading endpoints.
+- ``api.py`` for the core API machinery which is ``trio``-ized
- .feed: for real-time and historical data query endpoints.
+  wrapping around ``ib_insync``.
 - .ledger: for transaction processing as it pertains to accounting.
 - .symbols: for market (name) search and symbology meta-defs.
 '''
 from .symbols import (
    Pair,  # for symcache
    open_symbol_search,
    # required by `.accounting`, `.data`
    get_mkt_info,
 )
 # required by `.brokers`
 from .api import (
    get_client,
 )
 from .feed import (
-    # required by `.data`
+    get_mkt_info,
    stream_quotes,
    open_history_client,
    open_symbol_search,
    stream_quotes,
 )
 from .broker import (
    # required by `.clearing`
    open_trade_dialog,
 )
 from .ledger import (
    # required by `.accounting`
    norm_trade,
    norm_trade_records,
 )
@ -56,20 +43,17 @@ from .ledger import (
 __all__ = [
    'get_client',
    'get_mkt_info',
    'Pair',
    'open_trade_dialog',
    'open_history_client',
    'open_symbol_search',
    'stream_quotes',
    'norm_trade_records',
    'norm_trade',
 ]
 # tractor RPC enable arg
 __enable_modules__: list[str] = [
    'api',
    'broker',
    'feed',
-    'symbols',
+    'broker',
 ]
--- a/piker/brokers/kraken/api.py
+++ b/piker/brokers/kraken/api.py
@ -15,11 +15,12 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-Core (web) API client
+Kraken web API wrapping.
 '''
 from contextlib import asynccontextmanager as acm
 from datetime import datetime
 from decimal import Decimal
 import itertools
 from typing import (
    Any,
@ -27,8 +28,10 @@ from typing import (
 )
 import time
-import httpx
+from bidict import bidict
 import pendulum
 import asks
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 import urllib.parse
 import hashlib
@ -37,14 +40,11 @@ import base64
 import trio
 from piker import config
-from piker.data import (
+from piker.data.types import Struct
-    def_iohlcv_fields,
+from piker.data import def_iohlcv_fields
    match_from_pairs,
 )
 from piker.accounting._mktinfo import (
    Asset,
    digits_to_dec,
    dec_digits,
 )
 from piker.brokers._util import (
    resproc,
@ -54,17 +54,11 @@ from piker.brokers._util import (
 )
 from piker.accounting import Transaction
 from piker.log import get_logger
 from .symbols import Pair
 log = get_logger('piker.brokers.kraken')
 # <uri>/<version>/
 _url = 'https://api.kraken.com/0'
 _headers: dict[str, str] = {
    'User-Agent': 'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
 }
 # TODO: this is the only backend providing this right?
 # in which case we should drop it from the defaults and
 # instead make a custom fields descr in this module!
@ -75,18 +69,12 @@ _symbol_info_translation: dict[str, str] = {
 def get_config() -> dict[str, Any]:
    '''
    Load our section from `piker/brokers.toml`.
-    '''
+    conf, path = config.load()
-    conf, path = config.load(
+    section = conf.get('kraken')
-        conf_name='brokers',
+
-        touch_if_dne=True,
+    if section is None:
-    )
+        log.warning(f'No config section found for kraken in {path}')
    if (section := conf.get('kraken')) is None:
        log.warning(
            f'No config section found for kraken in {path}'
        )
        return {}
    return section
@ -117,51 +105,96 @@ class InvalidKey(ValueError):
    '''
 # https://www.kraken.com/features/api#get-tradable-pairs
 class Pair(Struct):
    altname: str  # alternate pair name
    wsname: str  # WebSocket pair name (if available)
    aclass_base: str  # asset class of base component
    base: str  # asset id of base component
    aclass_quote: str  # asset class of quote component
    quote: str  # asset id of quote component
    lot: str  # volume lot size
    cost_decimals: int
    costmin: float
    pair_decimals: int  # scaling decimal places for pair
    lot_decimals: int  # scaling decimal places for volume
    # amount to multiply lot volume by to get currency volume
    lot_multiplier: float
    # array of leverage amounts available when buying
    leverage_buy: list[int]
    # array of leverage amounts available when selling
    leverage_sell: list[int]
    # fee schedule array in [volume, percent fee] tuples
    fees: list[tuple[int, float]]
    # maker fee schedule array in [volume, percent fee] tuples (if on
    # maker/taker)
    fees_maker: list[tuple[int, float]]
    fee_volume_currency: str  # volume discount currency
    margin_call: str  # margin call level
    margin_stop: str  # stop-out/liquidation margin level
    ordermin: float  # minimum order volume for pair
    tick_size: float  # min price step size
    status: str
    short_position_limit: float = 0
    long_position_limit: float = float('inf')
    @property
    def price_tick(self) -> Decimal:
        return digits_to_dec(self.pair_decimals)
    @property
    def size_tick(self) -> Decimal:
        return digits_to_dec(self.lot_decimals)
    @property
    def bs_fqme(self) -> str:
        return f'{self.symbol}.SPOT'
 class Client:
-    # assets and mkt pairs are key-ed by kraken's ReST response
+    # symbol mapping from all names to the altname
-    # symbol-bs_mktids (we call them "X-keys" like fricking
+    _ntable: dict[str, str] = {}
    # "XXMRZEUR"). these keys used directly since ledger endpoints
    # return transaction sets keyed with the same set!
    _Assets: dict[str, Asset] = {}
    _AssetPairs: dict[str, Pair] = {}
-    # offer lookup tables for all .altname and .wsname
+    # 2-way map of symbol names to their "alt names" ffs XD
-    # to the equivalent .xname so that various symbol-schemas
+    _altnames: bidict[str, str] = bidict()
    # can be mapped to `Pair`s in the tables above.
    _altnames: dict[str, str] = {}
    _wsnames: dict[str, str] = {}
    # key-ed by `Pair.bs_fqme: str`, and thus used for search
    # allowing for lookup using piker's own FQME symbology sys.
    _pairs: dict[str, Pair] = {}
    _assets: dict[str, Asset] = {}
    def __init__(
        self,
        config: dict[str, str],
        httpx_client: httpx.AsyncClient,
        name: str = '',
        api_key: str = '',
        secret: str = ''
    ) -> None:
-
+        self._sesh = asks.Session(connections=4)
-        self._sesh: httpx.AsyncClient = httpx_client
+        self._sesh.base_location = _url
-
+        self._sesh.headers.update({
            'User-Agent':
                'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
        })
        self._name = name
        self._api_key = api_key
        self._secret = secret
        self.conf: dict[str, str] = config
        self.assets: dict[str, Asset] = {}
    @property
    def pairs(self) -> dict[str, Pair]:
        if self._pairs is None:
            raise RuntimeError(
-                "Client didn't run `.get_mkt_pairs()` on startup?!"
+                "Make sure to run `cache_symbols()` on startup!"
            )
            # retreive and cache all symbols
        return self._pairs
@ -170,9 +203,10 @@ class Client:
        method: str,
        data: dict,
    ) -> dict[str, Any]:
-        resp: httpx.Response = await self._sesh.post(
+        resp = await self._sesh.post(
-            url=f'/public/{method}',
+            path=f'/public/{method}',
            json=data,
            timeout=float('inf')
        )
        return resproc(resp, log)
@ -183,18 +217,18 @@ class Client:
        uri_path: str
    ) -> dict[str, Any]:
        headers = {
-            'Content-Type': 'application/x-www-form-urlencoded',
+            'Content-Type':
-            'API-Key': self._api_key,
+                'application/x-www-form-urlencoded',
-            'API-Sign': get_kraken_signature(
+            'API-Key':
-                uri_path,
+                self._api_key,
-                data,
+            'API-Sign':
-                self._secret,
+                get_kraken_signature(uri_path, data, self._secret)
            ),
        }
-        resp: httpx.Response = await self._sesh.post(
+        resp = await self._sesh.post(
-            url=f'/private/{method}',
+            path=f'/private/{method}',
            data=data,
            headers=headers,
            timeout=float('inf')
        )
        return resproc(resp, log)
@ -220,29 +254,17 @@ class Client:
            'Balance',
            {},
        )
-        by_bsmktid: dict[str, dict] = resp['result']
+        by_bsmktid = resp['result']
-        balances: dict = {}
+        # TODO: we need to pull out the "asset" decimals
-        for xname, bal in by_bsmktid.items():
+        # data and return a `decimal.Decimal` instead here!
-            asset: Asset = self._Assets[xname]
+        # using the underlying Asset
        return {
            self._altnames[sym].lower(): float(bal)
            for sym, bal in by_bsmktid.items()
        }
-            # TODO: which KEY should we use? it's used to index
+    async def get_assets(self) -> dict[str, Asset]:
            # the `Account.pps: dict` ..
            key: str = asset.name.lower()
            # TODO: should we just return a `Decimal` here
            # or is the rounded version ok?
            balances[key] = round(
                float(bal),
                ndigits=dec_digits(asset.tx_tick)
            )
        return balances
    async def get_assets(
        self,
        reload: bool = False,
    ) -> dict[str, Asset]:
        '''
        Load and cache all asset infos and pack into
        our native ``Asset`` struct.
@ -260,37 +282,21 @@ class Client:
            }
        '''
-        if (
+        resp = await self._public('Assets', {})
-            not self._assets
+        assets = resp['result']
            or reload
        ):
            resp = await self._public('Assets', {})
            assets: dict[str, dict] = resp['result']
-            for bs_mktid, info in assets.items():
+        for bs_mktid, info in assets.items():
            altname = self._altnames[bs_mktid] = info['altname']
            aclass: str = info['aclass']
-                altname: str = info['altname']
+            self.assets[bs_mktid] = Asset(
-                aclass: str = info['aclass']
+                name=altname.lower(),
-                asset = Asset(
+                atype=f'crypto_{aclass}',
-                    name=altname,
+                tx_tick=digits_to_dec(info['decimals']),
-                    atype=f'crypto_{aclass}',
+                info=info,
-                    tx_tick=digits_to_dec(info['decimals']),
+            )
                    info=info,
                )
                # NOTE: yes we keep 2 sets since kraken insists on
                # keeping 3 frickin sets bc apparently they have
                # no sane data engineers whol all like different
                # keys for their fricking symbology sets..
                self._Assets[bs_mktid] = asset
                self._assets[altname.lower()] = asset
                self._assets[altname] = asset
-        # we return the "most native" set merged with our preferred
+        return self.assets
        # naming (which i guess is the "altname" one) since that's
        # what the symcache loader will be storing, and we need the
        # keys that are easiest to match against in any trade
        # records.
        return self._Assets | self._assets
    async def get_trades(
        self,
@ -371,25 +377,23 @@ class Client:
        #     'amount': '0.00300726', 'fee': '0.00001000', 'time':
        #     1658347714, 'status': 'Success'}]}
        if xfers:
            import tractor
            await tractor.pp()
        trans: dict[str, Transaction] = {}
        for entry in xfers:
            # look up the normalized name and asset info
-            asset_key: str = entry['asset']
+            asset_key = entry['asset']
-            asset: Asset = self._Assets[asset_key]
+            asset = self.assets[asset_key]
-            asset_key: str = asset.name.lower()
+            asset_key = self._altnames[asset_key].lower()
            # XXX: this is in the asset units (likely) so it isn't
            # quite the same as a commisions cost necessarily..)
            # TODO: also round this based on `Pair` cost precision info?
            cost = float(entry['fee'])
-            # fqme: str = asset_key + '.kraken'
+
            fqme = asset_key + '.kraken'
            tx = Transaction(
-                fqme=asset_key,  # this must map to an entry in .assets!
+                fqme=fqme,
                sym=asset,
                tid=entry['txid'],
                dt=pendulum.from_timestamp(entry['time']),
                bs_mktid=f'{asset_key}{src_asset}',
@ -404,11 +408,6 @@ class Client:
                # XXX: see note above
                cost=cost,
                # not a trade but a withdrawal or deposit on the
                # asset (chain) system.
                etype='transfer',
            )
            trans[tx.tid] = tx
@ -459,7 +458,7 @@ class Client:
        # txid is a transaction id given by kraken
        return await self.endpoint('CancelOrder', {"txid": reqid})
-    async def asset_pairs(
+    async def pair_info(
        self,
        pair_patt: str | None = None,
@ -471,76 +470,64 @@ class Client:
        https://docs.kraken.com/rest/#tag/Market-Data/operation/getTradableAssetPairs
        '''
-        if not self._AssetPairs:
+        # get all pairs by default, or filter
-            # get all pairs by default, or filter
+        # to whatever pattern is provided as input.
-            # to whatever pattern is provided as input.
+        pairs: dict[str, str] | None = None
-            req_pairs: dict[str, str] | None = None
+        if pair_patt is not None:
-            if pair_patt is not None:
+            pairs = {'pair': pair_patt}
                req_pairs = {'pair': pair_patt}
-            resp = await self._public(
+        resp = await self._public(
-                'AssetPairs',
+            'AssetPairs',
-                req_pairs,
+            pairs,
-            )
+        )
-            err = resp['error']
+        err = resp['error']
-            if err:
+        if err:
-                raise SymbolNotFound(pair_patt)
+            raise SymbolNotFound(pair_patt)
-            # NOTE: we try to key pairs by our custom defined
+        pairs: dict[str, Pair] = {
            # `.bs_fqme` field since we want to offer search over
            # this pattern set, callers should fill out lookup
            # tables for kraken's bs_mktid keys to map to these
            # keys!
            # XXX: FURTHER kraken's data eng team decided to offer
            # 3 frickin market-pair-symbol key sets depending on
            # which frickin API is being used.
            # Example for the trading pair 'LTC<EUR'
            # - the "X-key" from rest eps 'XLTCZEUR'
            # - the "websocket key" from ws msgs is 'LTC/EUR'
            # - the "altname key" also delivered in pair info is 'LTCEUR'
            for xkey, data in resp['result'].items():
-                # NOTE: always cache in pairs tables for faster lookup
+            key: Pair(**data)
-                pair = Pair(xname=xkey, **data)
+            for key, data in resp['result'].items()
-
+        }
-                # register the above `Pair` structs for all
+        # always cache so we can possibly do faster lookup
-                # key-sets/monikers: a set of 4 (frickin) tables
+        self._pairs.update(pairs)
                # acting as a combined surjection of all possible
                # (and stupid) kraken names to their `Pair` obj.
                self._AssetPairs[xkey] = pair
                self._pairs[pair.bs_fqme] = pair
                self._altnames[pair.altname] = pair
                self._wsnames[pair.wsname] = pair
        if pair_patt is not None:
-            return next(iter(self._pairs.items()))[1]
+            return next(iter(pairs.items()))[1]
-        return self._AssetPairs
+        return pairs
-    async def get_mkt_pairs(
+    async def cache_symbols(self) -> dict:
        self,
        reload: bool = False,
    ) -> dict:
        '''
-        Load all market pair info build and cache it for downstream
+        Load all market pair info build and cache it for downstream use.
        use.
-        Multiple pair info lookup tables (like ``._altnames:
+        A ``._ntable: dict[str, str]`` is available for mapping the
-        dict[str, str]``) are created for looking up the
+        websocket pair name-keys and their http endpoint API (smh)
-        piker-native `Pair`-struct from any input of the three
+        equivalents to the "alternative name" which is generally the one
-        (yes, it's that idiotic..) available symbol/pair-key-sets
+        we actually want to use XD
        that kraken frickin offers depending on the API including
        the .altname, .wsname and the weird ass default set they
        return in ReST responses .xname..
        '''
-        if (
+        if not self._pairs:
-            not self._pairs
+            pairs = await self.pair_info()
-            or reload
+            assert self._pairs == pairs
        ):
            await self.asset_pairs()
-        return self._AssetPairs
+            # table of all ws and rest keys to their alt-name values.
            ntable: dict[str, str] = {}
            for rest_key in list(pairs.keys()):
                pair: Pair = pairs[rest_key]
                altname = pair.altname
                wsname = pair.wsname
                ntable[altname] = ntable[rest_key] = ntable[wsname] = altname
                # register the pair under all monikers, a giant flat
                # surjection of all possible names to each info obj.
                self._pairs[altname] = self._pairs[wsname] = pair
            self._ntable.update(ntable)
        return self._pairs
    async def search_symbols(
        self,
@ -556,20 +543,16 @@ class Client:
        '''
        if not len(self._pairs):
-            await self.get_mkt_pairs()
+            await self.cache_symbols()
-            assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
+            assert self._pairs, '`Client.cache_symbols()` was never called!?'
-        matches: dict[str, Pair] = match_from_pairs(
+        matches = fuzzy.extractBests(
-            pairs=self._pairs,
+            pattern,
-            query=pattern.upper(),
+            self._pairs,
            score_cutoff=50,
        )
-
+        # repack in dict form
-        # repack in .altname-keyed output table
+        return {item[0].altname: item[0] for item in matches}
        return {
            pair.altname: pair
            for pair in matches.values()
        }
    async def bars(
        self,
@ -649,10 +632,10 @@ class Client:
                raise BrokerError(errmsg)
    @classmethod
-    def to_bs_fqme(
+    def normalize_symbol(
        cls,
-        pair_str: str
+        ticker: str
-    ) -> str:
+    ) -> tuple[str, Pair]:
        '''
        Normalize symbol names to to a 3x3 pair from the global
        definition map which we build out from the data retreived from
@ -660,7 +643,7 @@ class Client:
        '''
        try:
-            return cls._altnames[pair_str.upper()].bs_fqme
+            return cls._ntable[ticker]
        except KeyError as ke:
            raise SymbolNotFound(f'kraken has no {ke.args[0]}')
@ -668,36 +651,21 @@ class Client:
@acm
 async def get_client() -> Client:
-    conf: dict[str, Any] = get_config()
+    conf = get_config()
-    async with httpx.AsyncClient(
+    if conf:
-        base_url=_url,
+        client = Client(
-        headers=_headers,
+            conf,
            name=conf['key_descr'],
            api_key=conf['api_key'],
            secret=conf['secret']
        )
    else:
        client = Client({})
-        # TODO: is there a way to numerate this?
+    # at startup, load all symbols, and asset info in
-        # https://www.python-httpx.org/advanced/clients/#why-use-a-client
+    # batch requests.
-        # connections=4
+    async with trio.open_nursery() as nurse:
-    ) as trio_client:
+        nurse.start_soon(client.get_assets)
-        if conf:
+        await client.cache_symbols()
            client = Client(
                conf,
                httpx_client=trio_client,
-                # TODO: don't break these up and just do internal
+    yield client
                # conf lookups instead..
                name=conf['key_descr'],
                api_key=conf['api_key'],
                secret=conf['secret']
            )
        else:
            client = Client(
                conf={},
                httpx_client=trio_client,
            )
        # at startup, load all symbols, and asset info in
        # batch requests.
        async with trio.open_nursery() as nurse:
            nurse.start_soon(client.get_assets)
            await client.get_mkt_pairs()
        yield client
--- a/piker/brokers/kraken/broker.py
+++ b/piker/brokers/kraken/broker.py
@ -24,6 +24,7 @@ from contextlib import (
 )
 from functools import partial
 from itertools import count
 import math
 from pprint import pformat
 import time
 from typing import (
@ -34,16 +35,21 @@ from typing import (
 )
 from bidict import bidict
 import pendulum
 import trio
 import tractor
 from piker.accounting import (
    Position,
-    Account,
+    PpTable,
    Transaction,
    TransactionLedger,
    open_trade_ledger,
-    open_account,
+    open_pps,
    get_likely_pair,
 )
 from piker.accounting._mktinfo import (
    MktPair,
 )
 from piker.clearing import(
    OrderDialogs,
@ -59,24 +65,18 @@ from piker.clearing._messages import (
    BrokerdPosition,
    BrokerdStatus,
 )
 from piker.brokers import (
    open_cached_client,
 )
 from piker.data import open_symcache
 from .api import (
    log,
    Client,
    BrokerError,
    get_client,
 )
 from .feed import (
    get_mkt_info,
    open_autorecon_ws,
    NoBsWs,
    stream_messages,
 )
 from .ledger import (
    norm_trade_records,
    verify_balances,
 )
 MsgUnion = Union[
    BrokerdCancel,
@ -371,8 +371,7 @@ async def subscribe(
 def trades2pps(
-    acnt: Account,
+    table: PpTable,
    ledger: TransactionLedger,
    acctid: str,
    new_trans: dict[str, Transaction] = {},
@ -380,14 +379,13 @@ def trades2pps(
 ) -> list[BrokerdPosition]:
    if new_trans:
-        updated = acnt.update_from_ledger(
+        updated = table.update_from_trans(
            new_trans,
            symcache=ledger.symcache,
        )
        log.info(f'Updated pps:\n{pformat(updated)}')
-    pp_entries, closed_pp_objs = acnt.dump_active()
+    pp_entries, closed_pp_objs = table.dump_active()
-    pp_objs: dict[Union[str, int], Position] = acnt.pps
+    pp_objs: dict[Union[str, int], Position] = table.pps
    pps: dict[int, Position]
    position_msgs: list[dict] = []
@ -401,13 +399,13 @@ def trades2pps(
                # backend suffix prefixed but when
                # reading accounts from ledgers we
                # don't need it and/or it's prefixed
-                # in the section acnt.. we should
+                # in the section table.. we should
                # just strip this from the message
                # right since `.broker` is already
                # included?
                account='kraken.' + acctid,
                symbol=p.mkt.fqme,
-                size=p.cumsize,
+                size=p.size,
                avg_price=p.ppu,
                currency='',
            )
@ -418,7 +416,7 @@ def trades2pps(
        # as little as possible. we need to either do
        # these writes in another actor, or try out `trio`'s
        # async file IO api?
-        acnt.write_config()
+        table.write_config()
    return position_msgs
@ -429,12 +427,7 @@ async def open_trade_dialog(
 ) -> AsyncIterator[dict[str, Any]]:
-    async with (
+    async with get_client() as client:
        # TODO: maybe bind these together and deliver
        # a tuple from `.open_cached_client()`?
        open_cached_client('kraken') as client,
        open_symcache('kraken') as symcache,
    ):
        # make ems flip to paper mode when no creds setup in
        # `brokers.toml` B0
        if not client._api_key:
@ -464,8 +457,8 @@ async def open_trade_dialog(
        # - delete the *ABSOLUTE LAST* entry from account's corresponding
        #   trade ledgers file (NOTE this MUST be the last record
        #   delivered from the api ledger),
-        # - open you ``account.kraken.spot.toml`` and find that
+        # - open you ``pps.toml`` and find that same tid and delete it
-        #   same tid and delete it from the pos's clears table,
+        #   from the pp's clears table,
        # - set this flag to `True`
        #
        # You should see an update come in after the order mode
@ -476,85 +469,172 @@ async def open_trade_dialog(
        # update things correctly.
        simulate_pp_update: bool = False
-        acnt: Account
+        table: PpTable
        ledger: TransactionLedger
        with (
-            open_account(
+            open_pps(
                'kraken',
                acctid,
                write_on_exit=True,
-            ) as acnt,
+            ) as table,
            open_trade_ledger(
                'kraken',
                acctid,
                symcache=symcache,
            ) as ledger,
        ):
-            # TODO: loading ledger entries should all be done
+            # transaction-ify the ledger entries
-            # within a newly implemented `async with open_account()
+            ledger_trans = await norm_trade_records(ledger)
            # as acnt` where `Account.ledger: TransactionLedger`
            # can be used to explicitily update and write the
            # offline TOML files!
            # ------ - ------
            # MOL the init sequence is:
            # - get `Account` (with presumed pre-loaded ledger done
            #   beind the scenes as part of ctx enter).
            # - pull new trades from API, update the ledger with
            #   normalized to `Transaction` entries of those
            #   records, presumably (and implicitly) update the
            #   acnt state including expiries, positions,
            #   transfers..), and finally of course existing
            #   per-asset balances.
            # - validate all pos and balances ensuring there's
            #   no seemingly noticeable discrepancies?
-            # LOAD and transaction-ify the EXISTING LEDGER
+            if not table.pps:
-            ledger_trans: dict[str, Transaction] = await norm_trade_records(
+                # NOTE: we can't use this since it first needs
-                ledger,
+                # broker: str input support!
-                client,
+                # table.update_from_trans(ledger.to_trans())
-                api_name_set='xname',
+                table.update_from_trans(ledger_trans)
-            )
+                table.write_config()
            if not acnt.pps:
                acnt.update_from_ledger(
                    ledger_trans,
                    symcache=ledger.symcache,
                )
                acnt.write_config()
            # TODO: eventually probably only load
            # as far back as it seems is not deliverd in the
            # most recent 50 trades and assume that by ordering we
-            # already have those records in the ledger?
+            # already have those records in the ledger.
-            tids2trades: dict[str, dict] = await client.get_trades()
+            tids2trades = await client.get_trades()
            ledger.update(tids2trades)
            if tids2trades:
                ledger.write_config()
-            api_trans: dict[str, Transaction] = await norm_trade_records(
+            api_trans = await norm_trade_records(tids2trades)
                tids2trades,
                client,
                api_name_set='xname',
            )
            # retrieve kraken reported balances
            # and do diff with ledger to determine
            # what amount of trades-transactions need
            # to be reloaded.
-            balances: dict[str, float] = await client.get_balances()
+            balances = await client.get_balances()
-            verify_balances(
+            for dst, size in balances.items():
                acnt,
                src_fiat,
                balances,
                client,
                ledger,
                ledger_trans,
                api_trans,
            )
-            # XXX NOTE: only for simulate-testing a "new fill" since
+                # we don't care about tracking positions
                # in the user's source fiat currency.
                if (
                    dst == src_fiat
                    or not any(
                        dst in bs_mktid for bs_mktid in table.pps
                    )
                ):
                    log.warning(
                        f'Skipping balance `{dst}`:{size} for position calcs!'
                    )
                    continue
                def has_pp(
                    dst: str,
                    size: float,
                ) -> Position | None:
                    src2dst: dict[str, str] = {}
                    for bs_mktid in table.pps:
                        likely_pair = get_likely_pair(
                            src_fiat,
                            dst,
                            bs_mktid,
                        )
                        if likely_pair:
                            src2dst[src_fiat] = dst
                    for src, dst in src2dst.items():
                        pair = f'{dst}{src_fiat}'
                        pp = table.pps.get(pair)
                        if (
                            pp
                            and math.isclose(pp.size, size)
                        ):
                            return pp
                        elif (
                            size == 0
                            and pp.size
                        ):
                            log.warning(
                                f'`kraken` account says you have  a ZERO '
                                f'balance for {bs_mktid}:{pair}\n'
                                f'but piker seems to think `{pp.size}`\n'
                                'This is likely a discrepancy in piker '
                                'accounting if the above number is'
                                "large,' though it's likely to due lack"
                                "f tracking xfers fees.."
                            )
                            return pp
                    return None  # signal no entry
                pos = has_pp(dst, size)
                if not pos:
                    # we have a balance for which there is no pp
                    # entry? so we have to likely update from the
                    # ledger.
                    updated = table.update_from_trans(ledger_trans)
                    log.info(f'Updated pps from ledger:\n{pformat(updated)}')
                    pos = has_pp(dst, size)
                    if (
                        not pos
                        and not simulate_pp_update
                    ):
                        # try reloading from API
                        table.update_from_trans(api_trans)
                        pos = has_pp(dst, size)
                        if not pos:
                            # get transfers to make sense of abs balances.
                            # NOTE: we do this after ledger and API
                            # loading since we might not have an entry
                            # in the ``pps.toml`` for the necessary pair
                            # yet and thus this likely pair grabber will
                            # likely fail.
                            for bs_mktid in table.pps:
                                likely_pair = get_likely_pair(
                                    src_fiat,
                                    dst,
                                    bs_mktid,
                                )
                                if likely_pair:
                                    break
                            else:
                                raise ValueError(
                                    'Could not find a position pair in '
                                    'ledger for likely widthdrawal '
                                    f'candidate: {dst}'
                                )
                            if likely_pair:
                                # this was likely pp that had a withdrawal
                                # from the dst asset out of the account.
                                xfer_trans = await client.get_xfers(
                                    dst,
                                    # TODO: not all src assets are
                                    # 3 chars long...
                                    src_asset=likely_pair[3:],
                                )
                                if xfer_trans:
                                    updated = table.update_from_trans(
                                        xfer_trans,
                                        cost_scalar=1,
                                    )
                                    log.info(
                                        f'Updated {dst} from transfers:\n'
                                        f'{pformat(updated)}'
                                    )
                        if has_pp(dst, size):
                            raise ValueError(
                                'Could not reproduce balance:\n'
                                f'dst: {dst}, {size}\n'
                            )
            # only for simulate-testing a "new fill" since
            # otherwise we have to actually conduct a live clear.
            if simulate_pp_update:
                tid = list(tids2trades)[0]
@ -563,27 +643,25 @@ async def open_trade_dialog(
                reqids2txids[0] = last_trade_dict['ordertxid']
            ppmsgs: list[BrokerdPosition] = trades2pps(
-                acnt,
+                table,
                ledger,
                acctid,
            )
            # sync with EMS delivering pps and accounts
            await ctx.started((ppmsgs, [acc_name]))
            # TODO: ideally this blocks the this task
            # as little as possible. we need to either do
            # these writes in another actor, or try out `trio`'s
            # async file IO api?
-            acnt.write_config()
+            table.write_config()
            # Get websocket token for authenticated data stream
            # Assert that a token was actually received.
            resp = await client.endpoint('GetWebSocketsToken', {})
-            if err := resp.get('error'):
+            err = resp.get('error')
            if err:
                raise BrokerError(err)
-            # resp token for ws init
+            token = resp['result']['token']
            token: str = resp['result']['token']
            ws: NoBsWs
            async with (
@ -612,35 +690,32 @@ async def open_trade_dialog(
                # enter relay loop
                await handle_order_updates(
-                    client=client,
+                    ws,
-                    ws=ws,
+                    stream,
-                    ws_stream=stream,
+                    ems_stream,
-                    ems_stream=ems_stream,
+                    apiflows,
-                    apiflows=apiflows,
+                    ids,
-                    ids=ids,
+                    reqids2txids,
-                    reqids2txids=reqids2txids,
+                    table,
-                    acnt=acnt,
+                    api_trans,
-                    ledger=ledger,
+                    acctid,
-                    acctid=acctid,
+                    acc_name,
-                    acc_name=acc_name,
+                    token,
                    token=token,
                )
 async def handle_order_updates(
    client: Client,  # only for pairs table needed in ledger proc
    ws: NoBsWs,
    ws_stream: AsyncIterator,
    ems_stream: tractor.MsgStream,
    apiflows: OrderDialogs,
    ids: bidict[str, int],
    reqids2txids: bidict[int, str],
-    acnt: Account,
+    table: PpTable,
    # transaction records which will be updated
    # on new trade clearing events (aka order "fills")
-    ledger: TransactionLedger,
+    ledger_trans: dict[str, Transaction],
    # ledger_trans: dict[str, Transaction],
    acctid: str,
    acc_name: str,
    token: str,
@ -658,7 +733,7 @@ async def handle_order_updates(
            # TODO: turns out you get the fill events from the
            # `openOrders` before you get this, so it might be better
-            # to do all fill/status/pos updates in that sub and just use
+            # to do all fill/status/pp updates in that sub and just use
            # this one for ledger syncs?
            # For eg. we could take the "last 50 trades" and do a diff
@ -700,8 +775,7 @@ async def handle_order_updates(
                    # if tid not in ledger_trans
                }
                for tid, trade in trades.items():
-                    # assert tid not in ledger_trans
+                    assert tid not in ledger_trans
                    assert tid not in ledger
                    txid = trade['ordertxid']
                    reqid = trade.get('userref')
@ -744,22 +818,12 @@ async def handle_order_updates(
                    )
                    await ems_stream.send(status_msg)
-                new_trans = await norm_trade_records(
+                new_trans = await norm_trade_records(trades)
-                    trades,
+                ppmsgs = trades2pps(
-                    client,
+                    table,
-                    api_name_set='wsname',
+                    acctid,
                    new_trans,
                )
                ppmsgs: list[BrokerdPosition] = trades2pps(
                    acnt=acnt,
                    ledger=ledger,
                    acctid=acctid,
                    new_trans=new_trans,
                )
                # ppmsgs = trades2pps(
                #     acnt,
                #     acctid,
                #     new_trans,
                # )
                for pp_msg in ppmsgs:
                    await ems_stream.send(pp_msg)
@ -1119,3 +1183,36 @@ async def handle_order_updates(
                        })
            case _:
                log.warning(f'Unhandled trades update msg: {msg}')
 async def norm_trade_records(
    ledger: dict[str, Any],
 ) -> dict[str, Transaction]:
    records: dict[str, Transaction] = {}
    for tid, record in ledger.items():
        size = float(record.get('vol')) * {
            'buy': 1,
            'sell': -1,
        }[record['type']]
        # we normalize to kraken's `altname` always..
        bs_mktid: str = Client.normalize_symbol(record['pair'])
        fqme = f'{bs_mktid.lower()}.kraken'
        mkt: MktPair = (await get_mkt_info(fqme))[0]
        records[tid] = Transaction(
            fqme=fqme,
            sym=mkt,
            tid=tid,
            size=size,
            price=float(record['price']),
            cost=float(record['fee']),
            dt=pendulum.from_timestamp(float(record['time'])),
            bs_mktid=bs_mktid,
        )
    return records
--- a/piker/brokers/kraken/feed.py
+++ b/piker/brokers/kraken/feed.py
@ -30,29 +30,38 @@ from typing import (
 )
 import time
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 import pendulum
 from trio_typing import TaskStatus
 import tractor
 import trio
 from piker.accounting._mktinfo import (
    Asset,
    MktPair,
    unpack_fqme,
 )
 from piker.brokers import (
    open_cached_client,
    SymbolNotFound,
 )
 from piker._cacheables import (
    async_lifo_cache,
 )
 from piker.brokers._util import (
    BrokerError,
    DataThrottle,
    DataUnavailable,
 )
-from piker.types import Struct
+from piker.data.types import Struct
 from piker.data.validate import FeedInit
 from piker.data._web_bs import open_autorecon_ws, NoBsWs
 from .api import (
    log,
    Client,
    Pair,
 )
 from .symbols import get_mkt_info
 class OHLC(Struct, frozen=True):
@ -258,6 +267,62 @@ async def open_history_client(
        yield get_ohlc, {'erlangs': 1, 'rate': 1}
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
 ) -> tuple[MktPair, Pair]:
    '''
    Query for and return a `MktPair` and backend-native `Pair` (or
    wtv else) info.
    If more then one fqme is provided return a ``dict`` of native
    key-strs to `MktPair`s.
    '''
    venue: str = 'spot'
    expiry: str = ''
    if '.kraken' in fqme:
        broker, pair, venue, expiry = unpack_fqme(fqme)
        venue: str = venue or 'spot'
    if venue != 'spot':
        raise SymbolNotFound(
            'kraken only supports spot markets right now!\n'
            f'{fqme}\n'
        )
    async with open_cached_client('kraken') as client:
        # uppercase since kraken bs_mktid is always upper
        bs_fqme, _, broker = fqme.partition('.')
        pair_str: str = bs_fqme.upper()
        bs_mktid: str = Client.normalize_symbol(pair_str)
        pair: Pair = await client.pair_info(pair_str)
        assets = client.assets
        dst_asset: Asset = assets[pair.base]
        src_asset: Asset = assets[pair.quote]
        mkt = MktPair(
            dst=dst_asset,
            src=src_asset,
            price_tick=pair.price_tick,
            size_tick=pair.size_tick,
            bs_mktid=bs_mktid,
            expiry=expiry,
            venue=venue or 'spot',
            # TODO: futes
            # _atype=_atype,
            broker='kraken',
        )
        return mkt, pair
 async def stream_quotes(
    send_chan: trio.abc.SendChannel,
@ -413,3 +478,30 @@ async def stream_quotes(
                        log.warning(f'Unknown WSS message: {typ}, {quote}')
                await send_chan.send({topic: quote})
@tractor.context
 async def open_symbol_search(
    ctx: tractor.Context,
 ) -> Client:
    async with open_cached_client('kraken') as client:
        # load all symbols locally for fast search
        cache = await client.cache_symbols()
        await ctx.started(cache)
        async with ctx.open_stream() as stream:
            async for pattern in stream:
                matches = fuzzy.extractBests(
                    pattern,
                    cache,
                    score_cutoff=50,
                )
                # repack in dict form
                await stream.send({
                    pair[0].altname: pair[0]
                    for pair in matches
                })
--- a/piker/brokers/kraken/ledger.py
+++ b/piker/brokers/kraken/ledger.py
@ -1,269 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Trade transaction accounting and normalization.
 '''
 import math
 from pprint import pformat
 from typing import (
    Any,
 )
 import pendulum
 from piker.accounting import (
    Transaction,
    Position,
    Account,
    get_likely_pair,
    TransactionLedger,
    # MktPair,
 )
 from piker.types import Struct
 from piker.data import (
    SymbologyCache,
 )
 from .api import (
    log,
    Client,
    Pair,
 )
 # from .feed import get_mkt_info
 def norm_trade(
    tid: str,
    record: dict[str, Any],
    # this is the dict that was returned from
    # `Client.get_mkt_pairs()` and when running offline ledger
    # processing from `.accounting`, this will be the table loaded
    # into `SymbologyCache.pairs`.
    pairs: dict[str, Struct],
    symcache: SymbologyCache | None = None,
 ) -> Transaction:
    size: float = float(record.get('vol')) * {
        'buy': 1,
        'sell': -1,
    }[record['type']]
    # NOTE: this value may be either the websocket OR the rest schema
    # so we need to detect the key format and then choose the
    # correct symbol lookup table to evetually get a ``Pair``..
    # See internals of `Client.asset_pairs()` for deats!
    src_pair_key: str = record['pair']
    # XXX: kraken's data engineering is soo bad they require THREE
    # different pair schemas (more or less seemingly tied to
    # transport-APIs)..LITERALLY they return different market id
    # pairs in the ledger endpoints vs. the websocket event subs..
    # lookup pair using appropriately provided tabled depending
    # on API-key-schema..
    pair: Pair = pairs[src_pair_key]
    fqme: str = pair.bs_fqme.lower() + '.kraken'
    return Transaction(
        fqme=fqme,
        tid=tid,
        size=size,
        price=float(record['price']),
        cost=float(record['fee']),
        dt=pendulum.from_timestamp(float(record['time'])),
        bs_mktid=pair.bs_mktid,
    )
 async def norm_trade_records(
    ledger: dict[str, Any],
    client: Client,
    api_name_set: str = 'xname',
 ) -> dict[str, Transaction]:
    '''
    Loop through an input ``dict`` of trade records
    and convert them to ``Transactions``.
    '''
    records: dict[str, Transaction] = {}
    for tid, record in ledger.items():
        # manual_fqme: str = f'{bs_mktid.lower()}.kraken'
        # mkt: MktPair = (await get_mkt_info(manual_fqme))[0]
        # fqme: str = mkt.fqme
        # assert fqme == manual_fqme
        pairs: dict[str, Pair] = {
            'xname': client._AssetPairs,
            'wsname': client._wsnames,
            'altname': client._altnames,
        }[api_name_set]
        records[tid] = norm_trade(
            tid,
            record,
            pairs=pairs,
        )
    return records
 def has_pp(
    acnt: Account,
    src_fiat: str,
    dst: str,
    size: float,
 ) -> Position | None:
    src2dst: dict[str, str] = {}
    for bs_mktid in acnt.pps:
        likely_pair = get_likely_pair(
            src_fiat,
            dst,
            bs_mktid,
        )
        if likely_pair:
            src2dst[src_fiat] = dst
    for src, dst in src2dst.items():
        pair: str = f'{dst}{src_fiat}'
        pos: Position = acnt.pps.get(pair)
        if (
            pos
            and math.isclose(pos.size, size)
        ):
            return pos
        elif (
            size == 0
            and pos.size
        ):
            log.warning(
                f'`kraken` account says you have  a ZERO '
                f'balance for {bs_mktid}:{pair}\n'
                f'but piker seems to think `{pos.size}`\n'
                'This is likely a discrepancy in piker '
                'accounting if the above number is'
                "large,' though it's likely to due lack"
                "f tracking xfers fees.."
            )
            return pos
    return None  # indicate no entry found
 # TODO: factor most of this "account updating from txns" into the
 # the `Account` impl so has to provide for hiding the mostly
 # cross-provider updates from txn sets
 async def verify_balances(
    acnt: Account,
    src_fiat: str,
    balances: dict[str, float],
    client: Client,
    ledger: TransactionLedger,
    ledger_trans: dict[str, Transaction],  # from toml
    api_trans: dict[str, Transaction],  # from API
    simulate_pp_update: bool = False,
 ) -> None:
    for dst, size in balances.items():
        # we don't care about tracking positions
        # in the user's source fiat currency.
        if (
            dst == src_fiat
            or not any(
                dst in bs_mktid for bs_mktid in acnt.pps
            )
        ):
            log.warning(
                f'Skipping balance `{dst}`:{size} for position calcs!'
            )
            continue
        # we have a balance for which there is no pos entry
        # - we have to likely update from the ledger?
        if not has_pp(acnt, src_fiat, dst, size):
            updated = acnt.update_from_ledger(
                ledger_trans,
                symcache=ledger.symcache,
            )
            log.info(f'Updated pps from ledger:\n{pformat(updated)}')
            # FIRST try reloading from API records
            if (
                not has_pp(acnt, src_fiat, dst, size)
                and not simulate_pp_update
            ):
                acnt.update_from_ledger(
                    api_trans,
                    symcache=ledger.symcache,
                )
                # get transfers to make sense of abs
                # balances.
                # NOTE: we do this after ledger and API
                # loading since we might not have an
                # entry in the
                # ``account.kraken.spot.toml`` for the
                # necessary pair yet and thus this
                # likely pair grabber will likely fail.
                if not has_pp(acnt, src_fiat, dst, size):
                    for bs_mktid in acnt.pps:
                        likely_pair: str | None = get_likely_pair(
                            src_fiat,
                            dst,
                            bs_mktid,
                        )
                        if likely_pair:
                            break
                    else:
                        raise ValueError(
                            'Could not find a position pair in '
                            'ledger for likely widthdrawal '
                            f'candidate: {dst}'
                        )
                    # this was likely pos that had a withdrawal
                    # from the dst asset out of the account.
                    if likely_pair:
                        xfer_trans = await client.get_xfers(
                            dst,
                            # TODO: not all src assets are
                            # 3 chars long...
                            src_asset=likely_pair[3:],
                        )
                        if xfer_trans:
                            updated = acnt.update_from_ledger(
                                xfer_trans,
                                cost_scalar=1,
                                symcache=ledger.symcache,
                            )
                            log.info(
                                f'Updated {dst} from transfers:\n'
                                f'{pformat(updated)}'
                            )
                if has_pp(acnt, src_fiat, dst, size):
                    raise ValueError(
                        'Could not reproduce balance:\n'
                        f'dst: {dst}, {size}\n'
                    )
--- a/piker/brokers/kraken/symbols.py
+++ b/piker/brokers/kraken/symbols.py
@ -1,206 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Symbology defs and search.
 '''
 from decimal import Decimal
 import tractor
 from rapidfuzz import process as fuzzy
 from piker._cacheables import (
    async_lifo_cache,
 )
 from piker.accounting._mktinfo import (
    digits_to_dec,
 )
 from piker.brokers import (
    open_cached_client,
    SymbolNotFound,
 )
 from piker.types import Struct
 from piker.accounting._mktinfo import (
    Asset,
    MktPair,
    unpack_fqme,
 )
 # https://www.kraken.com/features/api#get-tradable-pairs
 class Pair(Struct):
    xname: str  # idiotic bs_mktid equiv i guess?
    altname: str  # alternate pair name
    wsname: str  # WebSocket pair name (if available)
    aclass_base: str  # asset class of base component
    base: str  # asset id of base component
    aclass_quote: str  # asset class of quote component
    quote: str  # asset id of quote component
    lot: str  # volume lot size
    cost_decimals: int
    costmin: float
    pair_decimals: int  # scaling decimal places for pair
    lot_decimals: int  # scaling decimal places for volume
    # amount to multiply lot volume by to get currency volume
    lot_multiplier: float
    # array of leverage amounts available when buying
    leverage_buy: list[int]
    # array of leverage amounts available when selling
    leverage_sell: list[int]
    # fee schedule array in [volume, percent fee] tuples
    fees: list[tuple[int, float]]
    # maker fee schedule array in [volume, percent fee] tuples (if on
    # maker/taker)
    fees_maker: list[tuple[int, float]]
    fee_volume_currency: str  # volume discount currency
    margin_call: str  # margin call level
    margin_stop: str  # stop-out/liquidation margin level
    ordermin: float  # minimum order volume for pair
    tick_size: float  # min price step size
    status: str
    short_position_limit: float = 0
    long_position_limit: float = float('inf')
    # TODO: should we make this a literal NamespacePath ref?
    ns_path: str = 'piker.brokers.kraken:Pair'
    @property
    def bs_mktid(self) -> str:
        '''
        Kraken seems to index it's market symbol sets in
        transaction ledgers using the key returned from rest
        queries.. so use that since apparently they can't
        make up their minds on a better key set XD
        '''
        return self.xname
    @property
    def price_tick(self) -> Decimal:
        return digits_to_dec(self.pair_decimals)
    @property
    def size_tick(self) -> Decimal:
        return digits_to_dec(self.lot_decimals)
    @property
    def bs_dst_asset(self) -> str:
        dst, _ = self.wsname.split('/')
        return dst
    @property
    def bs_src_asset(self) -> str:
        _, src = self.wsname.split('/')
        return src
    @property
    def bs_fqme(self) -> str:
        '''
        Basically the `.altname` but with special '.' handling and
        `.SPOT` suffix appending (for future multi-venue support).
        '''
        dst, src = self.wsname.split('/')
        # XXX: omg for stupid shite like ETH2.S/ETH..
        dst = dst.replace('.', '-')
        return f'{dst}{src}.SPOT'
@tractor.context
 async def open_symbol_search(ctx: tractor.Context) -> None:
    async with open_cached_client('kraken') as client:
        # load all symbols locally for fast search
        cache = await client.get_mkt_pairs()
        await ctx.started(cache)
        async with ctx.open_stream() as stream:
            async for pattern in stream:
                await stream.send(
                    await client.search_symbols(pattern)
                )
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
 ) -> tuple[MktPair, Pair]:
    '''
    Query for and return a `MktPair` and backend-native `Pair` (or
    wtv else) info.
    If more then one fqme is provided return a ``dict`` of native
    key-strs to `MktPair`s.
    '''
    venue: str = 'spot'
    expiry: str = ''
    if '.kraken' not in fqme:
        fqme += '.kraken'
    broker, pair, venue, expiry = unpack_fqme(fqme)
    venue: str = venue or 'spot'
    if venue.lower() != 'spot':
        raise SymbolNotFound(
            'kraken only supports spot markets right now!\n'
            f'{fqme}\n'
        )
    async with open_cached_client('kraken') as client:
        # uppercase since kraken bs_mktid is always upper
        # bs_fqme, _, broker = fqme.partition('.')
        # pair_str: str = bs_fqme.upper()
        pair_str: str = f'{pair}.{venue}'
        pair: Pair | None = client._pairs.get(pair_str.upper())
        if not pair:
            bs_fqme: str = client.to_bs_fqme(pair_str)
            pair: Pair = client._pairs[bs_fqme]
        if not (assets := client._assets):
            assets: dict[str, Asset] = await client.get_assets()
        dst_asset: Asset = assets[pair.bs_dst_asset]
        src_asset: Asset = assets[pair.bs_src_asset]
        mkt = MktPair(
            dst=dst_asset,
            src=src_asset,
            price_tick=pair.price_tick,
            size_tick=pair.size_tick,
            bs_mktid=pair.bs_mktid,
            expiry=expiry,
            venue=venue or 'spot',
            # TODO: futes
            # _atype=_atype,
            broker='kraken',
        )
        return mkt, pair
--- a/piker/brokers/kucoin.py
+++ b/piker/brokers/kucoin.py
@ -16,9 +16,10 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-Kucoin cex API backend.
+Kucoin broker backend
 '''
 from contextlib import (
    asynccontextmanager as acm,
    aclosing,
@ -40,8 +41,9 @@ from typing import (
 import wsproto
 from uuid import uuid4
 from fuzzywuzzy import process as fuzzy
 from trio_typing import TaskStatus
-import httpx
+import asks
 from bidict import bidict
 import numpy as np
 import pendulum
@ -62,11 +64,8 @@ from piker._cacheables import (
 )
 from piker.log import get_logger
 from piker.data.validate import FeedInit
-from piker.types import Struct  # NOTE, this is already a `tractor.msg.Struct`
+from piker.data.types import Struct
-from piker.data import (
+from piker.data import def_iohlcv_fields
    def_iohlcv_fields,
    match_from_pairs,
 )
 from piker.data._web_bs import (
    open_autorecon_ws,
    NoBsWs,
@ -75,8 +74,6 @@ from ._util import DataUnavailable
 log = get_logger(__name__)
 _no_symcache: bool = True
 class KucoinMktPair(Struct, frozen=True):
    '''
@ -89,27 +86,18 @@ class KucoinMktPair(Struct, frozen=True):
    @property
    def price_tick(self) -> Decimal:
-        return Decimal(str(self.quoteIncrement))
+        return Decimal(str(self.baseIncrement))
    baseMaxSize: float
    baseMinSize: float
    @property
    def size_tick(self) -> Decimal:
-        return Decimal(str(self.quoteMinSize))
+        return Decimal(str(self.baseMinSize))
    callauctionFirstStageStartTime: None|float
    callauctionIsEnabled: bool
    callauctionPriceCeiling: float|None
    callauctionPriceFloor: float|None
    callauctionSecondStageStartTime: float|None
    callauctionThirdStageStartTime: float|None
    enableTrading: bool
    feeCategory: int
    feeCurrency: str
    isMarginEnabled: bool
    makerFeeCoefficient: float
    market: str
    minFunds: float
    name: str
@ -119,10 +107,7 @@ class KucoinMktPair(Struct, frozen=True):
    quoteIncrement: float
    quoteMaxSize: float
    quoteMinSize: float
    st: bool
    symbol: str  # our bs_mktid, kucoin's internal id
    takerFeeCoefficient: float
    tradingStartTime: float|None
 class AccountTrade(Struct, frozen=True):
@ -222,13 +207,8 @@ def get_config() -> BrokerConfig | None:
 class Client:
-
+    def __init__(self) -> None:
-    def __init__(
+        self._config: BrokerConfig | None = get_config()
        self,
        httpx_client: httpx.AsyncClient,
    ) -> None:
        self._http: httpx.AsyncClient = httpx_client
        self._config: BrokerConfig|None = get_config()
        self._pairs: dict[str, KucoinMktPair] = {}
        self._fqmes2mktids: bidict[str, str] = bidict()
        self._bars: list[list[float]] = []
@ -242,24 +222,18 @@ class Client:
    ) -> dict[str, str | bytes]:
        '''
-        Generate authenticated request headers:
+        Generate authenticated request headers
        https://docs.kucoin.com/#authentication
        https://www.kucoin.com/docs/basic-info/connection-method/authentication/creating-a-request
        https://www.kucoin.com/docs/basic-info/connection-method/authentication/signing-a-message
        '''
        if not self._config:
            raise ValueError(
-                'No config found when trying to send authenticated request'
+                'No config found when trying to send authenticated request')
            )
        str_to_sign = (
            str(int(time.time() * 1000))
-            +
+            + action + f'/api/{api}/{endpoint.lstrip("/")}'
            action
            +
            f'/api/{api}/{endpoint.lstrip("/")}'
        )
        signature = base64.b64encode(
@ -270,7 +244,6 @@ class Client:
            ).digest()
        )
        # TODO: can we cache this between calls?
        passphrase = base64.b64encode(
            hmac.new(
                self._config.key_secret.encode('utf-8'),
@ -292,10 +265,8 @@ class Client:
        self,
        action: Literal['POST', 'GET'],
        endpoint: str,
        api: str = 'v2',
        headers: dict = {},
    ) -> Any:
        '''
        Generic request wrapper for Kucoin API
@ -308,19 +279,14 @@ class Client:
                api,
            )
-        req_meth: Callable = getattr(
+        api_url = f'https://api.kucoin.com/api/{api}/{endpoint}'
-            self._http,
+
-            action.lower(),
+        res = await asks.request(action, api_url, headers=headers)
-        )
+
-        res = await req_meth(
+        json = res.json()
-            url=f'/{api}/{endpoint}',
+        if 'data' in json:
-            headers=headers,
+            return json['data']
        )
        json: dict = res.json()
        if (data := json.get('data')) is not None:
            return data
        else:
            api_url: str = self._http.base_url
            log.error(
                f'Error making request to {api_url} ->\n'
                f'{pformat(res)}'
@ -340,7 +306,7 @@ class Client:
        '''
        token_type = 'private' if private else 'public'
        try:
-            data: dict[str, Any]|None = await self._request(
+            data: dict[str, Any] | None = await self._request(
                'POST',
                endpoint=f'bullet-{token_type}',
                api='v1'
@ -378,8 +344,8 @@ class Client:
            currencies: dict[str, Currency] = {}
            entries: list[dict] = await self._request(
                'GET',
                endpoint='currencies',
                api='v1',
                endpoint='currencies',
            )
            for entry in entries:
                curr = Currency(**entry).copy()
@ -395,29 +361,20 @@ class Client:
        dict[str, KucoinMktPair],
        bidict[str, KucoinMktPair],
    ]:
-        entries = await self._request(
+        entries = await self._request('GET', 'symbols')
            'GET',
            endpoint='symbols',
        )
        log.info(f' {len(entries)} Kucoin market pairs fetched')
        pairs: dict[str, KucoinMktPair] = {}
        fqmes2mktids: bidict[str, str] = bidict()
        for item in entries:
-            try:
+            pair = pairs[item['name']] = KucoinMktPair(**item)
                pair = pairs[item['name']] = KucoinMktPair(**item)
            except TypeError as te:
                raise TypeError(
                    '`KucoinMktPair` and reponse fields do not match ??\n'
                    f'{KucoinMktPair.fields_diff(item)}\n'
                ) from te
            fqmes2mktids[
                item['name'].lower().replace('-', '')
            ] = pair.name
        return pairs, fqmes2mktids
-    async def get_mkt_pairs(
+    async def cache_pairs(
        self,
        update: bool = False,
@ -445,27 +402,16 @@ class Client:
    ) -> dict[str, KucoinMktPair]:
        '''
-        Use fuzzy search engine to match against pairs, deliver
+        Use fuzzy search to match against all market names.
        matching ones.
        '''
-        if not len(self._pairs):
+        data = await self.cache_pairs()
            await self.get_mkt_pairs()
            assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
-        matches: dict[str, KucoinMktPair] = match_from_pairs(
+        matches = fuzzy.extractBests(
-            pairs=self._pairs,
+            pattern, data, score_cutoff=35, limit=limit
            # query=pattern.upper(),
            query=pattern.upper(),
            score_cutoff=35,
            limit=limit,
        )
        # repack in dict form
-        return {
+        return {item[0].name: item[0] for item in matches}
            pair.name: pair
            for pair in matches.values()
        }
    async def last_trades(self, sym: str) -> list[AccountTrade]:
        trades = await self._request(
@ -605,21 +551,13 @@ def fqme_to_kucoin_sym(
@acm
 async def get_client() -> AsyncGenerator[Client, None]:
-    '''
+    client = Client()
    Load an API `Client` preconfigured from user settings
-    '''
+    async with trio.open_nursery() as n:
-    async with (
+        n.start_soon(client.cache_pairs)
-        httpx.AsyncClient(
+        await client.get_currencies()
            base_url='https://api.kucoin.com/api',
        ) as trio_client,
    ):
        client = Client(httpx_client=trio_client)
        async with trio.open_nursery() as tn:
            tn.start_soon(client.get_mkt_pairs)
            await client.get_currencies()
-        yield client
+    yield client
@tractor.context
@ -628,7 +566,7 @@ async def open_symbol_search(
 ) -> None:
    async with open_cached_client('kucoin') as client:
        # load all symbols locally for fast search
-        await client.get_mkt_pairs()
+        await client.cache_pairs()
        await ctx.started()
        async with ctx.open_stream() as stream:
@ -655,7 +593,7 @@ async def open_ping_task(
                await trio.sleep((ping_interval - 1000) / 1000)
                await ws.send_msg({'id': connect_id, 'type': 'ping'})
-        log.warning('Starting ping task for kucoin ws connection')
+        log.info('Starting ping task for kucoin ws connection')
        n.start_soon(ping_server)
        yield
@ -667,21 +605,16 @@ async def open_ping_task(
 async def get_mkt_info(
    fqme: str,
-) -> tuple[
+) -> tuple[MktPair, KucoinMktPair]:
    MktPair,
    KucoinMktPair,
 ]:
    '''
-    Query for and return both a `piker.accounting.MktPair` and
+    Query for and return a `MktPair` and `KucoinMktPair`.
    `KucoinMktPair` from provided `fqme: str`
    (fully-qualified-market-endpoint).
    '''
    async with open_cached_client('kucoin') as client:
        # split off any fqme broker part
        bs_fqme, _, broker = fqme.partition('.')
-        pairs: dict[str, KucoinMktPair] = await client.get_mkt_pairs()
+        pairs: dict[str, KucoinMktPair] = await client.cache_pairs()
        try:
            # likely search result key which is already in native mkt symbol form
@ -749,8 +682,6 @@ async def stream_quotes(
        log.info(f'Starting up quote stream(s) for {symbols}')
        for sym_str in symbols:
            mkt: MktPair
            pair: KucoinMktPair
            mkt, pair = await get_mkt_info(sym_str)
            init_msgs.append(
                FeedInit(mkt_info=mkt)
@ -758,11 +689,7 @@ async def stream_quotes(
        ws: NoBsWs
        token, ping_interval = await client._get_ws_token()
-        log.info('API reported ping_interval: {ping_interval}\n')
+        connect_id = str(uuid4())
        connect_id: str = str(uuid4())
        typ: str
        quote: dict
        async with (
            open_autorecon_ws(
                (
@ -776,37 +703,20 @@ async def stream_quotes(
                ),
            ) as ws,
            open_ping_task(ws, ping_interval, connect_id),
-            aclosing(
+            aclosing(stream_messages(ws, sym_str)) as msg_gen,
                iter_normed_quotes(
                    ws, sym_str
                )
            ) as iter_quotes,
        ):
-            typ, quote = await anext(iter_quotes)
+            typ, quote = await anext(msg_gen)
-            # take care to not unblock here until we get a real
+            while typ != 'trade':
-            # trade quote?
+                # take care to not unblock here until we get a real
-            # ^TODO, remove this right?
+                # trade quote
-            # -[ ] what often blocks chart boot/new-feed switching
+                typ, quote = await anext(msg_gen)
            #   since we'ere waiting for a live quote instead of just
            #   loading history afap..
            #  |_ XXX, not sure if we require a bit of rework to core
            #    feed init logic or if backends justg gotta be
            #    changed up.. feel like there was some causality
            #    dilema prolly only seen with IB too..
            # while typ != 'trade':
            #     typ, quote = await anext(iter_quotes)
            task_status.started((init_msgs, quote))
            feed_is_live.set()
-            # XXX NOTE, DO NOT include the `.<backend>` suffix!
+            async for typ, msg in msg_gen:
-            # OW the sampling loop will not broadcast correctly..
+                await send_chan.send({sym_str: msg})
            # since `bus._subscribers.setdefault(bs_fqme, set())`
            # is used inside `.data.open_feed_bus()` !!!
            topic: str = mkt.bs_fqme
            async for typ, quote in iter_quotes:
                await send_chan.send({topic: quote})
@acm
@ -861,7 +771,7 @@ async def subscribe(
            )
-async def iter_normed_quotes(
+async def stream_messages(
    ws: NoBsWs,
    sym: str,
@ -892,9 +802,6 @@ async def iter_normed_quotes(
                yield 'trade', {
                    'symbol': sym,
                    # TODO, is 'last' even used elsewhere/a-good
                    # semantic? can't we just read the ticks with our
                    # .data.ticktools.frame_ticks()`/
                    'last': trade_data.price,
                    'brokerd_ts': last_trade_ts,
                    'ticks': [
@ -987,7 +894,7 @@ async def open_history_client(
            if end_dt is None:
                inow = round(time.time())
-                log.debug(
+                print(
                    f'difference in time between load and processing'
                    f'{inow - times[-1]}'
                )
--- a/piker/clearing/README.rst
+++ b/piker/clearing/README.rst
@ -1,49 +0,0 @@
 piker.clearing
 ______________
 trade execution-n-control subsys for both live and paper trading as
 well as algo-trading manual override/interaction across any backend
 broker and data provider.
 avail UIs
 *********
 order ctl
 ---------
 the `piker.clearing` subsys is exposed mainly though
 the `piker chart` GUI as a "chart trader" style UX and
 is automatically enabled whenever a chart is opened.
 .. ^TODO, more prose here!
 the "manual" order control features are exposed via the
 `piker.ui.order_mode` API and can pretty much always be
 used (at least) in simulated-trading mode, aka "paper"-mode, and
 the micro-manual is as follows:
 ``order_mode`` (
    edge triggered activation by any of the following keys,
    ``mouse-click`` on y-level to submit at that price
    ):
    - ``f``/ ``ctl-f`` to stage buy
    - ``d``/ ``ctl-d`` to stage sell
    - ``a`` to stage alert
 ``search_mode`` (
    ``ctl-l`` or ``ctl-space`` to open,
    ``ctl-c`` or ``ctl-space`` to close
    ) :
    - begin typing to have symbol search automatically lookup
      symbols from all loaded backend (broker) providers
    - arrow keys and mouse click to navigate selection
    - vi-like ``ctl-[hjkl]`` for navigation
 position (pp) mgmt
 ------------------
 you can also configure your position allocation limits from the
 sidepane.
 .. ^TODO, explain and provide tut once more refined!
--- a/piker/clearing/init.py
+++ b/piker/clearing/init.py
@ -27,28 +27,13 @@ from ._ems import (
    open_brokerd_dialog,
 )
 from ._util import OrderDialogs
 from ._messages import(
    Order,
    Status,
    Cancel,
    # TODO: deprecate these and replace end-2-end with
    # client-side-dialog set above B)
    # https://github.com/pikers/piker/issues/514
    BrokerdPosition
 )
 __all__ = [
    'FeeModel',
    'open_ems',
    'OrderClient',
    'open_brokerd_dialog',
    'OrderDialogs',
    'Order',
    'Status',
    'Cancel',
    'BrokerdPosition'
 ]
--- a/piker/clearing/_client.py
+++ b/piker/clearing/_client.py
@ -30,7 +30,7 @@ from tractor.trionics import broadcast_receiver
 from ._util import (
    log,  # sub-sys logger
 )
-from piker.types import Struct
+from ..data.types import Struct
 from ..service import maybe_open_emsd
 from ._messages import (
    Order,
--- a/piker/clearing/_ems.py
+++ b/piker/clearing/_ems.py
@ -27,7 +27,7 @@ from contextlib import asynccontextmanager as acm
 from decimal import Decimal
 from math import isnan
 from pprint import pformat
-from time import time_ns
+import time
 from types import ModuleType
 from typing import (
    AsyncIterator,
@ -51,13 +51,12 @@ from ..accounting._mktinfo import (
    unpack_fqme,
    dec_digits,
 )
 from piker.types import Struct
 from ..ui._notify import notify_from_ems_status_msg
 from ..data import iterticks
 from ..data.types import Struct
 from ._messages import (
    Order,
    Status,
    Error,
    BrokerdCancel,
    BrokerdOrder,
    # BrokerdOrderAck,
@ -256,7 +255,7 @@ async def clear_dark_triggers(
                                action=action,
                                oid=oid,
                                account=account,
-                                time_ns=time_ns(),
+                                time_ns=time.time_ns(),
                                symbol=bfqme,
                                price=submit_price,
                                size=size,
@ -269,7 +268,7 @@ async def clear_dark_triggers(
                    # fallthrough logic
                    status = Status(
                        oid=oid,  # ems dialog id
-                        time_ns=time_ns(),
+                        time_ns=time.time_ns(),
                        resp=resp,
                        req=cmd,
                        brokerd_msg=brokerd_msg,
@ -827,8 +826,8 @@ async def translate_and_relay_brokerd_events(
                # keep pps per account up to date locally in ``emsd`` mem
                # sym, broker = pos_msg.symbol, pos_msg.broker
                # NOTE: translate to a FQME!
                relay.positions.setdefault(
                    # NOTE: translate to a FQSN!
                    (broker, pos_msg.account),
                    {}
                )[pos_msg.symbol] = pos_msg
@ -884,7 +883,7 @@ async def translate_and_relay_brokerd_events(
                        BrokerdCancel(
                            oid=oid,
                            reqid=reqid,
-                            time_ns=time_ns(),
+                            time_ns=time.time_ns(),
                            account=status_msg.req.account,
                        )
                    )
@ -899,75 +898,38 @@ async def translate_and_relay_brokerd_events(
                continue
            # BrokerdError
            # TODO: figure out how this will interact with EMS clients
            # for ex. on an error do we react with a dark orders
            # management response, like cancelling all dark orders?
            # This looks like a supervision policy for pending orders on
            # some unexpected failure - something we need to think more
            # about.  In most default situations, with composed orders
            # (ex.  brackets), most brokers seem to use a oca policy.
            case {
                'name': 'error',
                'oid': oid,  # ems order-dialog id
                'reqid': reqid,  # brokerd generated order-request id
            }:
-                if (
+                status_msg = book._active.get(oid)
                    not oid
                    # try to lookup any order dialog by
                    # brokerd-side id..
                    and not (
                        oid := book._ems2brokerd_ids.inverse.get(reqid)
                    )
                ):
                    log.warning(
                        f'Rxed unusable error-msg:\n'
                        f'{brokerd_msg}'
                    )
                    continue
                msg = BrokerdError(**brokerd_msg)
                log.error(fmsg)  # XXX make one when it's blank?
-                # NOTE: retreive the last client-side response
+                # TODO: figure out how this will interact with EMS clients
-                # OR create an error when we have no last msg /dialog
+                # for ex. on an error do we react with a dark orders
-                # on record
+                # management response, like cancelling all dark orders?
-                status_msg: Status
+                # This looks like a supervision policy for pending orders on
-                if not (status_msg := book._active.get(oid)):
+                # some unexpected failure - something we need to think more
-                    status_msg = Error(
+                # about.  In most default situations, with composed orders
-                        time_ns=time_ns(),
+                # (ex.  brackets), most brokers seem to use a oca policy.
-                        oid=oid,
+
-                        reqid=reqid,
+                # only relay to client side if we have an active
-                        brokerd_msg=msg,
+                # ongoing dialog
-                    )
+                if status_msg:
                else:
                    # only modify last status if we have an active
                    # ongoing dialog..
                    status_msg.resp = 'error'
                    status_msg.brokerd_msg = msg
                    book._active[oid] = status_msg
-                book._active[oid] = status_msg
+                    await router.client_broadcast(
-
+                        status_msg.req.symbol,
-                log.error(
+                        status_msg,
                    'Translating brokerd error to status:\n'
                    f'{fmsg}'
                    f'{status_msg.to_dict()}'
                )
                if req := status_msg.req:
                    fqme: str = req.symbol
                else:
                    bdmsg: Struct = status_msg.brokerd_msg
                    fqme: str = (
                        bdmsg.symbol  # might be None
                        or
                        bdmsg.broker_details['flow']
                        # NOTE: what happens in empty case in the
                        # broadcast below? it's a problem?
                        .get('symbol', '')
                    )
-                await router.client_broadcast(
+                else:
-                    fqme,
+                    log.error(f'Error for unknown order flow:\n{msg}')
-                    status_msg,
+                    continue
                )
            # BrokerdStatus
            case {
@ -1108,7 +1070,7 @@ async def translate_and_relay_brokerd_events(
                    status_msg.req = order
                    assert status_msg.src  # source tag?
-                    oid: str = str(status_msg.reqid)
+                    oid = str(status_msg.reqid)
                    # attempt to avoid collisions
                    status_msg.reqid = oid
@ -1125,28 +1087,38 @@ async def translate_and_relay_brokerd_events(
                        status_msg,
                    )
                # don't fall through
                continue
            # brokerd error
            case {
                'name': 'status',
                'status': 'error',
            }:
                log.error(f'Broker error:\n{fmsg}')
                # XXX: we presume the brokerd cancels its own order
                continue
            # TOO FAST ``BrokerdStatus`` that arrives
            # before the ``BrokerdAck``.
            # NOTE XXX: sometimes there is a race with the backend (like
            # `ib` where the pending status will be relayed *before*
            # the ack msg, in which case we just ignore the faster
            # pending msg and wait for our expected ack to arrive
            # later (i.e. the first block below should enter).
            case {
                # XXX: sometimes there is a race with the backend (like
                # `ib` where the pending stauts will be related before
                # the ack, in which case we just ignore the faster
                # pending msg and wait for our expected ack to arrive
                # later (i.e. the first block below should enter).
                'name': 'status',
                'status': status,
                'reqid': reqid,
            }:
-                msg = (
+                oid = book._ems2brokerd_ids.inverse.get(reqid)
-                    f'Unhandled broker status for dialog {reqid}:\n'
+                msg = f'Unhandled broker status for dialog {reqid}:\n'
-                    f'{pformat(brokerd_msg)}'
+                if oid:
-                )
+                    status_msg = book._active.get(oid)
-                if (
+                    # status msg may not have been set yet or popped?
                    oid := book._ems2brokerd_ids.inverse.get(reqid)
                ):
                    # NOTE: have seen a key error here on kraken
                    # clearable limits..
-                    if status_msg := book._active.get(oid):
+                    if status_msg:
                        msg += (
                            f'last status msg: {pformat(status_msg)}\n\n'
                            f'this msg:{fmsg}\n'
@ -1242,7 +1214,7 @@ async def process_client_order_cmds(
                        BrokerdCancel(
                            oid=oid,
                            reqid=reqid,
-                            time_ns=time_ns(),
+                            time_ns=time.time_ns(),
                            account=order.account,
                        )
                    )
@ -1317,7 +1289,7 @@ async def process_client_order_cmds(
                msg = BrokerdOrder(
                    oid=oid,  # no ib support for oids...
-                    time_ns=time_ns(),
+                    time_ns=time.time_ns(),
                    # if this is None, creates a new order
                    # otherwise will modify any existing one
@ -1335,7 +1307,7 @@ async def process_client_order_cmds(
                        oid=oid,
                        reqid=reqid,
                        resp='pending',
-                        time_ns=time_ns(),
+                        time_ns=time.time_ns(),
                        brokerd_msg=msg,
                        req=req,
                    )
@ -1452,7 +1424,7 @@ async def process_client_order_cmds(
                status = Status(
                    resp=resp,
                    oid=oid,
-                    time_ns=time_ns(),
+                    time_ns=time.time_ns(),
                    req=req,
                    src='dark',
                )
--- a/piker/clearing/_messages.py
+++ b/piker/clearing/_messages.py
@ -18,14 +18,39 @@
 Clearing sub-system message and protocols.
 """
-from __future__ import annotations
+# from collections import (
 #     ChainMap,
 #     deque,
 # )
 from typing import (
    Literal,
 )
 from msgspec import field
-from piker.types import Struct
+from ..data.types import Struct
 # TODO: a composite for tracking msg flow on 2-legged
 # dialogs.
 # class Dialog(ChainMap):
 #     '''
 #     Msg collection abstraction to easily track the state changes of
 #     a msg flow in one high level, query-able and immutable construct.
 #     The main use case is to query data from a (long-running)
 #     msg-transaction-sequence
 #     '''
 #     def update(
 #         self,
 #         msg,
 #     ) -> None:
 #         self.maps.insert(0, msg.to_dict())
 #     def flatten(self) -> dict:
 #         return dict(self)
 # TODO: ``msgspec`` stuff worth paying attention to:
@ -138,18 +163,6 @@ class Status(Struct):
    brokerd_msg: dict = {}
 class Error(Status):
    resp: str = 'error'
    # TODO: allow re-wrapping from existing (last) status?
    @classmethod
    def from_status(
        cls,
        msg: Status,
    ) -> Error:
        ...
 # ---------------
 # emsd -> brokerd
 # ---------------
@ -213,7 +226,6 @@ class BrokerdOrderAck(Struct):
    # emsd id originally sent in matching request msg
    oid: str
    # TODO: do we need this?
    account: str = ''
    name: str = 'ack'
@ -226,14 +238,13 @@ class BrokerdStatus(Struct):
        'open',
        'canceled',
        'pending',
-        # 'error',  # NOTE: use `BrokerdError`
+        'error',
        'closed',
    ]
    name: str = 'status'
    oid: str = ''
    # TODO: do we need this?
    account: str | None = None,
    name: str = 'status'
    filled: float = 0.0
    reason: str = ''
    remaining: float = 0.0
@ -276,22 +287,20 @@ class BrokerdError(Struct):
    This is still a TODO thing since we're not sure how to employ it yet.
    '''
    oid: str
    reason: str
    # TODO: drop this right?
    symbol: str | None = None
    oid: str | None = None
    # if no brokerd order request was actually submitted (eg. we errored
    # at the ``pikerd`` layer) then there will be ``reqid`` allocated.
-    reqid: str | None = None
+    reqid: int | str | None = None
    name: str = 'error'
    broker_details: dict = {}
 # TODO: yeah, so we REALLY need to completely deprecate
 # this and use the `.accounting.Position` msg-type instead..
 class BrokerdPosition(Struct):
    '''
    Position update event from brokerd.
--- a/piker/clearing/_paper_engine.py
+++ b/piker/clearing/_paper_engine.py
@ -26,12 +26,10 @@ from contextlib import asynccontextmanager as acm
 from datetime import datetime
 from operator import itemgetter
 import itertools
 from pprint import pformat
 import time
 from typing import (
    Callable,
 )
 from types import ModuleType
 import uuid
 from bidict import bidict
@ -39,26 +37,22 @@ import pendulum
 import trio
 import tractor
-from piker.brokers import get_brokermod
+from ..brokers import get_brokermod
-from piker.service import find_service
+from .. import data
-from piker.accounting import (
+from ..data.types import Struct
-    Account,
+from ..accounting._mktinfo import (
    MktPair,
 )
 from ..accounting import (
    Position,
    PpTable,
    Transaction,
    TransactionLedger,
    open_account,
    open_trade_ledger,
-    unpack_fqme,
+    open_pps,
 )
-from piker.data import (
+from ..data import iterticks
-    Feed,
+from ..accounting import unpack_fqme
    SymbologyCache,
    iterticks,
    open_feed,
    open_symcache,
 )
 from piker.types import Struct
 from ._util import (
    log,  # sub-sys logger
    get_console_log,
@ -83,10 +77,11 @@ class PaperBoi(Struct):
    '''
    broker: str
    ems_trades_stream: tractor.MsgStream
-    acnt: Account
+
    ppt: PpTable
    ledger: TransactionLedger
    fees: Callable
    # map of paper "live" orders which be used
    # to simulate fills based on paper engine settings
@ -268,42 +263,29 @@ class PaperBoi(Struct):
        # we don't actually have any unique backend symbol ourselves
        # other then this thing, our fqme address.
        bs_mktid: str = fqme
        if fees := self.fees:
            cost: float = fees(price, size)
        else:
            cost: float = 0
        t = Transaction(
            fqme=fqme,
            sym=self._mkts[fqme],
            tid=oid,
            size=size,
            price=price,
-            cost=cost,
+            cost=0,  # TODO: cost model
            dt=pendulum.from_timestamp(fill_time_s),
            bs_mktid=bs_mktid,
        )
        # update in-mem ledger and pos table
        self.ledger.update_from_t(t)
-        self.acnt.update_from_ledger(
+        self.ppt.update_from_trans({oid: t})
            {oid: t},
            symcache=self.ledger._symcache,
            # XXX when a backend has no symcache support yet we can
            # simply pass in the gmi() retreived table created
            # during init :o
            _mktmap_table=self._mkts,
        )
        # transmit pp msg to ems
-        pp: Position = self.acnt.pps[bs_mktid]
+        pp = self.ppt.pps[bs_mktid]
        pp_msg = BrokerdPosition(
            broker=self.broker,
            account='paper',
            symbol=fqme,
-            size=pp.cumsize,
+            size=pp.size,
            avg_price=pp.ppu,
            # TODO: we need to look up the asset currency from
@ -314,7 +296,7 @@ class PaperBoi(Struct):
        # write all updates to filesys immediately
        # (adds latency but that works for simulation anyway)
        self.ledger.write_config()
-        self.acnt.write_config()
+        self.ppt.write_config()
        await self.ems_trades_stream.send(pp_msg)
@ -343,7 +325,6 @@ async def simulate_fills(
    # this stream may eventually contain multiple symbols
    async for quotes in quote_stream:
        for sym, quote in quotes.items():
            # print(sym)
            for tick in iterticks(
                quote,
                # dark order price filter(s)
@ -559,185 +540,139 @@ async def open_trade_dialog(
    # enable piker.clearing console log for *this* subactor
    get_console_log(loglevel)
-    symcache: SymbologyCache
+    ppt: PpTable
-    async with open_symcache(get_brokermod(broker)) as symcache:
+    ledger: TransactionLedger
    with (
        open_pps(
            broker,
            'paper',
            write_on_exit=True,
        ) as ppt,
-        acnt: Account
+        open_trade_ledger(
-        ledger: TransactionLedger
+            broker,
-        with (
+            'paper',
        ) as ledger
    ):
        # NOTE: retreive market(pair) info from the backend broker
        # since ledger entries (in their backend native format) often
        # don't contain necessary market info per trade record entry..
        # - if no fqme was passed in, we presume we're running in
        #   "ledger-sync-only mode" and thus we load mkt info for
        #   each symbol found in the ledger to a ppt table manually.
-            # TODO: probably do the symcache and ledger loading
+        # TODO: how to process ledger info from backends?
-            # implicitly behind this? Deliver an account, and ledger
+        # - should we be rolling our own actor-cached version of these
-            # pair or make the ledger an attr of the account?
+        # client API refs or using portal IPC to send requests to the
-            open_account(
+        # existing brokerd daemon?
-                broker,
+        # - alternatively we can possibly expect and use
-                'paper',
+        # a `.broker.norm_trade_records()` ep?
-                write_on_exit=True,
+        brokermod = get_brokermod(broker)
-            ) as acnt,
+        gmi = getattr(brokermod, 'get_mkt_info', None)
-            open_trade_ledger(
+        # update all transactions with mkt info before
-                broker,
+        # loading any pps
-                'paper',
+        mkt_by_fqme: dict[str, MktPair] = {}
-                symcache=symcache,
+        if fqme:
-            ) as ledger
+            bs_fqme, _, broker = fqme.rpartition('.')
-        ):
+            mkt, _ = await brokermod.get_mkt_info(bs_fqme)
-            # NOTE: WE MUST retreive market(pair) info from each
+            mkt_by_fqme[mkt.fqme] = mkt
            # backend broker since ledger entries (in their
            # provider-native format) often don't contain necessary
            # market info per trade record entry..
            # FURTHER, if no fqme was passed in, we presume we're
            # running in "ledger-sync-only mode" and thus we load
            # mkt info for each symbol found in the ledger to
            # an acnt table manually.
-            # TODO: how to process ledger info from backends?
+        # for each sym in the ledger load it's `MktPair` info
-            # - should we be rolling our own actor-cached version of these
+        for tid, txdict in ledger.data.items():
-            #   client API refs or using portal IPC to send requests to the
+            l_fqme: str = txdict.get('fqme') or txdict['fqsn']
            #   existing brokerd daemon?
            # - alternatively we can possibly expect and use
            #   a `.broker.ledger.norm_trade()` ep?
            brokermod: ModuleType = get_brokermod(broker)
            gmi: Callable = getattr(brokermod, 'get_mkt_info', None)
            # update all transactions with mkt info before
            # loading any pps
            mkt_by_fqme: dict[str, MktPair] = {}
            if (
-                fqme
+                gmi
-                and fqme not in symcache.mktmaps
+                and l_fqme not in mkt_by_fqme
            ):
-                log.warning(
+                mkt, pair = await brokermod.get_mkt_info(
-                    f'Symcache for {broker} has no `{fqme}` entry?\n'
+                    l_fqme.rstrip(f'.{broker}'),
                    'Manually requesting mkt map data via `.get_mkt_info()`..'
                )
                mkt_by_fqme[l_fqme] = mkt
-                bs_fqme, _, broker = fqme.rpartition('.')
+            # if an ``fqme: str`` input was provided we only
-                mkt, pair = await gmi(bs_fqme)
+            # need a ``MktPair`` for that one market, since we're
-                mkt_by_fqme[mkt.fqme] = mkt
+            # running in real simulated-clearing mode, not just ledger
            # syncing.
            if (
                fqme is not None
                and fqme in mkt_by_fqme
            ):
                break
-            # for each sym in the ledger load its `MktPair` info
+        # update pos table from ledger history and provide a ``MktPair``
-            for tid, txdict in ledger.data.items():
+        # lookup for internal position accounting calcs.
-                l_fqme: str = txdict.get('fqme') or txdict['fqsn']
+        ppt.update_from_trans(ledger.to_trans(mkt_by_fqme=mkt_by_fqme))
-                if (
+        pp_msgs: list[BrokerdPosition] = []
-                    gmi
+        pos: Position
-                    and l_fqme not in symcache.mktmaps
+        token: str  # f'{symbol}.{self.broker}'
-                    and l_fqme not in mkt_by_fqme
+        for token, pos in ppt.pps.items():
-                ):
+            pp_msgs.append(BrokerdPosition(
-                    log.warning(
+                broker=broker,
-                        f'Symcache for {broker} has no `{l_fqme}` entry?\n'
+                account='paper',
-                        'Manually requesting mkt map data via `.get_mkt_info()`..'
+                symbol=pos.mkt.fqme,
-                    )
+                size=pos.size,
-                    mkt, pair = await gmi(
+                avg_price=pos.ppu,
                        l_fqme.rstrip(f'.{broker}'),
                    )
                    mkt_by_fqme[l_fqme] = mkt
                # if an ``fqme: str`` input was provided we only
                # need a ``MktPair`` for that one market, since we're
                # running in real simulated-clearing mode, not just ledger
                # syncing.
                if (
                    fqme is not None
                    and fqme in mkt_by_fqme
                ):
                    break
            # update pos table from ledger history and provide a ``MktPair``
            # lookup for internal position accounting calcs.
            acnt.update_from_ledger(
                ledger,
                # NOTE: if the symcache fails on fqme lookup
                # (either sycache not yet supported or not filled
                # in) use manually constructed table from calling
                # the `.get_mkt_info()` provider EP above.
                _mktmap_table=mkt_by_fqme,
            )
            pp_msgs: list[BrokerdPosition] = []
            pos: Position
            token: str  # f'{symbol}.{self.broker}'
            for token, pos in acnt.pps.items():
                pp_msgs.append(BrokerdPosition(
                    broker=broker,
                    account='paper',
                    symbol=pos.mkt.fqme,
                    size=pos.cumsize,
                    avg_price=pos.ppu,
                ))
            await ctx.started((
                pp_msgs,
                ['paper'],
            ))
-            # write new positions state in case ledger was
+        await ctx.started((
-            # newer then that tracked in pps.toml
+            pp_msgs,
-            acnt.write_config()
+            ['paper'],
        ))
-            # exit early since no fqme was passed,
+        # write new positions state in case ledger was
-            # normally this case is just to load
+        # newer then that tracked in pps.toml
-            # positions "offline".
+        ppt.write_config()
-            if fqme is None:
+
-                log.warning(
+        # exit early since no fqme was passed,
-                    'Paper engine only running in position delivery mode!\n'
+        # normally this case is just to load
-                    'NO SIMULATED CLEARING LOOP IS ACTIVE!'
+        # positions "offline".
-                )
+        if fqme is None:
-                await trio.sleep_forever()
+            log.warning(
-                return
+                'Paper engine only running in position delivery mode!\n'
                'NO SIMULATED CLEARING LOOP IS ACTIVE!'
            )
            await trio.sleep_forever()
            return
        async with (
            data.open_feed(
                [fqme],
                loglevel=loglevel,
            ) as feed,
        ):
            # sanity check all the mkt infos
            for fqme, flume in feed.flumes.items():
                assert mkt_by_fqme[fqme] == flume.mkt
            feed: Feed
            async with (
-                open_feed(
+                ctx.open_stream() as ems_stream,
-                    [fqme],
+                trio.open_nursery() as n,
                    loglevel=loglevel,
                ) as feed,
            ):
-                # sanity check all the mkt infos
+                client = PaperBoi(
-                for fqme, flume in feed.flumes.items():
+                    broker=broker,
-                    mkt: MktPair = symcache.mktmaps.get(fqme) or mkt_by_fqme[fqme]
+                    ems_trades_stream=ems_stream,
-                    if mkt != flume.mkt:
+                    ppt=ppt,
-                        diff: tuple = mkt - flume.mkt
+                    ledger=ledger,
-                        log.warning(
+
-                            'MktPair sig mismatch?\n'
+                    _buys=_buys,
-                            f'{pformat(diff)}'
+                    _sells=_sells,
-                        )
+                    _reqids=_reqids,
                    _mkts=mkt_by_fqme,
                get_cost: Callable = getattr(
                    brokermod,
                    'get_cost',
                    None,
                )
-                async with (
+                n.start_soon(
-                    ctx.open_stream() as ems_stream,
+                    handle_order_requests,
-                    trio.open_nursery() as n,
+                    client,
-                ):
+                    ems_stream,
-                    client = PaperBoi(
+                )
                        broker=broker,
                        ems_trades_stream=ems_stream,
                        acnt=acnt,
                        ledger=ledger,
                        fees=get_cost,
-                        _buys=_buys,
+                # paper engine simulator clearing task
-                        _sells=_sells,
+                await simulate_fills(feed.streams[broker], client)
                        _reqids=_reqids,
                        _mkts=mkt_by_fqme,
                    )
                    n.start_soon(
                        handle_order_requests,
                        client,
                        ems_stream,
                    )
                    # paper engine simulator clearing task
                    await simulate_fills(feed.streams[broker], client)
@acm
@ -761,7 +696,7 @@ async def open_paperboi(
    service_name = f'paperboi.{broker}'
    async with (
-        find_service(service_name) as portal,
+        tractor.find_actor(service_name) as portal,
        tractor.open_nursery() as an,
    ):
        # NOTE: only spawn if no paperboi already is up since we likely
@ -784,59 +719,7 @@ async def open_paperboi(
        ) as (ctx, first):
            yield ctx, first
-            # ALWAYS tear down connection AND any newly spawned
+            # tear down connection and any spawned actor on exit
            # paperboi actor on exit!
            await ctx.cancel()
            if we_spawned:
                await portal.cancel_actor()
 def norm_trade(
    tid: str,
    txdict: dict,
    pairs: dict[str, Struct],
    symcache: SymbologyCache | None = None,
    brokermod: ModuleType | None = None,
 ) -> Transaction:
    from pendulum import (
        DateTime,
        parse,
    )
    # special field handling for datetimes
    # to ensure pendulum is used!
    dt: DateTime = parse(txdict['dt'])
    expiry: str | None = txdict.get('expiry')
    fqme: str = txdict.get('fqme') or txdict.pop('fqsn')
    price: float = txdict['price']
    size: float =  txdict['size']
    cost: float = txdict.get('cost', 0)
    if (
        brokermod
        and (get_cost := getattr(
            brokermod,
            'get_cost',
            False,
        ))
    ):
        cost = get_cost(
            price,
            size,
            is_taker=True,
        )
    return Transaction(
        fqme=fqme,
        tid=txdict['tid'],
        dt=dt,
        price=price,
        size=size,
        cost=cost,
        bs_mktid=txdict['bs_mktid'],
        expiry=parse(expiry) if expiry else None,
        etype='clear',
    )
--- a/piker/clearing/_util.py
+++ b/piker/clearing/_util.py
@ -25,7 +25,7 @@ from ..log import (
    get_logger,
    get_console_log,
 )
-from piker.types import Struct
+from piker.data.types import Struct
 subsys: str = 'piker.clearing'
 log = get_logger(subsys)
@ -36,6 +36,9 @@ get_console_log = partial(
 )
 # TODO: use this in other backends like kraken which currently has
 # a less formalized version more or less:
 # `apiflows[reqid].maps.append(status_msg.to_dict())`
 class OrderDialogs(Struct):
    '''
    Order control dialog (and thus transaction) tracking via
--- a/piker/cli/init.py
+++ b/piker/cli/init.py
@ -1,33 +1,30 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present Tyler Goodlet
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
 # (in stewardship for pikers, everywhere.)
-# This program is free software: you can redistribute it and/or
+# This program is free software: you can redistribute it and/or modify
-# modify it under the terms of the GNU Affero General Public
+# it under the terms of the GNU Affero General Public License as published by
-# License as published by the Free Software Foundation, either
+# the Free Software Foundation, either version 3 of the License, or
-# version 3 of the License, or (at your option) any later version.
+# (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# Affero General Public License for more details.
+# GNU Affero General Public License for more details.
-# You should have received a copy of the GNU Affero General Public
+# You should have received a copy of the GNU Affero General Public License
-# License along with this program.  If not, see
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
 # <https://www.gnu.org/licenses/>.
 '''
 CLI commons.
 '''
 import os
-# from contextlib import AsyncExitStack
+from contextlib import AsyncExitStack
 from types import ModuleType
 import click
 import trio
 import tractor
 from tractor._multiaddr import parse_maddr
 from ..log import (
    get_console_log,
@ -45,175 +42,89 @@ from .. import config
 log = get_logger('piker.cli')
 def load_trans_eps(
    network: dict | None = None,
    maddrs: list[tuple] | None = None,
 ) -> dict[str, dict[str, dict]]:
    # transport-oriented endpoint multi-addresses
    eps: dict[
        str,  # service name, eg. `pikerd`, `emsd`..
        # libp2p style multi-addresses parsed into prot layers
        list[dict[str, str | int]]
    ] = {}
    if (
        network
        and not maddrs
    ):
        # load network section and (attempt to) connect all endpoints
        # which are reachable B)
        for key, maddrs in network.items():
            match key:
                # TODO: resolve table across multiple discov
                # prots Bo
                case 'resolv':
                    pass
                case 'pikerd':
                    dname: str = key
                    for maddr in maddrs:
                        layers: dict = parse_maddr(maddr)
                        eps.setdefault(
                            dname,
                            [],
                        ).append(layers)
    elif maddrs:
        # presume user is manually specifying the root actor ep.
        eps['pikerd'] = [parse_maddr(maddr)]
    return eps
@click.command()
@click.option('--loglevel', '-l', default='warning', help='Logging level')
@click.option('--tl', is_flag=True, help='Enable tractor logging')
@click.option('--pdb', is_flag=True, help='Enable tractor debug mode')
@click.option('--host', '-h', default=None, help='Host addr to bind')
@click.option('--port', '-p', default=None, help='Port number to bind')
@click.option(
-    '--loglevel',
+    '--tsdb',
    '-l',
    default='warning',
    help='Logging level',
 )
@click.option(
    '--tl',
    is_flag=True,
-    help='Enable tractor-runtime logs',
+    help='Enable local ``marketstore`` instance'
 )
@click.option(
-    '--pdb',
+    '--es',
    is_flag=True,
-    help='Enable tractor debug mode',
+    help='Enable local ``elasticsearch`` instance'
 )
@click.option(
    '--maddr',
    '-m',
    default=None,
    help='Multiaddrs to bind or contact',
 )
 # @click.option(
 #     '--tsdb',
 #     is_flag=True,
 #     help='Enable local ``marketstore`` instance'
 # )
 # @click.option(
 #     '--es',
 #     is_flag=True,
 #     help='Enable local ``elasticsearch`` instance'
 # )
 def pikerd(
    maddr: list[str] | None,
    loglevel: str,
    host: str,
    port: int,
    tl: bool,
    pdb: bool,
-    # tsdb: bool,
+    tsdb: bool,
-    # es: bool,
+    es: bool,
 ):
    '''
    Spawn the piker broker-daemon.
    '''
-    from tractor.devx import maybe_open_crash_handler
+    log = get_console_log(loglevel, name='cli')
    with maybe_open_crash_handler(pdb=pdb):
        log = get_console_log(loglevel, name='cli')
-        if pdb:
+    if pdb:
-            log.warning((
+        log.warning((
-                "\n"
+            "\n"
-                "!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
+            "!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
-                "When a `piker` daemon crashes it will block the "
+            "When a `piker` daemon crashes it will block the "
-                "task-thread until resumed from console!\n"
+            "task-thread until resumed from console!\n"
-                "\n"
+            "\n"
-            ))
+        ))
-        # service-actor registry endpoint socket-address set
+    reg_addr: None | tuple[str, int] = None
-        regaddrs: list[tuple[str, int]] = []
+    if host or port:
-
+        reg_addr = (
-        conf, _ = config.load(
+            host or _default_registry_host,
-            conf_name='conf',
+            int(port) or _default_registry_port,
        )
-        network: dict = conf.get('network')
+
-        if (
+    from .. import service
-            network is None
+
-            and not maddr
+    async def main():
        service_mngr: service.Services
        async with (
            service.open_pikerd(
                loglevel=loglevel,
                debug_mode=pdb,
                registry_addr=reg_addr,
            ) as service_mngr,  # normally delivers a ``Services`` handle
            AsyncExitStack() as stack,
        ):
-            regaddrs = [(
+            if tsdb:
-                _default_registry_host,
+                dname, conf = await stack.enter_async_context(
-                _default_registry_port,
+                    service.marketstore.start_ahab_daemon(
-            )]
+                        service_mngr,
                        loglevel=loglevel,
                    )
                )
                log.info(f'TSDB `{dname}` up with conf:\n{conf}')
-        else:
+            if es:
-            eps: dict = load_trans_eps(
+                dname, conf = await stack.enter_async_context(
-                network,
+                    service.elastic.start_ahab_daemon(
-                maddr,
+                        service_mngr,
-            )
+                        loglevel=loglevel,
-            for layers in eps['pikerd']:
+                    )
-                regaddrs.append((
+                )
-                    layers['ipv4']['addr'],
+                log.info(f'DB `{dname}` up with conf:\n{conf}')
                    layers['tcp']['port'],
                ))
-        from .. import service
+            await trio.sleep_forever()
-        async def main():
+    trio.run(main)
            service_mngr: service.Services
            async with (
                service.open_pikerd(
                    registry_addrs=regaddrs,
                    loglevel=loglevel,
                    debug_mode=pdb,
                ) as service_mngr,  # normally delivers a ``Services`` handle
                # AsyncExitStack() as stack,
            ):
                # TODO: spawn all other sub-actor daemons according to
                # multiaddress endpoint spec defined by user config
                assert service_mngr
                # if tsdb:
                #     dname, conf = await stack.enter_async_context(
                #         service.marketstore.start_ahab_daemon(
                #             service_mngr,
                #             loglevel=loglevel,
                #         )
                #     )
                #     log.info(f'TSDB `{dname}` up with conf:\n{conf}')
                # if es:
                #     dname, conf = await stack.enter_async_context(
                #         service.elastic.start_ahab_daemon(
                #             service_mngr,
                #             loglevel=loglevel,
                #         )
                #     )
                #     log.info(f'DB `{dname}` up with conf:\n{conf}')
                await trio.sleep_forever()
        trio.run(main)
@click.group(context_settings=config._context_defaults)
@ -226,24 +137,8 @@ def pikerd(
@click.option('--loglevel', '-l', default='warning', help='Logging level')
@click.option('--tl', is_flag=True, help='Enable tractor logging')
@click.option('--configdir', '-c', help='Configuration directory')
-@click.option(
+@click.option('--host', '-h', default=None, help='Host addr to bind')
-    '--pdb',
+@click.option('--port', '-p', default=None, help='Port number to bind')
    is_flag=True,
    help='Enable runtime debug mode ',
 )
@click.option(
    '--maddr',
    '-m',
    default=None,
    multiple=True,
    help='Multiaddr to bind',
 )
@click.option(
    '--regaddr',
    '-r',
    default=None,
    help='Registrar addr to contact',
 )
@click.pass_context
 def cli(
    ctx: click.Context,
@ -251,11 +146,8 @@ def cli(
    loglevel: str,
    tl: bool,
    configdir: str,
-    pdb: bool,
+    host: str,
-
+    port: int,
    # TODO: make these list[str] with multiple -m maddr0 -m maddr1
    maddr: list[str],
    regaddr: str,
 ) -> None:
    if configdir is not None:
@ -276,20 +168,12 @@ def cli(
    }
    assert brokermods
-    # TODO: load endpoints from `conf::[network].pikerd`
+    reg_addr: None | tuple[str, int] = None
-    # - pikerd vs. regd, separate registry daemon?
+    if host or port:
-    # - expose datad vs. brokerd?
+        reg_addr = (
-    # - bind emsd with certain perms on public iface?
+            host or _default_registry_host,
-    regaddrs: list[tuple[str, int]] = regaddr or [(
+            int(port) or _default_registry_port,
-        _default_registry_host,
+        )
        _default_registry_port,
    )]
    # TODO: factor [network] section parsing out from pikerd
    # above and call it here as well.
    # if maddr:
    #     for addr in maddr:
    #         layers: dict = parse_maddr(addr)
    ctx.obj.update({
        'brokers': brokers,
@ -299,12 +183,7 @@ def cli(
        'log': get_console_log(loglevel),
        'confdir': config._config_dir,
        'wl_path': config._watchlists_data_path,
-        'registry_addrs': regaddrs,
+        'registry_addr': reg_addr,
        'pdb': pdb,  # debug mode flag
        # TODO: endpoint parsing, pinging and binding
        # on no existing server.
        # 'maddrs': maddr,
    })
    # allow enabling same loglevel in ``tractor`` machinery
@ -351,7 +230,8 @@ def services(config, tl, ports):
 def _load_clis() -> None:
-    # from ..service import elastic  # noqa
+    from ..service import marketstore  # noqa
    from ..service import elastic  # noqa
    from ..brokers import cli  # noqa
    from ..ui import cli  # noqa
    from ..watchlists import cli  # noqa
--- a/piker/config.py
+++ b/piker/config.py
@ -104,15 +104,14 @@ def get_app_dir(
    # `tractor`) with the testing dir and check for it whenever we
    # detect `pytest` is being used (which it isn't under normal
    # operation).
-    # if "pytest" in sys.modules:
+    if "pytest" in sys.modules:
-    #     import tractor
+        import tractor
-    #     actor = tractor.current_actor(err_on_no_runtime=False)
+        actor = tractor.current_actor(err_on_no_runtime=False)
-    #     if actor:  # runtime is up
+        if actor:  # runtime is up
-    #         rvs = tractor._state._runtime_vars
+            rvs = tractor._state._runtime_vars
-    #         import pdbp; pdbp.set_trace()
+            testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
-    #         testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
+            assert testdirpath.exists(), 'piker test harness might be borked!?'
-    #         assert testdirpath.exists(), 'piker test harness might be borked!?'
+            app_name = str(testdirpath)
    #         app_name = str(testdirpath)
    if platform.system() == 'Windows':
        key = "APPDATA" if roaming else "LOCALAPPDATA"
@ -135,19 +134,14 @@ def get_app_dir(
 _click_config_dir: Path = Path(get_app_dir('piker'))
 _config_dir: Path = _click_config_dir
 _parent_user: str = os.environ.get('SUDO_USER')
-# NOTE: when using `sudo` we attempt to determine the non-root user
+if _parent_user:
 # and still use their normal config dir.
 if (
    (_parent_user := os.environ.get('SUDO_USER'))
    and
    _parent_user != 'root'
 ):
    non_root_user_dir = Path(
        os.path.expanduser(f'~{_parent_user}')
    )
    root: str = 'root'
-    _ccds: str = str(_click_config_dir)  # click config dir as string
+    _ccds: str = str(_click_config_dir)  # click config dir string
    i_tail: int = int(_ccds.rfind(root) + len(root))
    _config_dir = (
        non_root_user_dir
@ -252,8 +246,7 @@ def repodir() -> Path:
 def load(
-    # NOTE: always appended with .toml suffix
+    conf_name: str = 'brokers',  # appended with .toml suffix
    conf_name: str = 'conf',
    path: Path | None = None,
    decode: Callable[
@ -364,9 +357,7 @@ def load_accounts(
 ) -> bidict[str, str | None]:
-    conf, path = load(
+    conf, path = load()
        conf_name='brokers',
    )
    accounts = bidict()
    for provider_name, section in conf.items():
        accounts_section = section.get('accounts')
@ -387,3 +378,51 @@ def load_accounts(
    accounts['paper'] = None
    return accounts
 # XXX: Recursive getting & setting
 def get_value(_dict, _section):
    subs = _section.split('.')
    if len(subs) > 1:
        return get_value(
            _dict[subs[0]],
            '.'.join(subs[1:]),
        )
    else:
        return _dict[_section]
 def set_value(_dict, _section, val):
    subs = _section.split('.')
    if len(subs) > 1:
        if subs[0] not in _dict:
            _dict[subs[0]] = {}
        return set_value(
            _dict[subs[0]],
            '.'.join(subs[1:]),
            val
        )
    else:
        _dict[_section] = val
 def del_value(_dict, _section):
    subs = _section.split('.')
    if len(subs) > 1:
        if subs[0] not in _dict:
            return
        return del_value(
            _dict[subs[0]],
            '.'.join(subs[1:])
        )
    else:
        if _section not in _dict:
            return
        del _dict[_section]
--- a/piker/data/init.py
+++ b/piker/data/init.py
@ -39,33 +39,18 @@ from .feed import (
    open_feed,
 )
 from .flows import Flume
 from ._symcache import (
    SymbologyCache,
    open_symcache,
    get_symcache,
    match_from_pairs,
 )
 from ._sampling import open_sample_stream
 from ..types import Struct
-__all__: list[str] = [
+__all__ = [
    'Flume',
    'Feed',
    'open_feed',
    'ShmArray',
    'iterticks',
    'maybe_open_shm_array',
    'match_from_pairs',
    'attach_shm_array',
    'open_shm_array',
    'get_shm_token',
    'def_iohlcv_fields',
    'def_ohlcv_fields',
    'open_symcache',
    'open_sample_stream',
    'get_symcache',
    'Struct',
    'SymbologyCache',
    'types',
 ]
--- a/piker/data/_formatters.py
+++ b/piker/data/_formatters.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -13,10 +13,10 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 Pre-(path)-graphics formatted x/y nd/1d rendering subsystem.
-'''
+"""
 from __future__ import annotations
 from typing import (
    Optional,
@ -39,12 +39,7 @@ if TYPE_CHECKING:
    from ._dataviz import (
        Viz,
    )
-    from piker.toolz import Profiler
+    from .._profile import Profiler
 # default gap between bars: "bar gap multiplier"
 # - 0.5 is no overlap between OC arms,
 # - 1.0 is full overlap on each neighbor sample
 BGM: float = 0.16
 class IncrementalFormatter(msgspec.Struct):
@ -518,7 +513,6 @@ class IncrementalFormatter(msgspec.Struct):
 class OHLCBarsFmtr(IncrementalFormatter):
    x_offset: np.ndarray = np.array([
        -0.5,
        0,
@ -610,9 +604,8 @@ class OHLCBarsFmtr(IncrementalFormatter):
        vr: tuple[int, int],
        start: int = 0,  # XXX: do we need this?
        # 0.5 is no overlap between arms, 1.0 is full overlap
-        gap: float = BGM,
+        w: float = 0.16,
    ) -> tuple[
        np.ndarray,
@ -629,7 +622,7 @@ class OHLCBarsFmtr(IncrementalFormatter):
            array[:-1],
            start,
            bar_w=self.index_step_size,
-            bar_gap=gap * self.index_step_size,
+            bar_gap=w * self.index_step_size,
            # XXX: don't ask, due to a ``numba`` bug..
            use_time_index=(self.index_field == 'time'),
--- a/piker/data/_pathops.py
+++ b/piker/data/_pathops.py
@ -17,6 +17,11 @@
 Super fast ``QPainterPath`` generation related operator routines.
 """
 from math import (
    ceil,
    floor,
 )
 import numpy as np
 from numpy.lib import recfunctions as rfn
 from numba import (
@ -30,6 +35,11 @@ from numba import (
 # TODO: for ``numba`` typing..
 # from ._source import numba_ohlc_dtype
 from ._m4 import ds_m4
 from .._profile import (
    Profiler,
    pg_profile_enabled,
    ms_slower_then,
 )
 def xy_downsample(
@ -125,7 +135,7 @@ def path_arrays_from_ohlc(
    half_w: float = bar_w/2
    # TODO: report bug for assert @
-    # ../piker/env/lib/python3.8/site-packages/numba/core/typing/builtins.py:991
+    # /home/goodboy/repos/piker/env/lib/python3.8/site-packages/numba/core/typing/builtins.py:991
    for i, q in enumerate(data[start:], start):
        open = q['open']
@ -227,20 +237,20 @@ def trace_hl(
    for i in range(hl.size):
        row = hl[i]
-        lo, hi = row['low'], row['high']
+        l, h = row['low'], row['high']
-        up_diff = hi - last_l
+        up_diff = h - last_l
-        down_diff = last_h - lo
+        down_diff = last_h - l
        if up_diff > down_diff:
-            out[2*i + 1] = hi
+            out[2*i + 1] = h
            out[2*i] = last_l
        else:
-            out[2*i + 1] = lo
+            out[2*i + 1] = l
            out[2*i] = last_h
-        last_l = lo
+        last_l = l
-        last_h = hi
+        last_h = h
        x[2*i] = int(i) - margin
        x[2*i + 1] = int(i) + margin
--- a/piker/data/_sampling.py
+++ b/piker/data/_sampling.py
@ -33,11 +33,6 @@ from typing import (
 )
 import tractor
 from tractor import (
    Context,
    MsgStream,
    Channel,
 )
 from tractor.trionics import (
    maybe_open_nursery,
 )
@ -58,10 +53,7 @@ if TYPE_CHECKING:
    from ._sharedmem import (
        ShmArray,
    )
-    from .feed import (
+    from .feed import _FeedsBus
        _FeedsBus,
        Sub,
    )
 # highest frequency sample step is 1 second by default, though in
@ -102,7 +94,7 @@ class Sampler:
        float,
        list[
            float,
-            set[MsgStream]
+            set[tractor.MsgStream]
        ],
    ] = defaultdict(
        lambda: [
@ -266,8 +258,8 @@ class Sampler:
            f'broadcasting {period_s} -> {last_ts}\n'
            # f'consumers: {subs}'
        )
-        borked: set[MsgStream] = set()
+        borked: set[tractor.MsgStream] = set()
-        sent: set[MsgStream] = set()
+        sent: set[tractor.MsgStream] = set()
        while True:
            try:
                for stream in (subs - sent):
@ -322,7 +314,7 @@ class Sampler:
@tractor.context
 async def register_with_sampler(
-    ctx: Context,
+    ctx: tractor.Context,
    period_s: float,
    shms_by_period: dict[float, dict] | None = None,
@ -657,7 +649,12 @@ async def sample_and_broadcast(
            # eventually block this producer end of the feed and
            # thus other consumers still attached.
            sub_key: str = broker_symbol.lower()
-            subs: set[Sub] = bus.get_subs(sub_key)
+            subs: list[
                tuple[
                    tractor.MsgStream | trio.MemorySendChannel,
                    float | None,  # tick throttle in Hz
                ]
            ] = bus.get_subs(sub_key)
            # NOTE: by default the broker backend doesn't append
            # it's own "name" into the fqme schema (but maybe it
@ -666,40 +663,34 @@ async def sample_and_broadcast(
            fqme: str = f'{broker_symbol}.{brokername}'
            lags: int = 0
-            # XXX TODO XXX: speed up this loop in an AOT compiled
+            # TODO: speed up this loop in an AOT compiled lang (like
-            # lang (like rust or nim or zig)!
+            # rust or nim or zig) and/or instead of doing a fan out to
-            # AND/OR instead of doing a fan out to TCP sockets
+            # TCP sockets here, we add a shm-style tick queue which
-            # here, we add a shm-style tick queue which readers can
+            # readers can pull from instead of placing the burden of
-            # pull from instead of placing the burden of broadcast
+            # broadcast on solely on this `brokerd` actor. see issues:
            # on solely on this `brokerd` actor. see issues:
            # - https://github.com/pikers/piker/issues/98
            # - https://github.com/pikers/piker/issues/107
-            # for (stream, tick_throttle) in subs.copy():
+            for (stream, tick_throttle) in subs.copy():
            for sub in subs.copy():
                ipc: MsgStream = sub.ipc
                throttle: float = sub.throttle_rate
                try:
                    with trio.move_on_after(0.2) as cs:
-                        if throttle:
+                        if tick_throttle:
                            send_chan: trio.abc.SendChannel = sub.send_chan
                            # this is a send mem chan that likely
                            # pushes to the ``uniform_rate_send()`` below.
                            try:
-                                send_chan.send_nowait(
+                                stream.send_nowait(
                                    (fqme, quote)
                                )
                            except trio.WouldBlock:
                                overruns[sub_key] += 1
-                                ctx: Context = ipc._ctx
+                                ctx = stream._ctx
-                                chan: Channel = ctx.chan
+                                chan = ctx.chan
                                log.warning(
                                    f'Feed OVERRUN {sub_key}'
                                    '@{bus.brokername} -> \n'
                                    f'feed @ {chan.uid}\n'
-                                    f'throttle = {throttle} Hz'
+                                    f'throttle = {tick_throttle} Hz'
                                )
                                if overruns[sub_key] > 6:
@ -716,33 +707,33 @@ async def sample_and_broadcast(
                                            f'{sub_key}:'
                                            f'{ctx.cid}@{chan.uid}'
                                        )
-                                        await ipc.aclose()
+                                        await stream.aclose()
                                        raise trio.BrokenResourceError
                        else:
-                            await ipc.send(
+                            await stream.send(
                                {fqme: quote}
                            )
                    if cs.cancelled_caught:
                        lags += 1
                        if lags > 10:
-                            await tractor.pause()
+                            await tractor.breakpoint()
                except (
                    trio.BrokenResourceError,
                    trio.ClosedResourceError,
                    trio.EndOfChannel,
                ):
-                    ctx: Context = ipc._ctx
+                    ctx = stream._ctx
-                    chan: Channel = ctx.chan
+                    chan = ctx.chan
                    if ctx:
                        log.warning(
                            'Dropped `brokerd`-quotes-feed connection:\n'
                            f'{broker_symbol}:'
                            f'{ctx.cid}@{chan.uid}'
                        )
-                    if sub.throttle_rate:
+                    if tick_throttle:
-                        assert ipc._closed
+                        assert stream._closed
                    # XXX: do we need to deregister here
                    # if it's done in the fee bus code?
@ -751,7 +742,7 @@ async def sample_and_broadcast(
                    # since there seems to be some kinda race..
                    bus.remove_subs(
                        sub_key,
-                        {sub},
+                        {(stream, tick_throttle)},
                    )
@ -759,7 +750,7 @@ async def uniform_rate_send(
    rate: float,
    quote_stream: trio.abc.ReceiveChannel,
-    stream: MsgStream,
+    stream: tractor.MsgStream,
    task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
--- a/piker/data/_sharedmem.py
+++ b/piker/data/_sharedmem.py
@ -34,7 +34,7 @@ import tractor
 from ._util import log
 from ._source import def_iohlcv_fields
-from piker.types import Struct
+from .types import Struct
 def cuckoff_mantracker():
--- a/piker/data/_symcache.py
+++ b/piker/data/_symcache.py
@ -1,510 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Mega-simple symbology cache via TOML files.
 Allow backend data providers and/or brokers to stash their
 symbology sets (aka the meta data we normalize into our
 `.accounting.MktPair` type) to the filesystem for faster lookup and
 offline usage.
 '''
 from __future__ import annotations
 from contextlib import (
    asynccontextmanager as acm,
 )
 from pathlib import Path
 from pprint import pformat
 from typing import (
    Any,
    Sequence,
    Hashable,
    TYPE_CHECKING,
 )
 from types import ModuleType
 from rapidfuzz import process as fuzzy
 import tomli_w  # for fast symbol cache writing
 import tractor
 import trio
 try:
    import tomllib
 except ModuleNotFoundError:
    import tomli as tomllib
 from msgspec import field
 from piker.log import get_logger
 from piker import config
 from piker.types import Struct
 from piker.brokers import (
    open_cached_client,
    get_brokermod,
 )
 if TYPE_CHECKING:
    from ..accounting import (
        Asset,
        MktPair,
    )
 log = get_logger('data.cache')
 class SymbologyCache(Struct):
    '''
    Asset meta-data cache which holds lookup tables for 3 sets of
    market-symbology related struct-types required by the
    `.accounting` and `.data` subsystems.
    '''
    mod: ModuleType
    fp: Path
    # all asset-money-systems descriptions as minimally defined by
    # in `.accounting.Asset`
    assets: dict[str, Asset] = field(default_factory=dict)
    # backend-system pairs loaded in provider (schema) specific
    # structs.
    pairs: dict[str, Struct] = field(default_factory=dict)
    # serialized namespace path to the backend's pair-info-`Struct`
    # defn B)
    pair_ns_path: tractor.msg.NamespacePath | None = None
    # TODO: piker-normalized `.accounting.MktPair` table?
    # loaded from the `.pairs` and a normalizer
    # provided by the backend pkg.
    mktmaps: dict[str, MktPair] = field(default_factory=dict)
    def write_config(self) -> None:
        # put the backend's pair-struct type ref at the top
        # of file if possible.
        cachedict: dict[str, Any] = {
            'pair_ns_path': str(self.pair_ns_path) or '',
        }
        # serialize all tables as dicts for TOML.
        for key, table in {
            'assets': self.assets,
            'pairs': self.pairs,
            'mktmaps': self.mktmaps,
        }.items():
            if not table:
                log.warning(
                    f'Asset cache table for `{key}` is empty?'
                )
                continue
            dct = cachedict[key] = {}
            for key, struct in table.items():
                dct[key] = struct.to_dict(include_non_members=False)
        try:
            with self.fp.open(mode='wb') as fp:
                tomli_w.dump(cachedict, fp)
        except TypeError:
            self.fp.unlink()
            raise
    async def load(self) -> None:
        '''
        Explicitly load the "symbology set" for this provider by using
        2 required `Client` methods:
          - `.get_assets()`: returning a table of `Asset`s
          - `.get_mkt_pairs()`: returning a table of pair-`Struct`
            types, custom defined by the particular backend.
        AND, the required `.get_mkt_info()` module-level endpoint
        which maps `fqme: str` -> `MktPair`s.
        These tables are then used to fill out the `.assets`, `.pairs` and
        `.mktmaps` tables on this cache instance, respectively.
        '''
        async with open_cached_client(self.mod.name) as client:
            if get_assets := getattr(client, 'get_assets', None):
                assets: dict[str, Asset] = await get_assets()
                for bs_mktid, asset in assets.items():
                    self.assets[bs_mktid] = asset
            else:
                log.warning(
                    'No symbology cache `Asset` support for `{provider}`..\n'
                    'Implement `Client.get_assets()`!'
                )
            if get_mkt_pairs := getattr(client, 'get_mkt_pairs', None):
                pairs: dict[str, Struct] = await get_mkt_pairs()
                for bs_fqme, pair in pairs.items():
                    # NOTE: every backend defined pair should
                    # declare it's ns path for roundtrip
                    # serialization lookup.
                    if not getattr(pair, 'ns_path', None):
                        raise TypeError(
                            f'Pair-struct for {self.mod.name} MUST define a '
                            '`.ns_path: str`!\n'
                            f'{pair}'
                        )
                    entry = await self.mod.get_mkt_info(pair.bs_fqme)
                    if not entry:
                        continue
                    mkt: MktPair
                    pair: Struct
                    mkt, _pair = entry
                    assert _pair is pair, (
                        f'`{self.mod.name}` backend probably has a '
                        'keying-symmetry problem between the pair-`Struct` '
                        'returned from `Client.get_mkt_pairs()`and the '
                        'module level endpoint: `.get_mkt_info()`\n\n'
                        "Here's the struct diff:\n"
                        f'{_pair - pair}'
                    )
                    # NOTE XXX: this means backends MUST implement
                    # a `Struct.bs_mktid: str` field to provide
                    # a native-keyed map to their own symbol
                    # set(s).
                    self.pairs[pair.bs_mktid] = pair
                    # NOTE: `MktPair`s are keyed here using piker's
                    # internal FQME schema so that search,
                    # accounting and feed init can be accomplished
                    # a sane, uniform, normalized basis.
                    self.mktmaps[mkt.fqme] = mkt
                self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
                    pair,
                )
            else:
                log.warning(
                    'No symbology cache `Pair` support for `{provider}`..\n'
                    'Implement `Client.get_mkt_pairs()`!'
                )
        return self
    @classmethod
    def from_dict(
        cls: type,
        data: dict,
        **kwargs,
    ) -> SymbologyCache:
        # normal init inputs
        cache = cls(**kwargs)
        # XXX WARNING: this may break if backend namespacing
        # changes (eg. `Pair` class def is moved to another
        # module) in which case you can manually update the
        # `pair_ns_path` in the symcache file and try again.
        # TODO: probably a verbose error about this?
        Pair: type = tractor.msg.NamespacePath(
            str(data['pair_ns_path'])
        ).load_ref()
        pairtable = data.pop('pairs')
        for key, pairtable in pairtable.items():
            # allow each serialized pair-dict-table to declare its
            # specific struct type's path in cases where a backend
            # supports multiples (normally with different
            # schemas..) and we are storing them in a flat `.pairs`
            # table.
            ThisPair = Pair
            if this_pair_type := pairtable.get('ns_path'):
                ThisPair: type = tractor.msg.NamespacePath(
                    str(this_pair_type)
                ).load_ref()
            pair: Struct = ThisPair(**pairtable)
            cache.pairs[key] = pair
        from ..accounting import (
            Asset,
            MktPair,
        )
        # load `dict` -> `Asset`
        assettable = data.pop('assets')
        for name, asdict in assettable.items():
            cache.assets[name] = Asset.from_msg(asdict)
        # load `dict` -> `MktPair`
        dne: list[str] = []
        mkttable = data.pop('mktmaps')
        for fqme, mktdict in mkttable.items():
            mkt = MktPair.from_msg(mktdict)
            assert mkt.fqme == fqme
            # sanity check asset refs from those (presumably)
            # loaded asset set above.
            src: Asset = cache.assets[mkt.src.name]
            assert src == mkt.src
            dst: Asset
            if not (dst := cache.assets.get(mkt.dst.name)):
                dne.append(mkt.dst.name)
                continue
            else:
                assert dst.name == mkt.dst.name
            cache.mktmaps[fqme] = mkt
        log.warning(
            f'These `MktPair.dst: Asset`s DNE says `{cache.mod.name}`?\n'
            f'{pformat(dne)}'
        )
        return cache
    @staticmethod
    async def from_scratch(
        mod: ModuleType,
        fp: Path,
        **kwargs,
    ) -> SymbologyCache:
        '''
        Generate (a) new symcache (contents) entirely from scratch
        including all (TOML) serialized data and file.
        '''
        log.info(f'GENERATING symbology cache for `{mod.name}`')
        cache = SymbologyCache(
            mod=mod,
            fp=fp,
            **kwargs,
        )
        await cache.load()
        cache.write_config()
        return cache
    def search(
        self,
        pattern: str,
        table: str = 'mktmaps'
    ) -> dict[str, Struct]:
        '''
        (Fuzzy) search this cache's `.mktmaps` table, which is
        keyed by FQMEs, for `pattern: str` and return the best
        matches in a `dict` including the `MktPair` values.
        '''
        matches = fuzzy.extract(
            pattern,
            getattr(self, table),
            score_cutoff=50,
        )
        # repack in dict[fqme, MktPair] form
        return {
            item[0].fqme: item[0]
            for item in matches
        }
 # actor-process-local in-mem-cache of symcaches (by backend).
 _caches: dict[str, SymbologyCache] = {}
 def mk_cachefile(
    provider: str,
 ) -> Path:
    cachedir: Path = config.get_conf_dir() / '_cache'
    if not cachedir.is_dir():
        log.info(f'Creating `nativedb` director: {cachedir}')
        cachedir.mkdir()
    cachefile: Path = cachedir / f'{str(provider)}.symcache.toml'
    cachefile.touch()
    return cachefile
@acm
 async def open_symcache(
    mod_or_name: ModuleType | str,
    reload: bool = False,
    only_from_memcache: bool = False,  # no API req
    _no_symcache: bool = False,  # no backend support
 ) -> SymbologyCache:
    if isinstance(mod_or_name, str):
        mod = get_brokermod(mod_or_name)
    else:
        mod: ModuleType = mod_or_name
    provider: str = mod.name
    cachefile: Path = mk_cachefile(provider)
    # NOTE: certain backends might not support a symbology cache
    # (easily) and thus we allow for an empty instance to be loaded
    # and manually filled in at the whim of the caller presuming
    # the backend pkg-module is annotated appropriately.
    if (
        getattr(mod, '_no_symcache', False)
        or _no_symcache
    ):
        yield SymbologyCache(
            mod=mod,
            fp=cachefile,
        )
        # don't do nuttin
        return
    # actor-level cache-cache XD
    global _caches
    if not reload:
        try:
            yield _caches[provider]
        except KeyError:
            msg: str = (
                f'No asset info cache exists yet for `{provider}`'
            )
            if only_from_memcache:
                raise RuntimeError(msg)
            else:
                log.warning(msg)
    # if no cache exists or an explicit reload is requested, load
    # the provider API and call appropriate endpoints to populate
    # the mkt and asset tables.
    if (
        reload
        or not cachefile.is_file()
    ):
        cache = await SymbologyCache.from_scratch(
            mod=mod,
            fp=cachefile,
        )
    else:
        log.info(
            f'Loading EXISTING `{mod.name}` symbology cache:\n'
            f'> {cachefile}'
        )
        import time
        now = time.time()
        with cachefile.open('rb') as existing_fp:
            data: dict[str, dict] = tomllib.load(existing_fp)
            log.runtime(f'SYMCACHE TOML LOAD TIME: {time.time() - now}')
            # if there's an empty file for some reason we need
            # to do a full reload as well!
            if not data:
                cache = await SymbologyCache.from_scratch(
                    mod=mod,
                    fp=cachefile,
                )
            else:
                cache = SymbologyCache.from_dict(
                    data,
                    mod=mod,
                    fp=cachefile,
                )
        # TODO: use a real profiling sys..
        # https://github.com/pikers/piker/issues/337
        log.info(f'SYMCACHE LOAD TIME: {time.time() - now}')
    yield cache
    # TODO: write only when changes detected? but that should
    # never happen right except on reload?
    # cache.write_config()
 def get_symcache(
    provider: str,
    force_reload: bool = False,
 ) -> SymbologyCache:
    '''
    Get any available symbology/assets cache from sync code by
    (maybe) manually running `trio` to do the work.
    '''
    # spawn tractor runtime and generate cache
    # if not existing.
    async def sched_gen_symcache():
        async with (
            # only for runtime's debug mode
            tractor.open_nursery(debug_mode=True),
            open_symcache(
                get_brokermod(provider),
                reload=force_reload,
            ) as symcache,
        ):
            return symcache
    try:
        symcache: SymbologyCache = trio.run(sched_gen_symcache)
        assert symcache
    except BaseException:
        import pdbp
        pdbp.xpm()
    return symcache
 def match_from_pairs(
    pairs: dict[str, Struct],
    query: str,
    score_cutoff: int = 50,
    **extract_kwargs,
 ) -> dict[str, Struct]:
    '''
    Fuzzy search over a "pairs table" maintained by most backends
    as part of their symbology-info caching internals.
    Scan the native symbol key set and return best ranked
    matches back in a new `dict`.
    '''
    # TODO: somehow cache this list (per call) like we were in
    # `open_symbol_search()`?
    keys: list[str] = list(pairs)
    matches: list[tuple[
        Sequence[Hashable],  # matching input key
        Any,  # scores
        Any,
    ]] = fuzzy.extract(
        # NOTE: most backends provide keys uppercased
        query=query,
        choices=keys,
        score_cutoff=score_cutoff,
        **extract_kwargs,
    )
    # pop and repack pairs in output dict
    matched_pairs: dict[str, Struct] = {}
    for item in matches:
        pair_key: str = item[0]
        matched_pairs[pair_key] = pairs[pair_key]
    return matched_pairs
--- a/piker/data/_timeseries.py
+++ b/piker/data/_timeseries.py
@ -0,0 +1,326 @@
 # piker: trading gear for hackers
 # Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Financial time series processing utilities usually
 pertaining to OHLCV style sampled data.
 Routines are generally implemented in either ``numpy`` or ``polars`` B)
 '''
 from __future__ import annotations
 from typing import Literal
 from math import (
    ceil,
    floor,
 )
 import numpy as np
 import polars as pl
 from ._sharedmem import ShmArray
 from .._profile import (
    Profiler,
    pg_profile_enabled,
    ms_slower_then,
 )
 def slice_from_time(
    arr: np.ndarray,
    start_t: float,
    stop_t: float,
    step: float,  # sampler period step-diff
 ) -> slice:
    '''
    Calculate array indices mapped from a time range and return them in
    a slice.
    Given an input array with an epoch `'time'` series entry, calculate
    the indices which span the time range and return in a slice. Presume
    each `'time'` step increment is uniform and when the time stamp
    series contains gaps (the uniform presumption is untrue) use
    ``np.searchsorted()`` binary search to look up the appropriate
    index.
    '''
    profiler = Profiler(
        msg='slice_from_time()',
        disabled=not pg_profile_enabled(),
        ms_threshold=ms_slower_then,
    )
    times = arr['time']
    t_first = floor(times[0])
    t_last = ceil(times[-1])
    # the greatest index we can return which slices to the
    # end of the input array.
    read_i_max = arr.shape[0]
    # compute (presumed) uniform-time-step index offsets
    i_start_t = floor(start_t)
    read_i_start = floor(((i_start_t - t_first) // step)) - 1
    i_stop_t = ceil(stop_t)
    # XXX: edge case -> always set stop index to last in array whenever
    # the input stop time is detected to be greater then the equiv time
    # stamp at that last entry.
    if i_stop_t >= t_last:
        read_i_stop = read_i_max
    else:
        read_i_stop = ceil((i_stop_t - t_first) // step) + 1
    # always clip outputs to array support
    # for read start:
    # - never allow a start < the 0 index
    # - never allow an end index > the read array len
    read_i_start = min(
        max(0, read_i_start),
        read_i_max - 1,
    )
    read_i_stop = max(
        0,
        min(read_i_stop, read_i_max),
    )
    # check for larger-then-latest calculated index for given start
    # time, in which case we do a binary search for the correct index.
    # NOTE: this is usually the result of a time series with time gaps
    # where it is expected that each index step maps to a uniform step
    # in the time stamp series.
    t_iv_start = times[read_i_start]
    if (
        t_iv_start > i_start_t
    ):
        # do a binary search for the best index mapping to ``start_t``
        # given we measured an overshoot using the uniform-time-step
        # calculation from above.
        # TODO: once we start caching these per source-array,
        # we can just overwrite ``read_i_start`` directly.
        new_read_i_start = np.searchsorted(
            times,
            i_start_t,
            side='left',
        )
        # TODO: minimize binary search work as much as possible:
        # - cache these remap values which compensate for gaps in the
        #   uniform time step basis where we calc a later start
        #   index for the given input ``start_t``.
        # - can we shorten the input search sequence by heuristic?
        #   up_to_arith_start = index[:read_i_start]
        if (
            new_read_i_start <= read_i_start
        ):
            # t_diff = t_iv_start - start_t
            # print(
            #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
            #     f'start_t:{start_t} -> 0index start_t:{t_iv_start}\n'
            #     f'diff: {t_diff}\n'
            #     f'REMAPPED START i: {read_i_start} -> {new_read_i_start}\n'
            # )
            read_i_start = new_read_i_start
    t_iv_stop = times[read_i_stop - 1]
    if (
        t_iv_stop > i_stop_t
    ):
        # t_diff = stop_t - t_iv_stop
        # print(
        #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
        #     f'calced iv stop:{t_iv_stop} -> stop_t:{stop_t}\n'
        #     f'diff: {t_diff}\n'
        #     # f'SHOULD REMAP STOP: {read_i_start} -> {new_read_i_start}\n'
        # )
        new_read_i_stop = np.searchsorted(
            times[read_i_start:],
            # times,
            i_stop_t,
            side='right',
        )
        if (
            new_read_i_stop <= read_i_stop
        ):
            read_i_stop = read_i_start + new_read_i_stop + 1
    # sanity checks for range size
    # samples = (i_stop_t - i_start_t) // step
    # index_diff = read_i_stop - read_i_start + 1
    # if index_diff > (samples + 3):
    #     breakpoint()
    # read-relative indexes: gives a slice where `shm.array[read_slc]`
    # will be the data spanning the input time range `start_t` ->
    # `stop_t`
    read_slc = slice(
        int(read_i_start),
        int(read_i_stop),
    )
    profiler(
        'slicing complete'
        # f'{start_t} -> {abs_slc.start} | {read_slc.start}\n'
        # f'{stop_t} -> {abs_slc.stop} | {read_slc.stop}\n'
    )
    # NOTE: if caller needs absolute buffer indices they can
    # slice the buffer abs index like so:
    # index = arr['index']
    # abs_indx = index[read_slc]
    # abs_slc = slice(
    #     int(abs_indx[0]),
    #     int(abs_indx[-1]),
    # )
    return read_slc
 def detect_null_time_gap(
    shm: ShmArray,
    imargin: int = 1,
 ) -> tuple[float, float] | None:
    '''
    Detect if there are any zero-epoch stamped rows in
    the presumed 'time' field-column.
    Filter to the gap and return a surrounding index range.
    NOTE: for now presumes only ONE gap XD
    '''
    zero_pred: np.ndarray = shm.array['time'] == 0
    zero_t: np.ndarray = shm.array[zero_pred]
    if zero_t.size:
        istart, iend = zero_t['index'][[0, -1]]
        start, end = shm._array['time'][
            [istart - imargin, iend + imargin]
        ]
        return (
            istart - imargin,
            start,
            end,
            iend + imargin,
        )
    return None
 t_unit: Literal[
    'days',
    'hours',
    'minutes',
    'seconds',
    'miliseconds',
    'microseconds',
    'nanoseconds',
 ]
 def with_dts(
    df: pl.DataFrame,
    time_col: str = 'time',
 ) -> pl.DataFrame:
    '''
    Insert datetime (casted) columns to a (presumably) OHLC sampled
    time series with an epoch-time column keyed by ``time_col``.
    '''
    return df.with_columns([
        pl.col(time_col).shift(1).suffix('_prev'),
        pl.col(time_col).diff().alias('s_diff'),
        pl.from_epoch(pl.col(time_col)).alias('dt'),
    ]).with_columns([
        pl.from_epoch(pl.col(f'{time_col}_prev')).alias('dt_prev'),
        pl.col('dt').diff().alias('dt_diff'),
    ]) #.with_columns(
        # pl.col('dt').diff().dt.days().alias('days_dt_diff'),
    # )
 def detect_time_gaps(
    df: pl.DataFrame,
    time_col: str = 'time',
    # epoch sampling step diff
    expect_period: float = 60,
    # datetime diff unit and gap value
    # crypto mkts
    # gap_dt_unit: t_unit = 'minutes',
    # gap_thresh: int = 1,
    # legacy stock mkts
    gap_dt_unit: t_unit = 'days',
    gap_thresh: int = 2,
 ) -> pl.DataFrame:
    '''
    Filter to OHLC datums which contain sample step gaps.
    For eg. legacy markets which have venue close gaps and/or
    actual missing data segments.
    '''
    dt_gap_col: str = f'{gap_dt_unit}_diff'
    return with_dts(
        df
    ).filter(
        pl.col('s_diff').abs() > expect_period
    ).with_columns(
        getattr(
            pl.col('dt_diff').dt,
            gap_dt_unit,  # NOTE: must be valid ``Expr.dt.<name>``
        )().alias(dt_gap_col)
    ).filter(
        pl.col(dt_gap_col).abs() > gap_thresh
    )
 def detect_price_gaps(
    df: pl.DataFrame,
    gt_multiplier: float = 2.,
    price_fields: list[str] = ['high', 'low'],
 ) -> pl.DataFrame:
    '''
    Detect gaps in clearing price over an OHLC series.
    2 types of gaps generally exist; up gaps and down gaps:
    - UP gap: when any next sample's lo price is strictly greater
      then the current sample's hi price.
    - DOWN gap: when any next sample's hi price is strictly
      less then the current samples lo price.
    '''
    # return df.filter(
    #     pl.col('high') - ) > expect_period,
    # ).select([
    #     pl.dt.datetime(pl.col(time_col).shift(1)).suffix('_previous'),
    #     pl.all(),
    # ]).select([
    #     pl.all(),
    #     (pl.col(time_col) - pl.col(f'{time_col}_previous')).alias('diff'),
    # ])
    ...
--- a/piker/data/_web_bs.py
+++ b/piker/data/_web_bs.py
@ -50,8 +50,8 @@ from trio_websocket._impl import (
    ConnectionTimeout,
 )
 from piker.types import Struct
 from ._util import log
 from .types import Struct
 class NoBsWs:
@ -234,13 +234,10 @@ async def _reconnect_forever(
            f'{url} trying (RE)CONNECT'
        )
-        ws: WebSocketConnection
+        async with trio.open_nursery() as n:
-        try:
+            cs = nobsws._cs = n.cancel_scope
-            async with (
+            ws: WebSocketConnection
-                trio.open_nursery() as n,
+            async with open_websocket_url(url) as ws:
                open_websocket_url(url) as ws,
            ):
                cs = nobsws._cs = n.cancel_scope
                nobsws._ws = ws
                log.info(
                    f'{src_mod}\n'
@ -272,11 +269,9 @@ async def _reconnect_forever(
                # to let tasks run **inside** the ws open block above.
                nobsws._connected.set()
                await trio.sleep_forever()
        except HandshakeError:
            log.exception('Retrying connection')
        # ws & nursery block ends
            # ws open block end
        # nursery block end
        nobsws._connected = trio.Event()
        if cs.cancelled_caught:
            log.cancel(
@ -289,8 +284,7 @@ async def _reconnect_forever(
                and not nobsws._connected.is_set()
            )
-        # -> from here, move to next reconnect attempt iteration
+        # -> from here, move to next reconnect attempt
        # in the while loop above Bp
    else:
        log.exception(
@ -359,8 +353,8 @@ async def open_autorecon_ws(
 '''
-JSONRPC response-request style machinery for transparent multiplexing
+JSONRPC response-request style machinery for transparent multiplexing of msgs
-of msgs over a `NoBsWs`.
+over a NoBsWs.
 '''
@ -377,82 +371,43 @@ async def open_jsonrpc_session(
    url: str,
    start_id: int = 0,
    response_type: type = JSONRPCResult,
-    msg_recv_timeout: float = float('inf'),
+    request_type: Optional[type] = None,
-    # ^NOTE, since only `deribit` is using this jsonrpc stuff atm
+    request_hook: Optional[Callable] = None,
-    # and options mkts are generally "slow moving"..
+    error_hook: Optional[Callable] = None,
    #
    # FURTHER if we break the underlying ws connection then since we
    # don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
    # `_reconnect_forever()`, the jsonrpc "transport pipe" get's
    # broken and never restored with wtv init sequence is required to
    # re-establish a working req-resp session.
 ) -> Callable[[str, dict], dict]:
    '''
    Init a json-RPC-over-websocket connection to the provided `url`.
    A `json_rpc: Callable[[str, dict], dict` is delivered to the
    caller for sending requests and a bg-`trio.Task` handles
    processing of response msgs including error reporting/raising in
    the parent/caller task.
    '''
    # NOTE, store all request msgs so we can raise errors on the
    # caller side!
    req_msgs: dict[int, dict] = {}
    async with (
-        trio.open_nursery() as tn,
+        trio.open_nursery() as n,
-        open_autorecon_ws(
+        open_autorecon_ws(url) as ws
            url=url,
            msg_recv_timeout=msg_recv_timeout,
        ) as ws
    ):
-        rpc_id: Iterable[int] = count(start_id)
+        rpc_id: Iterable = count(start_id)
        rpc_results: dict[int, dict] = {}
-        async def json_rpc(
+        async def json_rpc(method: str, params: dict) -> dict:
            method: str,
            params: dict,
        ) -> dict:
            '''
            perform a json rpc call and wait for the result, raise exception in
            case of error field present on response
            '''
            nonlocal req_msgs
            req_id: int = next(rpc_id)
            msg = {
                'jsonrpc': '2.0',
-                'id': req_id,
+                'id': next(rpc_id),
                'method': method,
                'params': params
            }
            _id = msg['id']
-            result = rpc_results[_id] = {
+            rpc_results[_id] = {
                'result': None,
-                'error': None,
+                'event': trio.Event()
                'event': trio.Event(),  # signal caller resp arrived
            }
            req_msgs[_id] = msg
            await ws.send_msg(msg)
            # wait for reponse before unblocking requester code
            await rpc_results[_id]['event'].wait()
-            if (maybe_result := result['result']):
+            ret = rpc_results[_id]['result']
                ret = maybe_result
                del rpc_results[_id]
-            else:
+            del rpc_results[_id]
                err = result['error']
                raise Exception(
                    f'JSONRPC request failed\n'
                    f'req: {msg}\n'
                    f'resp: {err}\n'
                )
            if ret.error is not None:
                raise Exception(json.dumps(ret.error, indent=4))
@ -467,7 +422,6 @@ async def open_jsonrpc_session(
            the server side.
            '''
            nonlocal req_msgs
            async for msg in ws:
                match msg:
                    case {
@ -491,28 +445,19 @@ async def open_jsonrpc_session(
                        'params': _,
                    }:
                        log.debug(f'Recieved\n{msg}')
                        if request_hook:
                            await request_hook(request_type(**msg))
                    case {
                        'error': error
                    }:
-                        # retreive orig request msg, set error
+                        log.warning(f'Recieved\n{error}')
-                        # response in original "result" msg,
+                        if error_hook:
-                        # THEN FINALLY set the event to signal caller
+                            await error_hook(response_type(**msg))
                        # to raise the error in the parent task.
                        req_id: int = error['id']
                        req_msg: dict = req_msgs[req_id]
                        result: dict = rpc_results[req_id]
                        result['error'] = error
                        result['event'].set()
                        log.error(
                            f'JSONRPC request failed\n'
                            f'req: {req_msg}\n'
                            f'resp: {error}\n'
                        )
                    case _:
                        log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
-        tn.start_soon(recv_task)
+        n.start_soon(recv_task)
        yield json_rpc
-        tn.cancel_scope.cancel()
+        n.cancel_scope.cancel()
--- a/piker/data/feed.py
+++ b/piker/data/feed.py
@ -28,7 +28,6 @@ module.
 from __future__ import annotations
 from collections import (
    defaultdict,
    abc,
 )
 from contextlib import asynccontextmanager as acm
 from functools import partial
@ -37,6 +36,7 @@ from types import ModuleType
 from typing import (
    Any,
    AsyncContextManager,
    Optional,
    Awaitable,
    Sequence,
 )
@ -45,62 +45,41 @@ import trio
 from trio.abc import ReceiveChannel
 from trio_typing import TaskStatus
 import tractor
-from tractor import trionics
+from tractor.trionics import (
    maybe_open_context,
    gather_contexts,
 )
-from piker.accounting import (
+from ..brokers import get_brokermod
-    MktPair,
+from ..calc import humanize
    unpack_fqme,
 )
 from piker.types import Struct
 from piker.brokers import get_brokermod
 from piker.service import (
    maybe_spawn_brokerd,
 )
 from piker.calc import humanize
 from ._util import (
    log,
    get_console_log,
 )
 from ..service import (
    maybe_spawn_brokerd,
 )
 from .flows import Flume
 from .validate import (
    FeedInit,
    validate_backend,
 )
-from ..tsp import (
+from .history import (
    manage_history,
 )
 from .ingest import get_ingestormod
 from .types import Struct
 from ..accounting import (
    MktPair,
    unpack_fqme,
 )
 from ..ui import _search
 from ._sampling import (
    sample_and_broadcast,
    uniform_rate_send,
 )
 class Sub(Struct, frozen=True):
    '''
    A live feed subscription entry.
    Contains meta-data on the remote-actor type (in functionality
    terms) as well as refs to IPC streams and sampler runtime
    params.
    '''
    ipc: tractor.MsgStream
    send_chan: trio.abc.SendChannel | None = None
    # tick throttle rate in Hz; determines how live
    # quotes/ticks should be downsampled before relay
    # to the receiving remote consumer (process).
    throttle_rate:  float | None  = None
    _throttle_cs: trio.CancelScope | None = None
    # TODO: actually stash comms info for the far end to allow
    # `.tsp`, `.fsp` and `.data._sampling` sub-systems to re-render
    # the data view as needed via msging with the `._remote_ctl`
    # ipc ctx.
    rc_ui: bool = False
 class _FeedsBus(Struct):
    '''
    Data feeds broadcaster and persistence management.
@ -125,7 +104,13 @@ class _FeedsBus(Struct):
    _subscribers: defaultdict[
        str,
-        set[Sub]
+        set[
            tuple[
                tractor.MsgStream | trio.MemorySendChannel,
                # tractor.Context,
                float | None,  # tick throttle in Hz
            ]
        ]
    ] = defaultdict(set)
    async def start_task(
@ -140,8 +125,6 @@ class _FeedsBus(Struct):
                trio.CancelScope] = trio.TASK_STATUS_IGNORED,
        ) -> None:
            with trio.CancelScope() as cs:
                # TODO: shouldn't this be a direct await to avoid
                # cancellation contagion to the bus nursery!?!?!
                await self.nursery.start(
                    target,
                    *args,
@ -159,28 +142,31 @@ class _FeedsBus(Struct):
    def get_subs(
        self,
        key: str,
-
+    ) -> set[
-    ) -> set[Sub]:
+        tuple[
            tractor.MsgStream | trio.MemorySendChannel,
            float | None,  # tick throttle in Hz
        ]
    ]:
        '''
        Get the ``set`` of consumer subscription entries for the given key.
        '''
        return self._subscribers[key]
    def subs_items(self) -> abc.ItemsView[str, set[Sub]]:
        return self._subscribers.items()
    def add_subs(
        self,
        key: str,
-        subs: set[Sub],
+        subs: set[tuple[
-
+            tractor.MsgStream | trio.MemorySendChannel,
-    ) -> set[Sub]:
+            float | None,  # tick throttle in Hz
        ]],
    ) -> set[tuple]:
        '''
        Add a ``set`` of consumer subscription entries for the given key.
        '''
-        _subs: set[Sub] = self._subscribers.setdefault(key, set())
+        _subs: set[tuple] = self._subscribers[key]
        _subs.update(subs)
        return _subs
@ -345,6 +331,7 @@ async def allocate_persistent_feed(
    ) = await bus.nursery.start(
        manage_history,
        mod,
        bus,
        mkt,
        some_data_ready,
        feed_is_live,
@ -421,13 +408,7 @@ async def allocate_persistent_feed(
        rt_shm.array['time'][1] = ts + 1
    elif hist_shm.array.size == 0:
-        for i in range(100):
+        raise RuntimeError(f'History (1m) Shm for {fqme} is empty!?')
            await trio.sleep(0.1)
            if hist_shm.array.size > 0:
                break
        else:
            await tractor.pause()
            raise RuntimeError(f'History (1m) Shm for {fqme} is empty!?')
    # wait the spawning parent task to register its subscriber
    # send-stream entry before we start the sample loop.
@ -457,9 +438,8 @@ async def open_feed_bus(
    symbols: list[str],  # normally expected to the broker-specific fqme
    loglevel: str = 'error',
-    tick_throttle: float | None = None,
+    tick_throttle: Optional[float] = None,
    start_stream: bool = True,
    allow_remote_ctl_ui: bool = False,
 ) -> dict[
    str,  # fqme
@ -474,12 +454,8 @@ async def open_feed_bus(
    if loglevel is None:
        loglevel = tractor.current_actor().loglevel
-    # XXX: required to propagate ``tractor`` loglevel to piker
+    # XXX: required to propagate ``tractor`` loglevel to piker logging
-    # logging
+    get_console_log(loglevel or tractor.current_actor().loglevel)
    get_console_log(
        loglevel
        or tractor.current_actor().loglevel
    )
    # local state sanity checks
    # TODO: check for any stale shm entries for this symbol
@ -489,7 +465,7 @@ async def open_feed_bus(
    assert 'brokerd' in servicename
    assert brokername in servicename
-    bus: _FeedsBus = get_feed_bus(brokername)
+    bus = get_feed_bus(brokername)
    sub_registered = trio.Event()
    flumes: dict[str, Flume] = {}
@ -536,10 +512,10 @@ async def open_feed_bus(
        # pack for ``.started()`` sync msg
        flumes[fqme] = flume
-        # we use the broker-specific fqme (bs_fqme) for the sampler
+        # we use the broker-specific fqme (bs_fqme) for the
-        # subscription since the backend isn't (yet) expected to
+        # sampler subscription since the backend isn't (yet) expected to
-        # append it's own name to the fqme, so we filter on keys
+        # append it's own name to the fqme, so we filter on keys which
-        # which *do not* include that name (e.g .ib) .
+        # *do not* include that name (e.g .ib) .
        bus._subscribers.setdefault(bs_fqme, set())
    # sync feed subscribers with flume handles
@ -578,60 +554,49 @@ async def open_feed_bus(
            # that the ``sample_and_broadcast()`` task (spawned inside
            # ``allocate_persistent_feed()``) will push real-time quote
            # (ticks) to this new consumer.
-            cs: trio.CancelScope | None = None
+
            send: trio.MemorySendChannel | None = None
            if tick_throttle:
                flume.throttle_rate = tick_throttle
-                # open a bg task which receives quotes over a mem
+                # open a bg task which receives quotes over a mem chan
-                # chan and only pushes them to the target
+                # and only pushes them to the target actor-consumer at
-                # actor-consumer at a max ``tick_throttle``
+                # a max ``tick_throttle`` instantaneous rate.
                # (instantaneous) rate.
                send, recv = trio.open_memory_channel(2**10)
-                # NOTE: the ``.send`` channel here is a swapped-in
+                cs = await bus.start_task(
                # trio mem chan which gets `.send()`-ed by the normal
                # sampler task but instead of being sent directly
                # over the IPC msg stream it's the throttle task
                # does the work of incrementally forwarding to the
                # IPC stream at the throttle rate.
                cs: trio.CancelScope = await bus.start_task(
                    uniform_rate_send,
                    tick_throttle,
                    recv,
                    stream,
                )
                # NOTE: so the ``send`` channel here is actually a swapped
                # in trio mem chan which gets pushed by the normal sampler
                # task but instead of being sent directly over the IPC msg
                # stream it's the throttle task does the work of
                # incrementally forwarding to the IPC stream at the throttle
                # rate.
                send._ctx = ctx  # mock internal ``tractor.MsgStream`` ref
                sub = (send, tick_throttle)
-            sub = Sub(
+            else:
-                ipc=stream,
+                sub = (stream, tick_throttle)
                send_chan=send,
                throttle_rate=tick_throttle,
                _throttle_cs=cs,
                rc_ui=allow_remote_ctl_ui,
            )
            # TODO: add an api for this on the bus?
            # maybe use the current task-id to key the sub list that's
            # added / removed? Or maybe we can add a general
            # pause-resume by sub-key api?
            bs_fqme = fqme.removesuffix(f'.{brokername}')
-            local_subs.setdefault(
+            local_subs.setdefault(bs_fqme, set()).add(sub)
-                bs_fqme,
+            bus.add_subs(bs_fqme, {sub})
                set()
            ).add(sub)
            bus.add_subs(
                bs_fqme,
                {sub}
            )
        # sync caller with all subs registered state
        sub_registered.set()
-        uid: tuple[str, str] = ctx.chan.uid
+        uid = ctx.chan.uid
        try:
-            # ctrl protocol for start/stop of live quote streams
+            # ctrl protocol for start/stop of quote streams based on UI
-            # based on UI state (eg. don't need a stream when
+            # state (eg. don't need a stream when a symbol isn't being
-            # a symbol isn't being displayed).
+            # displayed).
            async for msg in stream:
                if msg == 'pause':
@ -769,7 +734,6 @@ async def install_brokerd_search(
                except trio.EndOfChannel:
                    return {}
            from piker.ui import _search
            async with _search.register_symbol_search(
                provider_name=brokermod.name,
@ -788,7 +752,7 @@ async def install_brokerd_search(
 async def maybe_open_feed(
    fqmes: list[str],
-    loglevel: str | None = None,
+    loglevel: Optional[str] = None,
    **kwargs,
@ -804,7 +768,7 @@ async def maybe_open_feed(
    '''
    fqme = fqmes[0]
-    async with trionics.maybe_open_context(
+    async with maybe_open_context(
        acm_func=open_feed,
        kwargs={
            'fqmes': fqmes,
@ -824,7 +788,7 @@ async def maybe_open_feed(
            # add a new broadcast subscription for the quote stream
            # if this feed is likely already in use
-            async with trionics.gather_contexts(
+            async with gather_contexts(
                mngrs=[stream.subscribe() for stream in feed.streams.values()]
            ) as bstreams:
                for bstream, flume in zip(bstreams, feed.flumes.values()):
@ -848,8 +812,6 @@ async def open_feed(
    start_stream: bool = True,
    tick_throttle: float | None = None,  # Hz
    allow_remote_ctl_ui: bool = False,
 ) -> Feed:
    '''
    Open a "data feed" which provides streamed real-time quotes.
@ -886,7 +848,7 @@ async def open_feed(
        )
    portals: tuple[tractor.Portal]
-    async with trionics.gather_contexts(
+    async with gather_contexts(
        brokerd_ctxs,
    ) as portals:
@ -932,19 +894,13 @@ async def open_feed(
                    # of these stream open sequences sequentially per
                    # backend? .. need some thot!
                    allow_overruns=True,
                    # NOTE: UI actors (like charts) can allow
                    # remote control of certain graphics rendering
                    # capabilities via the
                    # `.ui._remote_ctl.remote_annotate()` msg loop.
                    allow_remote_ctl_ui=allow_remote_ctl_ui,
                )
            )
        assert len(feed.mods) == len(feed.portals)
        async with (
-            trionics.gather_contexts(bus_ctxs) as ctxs,
+            gather_contexts(bus_ctxs) as ctxs,
        ):
            stream_ctxs: list[tractor.MsgStream] = []
            for (
@ -986,7 +942,7 @@ async def open_feed(
            brokermod: ModuleType
            fqmes: list[str]
            async with (
-                trionics.gather_contexts(stream_ctxs) as streams,
+                gather_contexts(stream_ctxs) as streams,
            ):
                for (
                    stream,
@ -1002,12 +958,6 @@ async def open_feed(
                        if brokermod.name == flume.mkt.broker:
                            flume.stream = stream
-                assert (
+                assert len(feed.mods) == len(feed.portals) == len(feed.streams)
                    len(feed.mods)
                    ==
                    len(feed.portals)
                    ==
                    len(feed.streams)
                )
                yield feed
--- a/piker/data/flows.py
+++ b/piker/data/flows.py
@ -30,27 +30,53 @@ import tractor
 import pendulum
 import numpy as np
-from piker.types import Struct
+from ..accounting import MktPair
 from ._util import log
 from .types import Struct
 from ._sharedmem import (
    attach_shm_array,
    ShmArray,
    _Token,
 )
 # from .._profile import (
 #     Profiler,
 #     pg_profile_enabled,
 # )
 if TYPE_CHECKING:
-    from ..accounting import MktPair
+    # from pyqtgraph import PlotItem
    from .feed import Feed
 # TODO: ideas for further abstractions as per
 # https://github.com/pikers/piker/issues/216 and
 # https://github.com/pikers/piker/issues/270:
 # - a ``Cascade`` would be the minimal "connection" of 2 ``Flumes``
 #   as per circuit parlance:
 #   https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
 #     - could cover the combination of our `FspAdmin` and the
 #       backend `.fsp._engine` related machinery to "connect" one flume
 #       to another?
 # - a (financial signal) ``Flow`` would be the a "collection" of such
 #    minmial cascades. Some engineering based jargon concepts:
 #     - https://en.wikipedia.org/wiki/Signal_chain
 #     - https://en.wikipedia.org/wiki/Daisy_chain_(electrical_engineering)
 #     - https://en.wikipedia.org/wiki/Audio_signal_flow
 #     - https://en.wikipedia.org/wiki/Digital_signal_processing#Implementation
 #     - https://en.wikipedia.org/wiki/Dataflow_programming
 #     - https://en.wikipedia.org/wiki/Signal_programming
 #     - https://en.wikipedia.org/wiki/Incremental_computing
 class Flume(Struct):
    '''
-    Composite reference type which points to all the addressing
+    Composite reference type which points to all the addressing handles
-    handles and other meta-data necessary for the read, measure and
+    and other meta-data necessary for the read, measure and management
-    management of a set of real-time updated data flows.
+    of a set of real-time updated data flows.
    Can be thought of as a "flow descriptor" or "flow frame" which
-    describes the high level properties of a set of data flows that
+    describes the high level properties of a set of data flows that can
-    can be used seamlessly across process-memory boundaries.
+    be used seamlessly across process-memory boundaries.
    Each instance's sub-components normally includes:
     - a msg oriented quote stream provided via an IPC transport
@ -73,7 +99,6 @@ class Flume(Struct):
    # private shm refs loaded dynamically from tokens
    _hist_shm: ShmArray | None = None
    _rt_shm: ShmArray | None = None
    _readonly: bool = True
    stream: tractor.MsgStream | None = None
    izero_hist: int = 0
@ -90,7 +115,7 @@ class Flume(Struct):
        if self._rt_shm is None:
            self._rt_shm = attach_shm_array(
                token=self._rt_shm_token,
-                readonly=self._readonly,
+                readonly=True,
            )
        return self._rt_shm
@ -103,10 +128,12 @@ class Flume(Struct):
                'No shm token has been set for the history buffer?'
            )
-        if self._hist_shm is None:
+        if (
            self._hist_shm is None
        ):
            self._hist_shm = attach_shm_array(
                token=self._hist_shm_token,
-                readonly=self._readonly,
+                readonly=True,
            )
        return self._hist_shm
@ -125,10 +152,10 @@ class Flume(Struct):
        period and ratio between them.
        '''
-        times: np.ndarray = self.hist_shm.array['time']
+        times = self.hist_shm.array['time']
-        end: float | int = pendulum.from_timestamp(times[-1])
+        end = pendulum.from_timestamp(times[-1])
-        start: float | int = pendulum.from_timestamp(times[times != times[-1]][-1])
+        start = pendulum.from_timestamp(times[times != times[-1]][-1])
-        hist_step_size_s: float = (end - start).seconds
+        hist_step_size_s = (end - start).seconds
        times = self.rt_shm.array['time']
        end = pendulum.from_timestamp(times[-1])
@ -148,25 +175,17 @@ class Flume(Struct):
        msg = self.to_dict()
        msg['mkt'] = self.mkt.to_dict()
-        # NOTE: pop all un-msg-serializable fields:
+        # can't serialize the stream or feed objects, it's expected
-        # - `tractor.MsgStream`
+        # you'll have a ref to it since this msg should be rxed on
-        # - `Feed`
+        # a stream on whatever far end IPC..
        # - `Shmarray`
        # it's expected the `.from_msg()` on the other side
        # will get instead some kind of msg-compat version
        # that it can load.
        msg.pop('stream')
        msg.pop('feed')
        msg.pop('_rt_shm')
        msg.pop('_hist_shm')
        return msg
    @classmethod
    def from_msg(
        cls,
        msg: dict,
        readonly: bool = True,
    ) -> dict:
        '''
@ -175,13 +194,8 @@ class Flume(Struct):
        '''
        mkt_msg = msg.pop('mkt')
        from ..accounting import MktPair  # cycle otherwise..
        mkt = MktPair.from_msg(mkt_msg)
-        msg |= {'_readonly': readonly}
+        return cls(mkt=mkt, **msg)
        return cls(
            mkt=mkt,
            **msg,
        )
    def get_index(
        self,
@ -219,3 +233,5 @@ class Flume(Struct):
            np.all(np.isin(vlm, -1))
            or np.all(np.isnan(vlm))
        )
--- a/piker/data/history.py
+++ b/piker/data/history.py
@ -0,0 +1,967 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Historical data business logic for load, backfill and tsdb storage.
 '''
 from __future__ import annotations
 # from collections import (
 #     Counter,
 # )
 from datetime import datetime
 from functools import partial
 # import time
 from types import ModuleType
 from typing import (
    Callable,
    TYPE_CHECKING,
 )
 import trio
 from trio_typing import TaskStatus
 import tractor
 from pendulum import (
    Duration,
    from_timestamp,
 )
 import numpy as np
 from ..accounting import (
    MktPair,
 )
 from ._util import (
    log,
 )
 from ._sharedmem import (
    maybe_open_shm_array,
    ShmArray,
 )
 from ._source import def_iohlcv_fields
 from ._sampling import (
    open_sample_stream,
 )
 from ..brokers._util import (
    DataUnavailable,
 )
 if TYPE_CHECKING:
    from bidict import bidict
    from ..service.marketstore import StorageClient
    from .feed import _FeedsBus
 # `ShmArray` buffer sizing configuration:
 _mins_in_day = int(60 * 24)
 # how much is probably dependent on lifestyle
 # but we reco a buncha times (but only on a
 # run-every-other-day kinda week).
 _secs_in_day = int(60 * _mins_in_day)
 _days_in_week: int = 7
 _days_worth: int = 3
 _default_hist_size: int = 6 * 365 * _mins_in_day
 _hist_buffer_start = int(
    _default_hist_size - round(7 * _mins_in_day)
 )
 _default_rt_size: int = _days_worth * _secs_in_day
 # NOTE: start the append index in rt buffer such that 1 day's worth
 # can be appenened before overrun.
 _rt_buffer_start = int((_days_worth - 1) * _secs_in_day)
 def diff_history(
    array: np.ndarray,
    append_until_dt: datetime | None = None,
    prepend_until_dt: datetime | None = None,
 ) -> np.ndarray:
    # no diffing with tsdb dt index possible..
    if (
        prepend_until_dt is None
        and append_until_dt is None
    ):
        return array
    times = array['time']
    if append_until_dt:
        return array[times < append_until_dt.timestamp()]
    else:
        return array[times >= prepend_until_dt.timestamp()]
 async def shm_push_in_between(
    shm: ShmArray,
    to_push: np.ndarray,
    prepend_index: int,
    update_start_on_prepend: bool = False,
 ) -> int:
    shm.push(
        to_push,
        prepend=True,
        # XXX: only update the ._first index if no tsdb
        # segment was previously prepended by the
        # parent task.
        update_first=update_start_on_prepend,
        # XXX: only prepend from a manually calculated shm
        # index if there was already a tsdb history
        # segment prepended (since then the
        # ._first.value is going to be wayyy in the
        # past!)
        start=(
            prepend_index
            if not update_start_on_prepend
            else None
        ),
    )
    # XXX: extremely important, there can be no checkpoints
    # in the block above to avoid entering new ``frames``
    # values while we're pipelining the current ones to
    # memory...
    array = shm.array
    zeros = array[array['low'] == 0]
    if (
        0 < zeros.size < 1000
    ):
        tractor.breakpoint()
 async def start_backfill(
    get_hist,
    mod: ModuleType,
    mkt: MktPair,
    shm: ShmArray,
    timeframe: float,
    backfill_from_shm_index: int,
    backfill_from_dt: datetime,
    sampler_stream: tractor.MsgStream,
    backfill_until_dt: datetime | None = None,
    storage: StorageClient | None = None,
    write_tsdb: bool = True,
    task_status: TaskStatus[tuple] = trio.TASK_STATUS_IGNORED,
 ) -> int:
        # let caller unblock and deliver latest history frame
        # and use to signal that backfilling the shm gap until
        # the tsdb end is complete!
        bf_done = trio.Event()
        task_status.started(bf_done)
        # based on the sample step size, maybe load a certain amount history
        update_start_on_prepend: bool = False
        if backfill_until_dt is None:
            # TODO: drop this right and just expose the backfill
            # limits inside a [storage] section in conf.toml?
            # when no tsdb "last datum" is provided, we just load
            # some near-term history.
            # periods = {
            #     1: {'days': 1},
            #     60: {'days': 14},
            # }
            # do a decently sized backfill and load it into storage.
            periods = {
                1: {'days': 6},
                60: {'years': 6},
            }
            period_duration: int = periods[timeframe]
            update_start_on_prepend = True
            # NOTE: manually set the "latest" datetime which we intend to
            # backfill history "until" so as to adhere to the history
            # settings above when the tsdb is detected as being empty.
            backfill_until_dt = backfill_from_dt.subtract(**period_duration)
        # TODO: can we drop this? without conc i don't think this
        # is necessary any more?
        # configure async query throttling
        # rate = config.get('rate', 1)
        # XXX: legacy from ``trimeter`` code but unsupported now.
        # erlangs = config.get('erlangs', 1)
        # avoid duplicate history frames with a set of datetime frame
        # starts and associated counts of how many duplicates we see
        # per time stamp.
        # starts: Counter[datetime] = Counter()
        # conduct "backward history gap filling" where we push to
        # the shm buffer until we have history back until the
        # latest entry loaded from the tsdb's table B)
        last_start_dt: datetime = backfill_from_dt
        next_prepend_index: int = backfill_from_shm_index
        while last_start_dt > backfill_until_dt:
            log.debug(
                f'Requesting {timeframe}s frame ending in {last_start_dt}'
            )
            try:
                (
                    array,
                    next_start_dt,
                    next_end_dt,
                ) = await get_hist(
                    timeframe,
                    end_dt=last_start_dt,
                )
            # broker says there never was or is no more history to pull
            except DataUnavailable:
                log.warning(
                    f'NO-MORE-DATA: backend {mod.name} halted history!?'
                )
                # ugh, what's a better way?
                # TODO: fwiw, we probably want a way to signal a throttle
                # condition (eg. with ib) so that we can halt the
                # request loop until the condition is resolved?
                return
            # TODO: drop this? see todo above..
            # if (
            #     next_start_dt in starts
            #     and starts[next_start_dt] <= 6
            # ):
            #     start_dt = min(starts)
            #     log.warning(
            #         f"{mkt.fqme}: skipping duplicate frame @ {next_start_dt}"
            #     )
            #     starts[start_dt] += 1
            #     await tractor.breakpoint()
            #     continue
            # elif starts[next_start_dt] > 6:
            #     log.warning(
            #         f'NO-MORE-DATA: backend {mod.name} before {next_start_dt}?'
            #     )
            #     return
            # # only update new start point if not-yet-seen
            # starts[next_start_dt] += 1
            assert array['time'][0] == next_start_dt.timestamp()
            diff = last_start_dt - next_start_dt
            frame_time_diff_s = diff.seconds
            # frame's worth of sample-period-steps, in seconds
            frame_size_s = len(array) * timeframe
            expected_frame_size_s = frame_size_s + timeframe
            if frame_time_diff_s > expected_frame_size_s:
                # XXX: query result includes a start point prior to our
                # expected "frame size" and thus is likely some kind of
                # history gap (eg. market closed period, outage, etc.)
                # so just report it to console for now.
                log.warning(
                    f'History frame ending @ {last_start_dt} appears to have a gap:\n'
                    f'{diff} ~= {frame_time_diff_s} seconds'
                )
            to_push = diff_history(
                array,
                prepend_until_dt=backfill_until_dt,
            )
            ln = len(to_push)
            if ln:
                log.info(f'{ln} bars for {next_start_dt} -> {last_start_dt}')
            else:
                log.warning(
                    '0 BARS TO PUSH after diff!?\n'
                    f'{next_start_dt} -> {last_start_dt}'
                )
            # bail gracefully on shm allocation overrun/full
            # condition
            try:
                await shm_push_in_between(
                    shm,
                    to_push,
                    prepend_index=next_prepend_index,
                    update_start_on_prepend=update_start_on_prepend,
                )
                await sampler_stream.send({
                    'broadcast_all': {
                        'backfilling': (mkt.fqme, timeframe),
                    },
                })
                # decrement next prepend point
                next_prepend_index = next_prepend_index - ln
                last_start_dt = next_start_dt
            except ValueError as ve:
                _ve = ve
                log.error(
                    f'Shm prepend OVERRUN on: {next_start_dt} -> {last_start_dt}?'
                )
                if next_prepend_index < ln:
                    log.warning(
                        f'Shm buffer can only hold {next_prepend_index} more rows..\n'
                        f'Appending those from recent {ln}-sized frame, no more!'
                    )
                to_push = to_push[-next_prepend_index + 1:]
                await shm_push_in_between(
                    shm,
                    to_push,
                    prepend_index=next_prepend_index,
                    update_start_on_prepend=update_start_on_prepend,
                )
                await sampler_stream.send({
                    'broadcast_all': {
                        'backfilling': (mkt.fqme, timeframe),
                    },
                })
                # can't push the entire frame? so
                # push only the amount that can fit..
                break
            log.info(
                f'Shm pushed {ln} frame:\n'
                f'{next_start_dt} -> {last_start_dt}'
            )
            # FINALLY, maybe write immediately to the tsdb backend for
            # long-term storage.
            if (
                storage is not None
                and write_tsdb
            ):
                log.info(
                    f'Writing {ln} frame to storage:\n'
                    f'{next_start_dt} -> {last_start_dt}'
                )
                if mkt.dst.atype not in {'crypto', 'crypto_currency'}:
                    # for now, our table key schema is not including
                    # the dst[/src] source asset token.
                    col_sym_key: str = mkt.get_fqme(
                        delim_char='',
                        without_src=True,
                    )
                else:
                    col_sym_key: str = mkt.get_fqme(delim_char='')
                # TODO: implement parquet append!?
                await storage.write_ohlcv(
                    col_sym_key,
                    shm.array,
                    timeframe,
                )
        else:
            # finally filled gap
            log.info(
                f'Finished filling gap to tsdb start @ {backfill_until_dt}!'
            )
            # conduct tsdb timestamp gap detection and backfill any
            # seemingly missing sequence segments..
            # TODO: ideally these never exist but somehow it seems
            # sometimes we're writing zero-ed segments on certain
            # (teardown) cases?
            from ._timeseries import detect_null_time_gap
            gap_indices: tuple | None = detect_null_time_gap(shm)
            while gap_indices:
                (
                    istart,
                    start,
                    end,
                    iend,
                ) = gap_indices
                start_dt = from_timestamp(start)
                end_dt = from_timestamp(end)
                (
                    array,
                    next_start_dt,
                    next_end_dt,
                ) = await get_hist(
                    timeframe,
                    start_dt=start_dt,
                    end_dt=end_dt,
                )
                # XXX TODO: pretty sure if i plot tsla, btcusdt.binance
                # and mnq.cme.ib this causes a Qt crash XXDDD
                # make sure we don't overrun the buffer start
                len_to_push: int = min(iend, array.size)
                to_push: np.ndarray = array[-len_to_push:]
                await shm_push_in_between(
                    shm,
                    to_push,
                    prepend_index=iend,
                    update_start_on_prepend=False,
                )
                # TODO: UI side needs IPC event to update..
                # - make sure the UI actually always handles
                #  this update!
                # - remember that in the display side, only refersh this
                #   if the respective history is actually "in view".
                #   loop
                await sampler_stream.send({
                    'broadcast_all': {
                        'backfilling': (mkt.fqme, timeframe),
                    },
                })
                gap_indices: tuple | None = detect_null_time_gap(shm)
        # XXX: extremely important, there can be no checkpoints
        # in the block above to avoid entering new ``frames``
        # values while we're pipelining the current ones to
        # memory...
        # await sampler_stream.send('broadcast_all')
        # short-circuit (for now)
        bf_done.set()
 async def back_load_from_tsdb(
    storemod: ModuleType,
    storage: StorageClient,
    fqme: str,
    tsdb_history: np.ndarray,
    last_tsdb_dt: datetime,
    latest_start_dt: datetime,
    latest_end_dt: datetime,
    bf_done: trio.Event,
    timeframe: int,
    shm: ShmArray,
 ):
    assert len(tsdb_history)
    # sync to backend history task's query/load completion
    # if bf_done:
    #     await bf_done.wait()
    # TODO: eventually it'd be nice to not require a shm array/buffer
    # to accomplish this.. maybe we can do some kind of tsdb direct to
    # graphics format eventually in a child-actor?
    if storemod.name == 'nativedb':
        return
        await tractor.breakpoint()
        assert shm._first.value == 0
    array = shm.array
    # if timeframe == 1:
    #     times = shm.array['time']
    #     assert (times[1] - times[0]) == 1
    if len(array):
        shm_last_dt = from_timestamp(
            shm.array[0]['time']
        )
    else:
        shm_last_dt = None
    if last_tsdb_dt:
        assert shm_last_dt >= last_tsdb_dt
    # do diff against start index of last frame of history and only
    # fill in an amount of datums from tsdb allows for most recent
    # to be loaded into mem *before* tsdb data.
    if (
        last_tsdb_dt
        and latest_start_dt
    ):
        backfilled_size_s = (
            latest_start_dt - last_tsdb_dt
        ).seconds
        # if the shm buffer len is not large enough to contain
        # all missing data between the most recent backend-queried frame
        # and the most recent dt-index in the db we warn that we only
        # want to load a portion of the next tsdb query to fill that
        # space.
        log.info(
            f'{backfilled_size_s} seconds worth of {timeframe}s loaded'
        )
    # Load TSDB history into shm buffer (for display) if there is
    # remaining buffer space.
    time_key: str = 'time'
    if getattr(storemod, 'ohlc_key_map', False):
        keymap: bidict = storemod.ohlc_key_map
        time_key: str = keymap.inverse['time']
    # if (
    #     not len(tsdb_history)
    # ):
    #     return
    tsdb_last_frame_start: datetime = last_tsdb_dt
    # load as much from storage into shm possible (depends on
    # user's shm size settings).
    while shm._first.value > 0:
        tsdb_history = await storage.read_ohlcv(
            fqme,
            timeframe=timeframe,
            end=tsdb_last_frame_start,
        )
        # # empty query
        # if not len(tsdb_history):
        #     break
        next_start = tsdb_history[time_key][0]
        if next_start >= tsdb_last_frame_start:
            # no earlier data detected
            break
        else:
            tsdb_last_frame_start = next_start
        # TODO: see if there's faster multi-field reads:
        # https://numpy.org/doc/stable/user/basics.rec.html#accessing-multiple-fields
        # re-index  with a `time` and index field
        prepend_start = shm._first.value
        to_push = tsdb_history[-prepend_start:]
        shm.push(
            to_push,
            # insert the history pre a "days worth" of samples
            # to leave some real-time buffer space at the end.
            prepend=True,
            # update_first=False,
            # start=prepend_start,
            field_map=storemod.ohlc_key_map,
        )
        log.info(f'Loaded {to_push.shape} datums from storage')
        tsdb_last_frame_start = tsdb_history[time_key][0]
        # manually trigger step update to update charts/fsps
        # which need an incremental update.
        # NOTE: the way this works is super duper
        # un-intuitive right now:
        # - the broadcaster fires a msg to the fsp subsystem.
        # - fsp subsys then checks for a sample step diff and
        #   possibly recomputes prepended history.
        # - the fsp then sends back to the parent actor
        #   (usually a chart showing graphics for said fsp)
        #   which tells the chart to conduct a manual full
        #   graphics loop cycle.
        # await sampler_stream.send('broadcast_all')
 async def tsdb_backfill(
    mod: ModuleType,
    storemod: ModuleType,
    tn: trio.Nursery,
    storage: StorageClient,
    mkt: MktPair,
    shm: ShmArray,
    timeframe: float,
    sampler_stream: tractor.MsgStream,
    task_status: TaskStatus[
        tuple[ShmArray, ShmArray]
    ] = trio.TASK_STATUS_IGNORED,
 ) -> None:
    get_hist: Callable[
        [int, datetime, datetime],
        tuple[np.ndarray, str]
    ]
    config: dict[str, int]
    async with mod.open_history_client(
        mkt,
    ) as (get_hist, config):
        log.info(f'{mod} history client returned backfill config: {config}')
        # get latest query's worth of history all the way
        # back to what is recorded in the tsdb
        try:
            array, mr_start_dt, mr_end_dt = await get_hist(
                timeframe,
                end_dt=None,
            )
        # XXX: timeframe not supported for backend (since
        # above exception type), terminate immediately since
        # there's no backfilling possible.
        except DataUnavailable:
            task_status.started()
            return
        times: np.ndarray = array['time']
        # sample period step size in seconds
        step_size_s = (
            from_timestamp(times[-1])
            - from_timestamp(times[-2])
        ).seconds
        if step_size_s not in (1, 60):
            log.error(f'Last 2 sample period is off!? -> {step_size_s}')
            step_size_s = (
                from_timestamp(times[-2])
                - from_timestamp(times[-3])
            ).seconds
        # NOTE: on the first history, most recent history
        # frame we PREPEND from the current shm ._last index
        # and thus a gap between the earliest datum loaded here
        # and the latest loaded from the tsdb may exist!
        log.info(f'Pushing {array.size} to shm!')
        shm.push(
            array,
            prepend=True,  # append on first frame
        )
        backfill_gap_from_shm_index: int = shm._first.value + 1
        # tell parent task to continue
        task_status.started()
        # loads a (large) frame of data from the tsdb depending
        # on the db's query size limit; our "nativedb" (using
        # parquet) generally can load the entire history into mem
        # but if not then below the remaining history can be lazy
        # loaded?
        fqme: str = mkt.fqme
        tsdb_entry: tuple | None =  await storage.load(
            fqme,
            timeframe=timeframe,
        )
        last_tsdb_dt: datetime | None = None
        if tsdb_entry:
            (
                tsdb_history,
                first_tsdb_dt,
                last_tsdb_dt,
            ) = tsdb_entry
            # calc the index from which the tsdb data should be
            # prepended, presuming there is a gap between the
            # latest frame (loaded/read above) and the latest
            # sample loaded from the tsdb.
            backfill_diff: Duration =  mr_start_dt - last_tsdb_dt
            offset_s: float = backfill_diff.in_seconds()
            offset_samples: int = round(offset_s / timeframe)
            # TODO: see if there's faster multi-field reads:
            # https://numpy.org/doc/stable/user/basics.rec.html#accessing-multiple-fields
            # re-index  with a `time` and index field
            prepend_start = shm._first.value - offset_samples + 1
            # tsdb history is so far in the past we can't fit it in
            # shm buffer space so simply don't load it!
            if prepend_start > 0:
                to_push = tsdb_history[-prepend_start:]
                shm.push(
                    to_push,
                    # insert the history pre a "days worth" of samples
                    # to leave some real-time buffer space at the end.
                    prepend=True,
                    # update_first=False,
                    start=prepend_start,
                    field_map=storemod.ohlc_key_map,
                )
                log.info(f'Loaded {to_push.shape} datums from storage')
        # TODO: maybe start history anal and load missing "history
        # gaps" via backend..
        if timeframe not in (1, 60):
            raise ValueError(
                '`piker` only needs to support 1m and 1s sampling '
                'but ur api is trying to deliver a longer '
                f'timeframe of {timeframe} seconds..\n'
                'So yuh.. dun do dat brudder.'
            )
        # if there is a gap to backfill from the first
        # history frame until the last datum loaded from the tsdb
        # continue that now in the background
        bf_done = await tn.start(
            partial(
                start_backfill,
                get_hist,
                mod,
                mkt,
                shm,
                timeframe,
                backfill_from_shm_index=backfill_gap_from_shm_index,
                backfill_from_dt=mr_start_dt,
                sampler_stream=sampler_stream,
                backfill_until_dt=last_tsdb_dt,
                storage=storage,
            )
        )
        # if len(hist_shm.array) < 2:
        # TODO: there's an edge case here to solve where if the last
        # frame before market close (at least on ib) was pushed and
        # there was only "1 new" row pushed from the first backfill
        # query-iteration, then the sample step sizing calcs will
        # break upstream from here since you can't diff on at least
        # 2 steps... probably should also add logic to compute from
        # the tsdb series and stash that somewhere as meta data on
        # the shm buffer?.. no se.
        # backload any further data from tsdb (concurrently per
        # timeframe) if not all data was able to be loaded (in memory)
        # from the ``StorageClient.load()`` call above.
        try:
            await trio.sleep_forever()
        finally:
            return
        # IF we need to continue backloading incrementall from the
        # tsdb client..
        tn.start_soon(
            back_load_from_tsdb,
            storemod,
            storage,
            fqme,
            tsdb_history,
            last_tsdb_dt,
            mr_start_dt,
            mr_end_dt,
            bf_done,
            timeframe,
            shm,
        )
 async def manage_history(
    mod: ModuleType,
    bus: _FeedsBus,
    mkt: MktPair,
    some_data_ready: trio.Event,
    feed_is_live: trio.Event,
    timeframe: float = 60,  # in seconds
    task_status: TaskStatus[
        tuple[ShmArray, ShmArray]
    ] = trio.TASK_STATUS_IGNORED,
 ) -> None:
    '''
    Load and manage historical data including the loading of any
    available series from any connected tsdb as well as conduct
    real-time update of both that existing db and the allocated
    shared memory buffer.
    Init sequence:
    - allocate shm (numpy array) buffers for 60s & 1s sample rates
    - configure "zero index" for each buffer: the index where
      history will prepended *to* and new live data will be
      appened *from*.
    - open a ``.storage.StorageClient`` and load any existing tsdb
      history as well as (async) start a backfill task which loads
      missing (newer) history from the data provider backend:
      - tsdb history is loaded first and pushed to shm ASAP.
      - the backfill task loads the most recent history before
        unblocking its parent task, so that the `ShmArray._last` is
        up to date to allow the OHLC sampler to begin writing new
        samples as the correct buffer index once the provider feed
        engages.
    '''
    # TODO: is there a way to make each shm file key
    # actor-tree-discovery-addr unique so we avoid collisions
    # when doing tests which also allocate shms for certain instruments
    # that may be in use on the system by some other running daemons?
    # from tractor._state import _runtime_vars
    # port = _runtime_vars['_root_mailbox'][1]
    uid = tractor.current_actor().uid
    name, uuid = uid
    service = name.rstrip(f'.{mod.name}')
    fqme: str = mkt.get_fqme(delim_char='')
    # (maybe) allocate shm array for this broker/symbol which will
    # be used for fast near-term history capture and processing.
    hist_shm, opened = maybe_open_shm_array(
        size=_default_hist_size,
        append_start_index=_hist_buffer_start,
        key=f'piker.{service}[{uuid[:16]}].{fqme}.hist',
        # use any broker defined ohlc dtype:
        dtype=getattr(mod, '_ohlc_dtype', def_iohlcv_fields),
        # we expect the sub-actor to write
        readonly=False,
    )
    hist_zero_index = hist_shm.index - 1
    # TODO: history validation
    if not opened:
        raise RuntimeError(
            "Persistent shm for sym was already open?!"
        )
    rt_shm, opened = maybe_open_shm_array(
        size=_default_rt_size,
        append_start_index=_rt_buffer_start,
        key=f'piker.{service}[{uuid[:16]}].{fqme}.rt',
        # use any broker defined ohlc dtype:
        dtype=getattr(mod, '_ohlc_dtype', def_iohlcv_fields),
        # we expect the sub-actor to write
        readonly=False,
    )
    # (for now) set the rt (hft) shm array with space to prepend
    # only a few days worth of 1s history.
    days = 2
    start_index = days*_secs_in_day
    rt_shm._first.value = start_index
    rt_shm._last.value = start_index
    rt_zero_index = rt_shm.index - 1
    if not opened:
        raise RuntimeError(
            "Persistent shm for sym was already open?!"
        )
    open_history_client = getattr(
        mod,
        'open_history_client',
        None,
    )
    assert open_history_client
    # TODO: maybe it should be a subpkg of `.data`?
    from piker import storage
    async with (
        storage.open_storage_client() as (storemod, client),
        trio.open_nursery() as tn,
    ):
        log.info(
            f'Connecting to storage backend `{storemod.name}`:\n'
            f'location: {client.address}\n'
            f'db cardinality: {client.cardinality}\n'
            # TODO: show backend config, eg:
            # - network settings
            # - storage size with compression
            # - number of loaded time series?
        )
        # NOTE: this call ONLY UNBLOCKS once the latest-most frame
        # (i.e. history just before the live feed latest datum) of
        # history has been loaded and written to the shm buffer:
        # - the backfiller task can write in reverse chronological
        #   to the shm and tsdb
        # - the tsdb data can be loaded immediately and the
        #   backfiller can do a single append from it's end datum and
        #   then prepends backward to that from the current time
        #   step.
        tf2mem: dict = {
            1: rt_shm,
            60: hist_shm,
        }
        async with open_sample_stream(
            period_s=1.,
            shms_by_period={
                1.: rt_shm.token,
                60.: hist_shm.token,
            },
            # NOTE: we want to only open a stream for doing
            # broadcasts on backfill operations, not receive the
            # sample index-stream (since there's no code in this
            # data feed layer that needs to consume it).
            open_index_stream=True,
            sub_for_broadcasts=False,
        ) as sample_stream:
            # register 1s and 1m buffers with the global incrementer task
            log.info(f'Connected to sampler stream: {sample_stream}')
            for timeframe in [60, 1]:
                await tn.start(
                    tsdb_backfill,
                    mod,
                    storemod,
                    tn,
                    # bus,
                    client,
                    mkt,
                    tf2mem[timeframe],
                    timeframe,
                    sample_stream,
                )
            # indicate to caller that feed can be delivered to
            # remote requesting client since we've loaded history
            # data that can be used.
            some_data_ready.set()
            # wait for a live feed before starting the sampler.
            await feed_is_live.wait()
            # yield back after client connect with filled shm
            task_status.started((
                hist_zero_index,
                hist_shm,
                rt_zero_index,
                rt_shm,
            ))
            # history retreival loop depending on user interaction
            # and thus a small RPC-prot for remotely controllinlg
            # what data is loaded for viewing.
            await trio.sleep_forever()
--- a/piker/data/types.py
+++ b/piker/data/types.py
@ -0,0 +1,104 @@
 # piker: trading gear for hackers
 # Copyright (C) (in stewardship for pikers)
 #  - Tyler Goodlet
 #  - Guillermo Rodriguez
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Extensions to built-in or (heavily used but 3rd party) friend-lib
 types.
 '''
 from pprint import pformat
 from msgspec import (
    msgpack,
    Struct,
    structs,
 )
 class Struct(
    Struct,
    # https://jcristharif.com/msgspec/structs.html#tagged-unions
    # tag='pikerstruct',
    # tag=True,
 ):
    '''
    A "human friendlier" (aka repl buddy) struct subtype.
    '''
    def to_dict(self) -> dict:
        '''
        Like it sounds.. direct delegation to:
        https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
        TODO: probably just drop this method since it's now a built-int method?
        '''
        return structs.asdict(self)
    def pformat(self) -> str:
        return f'Struct({pformat(self.to_dict())})'
    def copy(
        self,
        update: dict | None = None,
    ) -> Struct:
        '''
        Validate-typecast all self defined fields, return a copy of
        us with all such fields.
        NOTE: This is kinda like the default behaviour in
        `pydantic.BaseModel` except a copy of the object is
        returned making it compat with `frozen=True`.
        '''
        if update:
            for k, v in update.items():
                setattr(self, k, v)
        # NOTE: roundtrip serialize to validate
        # - enode to msgpack binary format,
        # - decode that back to a struct.
        return msgpack.Decoder(type=type(self)).decode(
            msgpack.Encoder().encode(self)
        )
    def typecast(
        self,
        # TODO: allow only casting a named subset?
        # fields: set[str] | None = None,
    ) -> None:
        '''
        Cast all fields using their declared type annotations
        (kinda like what `pydantic` does by default).
        NOTE: this of course won't work on frozen types, use
        ``.copy()`` above in such cases.
        '''
        # https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
        fi: structs.FieldInfo
        for fi in structs.fields(self):
            setattr(
                self,
                fi.name,
                fi.type(getattr(self, fi.name)),
            )
--- a/piker/data/validate.py
+++ b/piker/data/validate.py
@ -18,7 +18,6 @@ Data feed synchronization protocols, init msgs, and general
 data-provider-backend-agnostic schema definitions.
 '''
 from __future__ import annotations
 from decimal import Decimal
 from pprint import pformat
 from types import ModuleType
@ -29,8 +28,8 @@ from typing import (
 from msgspec import field
-from piker.types import Struct
+from .types import Struct
-from piker.accounting import (
+from ..accounting import (
    Asset,
    MktPair,
 )
@ -82,8 +81,8 @@ _eps: dict[str, list[str]] = {
    # live order control and trading
    'brokerd': [
        'trades_dialogue',
-        'open_trade_dialog',  # live order ctl
+        # TODO: ledger normalizer helper?
-        'norm_trade', # ledger normalizer for txns
+        # norm_trades(records: dict[str, Any]) -> TransactionLedger)
    ],
 }
--- a/piker/fsp/init.py
+++ b/piker/fsp/init.py
@ -22,40 +22,17 @@ from typing import AsyncIterator
 import numpy as np
-from ._api import (
+from ._engine import cascade
    maybe_mk_fsp_shm,
    Fsp,
 )
 from ._engine import (
    cascade,
    Cascade,
 )
 from ._volume import (
    dolla_vlm,
    flow_rates,
    tina_vwap,
 )
-__all__: list[str] = [
+__all__ = ['cascade']
    'cascade',
    'Cascade',
    'maybe_mk_fsp_shm',
    'Fsp',
    'dolla_vlm',
    'flow_rates',
    'tina_vwap',
 ]
 async def latency(
    source: 'TickStream[Dict[str, float]]',  # noqa
    ohlcv: np.ndarray
 ) -> AsyncIterator[np.ndarray]:
-    '''
+    """Latency measurements, broker to piker.
-    Latency measurements, broker to piker.
+    """
    '''
    # TODO: do we want to offer yielding this async
    # before the rt data connection comes up?
--- a/piker/fsp/_engine.py
+++ b/piker/fsp/_engine.py
@ -18,12 +18,13 @@
 core task logic for processing chains
 '''
-from __future__ import annotations
+from dataclasses import dataclass
 from contextlib import asynccontextmanager as acm
 from functools import partial
 from typing import (
    AsyncIterator,
    Callable,
    Optional,
    Union,
 )
 import numpy as np
@ -32,9 +33,9 @@ from trio_typing import TaskStatus
 import tractor
 from tractor.msg import NamespacePath
 from piker.types import Struct
 from ..log import get_logger, get_console_log
 from .. import data
 from ..data import attach_shm_array
 from ..data.feed import (
    Flume,
    Feed,
@ -50,11 +51,17 @@ from ._api import (
    _load_builtins,
    _Token,
 )
-from ..toolz import Profiler
+from .._profile import Profiler
 log = get_logger(__name__)
@dataclass
 class TaskTracker:
    complete: trio.Event
    cs: trio.CancelScope
 async def filter_quotes_by_sym(
    sym: str,
@ -75,168 +82,30 @@ async def filter_quotes_by_sym(
        if quote:
            yield quote
 # TODO: unifying the abstractions in this FSP subsys/layer:
 # -[ ] move the `.data.flows.Flume` type into this
 #   module/subsys/pkg?
 # -[ ] ideas for further abstractions as per
 #   - https://github.com/pikers/piker/issues/216,
 #   - https://github.com/pikers/piker/issues/270:
 #   - a (financial signal) ``Flow`` would be the a "collection" of such
 #     minmial cascades. Some engineering based jargon concepts:
 #     - https://en.wikipedia.org/wiki/Signal_chain
 #     - https://en.wikipedia.org/wiki/Daisy_chain_(electrical_engineering)
 #     - https://en.wikipedia.org/wiki/Audio_signal_flow
 #     - https://en.wikipedia.org/wiki/Digital_signal_processing#Implementation
 #     - https://en.wikipedia.org/wiki/Dataflow_programming
 #     - https://en.wikipedia.org/wiki/Signal_programming
 #     - https://en.wikipedia.org/wiki/Incremental_computing
 #     - https://en.wikipedia.org/wiki/Signal-flow_graph
 #     - https://en.wikipedia.org/wiki/Signal-flow_graph#Basic_components
-# -[ ] we probably want to eval THE BELOW design and unify with the
+async def fsp_compute(
 #   proto `TaskManager` in the `tractor` dev branch as well as with
 #   our below idea for `Cascade`:
 #   - https://github.com/goodboy/tractor/pull/363
 class Cascade(Struct):
    '''
    As per sig-proc engineering parlance, this is a chaining of
    `Flume`s, which are themselves collections of "Streams"
    implemented currently via `ShmArray`s.
    A `Cascade` is be the minimal "connection" of 2 `Flumes`
    as per circuit parlance:
    https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
    TODO:
      -[ ] could cover the combination of our `FspAdmin` and the
        backend `.fsp._engine` related machinery to "connect" one flume
        to another?
    '''
    # TODO: make these `Flume`s
    src: Flume
    dst: Flume
    tn: trio.Nursery
    fsp: Fsp  # UI-side middleware ctl API
    # filled during cascade/.bind_func() (fsp_compute) init phases
    bind_func: Callable | None = None
    complete: trio.Event | None = None
    cs: trio.CancelScope | None = None
    client_stream: tractor.MsgStream | None = None
    async def resync(self) -> int:
        # TODO: adopt an incremental update engine/approach
        # where possible here eventually!
        log.info(f're-syncing fsp {self.fsp.name} to source')
        self.cs.cancel()
        await self.complete.wait()
        index: int = await self.tn.start(self.bind_func)
        # always trigger UI refresh after history update,
        # see ``piker.ui._fsp.FspAdmin.open_chain()`` and
        # ``piker.ui._display.trigger_update()``.
        dst_shm: ShmArray = self.dst.rt_shm
        await self.client_stream.send({
            'fsp_update': {
                'key': dst_shm.token,
                'first': dst_shm._first.value,
                'last': dst_shm._last.value,
            }
        })
        return index
    def is_synced(self) -> tuple[bool, int, int]:
        '''
        Predicate to dertmine if a destination FSP
        output array is aligned to its source array.
        '''
        src_shm: ShmArray = self.src.rt_shm
        dst_shm: ShmArray = self.dst.rt_shm
        step_diff = src_shm.index - dst_shm.index
        len_diff = abs(len(src_shm.array) - len(dst_shm.array))
        synced: bool = not (
            # the source is likely backfilling and we must
            # sync history calculations
            len_diff > 2
            # we aren't step synced to the source and may be
            # leading/lagging by a step
            or step_diff > 1
            or step_diff < 0
        )
        if not synced:
            fsp: Fsp = self.fsp
            log.warning(
                '***DESYNCED FSP***\n'
                f'{fsp.ns_path}@{src_shm.token}\n'
                f'step_diff: {step_diff}\n'
                f'len_diff: {len_diff}\n'
            )
        return (
            synced,
            step_diff,
            len_diff,
        )
    async def poll_and_sync_to_step(self) -> int:
        synced, step_diff, _ = self.is_synced()
        while not synced:
            await self.resync()
            synced, step_diff, _ = self.is_synced()
        return step_diff
    @acm
    async def open_edge(
        self,
        bind_func: Callable,
    ) -> int:
        self.bind_func = bind_func
        index = await self.tn.start(bind_func)
        yield index
        # TODO: what do we want on teardown/error?
        # -[ ] dynamic reconnection after update?
 async def connect_streams(
    casc: Cascade,
    mkt: MktPair,
    flume: Flume,
    quote_stream: trio.abc.ReceiveChannel,
    src: Flume,
    dst: Flume,
-    edge_func: Callable,
+    src: ShmArray,
    dst: ShmArray,
    func: Callable,
    # attach_stream: bool = False,
    task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
 ) -> None:
    '''
    Stream and per-sample compute and write the cascade of
    2 `Flumes`/streams given some operating `func`.
    https://en.wikipedia.org/wiki/Signal-flow_graph#Basic_components
    Not literally, but something like:
        edge_func(Flume_in) -> Flume_out
    '''
    profiler = Profiler(
        delayed=False,
        disabled=True
    )
-    # TODO: just pull it from src.mkt.fqme no?
+    fqme = mkt.fqme
-    # fqme: str = mkt.fqme
+    out_stream = func(
    fqme: str = src.mkt.fqme
    # TODO: dynamic introspection of what the underlying (vertex)
    # function actually requires from input node (flumes) then
    # deliver those inputs as part of a graph "compilation" step?
    out_stream = edge_func(
        # TODO: do we even need this if we do the feed api right?
        # shouldn't a local stream do this before we get a handle
@ -244,21 +113,20 @@ async def connect_streams(
        # async itertools style?
        filter_quotes_by_sym(fqme, quote_stream),
-        # XXX: currently the ``ohlcv`` arg, but we should allow
+        # XXX: currently the ``ohlcv`` arg
-        # (dynamic) requests for src flume (node) streams?
+        flume.rt_shm,
        src.rt_shm,
    )
    # HISTORY COMPUTE PHASE
    # conduct a single iteration of fsp with historical bars input
    # and get historical output.
-    history_output: (
+    history_output: Union[
-        dict[str, np.ndarray]  # multi-output case
+        dict[str, np.ndarray],  # multi-output case
-        | np.ndarray,  # single output case
+        np.ndarray,  # single output case
-    )
+    ]
    history_output = await anext(out_stream)
-    func_name = edge_func.__name__
+    func_name = func.__name__
    profiler(f'{func_name} generated history')
    # build struct array with an 'index' field to push as history
@ -266,12 +134,10 @@ async def connect_streams(
    # TODO: push using a[['f0', 'f1', .., 'fn']] = .. syntax no?
    # if the output array is multi-field then push
    # each respective field.
-    dst_shm: ShmArray = dst.rt_shm
+    fields = getattr(dst.array.dtype, 'fields', None).copy()
    fields = getattr(dst_shm.array.dtype, 'fields', None).copy()
    fields.pop('index')
-    history_by_field: np.ndarray | None = None
+    history_by_field: Optional[np.ndarray] = None
-    src_shm: ShmArray = src.rt_shm
+    src_time = src.array['time']
    src_time = src_shm.array['time']
    if (
        fields and
@ -290,7 +156,7 @@ async def connect_streams(
                if history_by_field is None:
                    if output is None:
-                        length = len(src_shm.array)
+                        length = len(src.array)
                    else:
                        length = len(output)
@ -299,7 +165,7 @@ async def connect_streams(
                    # will be pushed to shm.
                    history_by_field = np.zeros(
                        length,
-                        dtype=dst_shm.array.dtype
+                        dtype=dst.array.dtype
                    )
                if output is None:
@ -316,13 +182,13 @@ async def connect_streams(
            )
        history_by_field = np.zeros(
            len(history_output),
-            dtype=dst_shm.array.dtype
+            dtype=dst.array.dtype
        )
        history_by_field[func_name] = history_output
    history_by_field['time'] = src_time[-len(history_by_field):]
-    history_output['time'] = src_shm.array['time']
+    history_output['time'] = src.array['time']
    # TODO: XXX:
    # THERE'S A BIG BUG HERE WITH THE `index` field since we're
@ -335,11 +201,11 @@ async def connect_streams(
    #   is `index` aware such that historical data can be indexed
    #   relative to the true first datum? Not sure if this is sane
    #   for incremental compuations.
-    first = dst_shm._first.value = src_shm._first.value
+    first = dst._first.value = src._first.value
    # TODO: can we use this `start` flag instead of the manual
    # setting above?
-    index = dst_shm.push(
+    index = dst.push(
        history_by_field,
        start=first,
    )
@ -350,9 +216,12 @@ async def connect_streams(
    # setup a respawn handle
    with trio.CancelScope() as cs:
-        casc.cs = cs
+        # TODO: might be better to just make a "restart" method where
-        casc.complete = trio.Event()
+        # the target task is spawned implicitly and then the event is
-        task_status.started(index)
+        # set via some higher level api? At that poing we might as well
        # be writing a one-cancels-one nursery though right?
        tracker = TaskTracker(trio.Event(), cs)
        task_status.started((tracker, index))
        profiler(f'{func_name} yield last index')
@ -366,12 +235,12 @@ async def connect_streams(
                log.debug(f"{func_name}: {processed}")
                key, output = processed
                # dst.array[-1][key] = output
-                dst_shm.array[[key, 'time']][-1] = (
+                dst.array[[key, 'time']][-1] = (
                    output,
                    # TODO: what about pushing ``time.time_ns()``
                    # in which case we'll need to round at the graphics
                    # processing / sampling layer?
-                    src_shm.array[-1]['time']
+                    src.array[-1]['time']
                )
                # NOTE: for now we aren't streaming this to the consumer
@ -383,7 +252,7 @@ async def connect_streams(
                # N-consumers who subscribe for the real-time output,
                # which we'll likely want to implement using local-mem
                # chans for the fan out?
-                # index = src_shm.index
+                # index = src.index
                # if attach_stream:
                #     await client_stream.send(index)
@ -393,7 +262,7 @@ async def connect_streams(
                #     log.info(f'FSP quote too fast: {hz}')
                # last = time.time()
        finally:
-            casc.complete.set()
+            tracker.complete.set()
@tractor.context
@ -404,15 +273,15 @@ async def cascade(
    # data feed key
    fqme: str,
-    # flume pair cascaded using an "edge function"
+    src_shm_token: dict,
-    src_flume_addr: dict,
+    dst_shm_token: tuple[str, np.dtype],
-    dst_flume_addr: dict,
+
    ns_path: NamespacePath,
    shm_registry: dict[str, _Token],
    zero_on_step: bool = False,
-    loglevel: str | None = None,
+    loglevel: Optional[str] = None,
 ) -> None:
    '''
@ -428,14 +297,8 @@ async def cascade(
    if loglevel:
        get_console_log(loglevel)
-    src: Flume = Flume.from_msg(src_flume_addr)
+    src = attach_shm_array(token=src_shm_token)
-    dst: Flume = Flume.from_msg(
+    dst = attach_shm_array(readonly=False, token=dst_shm_token)
        dst_flume_addr,
        readonly=False,
    )
    # src: ShmArray = attach_shm_array(token=src_shm_token)
    # dst: ShmArray = attach_shm_array(readonly=False, token=dst_shm_token)
    reg = _load_builtins()
    lines = '\n'.join([f'{key.rpartition(":")[2]} => {key}' for key in reg])
@ -443,11 +306,11 @@ async def cascade(
        f'Registered FSP set:\n{lines}'
    )
-    # NOTE XXX: update actorlocal flows table which registers
+    # update actorlocal flows table which registers
-    # readonly "instances" of this fsp for symbol/source so that
+    # readonly "instances" of this fsp for symbol/source
-    # consumer fsps can look it up by source + fsp.
+    # so that consumer fsps can look it up by source + fsp.
-    # TODO: ugh i hate this wind/unwind to list over the wire but
+    # TODO: ugh i hate this wind/unwind to list over the wire
-    # not sure how else to do it.
+    # but not sure how else to do it.
    for (token, fsp_name, dst_token) in shm_registry:
        Fsp._flow_registry[(
            _Token.from_msg(token),
@ -457,15 +320,12 @@ async def cascade(
    fsp: Fsp = reg.get(
        NamespacePath(ns_path)
    )
-    func: Callable = fsp.func
+    func = fsp.func
    if not func:
        # TODO: assume it's a func target path
        raise ValueError(f'Unknown fsp target: {ns_path}')
    _fqme: str = src.mkt.fqme
    assert _fqme == fqme
    # open a data feed stream with requested broker
    feed: Feed
    async with data.feed.maybe_open_feed(
@ -479,142 +339,177 @@ async def cascade(
    ) as feed:
-        flume: Flume = feed.flumes[fqme]
+        flume = feed.flumes[fqme]
-        # XXX: can't do this since flume.feed will be set XD
+        mkt = flume.mkt
-        # assert flume == src
+        assert src.token == flume.rt_shm.token
        assert flume.mkt == src.mkt
        mkt: MktPair = flume.mkt
        # NOTE: FOR NOW, sanity checks around the feed as being
        # always the src flume (until we get to fancier/lengthier
        # chains/graphs.
        assert src.rt_shm.token == flume.rt_shm.token
        # XXX: won't work bc the _hist_shm_token value will be
        # list[list] after IPC..
        # assert flume.to_msg() == src_flume_addr
        profiler(f'{func}: feed up')
-        func_name: str = func.__name__
+        func_name = func.__name__
        async with (
-            trio.open_nursery() as tn,
+            trio.open_nursery() as n,
        ):
            # TODO: might be better to just make a "restart" method where
            # the target task is spawned implicitly and then the event is
            # set via some higher level api? At that poing we might as well
            # be writing a one-cancels-one nursery though right?
            casc = Cascade(
                src,
                dst,
                tn,
                fsp,
            )
            # TODO: this seems like it should be wrapped somewhere?
            fsp_target = partial(
-                connect_streams,
+
-                casc=casc,
+                fsp_compute,
                mkt=mkt,
                flume=flume,
                quote_stream=flume.stream,
-                # flumes and shm passthrough
+                # shm
                src=src,
                dst=dst,
-                # chain function which takes src flume input(s)
+                # target
-                # and renders dst flume output(s)
+                func=func
                edge_func=func
            )
            async with casc.open_edge(
                bind_func=fsp_target,
            ) as index:
                # casc.bind_func = fsp_target
                # index = await tn.start(fsp_target)
                dst_shm: ShmArray = dst.rt_shm
                src_shm: ShmArray = src.rt_shm
-                if zero_on_step:
+            tracker, index = await n.start(fsp_target)
                    last = dst.rt_shm.array[-1:]
                    zeroed = np.zeros(last.shape, dtype=last.dtype)
-                profiler(f'{func_name}: fsp up')
+            if zero_on_step:
                last = dst.array[-1:]
                zeroed = np.zeros(last.shape, dtype=last.dtype)
-                # sync to client-side actor
+            profiler(f'{func_name}: fsp up')
                await ctx.started(index)
-                # XXX:  rt stream with client which we MUST
+            # sync client
-                # open here (and keep it open) in order to make
+            await ctx.started(index)
                # incremental "updates" as history prepends take
                # place.
                async with ctx.open_stream() as client_stream:
                    casc.client_stream: tractor.MsgStream = client_stream
-                    s, step, ld = casc.is_synced()
+            # XXX:  rt stream with client which we MUST
            # open here (and keep it open) in order to make
            # incremental "updates" as history prepends take
            # place.
            async with ctx.open_stream() as client_stream:
-                    # detect sample period step for subscription to increment
+                # TODO: these likely should all become
-                    # signal
+                # methods of this ``TaskLifetime`` or wtv
-                    times = src.rt_shm.array['time']
+                # abstraction..
-                    if len(times) > 1:
+                async def resync(
-                        last_ts = times[-1]
+                    tracker: TaskTracker,
                        delay_s: float = float(last_ts - times[times != last_ts][-1])
                    else:
                        # our default "HFT" sample rate.
                        delay_s: float = _default_delay_s
-                    # sub and increment the underlying shared memory buffer
+                ) -> tuple[TaskTracker, int]:
-                    # on every step msg received from the global `samplerd`
+                    # TODO: adopt an incremental update engine/approach
-                    # service.
+                    # where possible here eventually!
-                    async with open_sample_stream(
+                    log.info(f're-syncing fsp {func_name} to source')
-                        float(delay_s)
+                    tracker.cs.cancel()
-                    ) as istream:
+                    await tracker.complete.wait()
                    tracker, index = await n.start(fsp_target)
-                        profiler(f'{func_name}: sample stream up')
+                    # always trigger UI refresh after history update,
-                        profiler.finish()
+                    # see ``piker.ui._fsp.FspAdmin.open_chain()`` and
                    # ``piker.ui._display.trigger_update()``.
                    await client_stream.send({
                        'fsp_update': {
                            'key': dst_shm_token,
                            'first': dst._first.value,
                            'last': dst._last.value,
                        }
                    })
                    return tracker, index
-                        async for i in istream:
+                def is_synced(
-                            # print(f'FSP incrementing {i}')
+                    src: ShmArray,
                    dst: ShmArray
                ) -> tuple[bool, int, int]:
                    '''
                    Predicate to dertmine if a destination FSP
                    output array is aligned to its source array.
-                            # respawn the compute task if the source
+                    '''
-                            # array has been updated such that we compute
+                    step_diff = src.index - dst.index
-                            # new history from the (prepended) source.
+                    len_diff = abs(len(src.array) - len(dst.array))
-                            synced, step_diff, _ = casc.is_synced()
+                    return not (
-                            if not synced:
+                        # the source is likely backfilling and we must
-                                step_diff: int = await casc.poll_and_sync_to_step()
+                        # sync history calculations
                        len_diff > 2
-                                # skip adding a last bar since we should already
+                        # we aren't step synced to the source and may be
-                                # be step alinged
+                        # leading/lagging by a step
-                                if step_diff == 0:
+                        or step_diff > 1
-                                    continue
+                        or step_diff < 0
                    ), step_diff, len_diff
-                            # read out last shm row, copy and write new row
+                async def poll_and_sync_to_step(
-                            array = dst_shm.array
+                    tracker: TaskTracker,
                    src: ShmArray,
                    dst: ShmArray,
-                            # some metrics like vlm should be reset
+                ) -> tuple[TaskTracker, int]:
                            # to zero every step.
                            if zero_on_step:
                                last = zeroed
                            else:
                                last = array[-1:].copy()
-                            dst.rt_shm.push(last)
+                    synced, step_diff, _ = is_synced(src, dst)
                    while not synced:
                        tracker, index = await resync(tracker)
                        synced, step_diff, _ = is_synced(src, dst)
-                            # sync with source buffer's time step
+                    return tracker, step_diff
                            src_l2 = src_shm.array[-2:]
                            src_li, src_lt = src_l2[-1][['index', 'time']]
                            src_2li, src_2lt = src_l2[-2][['index', 'time']]
                            dst_shm._array['time'][src_li] = src_lt
                            dst_shm._array['time'][src_2li] = src_2lt
-                            # last2 = dst.array[-2:]
+                s, step, ld = is_synced(src, dst)
-                            # if (
+
-                            #     last2[-1]['index'] != src_li
+                # detect sample period step for subscription to increment
-                            #     or last2[-2]['index'] != src_2li
+                # signal
-                            # ):
+                times = src.array['time']
-                            #     dstl2 = list(last2)
+                if len(times) > 1:
-                            #     srcl2 = list(src_l2)
+                    last_ts = times[-1]
-                            #     print(
+                    delay_s = float(last_ts - times[times != last_ts][-1])
-                            #         # f'{dst.token}\n'
+                else:
-                            #         f'src: {srcl2}\n'
+                    # our default "HFT" sample rate.
-                            #         f'dst: {dstl2}\n'
+                    delay_s = _default_delay_s
-                            #     )
+
                # sub and increment the underlying shared memory buffer
                # on every step msg received from the global `samplerd`
                # service.
                async with open_sample_stream(float(delay_s)) as istream:
                    profiler(f'{func_name}: sample stream up')
                    profiler.finish()
                    async for i in istream:
                        # print(f'FSP incrementing {i}')
                        # respawn the compute task if the source
                        # array has been updated such that we compute
                        # new history from the (prepended) source.
                        synced, step_diff, _ = is_synced(src, dst)
                        if not synced:
                            tracker, step_diff = await poll_and_sync_to_step(
                                tracker,
                                src,
                                dst,
                            )
                            # skip adding a last bar since we should already
                            # be step alinged
                            if step_diff == 0:
                                continue
                        # read out last shm row, copy and write new row
                        array = dst.array
                        # some metrics like vlm should be reset
                        # to zero every step.
                        if zero_on_step:
                            last = zeroed
                        else:
                            last = array[-1:].copy()
                        dst.push(last)
                        # sync with source buffer's time step
                        src_l2 = src.array[-2:]
                        src_li, src_lt = src_l2[-1][['index', 'time']]
                        src_2li, src_2lt = src_l2[-2][['index', 'time']]
                        dst._array['time'][src_li] = src_lt
                        dst._array['time'][src_2li] = src_2lt
                        # last2 = dst.array[-2:]
                        # if (
                        #     last2[-1]['index'] != src_li
                        #     or last2[-2]['index'] != src_2li
                        # ):
                        #     dstl2 = list(last2)
                        #     srcl2 = list(src_l2)
                        #     print(
                        #         # f'{dst.token}\n'
                        #         f'src: {srcl2}\n'
                        #         f'dst: {dstl2}\n'
                        #     )
--- a/piker/service/init.py
+++ b/piker/service/init.py
@ -14,45 +14,49 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-Actor runtime primtives and (distributed) service APIs for,
+Actor-runtime service orchestration machinery.
- daemon-service mgmt: `_daemon` (i.e. low-level spawn and supervise machinery
+"""
-  for sub-actors like `brokerd`, `emsd`, datad`, etc.)
+from __future__ import annotations
- service-actor supervision (via `trio` tasks) API: `._mngr`
+from ._mngr import Services
-
+from ._registry import (  # noqa
- discovery interface (via light wrapping around `tractor`'s built-in
+    _tractor_kwargs,
-  prot): `._registry`
+    _default_reg_addr,
-
+    _default_registry_host,
- `docker` cntr SC supervision for use with `trio`: `_ahab`
+    _default_registry_port,
-  - wrappers for marketstore and elasticsearch dbs
+    open_registry,
-  => TODO: maybe to (re)move elsewhere?
+    find_service,
-
+    check_for_service,
 '''
 from ._mngr import Services as Services
 from ._registry import (
    _tractor_kwargs as _tractor_kwargs,
    _default_reg_addr as _default_reg_addr,
    _default_registry_host as _default_registry_host,
    _default_registry_port as _default_registry_port,
    open_registry as open_registry,
    find_service as find_service,
    check_for_service as check_for_service,
 )
-from ._daemon import (
+from ._daemon import (  # noqa
-    maybe_spawn_daemon as maybe_spawn_daemon,
+    maybe_spawn_daemon,
-    spawn_emsd as spawn_emsd,
+    spawn_emsd,
-    maybe_open_emsd as maybe_open_emsd,
+    maybe_open_emsd,
 )
 from ._actor_runtime import (
-    open_piker_runtime as open_piker_runtime,
+    open_piker_runtime,
-    maybe_open_pikerd as maybe_open_pikerd,
+    maybe_open_pikerd,
-    open_pikerd as open_pikerd,
+    open_pikerd,
-    get_runtime_vars as get_runtime_vars,
+    get_tractor_runtime_kwargs,
 )
 from ..brokers._daemon import (
-    spawn_brokerd as spawn_brokerd,
+    spawn_brokerd,
-    maybe_spawn_brokerd as maybe_spawn_brokerd,
+    maybe_spawn_brokerd,
 )
 __all__ = [
    'check_for_service',
    'Services',
    'maybe_spawn_daemon',
    'spawn_brokerd',
    'maybe_spawn_brokerd',
    'spawn_emsd',
    'maybe_open_emsd',
    'open_piker_runtime',
    'maybe_open_pikerd',
    'open_pikerd',
    'get_tractor_runtime_kwargs',
 ]
--- a/piker/service/_actor_runtime.py
+++ b/piker/service/_actor_runtime.py
@ -45,7 +45,7 @@ from ._registry import (  # noqa
 )
-def get_runtime_vars() -> dict[str, Any]:
+def get_tractor_runtime_kwargs() -> dict[str, Any]:
    '''
    Deliver ``tractor`` related runtime variables in a `dict`.
@ -56,8 +56,6 @@ def get_runtime_vars() -> dict[str, Any]:
@acm
 async def open_piker_runtime(
    name: str,
    registry_addrs: list[tuple[str, int]] = [],
    enable_modules: list[str] = [],
    loglevel: Optional[str] = None,
@ -65,6 +63,8 @@ async def open_piker_runtime(
    # for data daemons when running in production.
    debug_mode: bool = False,
    registry_addr: None | tuple[str, int] = None,
    # TODO: once we have `rsyscall` support we will read a config
    # and spawn the service tree distributed per that.
    start_method: str = 'trio',
@ -74,7 +74,7 @@ async def open_piker_runtime(
 ) -> tuple[
    tractor.Actor,
-    list[tuple[str, int]],
+    tuple[str, int],
 ]:
    '''
    Start a piker actor who's runtime will automatically sync with
@ -84,31 +84,21 @@ async def open_piker_runtime(
    a root actor.
    '''
    # check for existing runtime, boot it
    # if not already running.
    try:
-        actor = tractor.current_actor()
+        # check for existing runtime
        actor = tractor.current_actor().uid
    except tractor._exceptions.NoRuntime:
        tractor._state._runtime_vars[
-            'piker_vars'
+            'piker_vars'] = tractor_runtime_overrides
        ] = tractor_runtime_overrides
-        # NOTE: if no registrar list passed used the default of just
+        registry_addr = registry_addr or _default_reg_addr
        # setting it as the root actor on localhost.
        registry_addrs = (
            registry_addrs
            or [_default_reg_addr]
        )
        if ems := tractor_kwargs.pop('enable_modules', None):
            # import pdbp; pdbp.set_trace()
            enable_modules.extend(ems)
        async with (
            tractor.open_root_actor(
                # passed through to ``open_root_actor``
-                registry_addrs=registry_addrs,
+                arbiter_addr=registry_addr,
                name=name,
                loglevel=loglevel,
                debug_mode=debug_mode,
@ -120,30 +110,24 @@ async def open_piker_runtime(
                enable_modules=enable_modules,
                **tractor_kwargs,
-            ) as actor,
+            ) as _,
-            open_registry(
+            open_registry(registry_addr, ensure_exists=False) as addr,
                registry_addrs,
                ensure_exists=False,
            ) as addrs,
        ):
            assert actor is tractor.current_actor()
            yield (
-                actor,
+                tractor.current_actor(),
-                addrs,
+                addr,
            )
    else:
-        async with open_registry(
+        async with open_registry(registry_addr) as addr:
            registry_addrs
        ) as addrs:
            yield (
                actor,
-                addrs,
+                addr,
            )
-_root_dname: str = 'pikerd'
+_root_dname = 'pikerd'
-_root_modules: list[str] = [
+_root_modules = [
    __name__,
    'piker.service._daemon',
    'piker.brokers._daemon',
@ -157,13 +141,13 @@ _root_modules: list[str] = [
@acm
 async def open_pikerd(
    registry_addrs: list[tuple[str, int]],
    loglevel: str | None = None,
    # XXX: you should pretty much never want debug mode
    # for data daemons when running in production.
    debug_mode: bool = False,
    registry_addr: None | tuple[str, int] = None,
    **kwargs,
@ -175,39 +159,29 @@ async def open_pikerd(
    alive underling services (see below).
    '''
    # NOTE: for the root daemon we always enable the root
    # mod set and we `list.extend()` it into wtv the
    # caller requested.
    # TODO: make this mod set more strict?
    # -[ ] eventually we should be able to avoid
    #    having the root have more then permissions to spawn other
    #    specialized daemons I think?
    ems: list[str] = kwargs.setdefault('enable_modules', [])
    ems.extend(_root_modules)
    async with (
        open_piker_runtime(
            name=_root_dname,
            # TODO: eventually we should be able to avoid
            # having the root have more then permissions to
            # spawn other specialized daemons I think?
            enable_modules=_root_modules,
            loglevel=loglevel,
            debug_mode=debug_mode,
-            registry_addrs=registry_addrs,
+            registry_addr=registry_addr,
            **kwargs,
-        ) as (
+        ) as (root_actor, reg_addr),
            root_actor,
            reg_addrs,
        ),
        tractor.open_nursery() as actor_nursery,
        trio.open_nursery() as service_nursery,
    ):
-        for addr in reg_addrs:
+        if root_actor.accept_addr != reg_addr:
-            if addr not in root_actor.accept_addrs:
+            raise RuntimeError(
-                raise RuntimeError(
+                f'`pikerd` failed to bind on {reg_addr}!\n'
-                    f'`pikerd` failed to bind on {addr}!\n'
+                'Maybe you have another daemon already running?'
-                    'Maybe you have another daemon already running?'
+            )
                )
        # assign globally for future daemon/task creation
        Services.actor_n = actor_nursery
@ -251,9 +225,9 @@ async def open_pikerd(
@acm
 async def maybe_open_pikerd(
-    registry_addrs: list[tuple[str, int]] | None = None,
+    loglevel: Optional[str] = None,
    registry_addr: None | tuple = None,
    loglevel: str | None = None,
    **kwargs,
 ) -> tractor._portal.Portal | ClassVar[Services]:
@ -279,51 +253,32 @@ async def maybe_open_pikerd(
    #     async with open_portal(chan) as arb_portal:
    #         yield arb_portal
    registry_addrs: list[tuple[str, int]] = (
        registry_addrs
        or [_default_reg_addr]
    )
    pikerd_portal: tractor.Portal | None
    async with (
        open_piker_runtime(
            name=query_name,
-            registry_addrs=registry_addrs,
+            registry_addr=registry_addr,
            loglevel=loglevel,
            **kwargs,
-        ) as (actor, addrs),
+        ) as _,
        tractor.find_actor(
            _root_dname,
            arbiter_sockaddr=registry_addr,
        ) as portal
    ):
-        if _root_dname in actor.uid:
+        # connect to any existing daemon presuming
-            yield None
+        # its registry socket was selected.
        if (
            portal is not None
        ):
            yield portal
            return
        # NOTE: IFF running in disti mode, try to attach to any
        # existing (host-local) `pikerd`.
        else:
            async with tractor.find_actor(
                _root_dname,
                registry_addrs=registry_addrs,
                only_first=True,
                # raise_on_none=True,
            ) as pikerd_portal:
                # connect to any existing remote daemon presuming its
                # registry socket was selected.
                if pikerd_portal is not None:
                    # sanity check that we are actually connecting to
                    # a remote process and not ourselves.
                    assert actor.uid != pikerd_portal.channel.uid
                    assert registry_addrs
                    yield pikerd_portal
                    return
    # presume pikerd role since no daemon could be found at
    # configured address
    async with open_pikerd(
        loglevel=loglevel,
-        registry_addrs=registry_addrs,
+        registry_addr=registry_addr,
        # passthrough to ``tractor`` init
        **kwargs,
--- a/piker/service/_ahab.py
+++ b/piker/service/_ahab.py
@ -15,8 +15,8 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-Supervisor for ``docker`` with included async and SC wrapping to
+Supervisor for ``docker`` with included async and SC wrapping
-ensure a cancellable container lifetime system.
+to ensure a cancellable container lifetime system.
 '''
 from __future__ import annotations
--- a/piker/service/_daemon.py
+++ b/piker/service/_daemon.py
@ -70,10 +70,7 @@ async def maybe_spawn_daemon(
    lock = Services.locks[service_name]
    await lock.acquire()
-    async with find_service(
+    async with find_service(service_name) as portal:
        service_name,
        registry_addrs=[('127.0.0.1', 6116)],
    ) as portal:
        if portal is not None:
            lock.release()
            yield portal
--- a/piker/service/_mngr.py
+++ b/piker/service/_mngr.py
@ -27,12 +27,6 @@ from typing import (
 import trio
 from trio_typing import TaskStatus
 import tractor
 from tractor import (
    current_actor,
    ContextCancelled,
    Context,
    Portal,
 )
 from ._util import (
    log,  # sub-sys logger
@ -44,8 +38,6 @@ from ._util import (
 #   library.
 # - wrap a "remote api" wherein you can get a method proxy
 #   to the pikerd actor for starting services remotely!
 # - prolly rename this to ActorServicesNursery since it spawns
 #   new actors and supervises them to completion?
 class Services:
    actor_n: tractor._supervise.ActorNursery
@ -55,7 +47,7 @@ class Services:
        str,
        tuple[
            trio.CancelScope,
-            Portal,
+            tractor.Portal,
            trio.Event,
        ]
    ] = {}
@ -65,12 +57,12 @@ class Services:
    async def start_service_task(
        self,
        name: str,
-        portal: Portal,
+        portal: tractor.Portal,
        target: Callable,
        allow_overruns: bool = False,
        **ctx_kwargs,
-    ) -> (trio.CancelScope, Context):
+    ) -> (trio.CancelScope, tractor.Context):
        '''
        Open a context in a service sub-actor, add to a stack
        that gets unwound at ``pikerd`` teardown.
@ -109,30 +101,13 @@ class Services:
                        # wait on any context's return value
                        # and any final portal result from the
                        # sub-actor.
-                        ctx_res: Any = await ctx.result()
+                        ctx_res = await ctx.result()
                        # NOTE: blocks indefinitely until cancelled
                        # either by error from the target context
                        # function or by being cancelled here by the
                        # surrounding cancel scope.
                        return (await portal.result(), ctx_res)
                    except ContextCancelled as ctxe:
                        canceller: tuple[str, str] = ctxe.canceller
                        our_uid: tuple[str, str] = current_actor().uid
                        if (
                            canceller != portal.channel.uid
                            and
                            canceller != our_uid
                        ):
                            log.cancel(
                                f'Actor-service {name} was remotely cancelled?\n'
                                f'remote canceller: {canceller}\n'
                                f'Keeping {our_uid} alive, ignoring sub-actor cancel..\n'
                            )
                        else:
                            raise
                    finally:
                        await portal.cancel_actor()
--- a/piker/service/_registry.py
+++ b/piker/service/_registry.py
@ -27,7 +27,6 @@ from typing import (
 )
 import tractor
 from tractor import Portal
 from ._util import (
    log,  # sub-sys logger
@ -47,9 +46,7 @@ _registry: Registry | None = None
 class Registry:
-    # TODO: should this be a set or should we complain
+    addr: None | tuple[str, int] = None
    # on duplicates?
    addrs: list[tuple[str, int]] = []
    # TODO: table of uids to sockaddrs
    peers: dict[
@ -63,115 +60,69 @@ _tractor_kwargs: dict[str, Any] = {}
@acm
 async def open_registry(
-    addrs: list[tuple[str, int]],
+    addr: None | tuple[str, int] = None,
    ensure_exists: bool = True,
-) -> list[tuple[str, int]]:
+) -> tuple[str, int]:
    '''
    Open the service-actor-discovery registry by returning a set of
    tranport socket-addrs to registrar actors which may be
    contacted and queried for similar addresses for other
    non-registrar actors.
    '''
    global _tractor_kwargs
    actor = tractor.current_actor()
    uid = actor.uid
    preset_reg_addrs: list[tuple[str, int]] = Registry.addrs
    if (
-        preset_reg_addrs
+        Registry.addr is not None
-        and addrs
+        and addr
    ):
-        if preset_reg_addrs != addrs:
+        raise RuntimeError(
-            # if any(addr in preset_reg_addrs for addr in addrs):
+            f'`{uid}` registry addr already bound @ {_registry.sockaddr}'
-            diff: set[tuple[str, int]] = set(preset_reg_addrs) - set(addrs)
+        )
            if diff:
                log.warning(
                    f'`{uid}` requested only subset of registrars: {addrs}\n'
                    f'However there are more @{diff}'
                )
            else:
                raise RuntimeError(
                    f'`{uid}` has non-matching registrar addresses?\n'
                    f'request: {addrs}\n'
                    f'already set: {preset_reg_addrs}'
                )
    was_set: bool = False
    if (
        not tractor.is_root_process()
-        and not Registry.addrs
+        and Registry.addr is None
    ):
-        Registry.addrs.extend(actor.reg_addrs)
+        Registry.addr = actor._arb_addr
    if (
        ensure_exists
-        and not Registry.addrs
+        and Registry.addr is None
    ):
        raise RuntimeError(
-            f"`{uid}` registry should already exist but doesn't?"
+            f"`{uid}` registry should already exist bug doesn't?"
        )
    if (
-        not Registry.addrs
+        Registry.addr is None
    ):
        was_set = True
-        Registry.addrs = addrs or [_default_reg_addr]
+        Registry.addr = addr or _default_reg_addr
-    # NOTE: only spot this seems currently used is inside
+    _tractor_kwargs['arbiter_addr'] = Registry.addr
    # `.ui._exec` which is the (eventual qtloops) bootstrapping
    # with guest mode.
    _tractor_kwargs['registry_addrs'] = Registry.addrs
    try:
-        yield Registry.addrs
+        yield Registry.addr
    finally:
        # XXX: always clear the global addr if we set it so that the
        # next (set of) calls will apply whatever new one is passed
        # in.
        if was_set:
-            Registry.addrs = None
+            Registry.addr = None
@acm
 async def find_service(
    service_name: str,
-    registry_addrs: list[tuple[str, int]] | None = None,
+) -> tractor.Portal | None:
-    first_only: bool = True,
+    async with open_registry() as reg_addr:
 ) -> (
    Portal
    | list[Portal]
    | None
 ):
    reg_addrs: list[tuple[str, int]]
    async with open_registry(
        addrs=(
            registry_addrs
            # NOTE: if no addr set is passed assume the registry has
            # already been opened and use the previously applied
            # startup set.
            or Registry.addrs
        ),
    ) as reg_addrs:
        log.info(f'Scanning for service `{service_name}`')
        maybe_portals: list[Portal] | Portal | None
        # attach to existing daemon by name if possible
        async with tractor.find_actor(
            service_name,
-            registry_addrs=reg_addrs,
+            arbiter_sockaddr=reg_addr,
-            only_first=first_only,  # if set only returns single ref
+        ) as maybe_portal:
-        ) as maybe_portals:
+            yield maybe_portal
            if not maybe_portals:
                yield None
                return
            yield maybe_portals
 async def check_for_service(
@ -182,11 +133,9 @@ async def check_for_service(
    Service daemon "liveness" predicate.
    '''
-    async with (
+    async with open_registry(ensure_exists=False) as reg_addr:
-        open_registry(ensure_exists=False) as reg_addr,
+        async with tractor.query_actor(
        tractor.query_actor(
            service_name,
            arbiter_sockaddr=reg_addr,
-        ) as sockaddr,
+        ) as sockaddr:
-    ):
+            return sockaddr
        return sockaddr
--- a/piker/storage/init.py
+++ b/piker/storage/init.py
@ -139,13 +139,6 @@ class StorageClient(
        ...
 class TimeseriesNotFound(Exception):
    '''
    No timeseries entry can be found for this backend.
    '''
 class StorageConnectionError(ConnectionError):
    '''
    Can't connect to the desired tsdb subsys/service.
@ -176,13 +169,10 @@ async def open_storage_client(
    tsdb_host: str = 'localhost'
    # load root config and any tsdb user defined settings
-    conf, path = config.load(
+    conf, path = config.load('conf', touch_if_dne=True)
        conf_name='conf',
        touch_if_dne=True,
    )
    # TODO: maybe not under a "network" section.. since
-    # no more chitty `marketstore`..
+    # no more chitty mkts..
    tsdbconf: dict = {}
    service_section = conf.get('service')
    if (
@ -193,11 +183,8 @@ async def open_storage_client(
        # lookup backend tsdb module by name and load any user service
        # settings for connecting to the tsdb service.
-        backend: str = tsdbconf.pop(
+        backend: str = tsdbconf.pop('backend')
-            'name',
+        tsdb_host: str = tsdbconf['host']
            def_backend,
        )
        tsdb_host: str = tsdbconf.get('maddrs', [])
    if backend is None:
        backend: str = def_backend
@ -259,7 +246,7 @@ async def open_tsdb_client(
    #   * the original data feed arch blurb:
    #     - https://github.com/pikers/piker/issues/98
    #
-    from ..toolz import Profiler
+    from .._profile import Profiler
    profiler = Profiler(
        disabled=True,  # not pg_profile_enabled(),
        delayed=False,
--- a/piker/storage/cli.py
+++ b/piker/storage/cli.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -19,18 +19,10 @@ Storage middle-ware CLIs.
 """
 from __future__ import annotations
 # from datetime import datetime
 # from contextlib import (
 #     AsyncExitStack,
 # )
 from pathlib import Path
 from math import copysign
 import time
-from types import ModuleType
+from typing import Generator
-from typing import (
+# from typing import TYPE_CHECKING
    Any,
    TYPE_CHECKING,
 )
 import polars as pl
 import numpy as np
@ -43,21 +35,24 @@ import typer
 from piker.service import open_piker_runtime
 from piker.cli import cli
 from piker.config import get_conf_dir
 from piker.data import (
    maybe_open_shm_array,
    def_iohlcv_fields,
    ShmArray,
 )
-from piker import tsp
+from piker.data.history import (
-from piker.data._formatters import BGM
+    _default_hist_size,
-from . import log
+    _default_rt_size,
 )
 from . import (
    log,
 )
 from . import (
    __tsdbs__,
    open_storage_client,
    StorageClient,
 )
 if TYPE_CHECKING:
    from piker.ui._remote_ctl import AnnotCtl
 store = typer.Typer()
@ -82,6 +77,7 @@ def ls(
        async with (
            open_piker_runtime(
                'tsdb_storage',
                enable_modules=['piker.service._ahab'],
            ),
        ):
            for i, backend in enumerate(backends):
@ -103,18 +99,6 @@ def ls(
    trio.run(query_all)
 # TODO: like ls but takes in a pattern and matches
 # @store.command()
 # def search(
 #     patt: str,
 #     backends: list[str] = typer.Argument(
 #         default=None,
 #         help='Storage backends to query, default is all.'
 #     ),
 # ):
 #     ...
@store.command()
 def delete(
    symbols: list[str],
@ -137,6 +121,7 @@ def delete(
        async with (
            open_piker_runtime(
                'tsdb_storage',
                enable_modules=['piker.service._ahab']
            ),
            open_storage_client(backend) as (_, client),
            trio.open_nursery() as n,
@ -157,33 +142,20 @@ def delete(
 def anal(
    fqme: str,
    period: int = 60,
    pdb: bool = False,
 ) -> np.ndarray:
    '''
    Anal-ysis is when you take the data do stuff to it.
    NOTE: This ONLY loads the offline timeseries data (by default
    from a parquet file) NOT the in-shm version you might be seeing
    in a chart.
    '''
    async def main():
        async with (
            open_piker_runtime(
                # are you a bear or boi?
                'tsdb_polars_anal',
-                debug_mode=pdb,
+                # enable_modules=['piker.service._ahab']
            ),
            open_storage_client() as (
                mod,
                client,
            ),
            open_storage_client() as (mod, client),
        ):
            syms: list[str] = await client.list_keys()
-            log.info(f'{len(syms)} FOUND for {mod.name}')
+            print(f'{len(syms)} FOUND for {mod.name}')
            history: ShmArray  # np buffer format
            (
                history,
                first_dt,
@ -194,357 +166,166 @@ def anal(
            )
            assert first_dt < last_dt
-            null_segs: tuple = tsp.get_null_segs(
+            src_df = await client.as_df(fqme, period)
-                frame=history,
+            from piker.data import _timeseries as tsmod
-                period=period,
+            df = tsmod.with_dts(src_df)
-            )
+            gaps: pl.DataFrame = tsmod.detect_time_gaps(df)
            # TODO: do tsp queries to backcend to fill i missing
            # history and then prolly write it to tsdb!
-            shm_df: pl.DataFrame = await client.as_df(
+            if gaps:
-                fqme,
+                print(f'Gaps found:\n{gaps}')
                period,
            )
-            df: pl.DataFrame  # with dts
+            # TODO: something better with tab completion..
-            deduped: pl.DataFrame  # deduplicated dts
+            # is there something more minimal but nearly as
-            (
+            # functional as ipython?
-                df,
+            await tractor.breakpoint()
                deduped,
                diff,
            ) = tsp.dedupe(
                shm_df,
                period=period,
            )
            write_edits: bool = True
            if (
                write_edits
                and (
                    diff
                    or null_segs
                )
            ):
                await tractor.pause()
                await client.write_ohlcv(
                    fqme,
                    ohlcv=deduped,
                    timeframe=period,
                )
            else:
                # TODO: something better with tab completion..
                # is there something more minimal but nearly as
                # functional as ipython?
                await tractor.pause()
                assert not null_segs
    trio.run(main)
-async def markup_gaps(
+def iter_dfs_from_shms(fqme: str) -> Generator[
-    fqme: str,
+    tuple[Path, ShmArray, pl.DataFrame],
-    timeframe: float,
+    None,
-    actl: AnnotCtl,
+    None,
-    wdts: pl.DataFrame,
+]:
-    gaps: pl.DataFrame,
+    # shm buffer size table based on known sample rates
    sizes: dict[str, int] = {
        'hist': _default_hist_size,
        'rt': _default_rt_size,
    }
-) -> dict[int, dict]:
+    # load all detected shm buffer files which have the
-    '''
+    # passed FQME pattern in the file name.
-    Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
+    shmfiles: list[Path] = []
-    with rectangles.
+    shmdir = Path('/dev/shm/')
-    '''
+    for shmfile in shmdir.glob(f'*{fqme}*'):
-    aids: dict[int] = {}
+        filename: str = shmfile.name
    for i in range(gaps.height):
-        row: pl.DataFrame = gaps[i]
+        # skip index files
        if (
            '_first' in filename
            or '_last' in filename
        ):
            continue
-        # the gap's RIGHT-most bar's OPEN value
+        assert shmfile.is_file()
-        # at that time (sample) step.
+        log.debug(f'Found matching shm buffer file: {filename}')
-        iend: int = row['index'][0]
+        shmfiles.append(shmfile)
        # dt: datetime = row['dt'][0]
        # dt_prev: datetime = row['dt_prev'][0]
        # dt_end_t: float = dt.timestamp()
    for shmfile in shmfiles:
-        # TODO: can we eventually remove this
+        # lookup array buffer size based on file suffix
-        # once we figure out why the epoch cols
+        # being either .rt or .hist
-        # don't match?
+        size: int = sizes[shmfile.name.rsplit('.')[-1]]
        # TODO: FIX HOW/WHY these aren't matching
        # and are instead off by 4hours (EST
        # vs. UTC?!?!)
        # end_t: float = row['time']
        # assert (
        #     dt.timestamp()
        #     ==
        #     end_t
        # )
-        # the gap's LEFT-most bar's CLOSE value
+        # attach to any shm buffer, load array into polars df,
-        # at that time (sample) step.
+        # write to local parquet file.
-        prev_r: pl.DataFrame = wdts.filter(
+        shm, opened = maybe_open_shm_array(
-            pl.col('index') == iend - 1
+            key=shmfile.name,
            size=size,
            dtype=def_iohlcv_fields,
            readonly=True,
        )
-        # XXX: probably a gap in the (newly sorted or de-duplicated)
+        assert not opened
-        # dt-df, so we might need to re-index first..
+        ohlcv = shm.array
        if prev_r.is_empty():
            await tractor.pause()
-        istart: int = prev_r['index'][0]
+        start = time.time()
        # dt_start_t: float = dt_prev.timestamp()
-        # start_t: float = prev_r['time']
+        # XXX: thanks to this SO answer for this conversion tip:
-        # assert (
+        # https://stackoverflow.com/a/72054819
-        #     dt_start_t
+        df = pl.DataFrame({
-        #     ==
+            field_name: ohlcv[field_name]
-        #     start_t
+            for field_name in ohlcv.dtype.fields
-        # )
+        })
-
+        delay: float = round(
-        # TODO: implement px-col width measure
+            time.time() - start,
-        # and ensure at least as many px-cols
+            ndigits=6,
        # shown per rect as configured by user.
        # gap_w: float = abs((iend - istart))
        # if gap_w < 6:
        #     margin: float = 6
        #     iend += margin
        #     istart -= margin
        rect_gap: float = BGM*3/8
        opn: float = row['open'][0]
        ro: tuple[float, float] = (
            # dt_end_t,
            iend + rect_gap + 1,
            opn,
        )
-        cls: float = prev_r['close'][0]
+        log.info(
-        lc: tuple[float, float] = (
+            f'numpy -> polars conversion took {delay} secs\n'
-            # dt_start_t,
+            f'polars df: {df}'
            istart - rect_gap, # + 1 ,
            cls,
        )
-        color: str = 'dad_blue'
+        yield (
-        diff: float = cls - opn
+            shmfile,
-        sgn: float = copysign(1, diff)
+            shm,
-        color: str = {
+            df,
            -1: 'buy_green',
            1: 'sell_red',
        }[sgn]
        rect_kwargs: dict[str, Any] = dict(
            fqme=fqme,
            timeframe=timeframe,
            start_pos=lc,
            end_pos=ro,
            color=color,
        )
        aid: int = await actl.add_rect(**rect_kwargs)
        assert aid
        aids[aid] = rect_kwargs
    # tell chart to redraw all its
    # graphics view layers Bo
    await actl.redraw(
        fqme=fqme,
        timeframe=timeframe,
    )
    return aids
@store.command()
 def ldshm(
    fqme: str,
-    write_parquet: bool = True,
+
-    reload_parquet_to_shm: bool = True,
+    write_parquet: bool = False,
 ) -> None:
    '''
    Linux ONLY: load any fqme file name matching shm buffer from
    /dev/shm/ into an OHLCV numpy array and polars DataFrame,
-    optionally write to offline storage via `.parquet` file.
+    optionally write to .parquet file.
    '''
    async def main():
        from piker.ui._remote_ctl import (
            open_annot_ctl,
        )
        actl: AnnotCtl
        mod: ModuleType
        client: StorageClient
        async with (
            open_piker_runtime(
                'polars_boi',
                enable_modules=['piker.data._sharedmem'],
                debug_mode=True,
            ),
            open_storage_client() as (
                mod,
                client,
            ),
            open_annot_ctl() as actl,
        ):
            shm_df: pl.DataFrame | None = None
            tf2aids: dict[float, dict] = {}
-            for (
+            df: pl.DataFrame | None = None
-                shmfile,
+            for shmfile, shm, df in iter_dfs_from_shms(fqme):
                shm,
                # parquet_path,
                shm_df,
            ) in tsp.iter_dfs_from_shms(fqme):
                # compute ohlc properties for naming
                times: np.ndarray = shm.array['time']
-                d1: float = float(times[-1] - times[-2])
+                secs: float = times[-1] - times[-2]
-                d2: float = float(times[-2] - times[-3])
+                if secs < 1.:
-                med: float = np.median(np.diff(times))
+                    breakpoint()
                if (
                    d1 < 1.
                    and d2 < 1.
                    and med < 1.
                ):
                    raise ValueError(
                        f'Something is wrong with time period for {shm}:\n{times}'
                    )
-                period_s: float = float(max(d1, d2, med))
+                # TODO: maybe only optionally enter this depending
                # on some CLI flags and/or gap detection?
                await tractor.breakpoint()
-                null_segs: tuple = tsp.get_null_segs(
+                # write to parquet file?
-                    frame=shm.array,
+                if write_parquet:
-                    period=period_s,
+                    timeframe: str = f'{secs}s'
                )
-                # TODO: call null-seg fixer somehow?
+                    datadir: Path = get_conf_dir() / 'nativedb'
-                if null_segs:
+                    if not datadir.is_dir():
-                    await tractor.pause()
+                        datadir.mkdir()
                #     async with (
                #         trio.open_nursery() as tn,
                #         mod.open_history_client(
                #             mkt,
                #         ) as (get_hist, config),
                #     ):
                #         nulls_detected: trio.Event = await tn.start(partial(
                #             tsp.maybe_fill_null_segments,
-                #             shm=shm,
+                    path: Path = datadir / f'{fqme}.{timeframe}.parquet'
                #             timeframe=timeframe,
                #             get_hist=get_hist,
                #             sampler_stream=sampler_stream,
                #             mkt=mkt,
                #         ))
-                # over-write back to shm?
+                    # write to fs
-                wdts: pl.DataFrame  # with dts
+                    start = time.time()
-                deduped: pl.DataFrame  # deduplicated dts
+                    df.write_parquet(path)
-                (
+                    delay: float = round(
-                    wdts,
+                        time.time() - start,
-                    deduped,
+                        ndigits=6,
-                    diff,
+                    )
-                ) = tsp.dedupe(
+                    log.info(
-                    shm_df,
+                        f'parquet write took {delay} secs\n'
-                    period=period_s,
+                        f'file path: {path}'
                )
                # detect gaps from in expected (uniform OHLC) sample period
                step_gaps: pl.DataFrame = tsp.detect_time_gaps(
                    deduped,
                    expect_period=period_s,
                )
                # TODO: by default we always want to mark these up
                # with rects showing up/down gaps Bo
                venue_gaps: pl.DataFrame = tsp.detect_time_gaps(
                    deduped,
                    expect_period=period_s,
                    # TODO: actually pull the exact duration
                    # expected for each venue operational period?
                    gap_dt_unit='days',
                    gap_thresh=1,
                )
                # TODO: find the disjoint set of step gaps from
                # venue (closure) set!
                # -[ ] do a set diff by checking for the unique
                #    gap set only in the step_gaps?
                if (
                    not venue_gaps.is_empty()
                    or (
                        period_s < 60
                        and not step_gaps.is_empty()
                    )
                ):
                    # write repaired ts to parquet-file?
                    if write_parquet:
                        start: float = time.time()
                        path: Path = await client.write_ohlcv(
                            fqme,
                            ohlcv=deduped,
                            timeframe=period_s,
                        )
                        write_delay: float = round(
                            time.time() - start,
                            ndigits=6,
                        )
-                        # read back from fs
+                    # read back from fs
-                        start: float = time.time()
+                    start = time.time()
-                        read_df: pl.DataFrame = pl.read_parquet(path)
+                    read_df: pl.DataFrame = pl.read_parquet(path)
-                        read_delay: float = round(
+                    delay: float = round(
-                            time.time() - start,
+                        time.time() - start,
-                            ndigits=6,
+                        ndigits=6,
-                        )
+                    )
-                        log.info(
+                    print(
-                            f'parquet write took {write_delay} secs\n'
+                        f'parquet read took {delay} secs\n'
-                            f'file path: {path}'
+                        f'polars df: {read_df}'
-                            f'parquet read took {read_delay} secs\n'
+                    )
                            f'polars df: {read_df}'
                        )
-                        if reload_parquet_to_shm:
+            if df is None:
-                            new = tsp.pl2np(
+                log.error(f'No matching shm buffers for {fqme} ?')
                                deduped,
                                dtype=shm.array.dtype,
                            )
                            # since normally readonly
                            shm._array.setflags(
                                write=int(1),
                            )
                            shm.push(
                                new,
                                prepend=True,
                                start=new['index'][-1],
                                update_first=False,  # don't update ._first
                            )
                    do_markup_gaps: bool = True
                    if do_markup_gaps:
                        new_df: pl.DataFrame = tsp.np2pl(new)
                        aids: dict = await markup_gaps(
                            fqme,
                            period_s,
                            actl,
                            new_df,
                            step_gaps,
                        )
                        # last chance manual overwrites in REPL
                        # await tractor.pause()
                        assert aids
                        tf2aids[period_s] = aids
                else:
                    # allow interaction even when no ts problems.
                    assert not diff
            await tractor.pause()
            log.info('Exiting TSP shm anal-izer!')
            if shm_df is None:
                log.error(
                    f'No matching shm buffers for {fqme} ?'
                )
    trio.run(main)
--- a/piker/storage/marketstore/init.py
+++ b/piker/storage/marketstore/init.py
@ -59,6 +59,7 @@ from anyio_marketstore import (  # noqa
    Params,
 )
 from piker.log import get_logger
 # from .._profile import Profiler
 log = get_logger(__name__)
@ -204,7 +205,7 @@ class MktsStorageClient:
        #         break
        #     except purerpc.grpclib.exceptions.UnknownError as err:
        #         if 'snappy' in err.args:
-        #             await tractor.pause()
+        #             await tractor.breakpoint()
        #         # indicate there is no history for this timeframe
        #         log.exception(
@ -232,7 +233,7 @@ class MktsStorageClient:
                    'YOUR DATABASE LIKELY CONTAINS BAD DATA FROM AN OLD BUG '
                    f'WIPING HISTORY FOR {ts}s'
                )
-                await tractor.pause()
+                await tractor.breakpoint()
                # await self.delete_ts(fqme, timeframe)
                # try reading again..
--- a/piker/storage/nativedb.py
+++ b/piker/storage/nativedb.py
@ -19,8 +19,7 @@
            call a poor man's tsdb).
 AKA a `piker`-native file-system native "time series database"
-without needing an extra process and no standard TSDB features,
+without needing an extra process and no standard TSDB features, YET!
 YET!
 '''
 # TODO: like there's soo much..
@ -56,6 +55,8 @@ from datetime import datetime
 from pathlib import Path
 import time
 # from bidict import bidict
 # import tractor
 import numpy as np
 import polars as pl
 from pendulum import (
@ -63,18 +64,46 @@ from pendulum import (
 )
 from piker import config
-from piker import tsp
+from piker.data import def_iohlcv_fields
-from piker.data import (
+from piker.data import ShmArray
    def_iohlcv_fields,
    ShmArray,
 )
 from piker.log import get_logger
-from . import TimeseriesNotFound
+# from .._profile import Profiler
 log = get_logger('storage.nativedb')
 # NOTE: thanks to this SO answer for the below conversion routines
 # to go from numpy struct-arrays to polars dataframes and back:
 # https://stackoverflow.com/a/72054819
 def np2pl(array: np.ndarray) -> pl.DataFrame:
    return pl.DataFrame({
        field_name: array[field_name]
        for field_name in array.dtype.fields
    })
 def pl2np(
    df: pl.DataFrame,
    dtype: np.dtype,
 ) -> np.ndarray:
    # Create numpy struct array of the correct size and dtype
    # and loop through df columns to fill in array fields.
    array = np.empty(
        df.height,
        dtype,
    )
    for field, col in zip(
        dtype.fields,
        df.columns,
    ):
        array[field] = df.get_column(col).to_numpy()
    return array
 def detect_period(shm: ShmArray) -> float:
    '''
    Attempt to detect the series time step sampling period
@ -95,19 +124,16 @@ def detect_period(shm: ShmArray) -> float:
 def mk_ohlcv_shm_keyed_filepath(
    fqme: str,
-    period: float | int,  # ow known as the "timeframe"
+    period: float,  # ow known as the "timeframe"
    datadir: Path,
-) -> Path:
+) -> str:
    if period < 1.:
        raise ValueError('Sample period should be >= 1.!?')
-    path: Path = (
+    period_s: str = f'{period}s'
-        datadir
+    path: Path = datadir / f'{fqme}.ohlcv{period_s}.parquet'
        /
        f'{fqme}.ohlcv{int(period)}s.parquet'
    )
    return path
@ -161,13 +187,7 @@ class NativeStorageClient:
    def index_files(self):
        for path in self._datadir.iterdir():
-            if (
+            if 'borked' in path.name:
                path.is_dir()
                or
                '.parquet' not in str(path)
                # or
                # path.name in {'borked', 'expired',}
            ):
                continue
            key: str = path.name.rstrip('.parquet')
@ -209,21 +229,8 @@ class NativeStorageClient:
                fqme,
                timeframe,
            )
-        except FileNotFoundError as fnfe:
+        except FileNotFoundError:
-
+            return None
            bs_fqme, _, *_ = fqme.rpartition('.')
            possible_matches: list[str] = []
            for tskey in self._index:
                if bs_fqme in tskey:
                    possible_matches.append(tskey)
            match_str: str = '\n'.join(sorted(possible_matches))
            raise TimeseriesNotFound(
                f'No entry for `{fqme}`?\n'
                f'Maybe you need a more specific fqme-key like:\n\n'
                f'{match_str}'
            ) from fnfe
        times = array['time']
        return (
@ -236,7 +243,6 @@ class NativeStorageClient:
        self,
        fqme: str,
        period: float,
    ) -> Path:
        return mk_ohlcv_shm_keyed_filepath(
            fqme=fqme,
@ -244,23 +250,6 @@ class NativeStorageClient:
            datadir=self._datadir,
        )
    def _cache_df(
        self,
        fqme: str,
        df: pl.DataFrame,
        timeframe: float,
    ) -> None:
        # cache df for later usage since we (currently) need to
        # convert to np.ndarrays to push to our `ShmArray` rt
        # buffers subsys but later we may operate entirely on
        # pyarrow arrays/buffers so keeping the dfs around for
        # a variety of purposes is handy.
        self._dfs.setdefault(
            timeframe,
            {},
        )[fqme] = df
    async def read_ohlcv(
        self,
        fqme: str,
@ -269,20 +258,13 @@ class NativeStorageClient:
        # limit: int = int(200e3),
    ) -> np.ndarray:
-        path: Path = self.mk_path(
+        path: Path = self.mk_path(fqme, period=int(timeframe))
            fqme,
            period=int(timeframe),
        )
        df: pl.DataFrame = pl.read_parquet(path)
        self._dfs.setdefault(timeframe, {})[fqme] = df
        self._cache_df(
            fqme=fqme,
            df=df,
            timeframe=timeframe,
        )
        # TODO: filter by end and limit inputs
        # times: pl.Series = df['time']
-        array: np.ndarray = tsp.pl2np(
+        array: np.ndarray = pl2np(
            df,
            dtype=np.dtype(def_iohlcv_fields),
        )
@ -292,15 +274,11 @@ class NativeStorageClient:
        self,
        fqme: str,
        period: int = 60,
        load_from_offline: bool = True,
    ) -> pl.DataFrame:
        try:
            return self._dfs[period][fqme]
        except KeyError:
            if not load_from_offline:
                raise
            await self.read_ohlcv(fqme, period)
            return self._dfs[period][fqme]
@ -322,39 +300,32 @@ class NativeStorageClient:
            datadir=self._datadir,
        )
        if isinstance(ohlcv, np.ndarray):
-            df: pl.DataFrame = tsp.np2pl(ohlcv)
+            df: pl.DataFrame = np2pl(ohlcv)
        else:
            df = ohlcv
        self._cache_df(
            fqme=fqme,
            df=df,
            timeframe=timeframe,
        )
        # TODO: in terms of managing the ultra long term data
-        # -[ ] use a proper profiler to measure all this IO and
+        # - use a proper profiler to measure all this IO and
        #   roundtripping!
-        # -[ ] implement parquet append!? see issue:
+        # - try out ``fastparquet``'s append writing:
-        #   https://github.com/pikers/piker/issues/536
+        # https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
        #   -[ ] try out ``fastparquet``'s append writing:
        #     https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
        start = time.time()
        df.write_parquet(path)
        delay: float = round(
            time.time() - start,
            ndigits=6,
        )
-        log.info(
+        print(
            f'parquet write took {delay} secs\n'
            f'file path: {path}'
        )
        return path
    async def write_ohlcv(
        self,
        fqme: str,
-        ohlcv: np.ndarray | pl.DataFrame,
+        ohlcv: np.ndarray,
        timeframe: int,
    ) -> Path:
@ -406,8 +377,6 @@ class NativeStorageClient:
    #     ...
 # TODO: does this need to be async on average?
 # I guess for any IPC connected backend yes?
@acm
 async def get_client(
@ -425,7 +394,7 @@ async def get_client(
    '''
    datadir: Path = config.get_conf_dir() / 'nativedb'
    if not datadir.is_dir():
-        log.info(f'Creating `nativedb` dir: {datadir}')
+        log.info(f'Creating `nativedb` director: {datadir}')
        datadir.mkdir()
    client = NativeStorageClient(datadir)
--- a/piker/toolz/init.py
+++ b/piker/toolz/init.py
@ -1,29 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Toolz for debug, profile and trace of the distributed runtime :surfer:
 '''
 from tractor.devx import (
    open_crash_handler as open_crash_handler,
 )
 from .profile import (
    Profiler as Profiler,
    pg_profile_enabled as pg_profile_enabled,
    ms_slower_then as ms_slower_then,
    timeit as timeit,
 )
--- a/piker/trionics.py
+++ b/piker/trionics.py
@ -0,0 +1,80 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship of piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 sugarz for trio/tractor conc peeps.
 '''
 from typing import AsyncContextManager
 from typing import TypeVar
 from contextlib import asynccontextmanager as acm
 import trio
 # A regular invariant generic type
 T = TypeVar("T")
 async def _enter_and_sleep(
    mngr: AsyncContextManager[T],
    to_yield: dict[int, T],
    all_entered: trio.Event,
    # task_status: TaskStatus[T] = trio.TASK_STATUS_IGNORED,
 ) -> T:
    '''Open the async context manager deliver it's value
    to this task's spawner and sleep until cancelled.
    '''
    async with mngr as value:
        to_yield[id(mngr)] = value
        if all(to_yield.values()):
            all_entered.set()
        # sleep until cancelled
        await trio.sleep_forever()
@acm
 async def async_enter_all(
    *mngrs: list[AsyncContextManager[T]],
 ) -> tuple[T]:
    to_yield = {}.fromkeys(id(mngr) for mngr in mngrs)
    all_entered = trio.Event()
    async with trio.open_nursery() as n:
        for mngr in mngrs:
            n.start_soon(
                _enter_and_sleep,
                mngr,
                to_yield,
                all_entered,
            )
        # deliver control once all managers have started up
        await all_entered.wait()
        yield tuple(to_yield.values())
        # tear down all sleeper tasks thus triggering individual
        # mngr ``__aexit__()``s.
        n.cancel_scope.cancel()
--- a/piker/tsp/init.py
+++ b/piker/tsp/init.py
--- a/piker/tsp/_anal.py
+++ b/piker/tsp/_anal.py
@ -1,746 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Financial time series processing utilities usually
 pertaining to OHLCV style sampled data.
 Routines are generally implemented in either ``numpy`` or
 ``polars`` B)
 '''
 from __future__ import annotations
 from functools import partial
 from math import (
    ceil,
    floor,
 )
 import time
 from typing import (
    Literal,
    # AsyncGenerator,
    Generator,
 )
 import numpy as np
 import polars as pl
 from pendulum import (
    DateTime,
    from_timestamp,
 )
 from ..toolz.profile import (
    Profiler,
    pg_profile_enabled,
    ms_slower_then,
 )
 from ..log import (
    get_logger,
    get_console_log,
 )
 # for "time series processing"
 subsys: str = 'piker.tsp'
 log = get_logger(subsys)
 get_console_log = partial(
    get_console_log,
    name=subsys,
 )
 # NOTE: union type-defs to handle generic `numpy` and `polars` types
 # side-by-side Bo
 # |_ TODO: schema spec typing?
 #   -[ ] nptyping!
 #   -[ ] wtv we can with polars?
 Frame = pl.DataFrame | np.ndarray
 Seq = pl.Series | np.ndarray
 def slice_from_time(
    arr: np.ndarray,
    start_t: float,
    stop_t: float,
    step: float,  # sampler period step-diff
 ) -> slice:
    '''
    Calculate array indices mapped from a time range and return them in
    a slice.
    Given an input array with an epoch `'time'` series entry, calculate
    the indices which span the time range and return in a slice. Presume
    each `'time'` step increment is uniform and when the time stamp
    series contains gaps (the uniform presumption is untrue) use
    ``np.searchsorted()`` binary search to look up the appropriate
    index.
    '''
    profiler = Profiler(
        msg='slice_from_time()',
        disabled=not pg_profile_enabled(),
        ms_threshold=ms_slower_then,
    )
    times = arr['time']
    t_first = floor(times[0])
    t_last = ceil(times[-1])
    # the greatest index we can return which slices to the
    # end of the input array.
    read_i_max = arr.shape[0]
    # compute (presumed) uniform-time-step index offsets
    i_start_t = floor(start_t)
    read_i_start = floor(((i_start_t - t_first) // step)) - 1
    i_stop_t = ceil(stop_t)
    # XXX: edge case -> always set stop index to last in array whenever
    # the input stop time is detected to be greater then the equiv time
    # stamp at that last entry.
    if i_stop_t >= t_last:
        read_i_stop = read_i_max
    else:
        read_i_stop = ceil((i_stop_t - t_first) // step) + 1
    # always clip outputs to array support
    # for read start:
    # - never allow a start < the 0 index
    # - never allow an end index > the read array len
    read_i_start = min(
        max(0, read_i_start),
        read_i_max - 1,
    )
    read_i_stop = max(
        0,
        min(read_i_stop, read_i_max),
    )
    # check for larger-then-latest calculated index for given start
    # time, in which case we do a binary search for the correct index.
    # NOTE: this is usually the result of a time series with time gaps
    # where it is expected that each index step maps to a uniform step
    # in the time stamp series.
    t_iv_start = times[read_i_start]
    if (
        t_iv_start > i_start_t
    ):
        # do a binary search for the best index mapping to ``start_t``
        # given we measured an overshoot using the uniform-time-step
        # calculation from above.
        # TODO: once we start caching these per source-array,
        # we can just overwrite ``read_i_start`` directly.
        new_read_i_start = np.searchsorted(
            times,
            i_start_t,
            side='left',
        )
        # TODO: minimize binary search work as much as possible:
        # - cache these remap values which compensate for gaps in the
        #   uniform time step basis where we calc a later start
        #   index for the given input ``start_t``.
        # - can we shorten the input search sequence by heuristic?
        #   up_to_arith_start = index[:read_i_start]
        if (
            new_read_i_start <= read_i_start
        ):
            # t_diff = t_iv_start - start_t
            # print(
            #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
            #     f'start_t:{start_t} -> 0index start_t:{t_iv_start}\n'
            #     f'diff: {t_diff}\n'
            #     f'REMAPPED START i: {read_i_start} -> {new_read_i_start}\n'
            # )
            read_i_start = new_read_i_start
    t_iv_stop = times[read_i_stop - 1]
    if (
        t_iv_stop > i_stop_t
    ):
        # t_diff = stop_t - t_iv_stop
        # print(
        #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
        #     f'calced iv stop:{t_iv_stop} -> stop_t:{stop_t}\n'
        #     f'diff: {t_diff}\n'
        #     # f'SHOULD REMAP STOP: {read_i_start} -> {new_read_i_start}\n'
        # )
        new_read_i_stop = np.searchsorted(
            times[read_i_start:],
            # times,
            i_stop_t,
            side='right',
        )
        if (
            new_read_i_stop <= read_i_stop
        ):
            read_i_stop = read_i_start + new_read_i_stop + 1
    # sanity checks for range size
    # samples = (i_stop_t - i_start_t) // step
    # index_diff = read_i_stop - read_i_start + 1
    # if index_diff > (samples + 3):
    #     breakpoint()
    # read-relative indexes: gives a slice where `shm.array[read_slc]`
    # will be the data spanning the input time range `start_t` ->
    # `stop_t`
    read_slc = slice(
        int(read_i_start),
        int(read_i_stop),
    )
    profiler(
        'slicing complete'
        # f'{start_t} -> {abs_slc.start} | {read_slc.start}\n'
        # f'{stop_t} -> {abs_slc.stop} | {read_slc.stop}\n'
    )
    # NOTE: if caller needs absolute buffer indices they can
    # slice the buffer abs index like so:
    # index = arr['index']
    # abs_indx = index[read_slc]
    # abs_slc = slice(
    #     int(abs_indx[0]),
    #     int(abs_indx[-1]),
    # )
    return read_slc
 def get_null_segs(
    frame: Frame,
    period: float,  # sampling step in seconds
    imargin: int = 1,
    col: str = 'time',
 ) -> tuple[
    # Seq,  # TODO: can we make it an array-type instead?
    list[
        list[int, int],
    ],
    Seq,
    Frame
 ] | None:
    '''
    Detect if there are any zero(-epoch stamped) valued
    rows in for the provided `col: str` column; by default
    presume the 'time' field/column.
    Filter to all such zero (time) segments and return
    the corresponding frame zeroed segment's,
      - gap absolute (in buffer terms) indices-endpoints as
        `absi_zsegs`
      - abs indices of all rows with zeroed `col` values as `absi_zeros`
      - the corresponding frame's row-entries (view) which are
        zeroed for the `col` as `zero_t`
    '''
    times: Seq = frame['time']
    zero_pred: Seq = (times == 0)
    if isinstance(frame, np.ndarray):
        tis_zeros: int = zero_pred.any()
    else:
        tis_zeros: int = zero_pred.any()
    if not tis_zeros:
        return None
    # TODO: use ndarray for this?!
    absi_zsegs: list[list[int, int]] = []
    if isinstance(frame, np.ndarray):
        # view of ONLY the zero segments as one continuous chunk
        zero_t: np.ndarray = frame[zero_pred]
        # abs indices of said zeroed rows
        absi_zeros = zero_t['index']
        # diff of abs index steps between each zeroed row
        absi_zdiff: np.ndarray = np.diff(absi_zeros)
        # scan for all frame-indices where the
        # zeroed-row-abs-index-step-diff is greater then the
        # expected increment of 1.
        # data  1st zero seg  data  zeros
        # ----  ------------  ----  -----  ------  ----
        # ||||..000000000000..||||..00000..||||||..0000
        # ----  ------------  ----  -----  ------  ----
        #       ^zero_t[0]                            ^zero_t[-1]
        #                           ^fi_zgaps[0]   ^fi_zgaps[1]
        #       ^absi_zsegs[0][0]   ^---^ => absi_zsegs[1]: tuple
        #  absi_zsegs[0][1]^
        #
        # NOTE: the first entry in `fi_zgaps` is where
        # the first (absolute) index step diff is > 1.
        # and it is a frame-relative index into `zero_t`.
        fi_zgaps = np.argwhere(
            absi_zdiff > 1
            # NOTE: +1 here is ensure we index to the "start" of each
            # segment (if we didn't the below loop needs to be
            # re-written to expect `fi_end_rows`!
        ) + 1
        # the rows from the contiguous zeroed segments which have
        # abs-index steps >1 compared to the previous zero row
        # (indicating an end of zeroed segment).
        fi_zseg_start_rows = zero_t[fi_zgaps]
    # TODO: equiv for pl.DataFrame case!
    else:
        izeros: pl.Series = zero_pred.arg_true()
        zero_t: pl.DataFrame = frame[izeros]
        absi_zeros = zero_t['index']
        absi_zdiff: pl.Series = absi_zeros.diff()
        fi_zgaps = (absi_zdiff > 1).arg_true()
    # XXX: our goal (in this func) is to select out slice index
    # pairs (zseg0_start, zseg_end) in abs index units for each
    # null-segment portion detected throughout entire input frame.
    # only up to one null-segment in entire frame?
    num_gaps: int = fi_zgaps.size + 1
    if num_gaps < 1:
        if absi_zeros.size > 1:
            absi_zsegs = [[
                # TODO: maybe mk these max()/min() limits func
                # consts instead of called more then once?
                max(
                    absi_zeros[0] - 1,
                    0,
                ),
                # NOTE: need the + 1 to guarantee we index "up to"
                # the next non-null row-datum.
                min(
                    absi_zeros[-1] + 1,
                    frame['index'][-1],
                ),
            ]]
        else:
            # XXX EDGE CASE: only one null-datum found so
            # mark the start abs index as None to trigger
            # a full frame-len query to the respective backend?
            absi_zsegs = [[
                # see `get_hist()` in backend, should ALWAYS be
                # able to handle a `start_dt=None`!
                # None,
                None,
                absi_zeros[0] + 1,
            ]]
    # XXX NOTE XXX: if >= 2 zeroed segments are found, there should
    # ALWAYS be more then one zero-segment-abs-index-step-diff row
    # in `absi_zdiff`, so loop through all such
    # abs-index-step-diffs >1 (i.e. the entries of `absi_zdiff`)
    # and add them as the "end index" entries for each segment.
    # Then, iif NOT iterating the first such segment end, look back
    # for the prior segments zero-segment start indext by relative
    # indexing the `zero_t` frame by -1 and grabbing the abs index
    # of what should be the prior zero-segment abs start index.
    else:
        # NOTE: since `absi_zdiff` will never have a row
        # corresponding to the first zero-segment's row, we add it
        # manually here.
        absi_zsegs.append([
            max(
                absi_zeros[0] - 1,
                0,
            ),
            None,
        ])
        # TODO: can we do it with vec ops?
        for i, (
            fi,  # frame index of zero-seg start
            zseg_start_row,  # full row for ^
        ) in enumerate(zip(
            fi_zgaps,
            fi_zseg_start_rows,
        )):
            assert (zseg_start_row == zero_t[fi]).all()
            iabs: int = zseg_start_row['index'][0]
            absi_zsegs.append([
                iabs - 1,
                None,  # backfilled on next iter
            ])
            # final iter case, backfill FINAL end iabs!
            if (i + 1) == fi_zgaps.size:
                absi_zsegs[-1][1] = absi_zeros[-1] + 1
            # NOTE: only after the first segment (due to `.diff()`
            # usage above) can we do a lookback to the prior
            # segment's end row and determine it's abs index to
            # retroactively insert to the prior
            # `absi_zsegs[i-1][1]` entry Bo
            last_end: int = absi_zsegs[i][1]
            if last_end is None:
                prev_zseg_row = zero_t[fi - 1]
                absi_post_zseg = prev_zseg_row['index'][0] + 1
                # XXX: MUST BACKFILL previous end iabs!
                absi_zsegs[i][1] = absi_post_zseg
        else:
            if 0 < num_gaps < 2:
                absi_zsegs[-1][1] = min(
                    absi_zeros[-1] + 1,
                    frame['index'][-1],
                )
            iabs_first: int = frame['index'][0]
            for start, end in absi_zsegs:
                ts_start: float = times[start - iabs_first]
                ts_end: float = times[end - iabs_first]
                if (
                    (ts_start == 0 and not start == 0)
                    or
                    ts_end == 0
                ):
                    import pdbp
                    pdbp.set_trace()
                assert end
                assert start < end
    log.warning(
        f'Frame has {len(absi_zsegs)} NULL GAPS!?\n'
        f'period: {period}\n'
        f'total null samples: {len(zero_t)}\n'
    )
    return (
        absi_zsegs,  # [start, end] abs slice indices of seg
        absi_zeros,  # all abs indices within all null-segs
        zero_t,  # sliced-view of all null-segment rows-datums
    )
 def iter_null_segs(
    timeframe: float,
    frame: Frame | None = None,
    null_segs: tuple | None = None,
 ) -> Generator[
    tuple[
        int, int,
        int, int,
        float, float,
        float, float,
        # Seq,  # TODO: can we make it an array-type instead?
        # list[
        #     list[int, int],
        # ],
        # Seq,
        # Frame
    ],
    None,
 ]:
    if not (
        null_segs := get_null_segs(
            frame,
            period=timeframe,
        )
    ):
        return
    absi_pairs_zsegs: list[list[float, float]]
    izeros: Seq
    zero_t: Frame
    (
        absi_pairs_zsegs,
        izeros,
        zero_t,
    ) = null_segs
    absi_first: int = frame[0]['index']
    for (
        absi_start,
        absi_end,
    ) in absi_pairs_zsegs:
        fi_end: int = absi_end - absi_first
        end_row: Seq = frame[fi_end]
        end_t: float = end_row['time']
        end_dt: DateTime = from_timestamp(end_t)
        fi_start = None
        start_row = None
        start_t = None
        start_dt = None
        if (
            absi_start is not None
            and start_t != 0
        ):
            fi_start: int = absi_start - absi_first
            start_row: Seq = frame[fi_start]
            start_t: float = start_row['time']
            start_dt: DateTime = from_timestamp(start_t)
        if absi_start < 0:
            import pdbp
            pdbp.set_trace()
        yield (
            absi_start, absi_end,  # abs indices
            fi_start, fi_end,  # relative "frame" indices
            start_t, end_t,
            start_dt, end_dt,
        )
 def with_dts(
    df: pl.DataFrame,
    time_col: str = 'time',
 ) -> pl.DataFrame:
    '''
    Insert datetime (casted) columns to a (presumably) OHLC sampled
    time series with an epoch-time column keyed by `time_col: str`.
    '''
    return df.with_columns([
        pl.col(time_col).shift(1).suffix('_prev'),
        pl.col(time_col).diff().alias('s_diff'),
        pl.from_epoch(pl.col(time_col)).alias('dt'),
    ]).with_columns([
        pl.from_epoch(
            column=pl.col(f'{time_col}_prev'),
        ).alias('dt_prev'),
        pl.col('dt').diff().alias('dt_diff'),
    ])
 t_unit: Literal = Literal[
    'days',
    'hours',
    'minutes',
    'seconds',
    'miliseconds',
    'microseconds',
    'nanoseconds',
 ]
 def detect_time_gaps(
    w_dts: pl.DataFrame,
    time_col: str = 'time',
    # epoch sampling step diff
    expect_period: float = 60,
    # NOTE: legacy stock mkts have venue operating hours
    # and thus gaps normally no more then 1-2 days at
    # a time.
    gap_thresh: float = 1.,
    # TODO: allow passing in a frame of operating hours?
    # -[ ] durations/ranges for faster legit gap checks?
    # XXX -> must be valid ``polars.Expr.dt.<name>``
    # like 'days' which a sane default for venue closures
    # though will detect weekend gaps which are normal :o
    gap_dt_unit: t_unit | None = None,
 ) -> pl.DataFrame:
    '''
    Filter to OHLC datums which contain sample step gaps.
    For eg. legacy markets which have venue close gaps and/or
    actual missing data segments.
    '''
    # first select by any sample-period (in seconds unit) step size
    # greater then expected.
    step_gaps: pl.DataFrame = w_dts.filter(
        pl.col('s_diff').abs() > expect_period
    )
    if gap_dt_unit is None:
        return step_gaps
    # NOTE: this flag is to indicate that on this (sampling) time
    # scale we expect to only be filtering against larger venue
    # closures-scale time gaps.
    return step_gaps.filter(
        # Second by an arbitrary dt-unit step size
        getattr(
            pl.col('dt_diff').dt,
            gap_dt_unit,
        )().abs() > gap_thresh
    )
 def detect_price_gaps(
    df: pl.DataFrame,
    gt_multiplier: float = 2.,
    price_fields: list[str] = ['high', 'low'],
 ) -> pl.DataFrame:
    '''
    Detect gaps in clearing price over an OHLC series.
    2 types of gaps generally exist; up gaps and down gaps:
    - UP gap: when any next sample's lo price is strictly greater
      then the current sample's hi price.
    - DOWN gap: when any next sample's hi price is strictly
      less then the current samples lo price.
    '''
    # return df.filter(
    #     pl.col('high') - ) > expect_period,
    # ).select([
    #     pl.dt.datetime(pl.col(time_col).shift(1)).suffix('_previous'),
    #     pl.all(),
    # ]).select([
    #     pl.all(),
    #     (pl.col(time_col) - pl.col(f'{time_col}_previous')).alias('diff'),
    # ])
    ...
 # TODO: probably just use the null_segs impl above?
 def detect_vlm_gaps(
    df: pl.DataFrame,
    col: str = 'volume',
 ) -> pl.DataFrame:
    vnull: pl.DataFrame = w_dts.filter(
        pl.col(col) == 0
    )
    return vnull
 def dedupe(
    src_df: pl.DataFrame,
    time_gaps: pl.DataFrame | None = None,
    sort: bool = True,
    period: float = 60,
 ) -> tuple[
    pl.DataFrame,  # with dts
    pl.DataFrame,  # with deduplicated dts (aka gap/repeat removal)
    int,  # len diff between input and deduped
 ]:
    '''
    Check for time series gaps and if found
    de-duplicate any datetime entries, check for
    a frame height diff and return the newly
    dt-deduplicated frame.
    '''
    wdts: pl.DataFrame = with_dts(src_df)
    deduped = wdts
    # remove duplicated datetime samples/sections
    deduped: pl.DataFrame = wdts.unique(
        # subset=['dt'],
        subset=['time'],
        maintain_order=True,
    )
    # maybe sort on any time field
    if sort:
        deduped = deduped.sort(by='time')
        # TODO: detect out-of-order segments which were corrected!
        # -[ ] report in log msg
        # -[ ] possibly return segment sections which were moved?
    diff: int = (
        wdts.height
        -
        deduped.height
    )
    return (
        wdts,
        deduped,
        diff,
    )
 def sort_diff(
    src_df: pl.DataFrame,
    col: str = 'time',
 ) -> tuple[
    pl.DataFrame,  # with dts
    pl.DataFrame,  # sorted
    list[int],  # indices of segments that are out-of-order
 ]:
    ser: pl.Series = src_df[col]
    sortd: pl.DataFrame = ser.sort()
    diff: pl.Series = ser.diff()
    sortd_diff: pl.Series = sortd.diff()
    i_step_diff = (diff != sortd_diff).arg_true()
    frame_reorders: int = i_step_diff.len()
    if frame_reorders:
        log.warn(
            f'Resorted frame on col: {col}\n'
            f'{frame_reorders}'
        )
        # import pdbp; pdbp.set_trace()
 # NOTE: thanks to this SO answer for the below conversion routines
 # to go from numpy struct-arrays to polars dataframes and back:
 # https://stackoverflow.com/a/72054819
 def np2pl(array: np.ndarray) -> pl.DataFrame:
    start: float = time.time()
    # XXX: thanks to this SO answer for this conversion tip:
    # https://stackoverflow.com/a/72054819
    df = pl.DataFrame({
        field_name: array[field_name]
        for field_name in array.dtype.fields
    })
    delay: float = round(
        time.time() - start,
        ndigits=6,
    )
    log.info(
        f'numpy -> polars conversion took {delay} secs\n'
        f'polars df: {df}'
    )
    return df
 def pl2np(
    df: pl.DataFrame,
    dtype: np.dtype,
 ) -> np.ndarray:
    # Create numpy struct array of the correct size and dtype
    # and loop through df columns to fill in array fields.
    array = np.empty(
        df.height,
        dtype,
    )
    for field, col in zip(
        dtype.fields,
        df.columns,
    ):
        array[field] = df.get_column(col).to_numpy()
    return array
--- a/piker/types.py
+++ b/piker/types.py
@ -1,250 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) (in stewardship for pikers)
 #  - Tyler Goodlet
 #  - Guillermo Rodriguez
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Extensions to built-in or (heavily used but 3rd party) friend-lib
 types.
 '''
 from __future__ import annotations
 from collections import UserList
 from pprint import (
    saferepr,
 )
 from typing import Any
 from msgspec import (
    msgpack,
    Struct as _Struct,
    structs,
 )
 class DiffDump(UserList):
    '''
    Very simple list delegator that repr() dumps (presumed) tuple
    elements of the form `tuple[str, Any, Any]` in a nice
    multi-line readable form for analyzing `Struct` diffs.
    '''
    def __repr__(self) -> str:
        if not len(self):
            return super().__repr__()
        # format by displaying item pair's ``repr()`` on multiple,
        # indented lines such that they are more easily visually
        # comparable when printed to console when printed to
        # console.
        repstr: str = '[\n'
        for k, left, right in self:
            repstr += (
                f'({k},\n'
                f'\t{repr(left)},\n'
                f'\t{repr(right)},\n'
                ')\n'
            )
        repstr += ']\n'
        return repstr
 class Struct(
    _Struct,
    # https://jcristharif.com/msgspec/structs.html#tagged-unions
    # tag='pikerstruct',
    # tag=True,
 ):
    '''
    A "human friendlier" (aka repl buddy) struct subtype.
    '''
    def _sin_props(self) -> Iterator[
        tuple[
            structs.FieldIinfo,
            str,
            Any,
        ]
    ]:
        '''
        Iterate over all non-@property fields of this struct.
        '''
        fi: structs.FieldInfo
        for fi in structs.fields(self):
            key: str = fi.name
            val: Any = getattr(self, key)
            yield fi, key, val
    def to_dict(
        self,
        include_non_members: bool = True,
    ) -> dict:
        '''
        Like it sounds.. direct delegation to:
        https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
        BUT, by default we pop all non-member (aka not defined as
        struct fields) fields by default.
        '''
        asdict: dict = structs.asdict(self)
        if include_non_members:
            return asdict
        # only return a dict of the struct members
        # which were provided as input, NOT anything
        # added as type-defined `@property` methods!
        sin_props: dict = {}
        fi: structs.FieldInfo
        for fi, k, v in self._sin_props():
            sin_props[k] = asdict[k]
        return sin_props
    def pformat(
        self,
        field_indent: int = 2,
        indent: int = 0,
    ) -> str:
        '''
        Recursion-safe `pprint.pformat()` style formatting of
        a `msgspec.Struct` for sane reading by a human using a REPL.
        '''
        # global whitespace indent
        ws: str = ' '*indent
        # field whitespace indent
        field_ws: str = ' '*(field_indent + indent)
        # qtn: str = ws + self.__class__.__qualname__
        qtn: str = self.__class__.__qualname__
        obj_str: str = ''  # accumulator
        fi: structs.FieldInfo
        k: str
        v: Any
        for fi, k, v in self._sin_props():
            # TODO: how can we prefer `Literal['option1',  'option2,
            # ..]` over .__name__ == `Literal` but still get only the
            # latter for simple types like `str | int | None` etc..?
            ft: type = fi.type
            typ_name: str = getattr(ft, '__name__', str(ft))
            # recurse to get sub-struct's `.pformat()` output Bo
            if isinstance(v, Struct):
                val_str: str =  v.pformat(
                    indent=field_indent + indent,
                    field_indent=indent + field_indent,
                )
            else:  # the `pprint` recursion-safe format:
                # https://docs.python.org/3.11/library/pprint.html#pprint.saferepr
                val_str: str = saferepr(v)
            obj_str += (field_ws + f'{k}: {typ_name} = {val_str},\n')
        return (
            f'{qtn}(\n'
            f'{obj_str}'
            f'{ws})'
        )
    # TODO: use a pprint.PrettyPrinter instance around ONLY rendering
    # inside a known tty?
    # def __repr__(self) -> str:
    #     ...
    # __str__ = __repr__ = pformat
    __repr__ = pformat
    def copy(
        self,
        update: dict | None = None,
    ) -> Struct:
        '''
        Validate-typecast all self defined fields, return a copy of
        us with all such fields.
        NOTE: This is kinda like the default behaviour in
        `pydantic.BaseModel` except a copy of the object is
        returned making it compat with `frozen=True`.
        '''
        if update:
            for k, v in update.items():
                setattr(self, k, v)
        # NOTE: roundtrip serialize to validate
        # - enode to msgpack binary format,
        # - decode that back to a struct.
        return msgpack.Decoder(type=type(self)).decode(
            msgpack.Encoder().encode(self)
        )
    def typecast(
        self,
        # TODO: allow only casting a named subset?
        # fields: set[str] | None = None,
    ) -> None:
        '''
        Cast all fields using their declared type annotations
        (kinda like what `pydantic` does by default).
        NOTE: this of course won't work on frozen types, use
        ``.copy()`` above in such cases.
        '''
        # https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
        fi: structs.FieldInfo
        for fi in structs.fields(self):
            setattr(
                self,
                fi.name,
                fi.type(getattr(self, fi.name)),
            )
    def __sub__(
        self,
        other: Struct,
    ) -> DiffDump[tuple[str, Any, Any]]:
        '''
        Compare fields/items key-wise and return a ``DiffDump``
        for easy visual REPL comparison B)
        '''
        diffs: DiffDump[tuple[str, Any, Any]] = DiffDump()
        for fi in structs.fields(self):
            attr_name: str = fi.name
            ours: Any = getattr(self, attr_name)
            theirs: Any = getattr(other, attr_name)
            if ours != theirs:
                diffs.append((
                    attr_name,
                    ours,
                    theirs,
                ))
        return diffs
--- a/piker/ui/init.py
+++ b/piker/ui/init.py
@ -14,8 +14,9 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-UI components built using `Qt` with major versions swapped in via
+Stuff for your eyes, aka super hawt Qt UI components.
 the import indirection in the `.qt` sub-mod.
-'''
+Currently we only support PyQt5 due to this issue in Pyside2:
 https://bugreports.qt.io/projects/PYSIDE/issues/PYSIDE-1313
 """
--- a/piker/ui/_anchors.py
+++ b/piker/ui/_anchors.py
@ -21,10 +21,8 @@ Anchor funtions for UI placement of annotions.
 from __future__ import annotations
 from typing import Callable, TYPE_CHECKING
-from piker.ui.qt import (
+from PyQt5.QtCore import QPointF
-    QPointF,
+from PyQt5.QtWidgets import QGraphicsPathItem
    QGraphicsPathItem,
 )
 if TYPE_CHECKING:
    from ._chart import ChartPlotWidget
--- a/piker/ui/_annotate.py
+++ b/piker/ui/_annotate.py
@ -20,22 +20,12 @@ Annotations for ur faces.
 """
 from typing import Callable
-from pyqtgraph import (
+from PyQt5 import QtCore, QtGui, QtWidgets
-    Point,
+from PyQt5.QtCore import QPointF, QRectF
-    functions as fn,
+from PyQt5.QtWidgets import QGraphicsPathItem
-    Color,
+from pyqtgraph import Point, functions as fn, Color
 )
 import numpy as np
 from piker.ui.qt import (
    QtCore,
    QtGui,
    QtWidgets,
    QPointF,
    QRectF,
    QGraphicsPathItem,
 )
 def mk_marker_path(
--- a/piker/ui/_app.py
+++ b/piker/ui/_app.py
@ -21,16 +21,13 @@ Main app startup and run.
 from functools import partial
 from types import ModuleType
 from PyQt5.QtCore import QEvent
 import trio
 from piker.ui.qt import (
    QEvent,
 )
 from ..service import maybe_spawn_brokerd
 from . import _event
 from ._exec import run_qtractor
 from ..data.feed import install_brokerd_search
 from ..data._symcache import open_symcache
 from ..accounting import unpack_fqme
 from . import _search
 from ._chart import GodWidget
@ -59,13 +56,7 @@ async def load_provider_search(
            portal,
            brokermod,
        ),
        open_symcache(brokermod) as symcache,
    ):
        if not symcache.mktmaps:
            log.warning(
                f'BACKEND DOES NOT (yet) support symcaching: `{brokermod.name}`'
            )
        # keep search engine stream up until cancelled
        await trio.sleep_forever()
@ -108,8 +99,6 @@ async def _async_main(
    sbar = godwidget.window.status_bar
    starting_done = sbar.open_status('starting ze sexy chartz')
    # NOTE: by default we load all "builtin" backends for search
    # and that includes loading their symcaches if possible B)
    needed_brokermods: dict[str, ModuleType] = {}
    for fqme in syms:
        brokername, *_ = unpack_fqme(fqme)
--- a/piker/ui/_axes.py
+++ b/piker/ui/_axes.py
@ -23,24 +23,16 @@ from functools import lru_cache
 from typing import Callable
 from math import floor
-import polars as pl
+import numpy as np
 import pyqtgraph as pg
 from PyQt5 import QtCore, QtGui, QtWidgets
 from PyQt5.QtCore import QPointF
 from piker.ui.qt import (
    QtCore,
    QtGui,
    QtWidgets,
    QPointF,
    txt_flag,
    align_flag,
    px_cache_mode,
 )
 from . import _pg_overrides as pgo
 from ..accounting._mktinfo import float_digits
 from ._label import Label
 from ._style import DpiAwareFont, hcolor, _font
 from ._interaction import ChartView
 from ._dataviz import Viz
 _axis_pen = pg.mkPen(hcolor('bracket'))
@ -295,7 +287,9 @@ class DynamicDateAxis(Axis):
    # time formats mapped by seconds between bars
    tick_tpl = {
        60 * 60 * 24: '%Y-%b-%d',
-        60: '%Y-%b-%d(%H:%M)',
+        60: '%H:%M',
        30: '%H:%M:%S',
        5: '%H:%M:%S',
        1: '%H:%M:%S',
    }
@ -311,10 +305,10 @@ class DynamicDateAxis(Axis):
        # XX: ARGGGGG AG:LKSKDJF:LKJSDFD
        chart = self.pi.chart_widget
-        viz: Viz = chart._vizs[chart.name]
+        viz = chart._vizs[chart.name]
        shm = viz.shm
        array = shm.array
-        ifield: str = viz.index_field
+        ifield = viz.index_field
        index = array[ifield]
        i_0, i_l = index[0], index[-1]
@ -335,7 +329,7 @@ class DynamicDateAxis(Axis):
            arr_len = index.shape[0]
            first = shm._first.value
            times = array['time']
-            epochs: list[int] = times[
+            epochs = times[
                list(
                    map(
                        int,
@ -347,30 +341,23 @@ class DynamicDateAxis(Axis):
                )
            ]
        else:
-            epochs: list[int] = list(map(int, indexes))
+            epochs = list(map(int, indexes))
        # TODO: **don't** have this hard coded shift to EST
-        delay: float = viz.time_step()
+        # delay = times[-1] - times[-2]
-        if delay > 1:
+        dts = np.array(
            # NOTE: use less granular dt-str when using 1M+ OHLC
            fmtstr: str = self.tick_tpl[delay]
        else:
            fmtstr: str = '%Y-%m-%d(%H:%M:%S)'
        # https://pola-rs.github.io/polars/py-polars/html/reference/expressions/api/polars.from_epoch.html#polars-from-epoch
        pl_dts: pl.Series = pl.from_epoch(
            epochs,
-            time_unit='s',
+            dtype='datetime64[s]',
        # NOTE: kinda weird we can pass it to `.from_epoch()` no?
        ).dt.replace_time_zone(
            time_zone='UTC'
        ).dt.convert_time_zone(
            # TODO: pull this from either:
            # -[ ] the mkt venue tz by default
            # -[ ] the user's config under `sys.mkt_timezone: str`
            'EST'
        )
-        return pl_dts.dt.to_string(fmtstr).to_list()
+
        # see units listing:
        # https://numpy.org/devdocs/reference/arrays.datetime.html#datetime-units
        return list(np.datetime_as_string(dts))
        # TODO: per timeframe formatting?
        # - we probably need this based on zoom now right?
        # prec = self.np_dt_precision[delay]
        # return dts.strftime(self.tick_tpl[delay])
    def tickStrings(
        self,
@ -421,15 +408,11 @@ class AxisLabel(pg.GraphicsObject):
        super().__init__()
        self.setParentItem(parent)
-        self.setFlag(
+        self.setFlag(self.ItemIgnoresTransformations)
            self.GraphicsItemFlag.ItemIgnoresTransformations
        )
        self.setZValue(100)
        # XXX: pretty sure this is faster
-        self.setCacheMode(
+        self.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
            px_cache_mode.DeviceCoordinateCache
        )
        self._parent = parent
@ -566,14 +549,21 @@ class AxisLabel(pg.GraphicsObject):
        return (self.rect.width(), self.rect.height())
 # _common_text_flags = (
 #     QtCore.Qt.TextDontClip |
 #     QtCore.Qt.AlignCenter |
 #     QtCore.Qt.AlignTop |
 #     QtCore.Qt.AlignHCenter |
 #     QtCore.Qt.AlignVCenter
 # )
 class XAxisLabel(AxisLabel):
    _x_margin = 8
    text_flags = (
-        align_flag.AlignCenter
+        QtCore.Qt.TextDontClip
-        | txt_flag.TextDontClip
+        | QtCore.Qt.AlignCenter
    )
    def size_hint(self) -> tuple[float, float]:
@ -630,10 +620,10 @@ class YAxisLabel(AxisLabel):
    _y_margin: int = 4
    text_flags = (
-        align_flag.AlignLeft
+        QtCore.Qt.AlignLeft
-        | align_flag.AlignVCenter
+        # QtCore.Qt.AlignHCenter
-        # | align_flag.AlignHCenter
+        | QtCore.Qt.AlignVCenter
-        | txt_flag.TextDontClip
+        | QtCore.Qt.TextDontClip
    )
    def __init__(
--- a/piker/ui/_chart.py
+++ b/piker/ui/_chart.py
@ -28,20 +28,22 @@ from typing import (
    TYPE_CHECKING,
 )
-import pyqtgraph as pg
+from PyQt5 import QtCore, QtWidgets
-import trio
+from PyQt5.QtCore import (
 from piker.ui.qt import (
    QtCore,
    QtWidgets,
    Qt,
    QLineF,
    # QPointF,
 )
 from PyQt5.QtWidgets import (
    QFrame,
    QWidget,
    QHBoxLayout,
    QVBoxLayout,
    QSplitter,
 )
 import pyqtgraph as pg
 import trio
 from ._axes import (
    DynamicDateAxis,
    PriceAxis,
@ -280,7 +282,7 @@ class GodWidget(QWidget):
            # TODO: probably stick this in some kinda `LooknFeel` API?
            for tracker in self.rt_linked.mode.trackers.values():
                pp_nav = tracker.nav
-                if tracker.live_pp.cumsize:
+                if tracker.live_pp.size:
                    pp_nav.show()
                    pp_nav.hide_info()
                else:
@ -404,7 +406,6 @@ class ChartnPane(QFrame):
        )
        self._sidepane = sidepane
    @property
    def sidepane(self) -> FieldsForm | SearchWidget:
        return self._sidepane
@ -494,7 +495,7 @@ class LinkedSplits(QWidget):
        Set the proportion of space allocated for linked subcharts.
        '''
-        ln: int = len(self.subplots) or 1
+        ln = len(self.subplots) or 1
        # proportion allocated to consumer subcharts
        if not prop:
@ -568,8 +569,8 @@ class LinkedSplits(QWidget):
        # style?
        self.chart.setFrameStyle(
-            QFrame.Shape.StyledPanel |
+            QFrame.StyledPanel |
-            QFrame.Shadow.Plain
+            QFrame.Plain
        )
        return self.chart
@ -687,8 +688,8 @@ class LinkedSplits(QWidget):
        cpw.plotItem.vb.linked = self
        cpw.setFrameStyle(
-            QFrame.Shape.StyledPanel
+            QtWidgets.QFrame.StyledPanel
-            # | QFrame.Shadow.Plain
+            # | QtWidgets.QFrame.Plain
        )
        # don't show the little "autoscale" A label.
@ -924,7 +925,6 @@ class ChartPlotWidget(pg.PlotWidget):
        self.useOpenGL(use_open_gl)
        self.name = name
        self.data_key = data_key or name
        self.qframe: ChartnPane | None = None
        # scene-local placeholder for book graphics
        # sizing to avoid overlap with data contents
--- a/piker/ui/_cursor.py
+++ b/piker/ui/_cursor.py
@ -28,14 +28,9 @@ from typing import (
 import inspect
 import numpy as np
 import pyqtgraph as pg
 from PyQt5 import QtCore, QtWidgets
 from PyQt5.QtCore import QPointF, QRectF
 from piker.ui.qt import (
    QPointF,
    QRectF,
    QtCore,
    QtWidgets,
    px_cache_mode,
 )
 from ._style import (
    _xaxis_at,
    hcolor,
@ -109,9 +104,7 @@ class LineDot(pg.CurvePoint):
        dot.setParentItem(self)
        # keep a static size
-        self.setFlag(
+        self.setFlag(self.ItemIgnoresTransformations)
            self.GraphicsItemFlag.ItemIgnoresTransformations
        )
    def event(
        self,
@ -214,10 +207,9 @@ class ContentsLabel(pg.LabelItem):
        # this being "html" is the dumbest shit :eyeroll:
        self.setText(
-            "<b>i_arr</b>:{index}<br/>"
+            "<b>i</b>:{index}<br/>"
            # NB: these fields must be indexed in the correct order via
            # the slice syntax below.
            "<b>i_shm</b>:{}<br/>"
            "<b>epoch</b>:{}<br/>"
            "<b>O</b>:{}<br/>"
            "<b>H</b>:{}<br/>"
@ -227,7 +219,6 @@ class ContentsLabel(pg.LabelItem):
            # "<b>wap</b>:{}".format(
                *array[ix][
                    [
                        'index',
                        'time',
                        'open',
                        'high',
@ -279,15 +270,10 @@ class ContentsLabels:
        x_in: int,
    ) -> None:
-        for (
+        for chart, name, label, update in self._labels:
            chart,
            name,
            label,
            update,
        )in self._labels:
            viz = chart.get_viz(name)
-            array: np.ndarray = viz.shm._array
+            array = viz.shm.array
            index = array[viz.index_field]
            start = index[0]
            stop = index[-1]
@ -298,7 +284,7 @@ class ContentsLabels:
            ):
                # out of range
                print('WTF out of range?')
-                # continue
+                continue
            # call provided update func with data point
            try:
@ -306,7 +292,6 @@ class ContentsLabels:
                ix = np.searchsorted(index, x_in)
                if ix > len(array):
                    breakpoint()
                update(ix, array)
            except IndexError:
@ -431,10 +416,10 @@ class Cursor(pg.GraphicsObject):
        # vertical and horizonal lines and a y-axis label
        vl = plot.addLine(x=0, pen=self.lines_pen, movable=False)
-        vl.setCacheMode(px_cache_mode.DeviceCoordinateCache)
+        vl.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
        hl = plot.addLine(y=0, pen=self.lines_pen, movable=False)
-        hl.setCacheMode(px_cache_mode.DeviceCoordinateCache)
+        hl.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
        hl.hide()
        yl = YAxisLabel(
@ -518,10 +503,7 @@ class Cursor(pg.GraphicsObject):
            plot=chart
        )
        chart.addItem(cursor)
-        self.graphics[chart].setdefault(
+        self.graphics[chart].setdefault('cursors', []).append(cursor)
            'cursors',
            [],
        ).append(cursor)
        return cursor
    def mouseAction(
--- a/piker/ui/_curve.py
+++ b/piker/ui/_curve.py
@ -19,47 +19,42 @@ Fast, smooth, sexy curves.
 """
 from contextlib import contextmanager as cm
 from enum import EnumType
 from typing import Callable
 import numpy as np
 import pyqtgraph as pg
-
+from PyQt5 import QtWidgets
-from piker.ui.qt import (
+from PyQt5.QtWidgets import QGraphicsItem
-    QtWidgets,
+from PyQt5.QtCore import (
    QGraphicsItem,
    Qt,
    QLineF,
    QRectF,
 )
 from PyQt5.QtGui import (
    QPainter,
    QPainterPath,
    px_cache_mode,
 )
 from .._profile import pg_profile_enabled, ms_slower_then
 from ._style import hcolor
 from ..log import get_logger
-from ..toolz.profile import (
+from .._profile import Profiler
-    Profiler,
+
    pg_profile_enabled,
    ms_slower_then,
 )
 log = get_logger(__name__)
 pen_style: EnumType = Qt.PenStyle
 _line_styles: dict[str, int] = {
-    'solid': pen_style.SolidLine,
+    'solid': Qt.PenStyle.SolidLine,
-    'dash': pen_style.DashLine,
+    'dash': Qt.PenStyle.DashLine,
-    'dot': pen_style.DotLine,
+    'dot': Qt.PenStyle.DotLine,
-    'dashdot': pen_style.DashDotLine,
+    'dashdot': Qt.PenStyle.DashDotLine,
 }
 class FlowGraphic(pg.GraphicsObject):
    '''
-    Base class with minimal interface for `QPainterPath`
+    Base class with minimal interface for `QPainterPath` implemented,
-    implemented, real-time updated "data flow" graphics.
+    real-time updated "data flow" graphics.
    See subtypes below.
@ -71,12 +66,12 @@ class FlowGraphic(pg.GraphicsObject):
    # XXX-NOTE-XXX: graphics caching B)
    # see explanation for different caching modes:
    # https://stackoverflow.com/a/39410081
-    cache_mode: int = px_cache_mode.DeviceCoordinateCache
+    cache_mode: int = QGraphicsItem.DeviceCoordinateCache
    # XXX: WARNING item caching seems to only be useful
    # if we don't re-generate the entire QPainterPath every time
    # don't ever use this - it's a colossal nightmare of artefacts
    # and is disastrous for performance.
-    # cache_mode.ItemCoordinateCache
+    # QGraphicsItem.ItemCoordinateCache
    # TODO: still questions todo with coord-cacheing that we should
    # probably talk to a core dev about:
    # - if this makes trasform interactions slower (such as zooming)
@ -169,16 +164,15 @@ class FlowGraphic(pg.GraphicsObject):
        return None
    # XXX: due to a variety of weird jitter bugs and "smearing"
-    # artifacts when click-drag panning and viewing history time
+    # artifacts when click-drag panning and viewing history time series,
-    # series, we offer this ctx-mngr interface to allow temporarily
+    # we offer this ctx-mngr interface to allow temporarily disabling
-    # disabling Qt's graphics caching mode; this is now currently
+    # Qt's graphics caching mode; this is now currently used from
-    # used from ``ChartView.start/signal_ic()`` methods which also
+    # ``ChartView.start/signal_ic()`` methods which also disable the
-    # disable the rt-display loop when the user is moving around
+    # rt-display loop when the user is moving around a view.
    # a view.
    @cm
    def reset_cache(self) -> None:
        try:
-            none = px_cache_mode.NoCache
+            none = QGraphicsItem.NoCache
            log.debug(
                f'{self._name} -> CACHE DISABLE: {none}'
            )
--- a/piker/ui/_dataviz.py
+++ b/piker/ui/_dataviz.py
@ -36,12 +36,9 @@ from msgspec import (
    field,
 )
 import numpy as np
 from numpy import (
    ndarray,
 )
 import pyqtgraph as pg
 from PyQt5.QtCore import QLineF
 from piker.ui.qt import QLineF
 from ..data._sharedmem import (
    ShmArray,
 )
@ -52,7 +49,7 @@ from ..data._formatters import (
    OHLCBarsAsCurveFmtr,  # OHLC converted to line
    StepCurveFmtr,  # "step" curve (like for vlm)
 )
-from ..tsp import (
+from ..data._timeseries import (
    slice_from_time,
 )
 from ._ohlc import (
@ -65,7 +62,7 @@ from ._curve import (
 )
 from ._render import Renderer
 from ..log import get_logger
-from ..toolz.profile import (
+from .._profile import (
    Profiler,
    pg_profile_enabled,
    ms_slower_then,
@ -85,11 +82,10 @@ def render_baritems(
    viz: Viz,
    graphics: BarItems,
    read: tuple[
-        int, int, ndarray,
+        int, int, np.ndarray,
-        int, int, ndarray,
+        int, int, np.ndarray,
    ],
    profiler: Profiler,
    force_redraw: bool = False,
    **kwargs,
 ) -> None:
@ -220,11 +216,9 @@ def render_baritems(
    viz._in_ds = should_line
    should_redraw = (
-        force_redraw
+        changed_to_line
        or changed_to_line
        or not should_line
    )
    # print(f'should_redraw: {should_redraw}')
    return (
        graphics,
        r,
@ -256,7 +250,7 @@ class ViewState(Struct):
    ] | None = None
    # last in view ``ShmArray.array[read_slc]`` data
-    in_view: ndarray | None = None
+    in_view: np.ndarray | None = None
 class Viz(Struct):
@ -319,7 +313,6 @@ class Viz(Struct):
    _last_uppx: float = 0
    _in_ds: bool = False
    _index_step: float | None = None
    _time_step: float | None = None
    # map from uppx -> (downsampled data, incremental graphics)
    _src_r: Renderer | None = None
@ -366,8 +359,7 @@ class Viz(Struct):
    def index_step(
        self,
-        index_field: str | None = None,
+        reset: bool = False,
    ) -> float:
        '''
        Return the size between sample steps in the units of the
@ -375,17 +367,12 @@ class Viz(Struct):
        epoch time in seconds.
        '''
-        # attempt to detect the best step size by scanning a sample
+        # attempt to dectect the best step size by scanning a sample of
-        # of the source data.
+        # the source data.
-        if (
+        if self._index_step is None:
-            self._index_step is None
+
-            or index_field is not None
+            index: np.ndarray = self.shm.array[self.index_field]
-        ):
+            isample: np.ndarray = index[-16:]
            index: ndarray = self.shm.array[
                index_field
                or self.index_field
            ]
            isample: ndarray = index[-16:]
            mxdiff: None | float = None
            for step in np.diff(isample):
@ -399,15 +386,7 @@ class Viz(Struct):
                        )
                    mxdiff = step
-            step: float = max(mxdiff, 1)
+            self._index_step = max(mxdiff, 1)
            # only SET the internal index step if an explicit
            # field name is NOT passed, since in such cases this
            # is likely just being called from `.time_step()`.
            if index_field is not None:
                return step
            self._index_step = step
            if (
                mxdiff < 1
                or 1 < mxdiff < 60
@ -418,17 +397,6 @@ class Viz(Struct):
        return self._index_step
    def time_step(self) -> float:
        '''
        Attempt to determine the per-sample time-step period by 
        forcing an epoch-index and calling `.index_step()`.
        '''
        if self._time_step is None:
            self._time_step: float = self.index_step(index_field='time')
        return self._time_step
    def maxmin(
        self,
@ -436,9 +404,6 @@ class Viz(Struct):
        i_read_range: tuple[int, int] | None = None,
        use_caching: bool = True,
        # XXX: internal debug
        _do_print: bool = False
    ) -> tuple[float, float] | None:
        '''
        Compute the cached max and min y-range values for a given
@ -458,14 +423,15 @@ class Viz(Struct):
        if shm is None:
            return None
-        arr: ndarray = shm.array
+        do_print: bool = False
        arr = shm.array
        if i_read_range is not None:
            read_slc = slice(*i_read_range)
-            index: float | int = arr[read_slc][self.index_field]
+            index = arr[read_slc][self.index_field]
            if not index.size:
                return None
-            ixrng: tuple[int, int] = (index[0], index[-1])
+            ixrng = (index[0], index[-1])
        else:
            if x_range is None:
@ -483,24 +449,15 @@ class Viz(Struct):
            # TODO: hash the slice instead maybe?
            # https://stackoverflow.com/a/29980872
-            ixrng = lbar, rbar = (
+            ixrng = lbar, rbar = round(x_range[0]), round(x_range[1])
                round(x_range[0]),
                round(x_range[1]),
            )
        if (
            use_caching
            and self._mxmn_cache_enabled
        ):
            # TODO: is there a way to ONLY clear ranges containing
            # a certain sub-range?
            # -[ ] currently we have a problem where a previously
            #    cached mxmn will persist even if the viz is "hard
            #    re-rendered" (usually bc underlying data was
            #    corrected)
            cached_result = self._mxmns.get(ixrng)
            if cached_result:
-                if _do_print:
+                if do_print:
                    print(
                        f'{self.name} CACHED maxmin\n'
                        f'{ixrng} -> {cached_result}'
@ -530,7 +487,7 @@ class Viz(Struct):
                    (rbar - ifirst) + 1
                )
-        slice_view: ndarray = arr[read_slc]
+        slice_view = arr[read_slc]
        if not slice_view.size:
            log.warning(
@ -541,7 +498,7 @@ class Viz(Struct):
        elif self.ds_yrange:
            mxmn = self.ds_yrange
-            if _do_print:
+            if do_print:
                print(
                    f'{self.name} M4 maxmin:\n'
                    f'{ixrng} -> {mxmn}'
@ -558,7 +515,7 @@ class Viz(Struct):
            mxmn = ylow, yhigh
            if (
-                _do_print
+                do_print
            ):
                s = 3
                print(
@ -572,23 +529,14 @@ class Viz(Struct):
        # cache result for input range
        ylow, yhi = mxmn
        diff: float = yhi - ylow
        # order-of-magnitude check
        # TODO: really we should be checking the hi or low
        # against the previous sample to catch stuff like,
        # - rando stock (reverse-)split
        # - null-segments written by some prior
        #   crash-during-backfil
        if diff > 0:
            omg: float = abs(logf(diff, 10))
        else:
            omg: float = 0
        try:
            prolly_anomaly: bool = (
-                # diff == 0
+                (
-                (ylow and omg > 10)
+                    abs(logf(ylow, 10)) > 16
                    if ylow
                    else False
                )
                or (
                    isnan(ylow) or isnan(yhi)
                )
@ -615,8 +563,7 @@ class Viz(Struct):
    def view_range(self) -> tuple[int, int]:
        '''
-        Return the start and stop x-indexes for the managed
+        Return the start and stop x-indexes for the managed ``ViewBox``.
        ``ViewBox``.
        '''
        vr = self.plot.viewRect()
@ -629,7 +576,7 @@ class Viz(Struct):
        self,
        view_range: None | tuple[float, float] = None,
        index_field: str | None = None,
-        array: ndarray | None = None,
+        array: np.ndarray | None = None,
    ) -> tuple[
        int, int, int, int, int, int
@ -700,8 +647,8 @@ class Viz(Struct):
        profiler: None | Profiler = None,
    ) -> tuple[
-        int, int, ndarray,
+        int, int, np.ndarray,
-        int, int, ndarray,
+        int, int, np.ndarray,
    ]:
        '''
        Read the underlying shm array buffer and
@ -871,10 +818,6 @@ class Viz(Struct):
                graphics,
                read,
                profiler,
                # NOTE: only set when caller says to
                force_redraw=should_redraw,
                **kwargs,
            )
@ -1037,39 +980,6 @@ class Viz(Struct):
            graphics,
        )
    def reset_graphics(
        self,
        # TODO: allow only resetting within some x-domain range?
        # ixrng: tuple[int, int] | None = None,
    ) -> None:
        '''
        Hard reset all graphics (rendering) layers for this
        data viz including clearing the mxmn auto-y-range
        cache.
        Normally called when the underlying data set is modified
        (probably by some `.tsp` correcting/editing routine) and
        the (now cached) graphics need to be fully re-rendered from
        source.
        '''
        log.warning(
            f'Forcing hard Viz graphihcs RESET:\n'
            f'.name: {self.name}\n'
            f'.index_field: {self.index_field}\n'
            f'.index_step(): {self.index_step()}\n'
            f'.time_step(): {self.time_step()}\n'
        )
        # XXX: always clear the mxn y-range cache
        # to avoid old data (anomalies) from being
        # retained in auto-yrange output.
        self._mxmn_cache_enabled = False
        self._mxmns.clear()
        self.update_graphics(force_redraw=True)
        self._mxmn_cache_enabled = True
    def draw_last(
        self,
        array_key: str | None = None,
@ -1162,7 +1072,7 @@ class Viz(Struct):
        '''
        shm: ShmArray = self.shm
-        array: ndarray = shm.array
+        array: np.ndarray = shm.array
        view: ChartView = self.plot.vb
        (
            vl,
--- a/piker/ui/_display.py
+++ b/piker/ui/_display.py
@ -36,28 +36,25 @@ import pyqtgraph as pg
 from msgspec import field
 # from .. import brokers
-from piker.accounting import (
+from ..accounting import (
    MktPair,
 )
-from piker.types import Struct
+from ..data import (
 from piker.data import (
    open_feed,
    Feed,
    Flume,
    open_sample_stream,
    ShmArray,
 )
-from piker.data.ticktools import (
+from ..data.ticktools import (
    _tick_groups,
    _auction_ticks,
 )
-from piker.toolz import (
+from ..data.types import Struct
-    pg_profile_enabled,
+from ..data._sharedmem import (
-    ms_slower_then,
+    ShmArray,
-    Profiler,
+)
 from ..data._sampling import (
    open_sample_stream,
 )
 from piker.log import get_logger
 from piker import config
 # from ..data._source import tf_in_1s
 from ._axes import YAxisLabel
 from ._chart import (
@ -82,6 +79,12 @@ from .order_mode import (
    open_order_mode,
    OrderMode,
 )
 from .._profile import (
    pg_profile_enabled,
    ms_slower_then,
 )
 from ..log import get_logger
 from .._profile import Profiler
 if TYPE_CHECKING:
    from ._interaction import ChartView
@ -211,9 +214,9 @@ async def increment_history_view(
 ):
    hist_chart: ChartPlotWidget = ds.hist_chart
    hist_viz: Viz = ds.hist_viz
-    # viz: Viz = ds.viz
+    viz: Viz = ds.viz
    assert 'hist' in hist_viz.shm.token['shm_name']
-    # name: str = hist_viz.name
+    name: str = hist_viz.name
    # TODO: seems this is more reliable at keeping the slow
    # chart incremented in view more correctly?
@ -226,8 +229,7 @@ async def increment_history_view(
    # draw everything from scratch on first entry!
    for curve_name, hist_viz in hist_chart._vizs.items():
        log.info(f'Forcing hard redraw -> {curve_name}')
-        hist_viz.reset_graphics()
+        hist_viz.update_graphics(force_redraw=True)
        # hist_viz.update_graphics(force_redraw=True)
    async with open_sample_stream(1.) as min_istream:
        async for msg in min_istream:
@ -250,27 +252,17 @@ async def increment_history_view(
            # - samplerd could emit the actual update range via
            #   tuple and then we only enter the below block if that
            #   range is detected as in-view?
-            # match msg:
+            if (
-            #     case {
+                (bf_wut := msg.get('backfilling', False))
-            #         'backfilling': (viz_name, timeframe),
+            ):
-            #     } if (
+                viz_name, timeframe = bf_wut
-            #         viz_name == name
+                if viz_name == name:
-            #     ):
+                    log.info(f'Forcing hard redraw -> {name}@{timeframe}')
-            #         log.warning(
+                    match timeframe:
-            #             f'Forcing HARD REDRAW:\n'
+                        case 60:
-            #             f'name: {name}\n'
+                            hist_viz.update_graphics(force_redraw=True)
-            #             f'timeframe: {timeframe}\n'
+                        case 1:
-            #         )
+                            viz.update_graphics(force_redraw=True)
            #         # TODO: only allow this when the data is IN VIEW!
            #         # also, we probably can do this more efficiently
            #         # / smarter by only redrawing the portion of the
            #         # path necessary?
            #         {
            #             60: hist_viz,
            #             1: viz,
            #         }[timeframe].update_graphics(
            #             force_redraw=True
            #         )
            # check if slow chart needs an x-domain shift and/or
            # y-range resize.
@ -311,7 +303,6 @@ async def increment_history_view(
 async def graphics_update_loop(
    dss: dict[str, DisplayState],
    nurse: trio.Nursery,
    godwidget: GodWidget,
    feed: Feed,
@ -353,6 +344,8 @@ async def graphics_update_loop(
        'i_last_slow_t':  0,  # multiview-global slow (1m) step index
    }
    dss: dict[str, DisplayState] = {}
    for fqme, flume in feed.flumes.items():
        ohlcv = flume.rt_shm
        hist_ohlcv = flume.hist_shm
@ -469,20 +462,12 @@ async def graphics_update_loop(
        await trio.sleep(0)
        if ds.hist_vars['i_last'] < ds.hist_vars['i_last_append']:
-            await tractor.pause()
+            await tractor.breakpoint()
    # try:
    # XXX TODO: we need to do _dss UPDATE here so that when
    # a feed-view is switched you can still remote annotate the
    # prior view..
    from . import _remote_ctl
    _remote_ctl._dss.update(dss)
    # main real-time quotes update loop
    stream: tractor.MsgStream
    async with feed.open_multi_stream() as stream:
-        # assert stream
+        assert stream
        async for quotes in stream:
            quote_period = time.time() - last_quote_s
            quote_rate = round(
@ -498,7 +483,7 @@ async def graphics_update_loop(
                pass
                # log.warning(f'High quote rate {mkt.fqme}: {quote_rate}')
-            last_quote_s: float = time.time()
+            last_quote_s = time.time()
            for fqme, quote in quotes.items():
                ds = dss[fqme]
@ -528,12 +513,6 @@ async def graphics_update_loop(
                    quote,
                )
    # finally:
    #     # XXX: cancel any remote annotation control ctxs
    #     _remote_ctl._dss = None
    #     for cid, (ctx, aids) in _remote_ctl._ctxs.items():
    #         await ctx.cancel()
 def graphics_update_cycle(
    ds: DisplayState,
@ -1232,8 +1211,6 @@ async def link_views_with_region(
    # region.sigRegionChangeFinished.connect(update_pi_from_region)
 # NOTE: default is set to 60 FPS until the runtime delivers the
 # discoverd hw value below.
 _quote_throttle_rate: int = 60 - 6
@ -1252,7 +1229,7 @@ async def display_symbol_data(
    fast from a cached watch-list.
    '''
-    # sbar = godwidget.window.status_bar
+    sbar = godwidget.window.status_bar
    # historical data fetch
    # brokermod = brokers.get_brokermod(provider)
@ -1262,11 +1239,11 @@ async def display_symbol_data(
    #     group_key=loading_sym_key,
    # )
-    # for fqme in fqmes:
+    for fqme in fqmes:
-    #     loading_sym_key = sbar.open_status(
+        loading_sym_key = sbar.open_status(
-    #         f'loading {fqme} ->',
+            f'loading {fqme} ->',
-    #         group_key=True
+            group_key=True
-    #     )
+        )
    # (TODO: make this not so shit XD)
    # close group status once a symbol feed fully loads to view.
@ -1275,54 +1252,26 @@ async def display_symbol_data(
    # TODO: ctl over update loop's maximum frequency.
    # - load this from a config.toml!
    # - allow dyanmic configuration from chart UI?
    (
        conf,
        path,
    ) = config.load()
    ui_conf: dict = conf['ui']
    global _quote_throttle_rate
    from ._window import main_window
-
+    display_rate = main_window().current_screen().refreshRate()
-    display_rate: int = floor(
+    _quote_throttle_rate = floor(display_rate) - 6
        main_window().current_screen().refreshRate()
    ) - 6
    mx_redraw_rate: int = ui_conf.get(
        'max_redraw_rate',
        _quote_throttle_rate,
    )
    if mx_redraw_rate < display_rate:
        log.info(
            'Down-throttling redraw rate to config setting\n'
            f'display FPS: {display_rate}\n'
            'max_redraw_rate: {max_redraw_rate}\n'
        )
    else:
        _quote_throttle_rate = display_rate
    # TODO: we should be able to increase this if we use some
    # `mypyc` speedups elsewhere? 22ish seems to be the sweet
    # spot for single-feed chart.
    num_of_feeds = len(fqmes)
-    # if num_of_feeds > 1:
+    mx: int = 22
-
+    if num_of_feeds > 1:
-    # there will be more ctx switches with more than 1 feed so we
+        # there will be more ctx switches with more than 1 feed so we
-    # max throttle down a bit more.
+        # max throttle down a bit more.
-    mx_per_feed: int = (
+        mx = 16
        ui_conf.get(
        'per_feed_redraw_rate',
        mx_redraw_rate,
        )
        or 16
    )
    # limit to at least display's FPS
    # avoiding needless Qt-in-guest-mode context switches
    cycles_per_feed = min(
        round(_quote_throttle_rate/num_of_feeds),
-        mx_per_feed,
+        mx,
    )
    feed: Feed
@ -1467,7 +1416,7 @@ async def display_symbol_data(
                start_fsp_displays,
                rt_linked,
                flume,
-                # loading_sym_key,
+                loading_sym_key,
                loglevel,
            )
@ -1586,10 +1535,8 @@ async def display_symbol_data(
            )
            # start update loop task
            dss: dict[str, DisplayState] = {}
            ln.start_soon(
                graphics_update_loop,
                dss,
                ln,
                godwidget,
                feed,
@ -1603,31 +1550,15 @@ async def display_symbol_data(
            order_ctl_fqme: str = fqmes[0]
            mode: OrderMode
            async with (
                open_order_mode(
                    feed,
                    godwidget,
                    order_ctl_fqme,
                    order_mode_started,
                    loglevel=loglevel
-                ) as mode,
+                ) as mode
                # TODO: maybe have these startup sooner before
                # order mode fully boots? but we gotta,
                # -[ ] decouple the order mode bindings until
                #    the mode has fully booted..
                #    -[ ] maybe do an Event to sync?
                # start input handling for ``ChartView`` input
                # (i.e. kb + mouse handling loops)
                rt_chart.view.open_async_input_handler(
                    dss=dss,
                ),
                hist_chart.view.open_async_input_handler(
                    dss=dss,
                ),
            ):
                rt_linked.mode = mode
                rt_viz = rt_chart.get_viz(order_ctl_fqme)
--- a/piker/ui/_editors.py
+++ b/piker/ui/_editors.py
@ -21,8 +21,7 @@ Higher level annotation editors.
 from __future__ import annotations
 from collections import defaultdict
 from typing import (
-    Sequence,
+    TYPE_CHECKING
    TYPE_CHECKING,
 )
 import pyqtgraph as pg
@ -32,34 +31,24 @@ from pyqtgraph import (
    QtCore,
    QtWidgets,
 )
 from PyQt5.QtGui import (
    QColor,
 )
 from PyQt5.QtWidgets import (
    QLabel,
 )
 from pyqtgraph import functions as fn
 from PyQt5.QtCore import QPointF
 import numpy as np
-from piker.types import Struct
+from ._style import hcolor, _font
 from piker.ui.qt import (
    Qt,
    QPointF,
    QRectF,
    QGraphicsProxyWidget,
    QGraphicsScene,
    QLabel,
    QColor,
    QTransform,
 )
 from ._style import (
    hcolor,
    _font,
 )
 from ._lines import LevelLine
 from ..log import get_logger
 from ..data.types import Struct
 if TYPE_CHECKING:
-    from ._chart import (
+    from ._chart import GodWidget
        GodWidget,
        ChartPlotWidget,
    )
    from ._interaction import ChartView
 log = get_logger(__name__)
@ -76,7 +65,7 @@ class ArrowEditor(Struct):
        uid: str,
        x: float,
        y: float,
-        color: str = 'default',
+        color='default',
        pointing: str | None = None,
    ) -> pg.ArrowItem:
@ -262,75 +251,43 @@ class LineEditor(Struct):
        return lines
 def as_point(
    pair: Sequence[float, float] | QPointF,
 ) -> list[QPointF, QPointF]:
    '''
    Case any input tuple of floats to a a list of `QPoint` objects
    for use in Qt geometry routines.
    '''
    if isinstance(pair, QPointF):
        return pair
    return QPointF(pair[0], pair[1])
 # TODO: maybe implement better, something something RectItemProxy??
 # -[ ] dig into details of how proxy's work?
 #    https://doc.qt.io/qt-5/qgraphicsscene.html#addWidget
 # -[ ] consider using `.addRect()` maybe?
 class SelectRect(QtWidgets.QGraphicsRectItem):
    '''
    A data-view "selection rectangle": the most fundamental
    geometry for annotating data views.
    - https://doc.qt.io/qt-5/qgraphicsrectitem.html
    - https://doc.qt.io/qt-6/qgraphicsrectitem.html
    '''
    def __init__(
        self,
        viewbox: ViewBox,
-        color: str | None = None,
+        color: str = 'dad_blue',
    ) -> None:
        super().__init__(0, 0, 1, 1)
        # self.rbScaleBox = QGraphicsRectItem(0, 0, 1, 1)
-        self.vb: ViewBox = viewbox
+        self.vb = viewbox
        self._chart: 'ChartPlotWidget' = None  # noqa
-        self._chart: ChartPlotWidget | None = None  # noqa
+        # override selection box color
        # TODO: maybe allow this to be dynamic via a method?
        #l override selection box color
        color: str = color or 'dad_blue'
        color = QColor(hcolor(color))
        self.setPen(fn.mkPen(color, width=1))
        color.setAlpha(66)
        self.setBrush(fn.mkBrush(color))
        self.setZValue(1e9)
        self.hide()
        self._label = None
        label = self._label = QLabel()
-        label.setTextFormat(
+        label.setTextFormat(0)  # markdown
            Qt.TextFormat.MarkdownText
        )
        label.setFont(_font.font)
        label.setMargin(0)
        label.setAlignment(
            QtCore.Qt.AlignLeft
            # | QtCore.Qt.AlignVCenter
        )
        label.hide()  # always right after init
        # proxy is created after containing scene is initialized
-        self._label_proxy: QGraphicsProxyWidget | None = None
+        self._label_proxy = None
-        self._abs_top_right: Point | None = None
+        self._abs_top_right = None
-        # TODO: "swing %" might be handy here (data's max/min
+        # TODO: "swing %" might be handy here (data's max/min # % change)
-        # # % change)?
+        self._contents = [
        self._contents: list[str] = [
            'change: {pchng:.2f} %',
            'range: {rng:.2f}',
            'bars: {nbars}',
@ -340,31 +297,12 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
            'sigma: {std:.2f}',
        ]
        self.add_to_view(viewbox)
        self.hide()
    def add_to_view(
        self,
        view: ChartView,
    ) -> None:
        '''
        Self-defined view hookup impl which will
        also re-assign the internal ref.
        '''
        view.addItem(
            self,
            ignoreBounds=True,
        )
        if self.vb is not view:
            self.vb = view
    @property
-    def chart(self) -> ChartPlotWidget:  # noqa
+    def chart(self) -> 'ChartPlotWidget':  # noqa
        return self._chart
    @chart.setter
-    def chart(self, chart: ChartPlotWidget) -> None:  # noqa
+    def chart(self, chart: 'ChartPlotWidget') -> None:  # noqa
        self._chart = chart
        chart.sigRangeChanged.connect(self.update_on_resize)
        palette = self._label.palette()
@ -377,155 +315,57 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
        )
    def update_on_resize(self, vr, r):
-        '''
+        """Re-position measure label on view range change.
        Re-position measure label on view range change.
-        '''
+        """
        if self._abs_top_right:
            self._label_proxy.setPos(
                self.vb.mapFromView(self._abs_top_right)
            )
-    def set_scen_pos(
+    def mouse_drag_released(
        self,
-        scen_p1: QPointF,
+        p1: QPointF,
-        scen_p2: QPointF,
+        p2: QPointF
        update_label: bool = True,
    ) -> None:
-        '''
+        """Called on final button release for mouse drag with start and
-        Set position from scene coords of selection rect (normally
+        end positions.
        from mouse position) and accompanying label, move label to
        match.
-        '''
+        """
-        # NOTE XXX: apparently just setting it doesn't work!?
+        self.set_pos(p1, p2)
        # i have no idea why but it's pretty weird we have to do
        # this transform thing which was basically pulled verbatim
        # from the `pg.ViewBox.updateScaleBox()` method.
        view_rect: QRectF = self.vb.childGroup.mapRectFromScene(
            QRectF(
                scen_p1, 
                scen_p2,
            )
        )
        self.setPos(view_rect.topLeft())
        # XXX: does not work..!?!?
        # https://doc.qt.io/qt-5/qgraphicsrectitem.html#setRect
        # self.setRect(view_rect)
-        tr = QTransform.fromScale(
+    def set_pos(
            view_rect.width(),
            view_rect.height(),
        )
        self.setTransform(tr)
        # XXX: never got this working, was always offset
        # / transformed completely wrong (and off to the far right
        # from the cursor?)
        # self.set_view_pos(
        #     view_rect=view_rect,
        #     # self.vwqpToView(p1),
        #     # self.vb.mapToView(p2),
        #     # start_pos=self.vb.mapToScene(p1),
        #     # end_pos=self.vb.mapToScene(p2),
        # )
        self.show()
        if update_label:
            self.init_label(view_rect)
    def set_view_pos(
        self,
-
+        p1: QPointF,
-        start_pos: QPointF | Sequence[float, float] | None = None,
+        p2: QPointF
        end_pos: QPointF | Sequence[float, float] | None = None,
        view_rect: QRectF | None = None,
        update_label: bool = True,
    ) -> None:
-        '''
+        """Set position of selection rect and accompanying label, move
-        Set position from `ViewBox` coords (i.e. from the actual
+        label to match.
        data domain) of rect (and any accompanying label which is
        moved to match).
-        '''
+        """
        if self._chart is None:
            raise RuntimeError(
                'You MUST assign a `SelectRect.chart: ChartPlotWidget`!'
            )
        if view_rect is None:
            # ensure point casting
            start_pos: QPointF = as_point(start_pos)
            end_pos: QPointF = as_point(end_pos)
            # map to view coords and update area
            view_rect = QtCore.QRectF(
                start_pos,
                end_pos,
            )
        self.setPos(view_rect.topLeft())
        # NOTE: SERIOUSLY NO IDEA WHY THIS WORKS...
        # but it does and all the other commented stuff above
        # dint, dawg..
        # self.resetTransform()
        # self.setRect(view_rect)
        tr = QTransform.fromScale(
            view_rect.width(),
            view_rect.height(),
        )
        self.setTransform(tr)
        if update_label:
            self.init_label(view_rect)
        print(
            'SelectRect modify:\n'
            f'QRectF: {view_rect}\n'
            f'start_pos: {start_pos}\n'
            f'end_pos: {end_pos}\n'
        )
        self.show()
    def init_label(
        self,
        view_rect: QRectF,
   ) -> QLabel:
        # should be init-ed in `.__init__()`
        label: QLabel = self._label
        cv: ChartView = self.vb
        # https://doc.qt.io/qt-5/qgraphicsproxywidget.html
        if self._label_proxy is None:
-            scen: QGraphicsScene = cv.scene()
+            # https://doc.qt.io/qt-5/qgraphicsproxywidget.html
-            # NOTE: specifically this is passing a widget
+            self._label_proxy = self.vb.scene().addWidget(self._label)
            # pointer to the scene's `.addWidget()` as per,
            # https://doc.qt.io/qt-5/qgraphicsproxywidget.html#embedding-a-widget-with-qgraphicsproxywidget
            self._label_proxy: QGraphicsProxyWidget = scen.addWidget(label)
-        # get label startup coords
+        start_pos = self.vb.mapToView(p1)
-        tl: QPointF = view_rect.topLeft()
+        end_pos = self.vb.mapToView(p2)
        br: QPointF = view_rect.bottomRight()
-        x1, y1 = tl.x(), tl.y()
+        # map to view coords and update area
-        x2, y2 = br.x(), br.y()
+        r = QtCore.QRectF(start_pos, end_pos)
-        # TODO: to remove, previous label corner point unpacking
+        # old way; don't need right?
-        # x1, y1 = start_pos.x(), start_pos.y()
+        # lr = QtCore.QRectF(p1, p2)
-        # x2, y2 = end_pos.x(), end_pos.y()
+        # r = self.vb.childGroup.mapRectFromParent(lr)
        # y1, y2 = start_pos.y(), end_pos.y()
        # x1, x2 = start_pos.x(), end_pos.x()
-        # TODO: heh, could probably use a max-min streamin algo
+        self.setPos(r.topLeft())
-        # here too?
+        self.resetTransform()
        self.setRect(r)
        self.show()
        y1, y2 = start_pos.y(), end_pos.y()
        x1, x2 = start_pos.x(), end_pos.x()
        # TODO: heh, could probably use a max-min streamin algo here too
        _, xmn = min(y1, y2), min(x1, x2)
        ymx, xmx = max(y1, y2), max(x1, x2)
@ -535,35 +375,26 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
        ixmn, ixmx = round(xmn), round(xmx)
        nbars = ixmx - ixmn + 1
-        chart: ChartPlotWidget = self._chart
+        chart = self._chart
-        data: np.ndarray = chart.get_viz(
+        data = chart.get_viz(chart.name).shm.array[ixmn:ixmx]
            chart.name
        ).shm.array[ixmn:ixmx]
        if len(data):
-            std: float = data['close'].std()
+            std = data['close'].std()
-            dmx: float = data['high'].max()
+            dmx = data['high'].max()
-            dmn: float = data['low'].min()
+            dmn = data['low'].min()
        else:
            dmn = dmx = std = np.nan
        # update label info
-        label.setText('\n'.join(self._contents).format(
+        self._label.setText('\n'.join(self._contents).format(
-            pchng=pchng,
+            pchng=pchng, rng=rng, nbars=nbars,
-            rng=rng,
+            std=std, dmx=dmx, dmn=dmn,
            nbars=nbars,
            std=std,
            dmx=dmx,
            dmn=dmn,
        ))
        # print(f'x2, y2: {(x2, y2)}')
        # print(f'xmn, ymn: {(xmn, ymx)}')
-        label_anchor = Point(
+        label_anchor = Point(xmx + 2, ymx)
            xmx + 2,
            ymx,
        )
        # XXX: in the drag bottom-right -> top-left case we don't
        # want the label to overlay the box.
@ -572,40 +403,13 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
        #     # label_anchor = Point(x2, y2 + self._label.height())
        #     label_anchor = Point(xmn, ymn)
-        self._abs_top_right: Point = label_anchor
+        self._abs_top_right = label_anchor
-        self._label_proxy.setPos(
+        self._label_proxy.setPos(self.vb.mapFromView(label_anchor))
-            cv.mapFromView(label_anchor)
+        # self._label.show()
        )
        label.show()
-    def hide(self):
+    def clear(self):
-        '''
+        """Clear the selection box from view.
        Clear the selection box from its graphics scene but
        don't delete it permanently.
-        '''
+        """
        super().hide()
        self._label.hide()
-
+        self.hide()
    # TODO: ensure noone else using dis.
    clear = hide
    def delete(self) -> None:
        '''
        De-allocate this rect from its rendering graphics scene.
        Like a permanent hide.
        '''
        scen: QGraphicsScene = self.scene()
        if scen is None:
            return
        scen.removeItem(self)
        if (
            self._label
            and
            self._label_proxy
        ):
            scen.removeItem(self._label_proxy)
--- a/piker/ui/_event.py
+++ b/piker/ui/_event.py
@ -23,29 +23,28 @@ from typing import Callable
 import trio
 from tractor.trionics import gather_contexts
-
+from PyQt5 import QtCore
-from piker.ui.qt import (
+from PyQt5.QtCore import QEvent, pyqtBoundSignal
-    QtCore,
+from PyQt5.QtWidgets import QWidget
-    QWidget,
+from PyQt5.QtWidgets import (
-    QEvent,
+    QGraphicsSceneMouseEvent as gs_mouse,
    keys,
    gs_keys,
    pyqtBoundSignal,
 )
-from piker.types import Struct
+
 from ..data.types import Struct
 MOUSE_EVENTS = {
-    gs_keys.GraphicsSceneMousePress,
+    gs_mouse.GraphicsSceneMousePress,
-    gs_keys.GraphicsSceneMouseRelease,
+    gs_mouse.GraphicsSceneMouseRelease,
-    keys.MouseButtonPress,
+    QEvent.MouseButtonPress,
-    keys.MouseButtonRelease,
+    QEvent.MouseButtonRelease,
    # QtGui.QMouseEvent,
 }
 # TODO: maybe consider some constrained ints down the road?
 # https://pydantic-docs.helpmanual.io/usage/types/#constrained-types
 class KeyboardMsg(Struct):
    '''Unpacked Qt keyboard event data.
@ -115,10 +114,7 @@ class EventRelay(QtCore.QObject):
        # something to do with Qt internals and calling the
        # parent handler?
-        if etype in {
+        if etype in {QEvent.KeyPress, QEvent.KeyRelease}:
            QEvent.Type.KeyPress,
            QEvent.Type.KeyRelease,
        }:
            msg = KeyboardMsg(
                event=ev,
@ -164,9 +160,7 @@ class EventRelay(QtCore.QObject):
 async def open_event_stream(
    source_widget: QWidget,
-    event_types: set[QEvent] = {
+    event_types: set[QEvent] = {QEvent.KeyPress},
        QEvent.Type.KeyPress,
    },
    filter_auto_repeats: bool = True,
 ) -> trio.abc.ReceiveChannel:
@ -207,8 +201,8 @@ async def open_signal_handler(
        async for args in recv:
            await async_handler(*args)
-    async with trio.open_nursery() as tn:
+    async with trio.open_nursery() as n:
-        tn.start_soon(proxy_to_handler)
+        n.start_soon(proxy_to_handler)
        async with send:
            yield
@ -218,48 +212,18 @@ async def open_handlers(
    source_widgets: list[QWidget],
    event_types: set[QEvent],
-
+    async_handler: Callable[[QWidget, trio.abc.ReceiveChannel], None],
-    # NOTE: if you want to bind in additional kwargs to the handler
+    **kwargs,
    # pass in a `partial()` instead!
    async_handler: Callable[
        [QWidget, trio.abc.ReceiveChannel],  # required handler args
        None
    ],
    # XXX: these are ONLY inputs available to the
    # `open_event_stream()` event-relay to mem-chan factor above!
    **open_ev_stream_kwargs,
 ) -> None:
    '''
    Connect and schedule an async handler function to receive an
    arbitrary `QWidget`'s events with kb/mouse msgs repacked into
    structs (see above) and shuttled over a mem-chan to the input
    `async_handler` to allow interaction-IO processing from
    a `trio` func-as-task.
    '''
    widget: QWidget
    streams: list[trio.abc.ReceiveChannel]
    async with (
-        trio.open_nursery() as tn,
+        trio.open_nursery() as n,
        gather_contexts([
-            open_event_stream(
+            open_event_stream(widget, event_types, **kwargs)
                widget,
                event_types,
                **open_ev_stream_kwargs,
            )
            for widget in source_widgets
        ]) as streams,
    ):
-        for widget, event_recv_stream in zip(
+        for widget, event_recv_stream in zip(source_widgets, streams):
-            source_widgets,
+            n.start_soon(async_handler, widget, event_recv_stream)
            streams,
        ):
            tn.start_soon(
                async_handler,
                widget,
                event_recv_stream,
            )
        yield
--- a/piker/ui/_exec.py
+++ b/piker/ui/_exec.py
@ -30,35 +30,34 @@ from typing import (
 import platform
 import traceback
 # Qt specific
 import PyQt5  # noqa
 from PyQt5.QtWidgets import (
    QWidget,
    QMainWindow,
    QApplication,
 )
 from PyQt5 import QtCore
 from PyQt5.QtCore import (
    pyqtRemoveInputHook,
    Qt,
    QCoreApplication,
 )
 import qdarkstyle
 from qdarkstyle import DarkPalette
 # import qdarkgraystyle  # TODO: play with it
 import trio
 from outcome import Error
 # Qt version-agnostic
 from .qt import (
    QWidget,
    QMainWindow,
    QApplication,
    QtCore,
    pyqtRemoveInputHook,
    Qt,
    QCoreApplication,
 )
 from ..service import (
    maybe_open_pikerd,
-    get_runtime_vars,
+    get_tractor_runtime_kwargs,
 )
 from ..log import get_logger
 from ._pg_overrides import _do_overrides
 from . import _style
 if TYPE_CHECKING:
    from ._chart import GodWidget
 log = get_logger(__name__)
 # pyqtgraph global config
@ -147,7 +146,7 @@ def run_qtractor(
    # load dark theme
    stylesheet = qdarkstyle.load_stylesheet(
-        qt_api='pyqt6',
+        qt_api='pyqt5',
        palette=DarkPalette,
    )
    app.setStyleSheet(stylesheet)
@ -174,9 +173,7 @@ def run_qtractor(
    instance.window = window
    # override tractor's defaults
-    tractor_kwargs.update(
+    tractor_kwargs.update(get_tractor_runtime_kwargs())
        get_runtime_vars()
    )
    # define tractor entrypoint
    async def main():
--- a/piker/ui/_forms.py
+++ b/piker/ui/_forms.py
@ -21,6 +21,7 @@ Text entry "forms" widgets (mostly for configuration and UI user input).
 from __future__ import annotations
 from contextlib import asynccontextmanager
 from functools import partial
 from math import floor
 from typing import (
    Any,
    Callable,
@ -28,15 +29,9 @@ from typing import (
 )
 import trio
-
+from PyQt5 import QtGui
-from piker.ui.qt import (
+from PyQt5.QtCore import QSize, QModelIndex, Qt, QEvent
-    keys,
+from PyQt5.QtWidgets import (
    size_policy,
    QtGui,
    QSize,
    QModelIndex,
    Qt,
    QEvent,
    QWidget,
    QLabel,
    QComboBox,
@ -45,6 +40,7 @@ from piker.ui.qt import (
    QVBoxLayout,
    QFormLayout,
    QProgressBar,
    QSizePolicy,
    QStyledItemDelegate,
    QStyleOptionViewItem,
 )
@ -76,14 +72,14 @@ class Edit(QLineEdit):
        if width_in_chars:
            self._chars = int(width_in_chars)
-            x_size_policy = size_policy.Fixed
+            x_size_policy = QSizePolicy.Fixed
        else:
            # chart count which will be used to calculate
            # width of input field.
            self._chars: int = 6
            # fit to surroundingn frame width
-            x_size_policy = size_policy.Expanding
+            x_size_policy = QSizePolicy.Expanding
        super().__init__(parent)
@ -91,7 +87,7 @@ class Edit(QLineEdit):
        # https://doc.qt.io/qt-5/qsizepolicy.html#Policy-enum
        self.setSizePolicy(
            x_size_policy,
-            size_policy.Fixed,
+            QSizePolicy.Fixed,
        )
        self.setFont(font.font)
@ -185,13 +181,11 @@ class Selection(QComboBox):
        self._items: dict[str, int] = {}
        super().__init__(parent=parent)
-        self.setSizeAdjustPolicy(
+        self.setSizeAdjustPolicy(QComboBox.AdjustToContents)
            QComboBox.SizeAdjustPolicy.AdjustToContents,
        )
        # make line edit expand to surrounding frame
        self.setSizePolicy(
-            size_policy.Expanding,
+            QSizePolicy.Expanding,
-            size_policy.Fixed,
+            QSizePolicy.Fixed,
        )
        view = self.view()
        view.setUniformItemSizes(True)
@ -315,8 +309,8 @@ class FieldsForm(QWidget):
        # size it as we specify
        self.setSizePolicy(
-            size_policy.Expanding,
+            QSizePolicy.Expanding,
-            size_policy.Expanding,
+            QSizePolicy.Expanding,
        )
        # XXX: not sure why we have to create this here exactly
@ -423,8 +417,8 @@ class FieldsForm(QWidget):
        select.set_items(values)
        self.setSizePolicy(
-            size_policy.Fixed,
+            QSizePolicy.Fixed,
-            size_policy.Fixed,
+            QSizePolicy.Fixed,
        )
        select.show()
        self.form.addRow(label, select)
@ -444,10 +438,7 @@ async def handle_field_input(
    async for kbmsg in recv_chan:
-        if kbmsg.etype in {
+        if kbmsg.etype in {QEvent.KeyPress, QEvent.KeyRelease}:
            keys.KeyPress,
            keys.KeyRelease,
        }:
            event, etype, key, mods, txt = kbmsg.to_tuple()
            print(f'key: {kbmsg.key}, mods: {kbmsg.mods}, txt: {kbmsg.txt}')
@ -713,8 +704,7 @@ def mk_fill_status_bar(
    )
    bottom_label = form.add_field_label(
-        # 'x: {step_size}',
+        'x: {step_size}',
       '{unit_prefix}: {step_size}',
        font_size=bar_label_font_size,
        font_color='gunmetal',
    )
--- a/piker/ui/_fsp.py
+++ b/piker/ui/_fsp.py
@ -36,27 +36,16 @@ from tractor.trionics import maybe_open_context
 import trio
 from trio_typing import TaskStatus
-from piker.accounting import MktPair
+from piker.data.types import Struct
-from piker.fsp import (
+from ._axes import PriceAxis
-    cascade,
+from ..calc import humanize
-    maybe_mk_fsp_shm,
+from ..data._sharedmem import (
    Fsp,
    dolla_vlm,
    flow_rates,
 )
 from piker.data import (
    Flume,
    ShmArray,
 )
 from piker.data._sharedmem import (
    _Token,
    try_read,
 )
-from piker.log import get_logger
+from ..data.feed import Flume
-from piker.toolz import Profiler
+from ..accounting import MktPair
 from piker.types import Struct
 from ._axes import PriceAxis
 from ..calc import humanize
 from ._chart import (
    ChartPlotWidget,
    LinkedSplits,
@ -66,6 +55,18 @@ from ._forms import (
    mk_form,
    open_form_input_handling,
 )
 from ..fsp._api import (
    maybe_mk_fsp_shm,
    Fsp,
 )
 from ..fsp import cascade
 from ..fsp._volume import (
    # tina_vwap,
    dolla_vlm,
    flow_rates,
 )
 from ..log import get_logger
 from .._profile import Profiler
 log = get_logger(__name__)
@ -181,10 +182,7 @@ async def open_fsp_sidepane(
 async def open_fsp_actor_cluster(
    names: list[str] = ['fsp_0', 'fsp_1'],
-) -> AsyncGenerator[
+) -> AsyncGenerator[int, dict[str, tractor.Portal]]:
    int,
    dict[str, tractor.Portal]
 ]:
    from tractor._clustering import open_actor_cluster
@ -285,7 +283,6 @@ async def run_fsp_ui(
            name,
            array_key=array_key,
        )
        assert chart.qframe
        chart.linked.focus()
@ -393,7 +390,7 @@ class FspAdmin:
        complete: trio.Event,
        started: trio.Event,
        fqme: str,
-        dst_flume: Flume,
+        dst_fsp_flume: Flume,
        conf: dict,
        target: Fsp,
        loglevel: str,
@ -411,14 +408,16 @@ class FspAdmin:
                # chaining entrypoint
                cascade,
                # TODO: can't we just drop this and expect
                # far end to read the src flume's .mkt.fqme?
                # data feed key
                fqme=fqme,
-                src_flume_addr=self.flume.to_msg(),
+                # TODO: pass `Flume.to_msg()`s here?
-                dst_flume_addr=dst_flume.to_msg(),
+                # mems
-                ns_path=ns_path,  # edge-bind-func
+                src_shm_token=self.flume.rt_shm.token,
                dst_shm_token=dst_fsp_flume.rt_shm.token,
                # target
                ns_path=ns_path,
                loglevel=loglevel,
                zero_on_step=conf.get('zero_on_step', False),
@ -432,14 +431,14 @@ class FspAdmin:
            ctx.open_stream() as stream,
        ):
-            dst_flume.stream: tractor.MsgStream = stream
+            dst_fsp_flume.stream: tractor.MsgStream = stream
            # register output data
            self._registry[
                (fqme, ns_path)
            ] = (
                stream,
-                dst_flume.rt_shm,
+                dst_fsp_flume.rt_shm,
                complete
            )
@ -516,7 +515,7 @@ class FspAdmin:
            broker='piker',
            _atype='fsp',
        )
-        dst_flume = Flume(
+        dst_fsp_flume = Flume(
            mkt=mkt,
            _rt_shm_token=dst_shm.token,
            first_quote={},
@ -544,13 +543,13 @@ class FspAdmin:
            complete,
            started,
            fqme,
-            dst_flume,
+            dst_fsp_flume,
            conf,
            target,
            loglevel,
        )
-        return dst_flume, started
+        return dst_fsp_flume, started
    async def open_fsp_chart(
        self,
@ -560,7 +559,7 @@ class FspAdmin:
        conf: dict,  # yeah probably dumb..
        loglevel: str = 'error',
-    ) -> trio.Event:
+    ) -> (trio.Event, ChartPlotWidget):
        flume, started = await self.start_engine_task(
            target,
@ -927,7 +926,7 @@ async def start_fsp_displays(
    linked: LinkedSplits,
    flume: Flume,
-    # group_status_key: str,
+    group_status_key: str,
    loglevel: str,
 ) -> None:
@ -974,23 +973,21 @@ async def start_fsp_displays(
            flume,
        ) as admin,
    ):
-        statuses: list[trio.Event] = []
+        statuses = []
        for target, conf in fsp_conf.items():
-            started: trio.Event = await admin.open_fsp_chart(
+            started = await admin.open_fsp_chart(
                target,
                conf,
            )
-            # done = linked.window().status_bar.open_status(
+            done = linked.window().status_bar.open_status(
-            #     f'loading fsp, {target}..',
+                f'loading fsp, {target}..',
-            #     group_key=group_status_key,
+                group_key=group_status_key,
-            # )
+            )
-            # statuses.append((started, done))
+            statuses.append((started, done))
            statuses.append(started)
-        # for fsp_loaded, status_cb in statuses:
+        for fsp_loaded, status_cb in statuses:
        for fsp_loaded in statuses:
            await fsp_loaded.wait()
            profiler(f'attached to fsp portal: {target}')
-            # status_cb()
+            status_cb()
    # blocks on nursery until all fsp actors complete
--- a/piker/ui/_icons.py
+++ b/piker/ui/_icons.py
@ -15,18 +15,15 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-`QIcon` hackery.
+``QIcon`` hackery.
 Mostly dynamically loading pixmaps for use with `QGraphicsScene`.
 '''
-from piker.ui.qt import (
+from PyQt5.QtWidgets import QStyle
-    QSize,
+from PyQt5.QtGui import (
-    QStyle,
+    QIcon, QPixmap, QColor
    QIcon,
    QPixmap,
    QColor,
 )
 from PyQt5.QtCore import QSize
 from ._style import hcolor
 # https://www.pythonguis.com/faq/built-in-qicons-pyqt/
@ -47,8 +44,7 @@ def mk_icons(
    size: QSize,
 ) -> dict[str, QIcon]:
-    '''
+    '''This helper is indempotent.
    This helper is indempotent.
    '''
    global _icons, _icon_names
@ -60,11 +56,7 @@ def mk_icons(
    # load account selection using current style
    for name, icon_name in _icon_names.items():
-        stdpixmap = getattr(
+        stdpixmap = getattr(QStyle, icon_name)
            # https://www.pythonguis.com/faq/built-in-qicons-pyqt/
            QStyle.StandardPixmap,  # pyqt/pyside6
            icon_name,
        )
        stdicon = style.standardIcon(stdpixmap)
        pixmap = stdicon.pixmap(size)
--- a/piker/ui/_interaction.py
+++ b/piker/ui/_interaction.py
@ -23,7 +23,6 @@ from contextlib import (
    asynccontextmanager,
    ExitStack,
 )
 from functools import partial
 import time
 from typing import (
    Callable,
@ -31,70 +30,46 @@ from typing import (
 )
 import pyqtgraph as pg
-# NOTE XXX: pg is super annoying and re-implements it's own mouse
+# from pyqtgraph.GraphicsScene import mouseEvents
-# event subsystem.. we should really look into re-working/writing
+from PyQt5.QtWidgets import QGraphicsSceneMouseEvent as gs_mouse
-# this down the road.. Bo
+from PyQt5.QtCore import Qt, QEvent
-from pyqtgraph.GraphicsScene import mouseEvents as mevs
+from pyqtgraph import ViewBox, Point, QtCore
-# from pyqtgraph.GraphicsScene.mouseEvents import MouseDragEvent
+from pyqtgraph import functions as fn
 from pyqtgraph import (
    ViewBox,
    Point,
    QtCore,
    functions as fn,
 )
 import numpy as np
 import trio
 from piker.ui.qt import (
    QWheelEvent,
    QGraphicsSceneMouseEvent as gs_mouse,
    Qt,
    QEvent,
 )
 from ..log import get_logger
-from ..toolz import (
+from .._profile import Profiler
-    Profiler,
+from .._profile import pg_profile_enabled, ms_slower_then
    pg_profile_enabled,
    ms_slower_then,
 )
 from .view_mode import overlay_viewlists
 # from ._style import _min_points_to_show
 from ._editors import SelectRect
 from . import _event
 if TYPE_CHECKING:
-    # from ._search import (
+    from ._chart import ChartPlotWidget
    #     SearchWidget,
    # )
    from ._chart import (
        ChartnPane,
        ChartPlotWidget,
        GodWidget,
    )
    from ._dataviz import Viz
    from .order_mode import OrderMode
    from ._display import DisplayState
 log = get_logger(__name__)
 NUMBER_LINE = {
-    Qt.Key.Key_1,
+    Qt.Key_1,
-    Qt.Key.Key_2,
+    Qt.Key_2,
-    Qt.Key.Key_3,
+    Qt.Key_3,
-    Qt.Key.Key_4,
+    Qt.Key_4,
-    Qt.Key.Key_5,
+    Qt.Key_5,
-    Qt.Key.Key_6,
+    Qt.Key_6,
-    Qt.Key.Key_7,
+    Qt.Key_7,
-    Qt.Key.Key_8,
+    Qt.Key_8,
-    Qt.Key.Key_9,
+    Qt.Key_9,
-    Qt.Key.Key_0,
+    Qt.Key_0,
 }
 ORDER_MODE = {
-    Qt.Key.Key_A,
+    Qt.Key_A,
-    Qt.Key.Key_F,
+    Qt.Key_F,
-    Qt.Key.Key_D,
+    Qt.Key_D,
 }
@ -102,12 +77,10 @@ async def handle_viewmode_kb_inputs(
    view: ChartView,
    recv_chan: trio.abc.ReceiveChannel,
    dss: dict[str, DisplayState],
 ) -> None:
-    order_mode: OrderMode = view.order_mode
+    order_mode = view.order_mode
    godw: GodWidget = order_mode.godw  # noqa
    # track edge triggered keys
    # (https://en.wikipedia.org/wiki/Interrupt#Triggering_methods)
@ -171,55 +144,29 @@ async def handle_viewmode_kb_inputs(
            if mods == Qt.ControlModifier:
                ctrl = True
-            # UI REPL-shell, with ctrl-p (for "pause")
+            # UI REPL-shell
            if (
-                ctrl
+                ctrl and key in {
-                and key in {
+                    Qt.Key_U,
                    Qt.Key_P,
                }
            ):
                import tractor
                god = order_mode.godw  # noqa
                feed = order_mode.feed  # noqa
                chart = order_mode.chart  # noqa
                viz = chart.main_viz  # noqa
                vlm_chart = chart.linked.subplots['volume']  # noqa
                vlm_viz = vlm_chart.main_viz  # noqa
                dvlm_pi = vlm_chart._vizs['dolla_vlm'].plot  # noqa
-                import tractor
+                await tractor.breakpoint()
                await tractor.pause()
                view.interact_graphics_cycle()
-            # FORCE graphics reset-and-render of all currently
+            # SEARCH MODE #
            # shown data `Viz`s for the current chart app.
            if (
                ctrl
                and key in {
                    Qt.Key_R,
                }
            ):
                fqme: str
                ds: DisplayState
                for fqme, ds in dss.items():
                    viz: Viz
                    for tf, viz in {
                        60: ds.hist_viz,
                        1: ds.viz,
                    }.items():
                        # TODO: only allow this when the data is IN VIEW!
                        # also, we probably can do this more efficiently
                        # / smarter by only redrawing the portion of the
                        # path necessary?
                        viz.reset_graphics()
            # ------ - ------
            # SEARCH MODE
            # ------ - ------
            # ctlr-<space>/<l> for "lookup", "search" -> open search tree
            if (
-                ctrl
+                ctrl and key in {
                and key in {
                    Qt.Key_L,
-                    # Qt.Key_Space,
+                    Qt.Key_Space,
                }
            ):
                godw = view._chart.linked.godwidget
@ -227,56 +174,20 @@ async def handle_viewmode_kb_inputs(
                godw.search.focus()
            # esc and ctrl-c
-            if (
+            if key == Qt.Key_Escape or (ctrl and key == Qt.Key_C):
                key == Qt.Key_Escape
                or (
                    ctrl
                    and key == Qt.Key_C
                )
            ):
                # ctrl-c as cancel
                # https://forum.qt.io/topic/532/how-to-catch-ctrl-c-on-a-widget/9
                view.select_box.clear()
                view.linked.focus()
            # cancel order or clear graphics
-            if (
+            if key == Qt.Key_C or key == Qt.Key_Delete:
                key == Qt.Key_C
                or key == Qt.Key_Delete
            ):
                order_mode.cancel_orders_under_cursor()
            # View modes
-            if (
+            if key == Qt.Key_R:
                ctrl
                and (
                    key == Qt.Key_Equal
                    or key == Qt.Key_I
                )
            ):
                view.wheelEvent(
                    ev=None,
                    axis=None,
                    delta=view.def_delta,
                )
            elif (
                ctrl
                and (
                    key == Qt.Key_Minus
                    or key == Qt.Key_O
                )
            ):
                view.wheelEvent(
                    ev=None,
                    axis=None,
                    delta=-view.def_delta,
                )
            elif (
                not ctrl
                and key == Qt.Key_R
            ):
                # NOTE: seems that if we don't yield a Qt render
                # cycle then the m4 downsampled curves will show here
                # without another reset..
@ -321,47 +232,15 @@ async def handle_viewmode_kb_inputs(
        # Toggle position config pane
        if (
-            ctrl
+            ctrl and key in {
-            and key in {
+                Qt.Key_P,
                Qt.Key_Space,
            }
        ):
-            # searchw: SearchWidget = godw.search
+            pp_pane = order_mode.current_pp.pane
-            # pp_pane = order_mode.current_pp.pane
+            if pp_pane.isHidden():
-            qframes: list[ChartnPane] = []
+                pp_pane.show()
            for linked in (
                godw.rt_linked,
                godw.hist_linked,
            ):
                for chartw in (
                    [linked.chart]
                    +
                    list(linked.subplots.values())
                ):
                    qframes.append(
                        chartw.qframe
                    )
            # NOTE: place priority on FIRST hiding all
            # panes before showing them.
            # TODO: make this more "fancy"?
            # - maybe look at majority of hidden states and then
            #   flip based on that?
            # - move these loops into the chart APIs?
            # - store the UX-state for a given feed/symbol and
            #   apply when opening a new one (eg. if panes were
            #   hidden then also hide them on newly loaded mkt
            #   feeds).
            if not any(
                qf.sidepane.isHidden() for qf in qframes
            ):
                for qf in qframes:
                    qf.sidepane.hide()
            else:
-                for qf in qframes:
+                pp_pane.hide()
                    qf.sidepane.show()
        # ORDER MODE
        # ----------
@ -379,7 +258,7 @@ async def handle_viewmode_kb_inputs(
            # show the pp size label only if there is
            # a non-zero pos existing
            tracker = order_mode.current_pp
-            if tracker.live_pp.cumsize:
+            if tracker.live_pp.size:
                tracker.nav.show()
            # TODO: show pp config mini-params in status bar widget
@ -459,7 +338,6 @@ async def handle_viewmode_mouse(
    view: ChartView,
    recv_chan: trio.abc.ReceiveChannel,
    dss: dict[str, DisplayState],
 ) -> None:
@ -497,9 +375,6 @@ class ChartView(ViewBox):
    '''
    mode_name: str = 'view'
    def_delta: float = 616 * 6
    def_scale_factor: float = 1.016 ** (def_delta * -1 / 20)
    # annots: dict[int, GraphicsObject] = {}
    def __init__(
        self,
@ -520,7 +395,6 @@ class ChartView(ViewBox):
            # defaultPadding=0.,
            **kwargs
        )
        # for "known y-range style"
        self._static_yrange = static_yrange
@ -535,11 +409,7 @@ class ChartView(ViewBox):
        # add our selection box annotator
        self.select_box = SelectRect(self)
-        # self.select_box.add_to_view(self)
+        self.addItem(self.select_box, ignoreBounds=True)
        # self.addItem(
        #     self.select_box,
        #     ignoreBounds=True,
        # )
        self.mode = None
        self.order_mode: bool = False
@ -596,7 +466,6 @@ class ChartView(ViewBox):
    @asynccontextmanager
    async def open_async_input_handler(
        self,
        **handler_kwargs,
    ) -> ChartView:
@ -607,20 +476,14 @@ class ChartView(ViewBox):
                    QEvent.KeyPress,
                    QEvent.KeyRelease,
                },
-                async_handler=partial(
+                async_handler=handle_viewmode_kb_inputs,
                    handle_viewmode_kb_inputs,
                    **handler_kwargs,
                ),
            ),
            _event.open_handlers(
                [self],
                event_types={
                    gs_mouse.GraphicsSceneMousePress,
                },
-                async_handler=partial(
+                async_handler=handle_viewmode_mouse,
                    handle_viewmode_mouse,
                    **handler_kwargs,
                ),
            ),
        ):
            yield self
@ -636,9 +499,8 @@ class ChartView(ViewBox):
    def wheelEvent(
        self,
-        ev: QWheelEvent | None = None,
+        ev,
-        axis: int | None = None,
+        axis=None,
        delta: float | None = None,
    ):
        '''
        Override "center-point" location for scrolling.
@ -649,12 +511,6 @@ class ChartView(ViewBox):
        TODO: PR a method into ``pyqtgraph`` to make this configurable
        '''
        # NOTE: certain operations are only avail when this handler is
        # actually called on events.
        if ev is None:
            assert delta
            assert axis is None
        linked = self.linked
        if (
            not linked
@ -665,7 +521,7 @@ class ChartView(ViewBox):
            mask = [False, False]
            mask[axis] = self.state['mouseEnabled'][axis]
        else:
-            mask: list[bool] = self.state['mouseEnabled'][:]
+            mask = self.state['mouseEnabled'][:]
        chart = self.linked.chart
@ -686,15 +542,8 @@ class ChartView(ViewBox):
        #     return
        # actual scaling factor
-        delta: float = ev.delta() if ev else delta
+        s = 1.016 ** (ev.delta() * -1 / 20)  # self.state['wheelScaleFactor'])
-        scale_factor: float = 1.016 ** (delta * -1 / 20)
+        s = [(None if m is False else s) for m in mask]
        # NOTE: if elem is False -> None meaning "do not scale that
        # axis".
        scales: list[float | bool] = [
            (None if m is False else scale_factor)
            for m in mask
        ]
        if (
            # zoom happened on axis
@ -717,7 +566,7 @@ class ChartView(ViewBox):
                ).map(ev.pos())
            )
            # scale_y = 1.3 ** (center.y() * -1 / 20)
-            self.scaleBy(scales, center)
+            self.scaleBy(s, center)
        # zoom in view-box area
        else:
@ -732,7 +581,7 @@ class ChartView(ViewBox):
            # NOTE: scroll "around" the right most datum-element in view
            # gives the feeling of staying "pinned" in place.
-            self.scaleBy(scales, focal)
+            self.scaleBy(s, focal)
            # XXX: the order of the next 2 lines i'm pretty sure
            # matters, we want the resize to trigger before the graphics
@ -752,23 +601,21 @@ class ChartView(ViewBox):
            self.interact_graphics_cycle()
            self.interact_graphics_cycle()
-            if ev:
+            ev.accept()
                ev.accept()
    def mouseDragEvent(
        self,
-        ev: mevs.MouseDragEvent,
+        ev,
        axis: int | None = None,
    ) -> None:
-        pos: Point = ev.pos()
+        pos = ev.pos()
-        lastPos: Point = ev.lastPos()
+        lastPos = ev.lastPos()
-        dif: Point = (pos - lastPos) * -1
+        dif = pos - lastPos
-        # dif: Point = pos - lastPos
+        dif = dif * -1
        # dif: Point = dif * -1
        # NOTE: if axis is specified, event will only affect that axis.
-        btn = ev.button()
+        button = ev.button()
        # Ignore axes if mouse is disabled
        mouseEnabled = np.array(
@ -780,7 +627,7 @@ class ChartView(ViewBox):
            mask[1-axis] = 0.0
        # Scale or translate based on mouse button
-        if btn & (
+        if button & (
            QtCore.Qt.LeftButton | QtCore.Qt.MidButton
        ):
            # zoom y-axis ONLY when click-n-drag on it
@ -803,55 +650,34 @@ class ChartView(ViewBox):
                # XXX: WHY
                ev.accept()
-                down_pos: Point = ev.buttonDownPos(
+                down_pos = ev.buttonDownPos()
                    btn=btn,
                )
                scen_pos: Point = ev.scenePos()
                scen_down_pos: Point = ev.buttonDownScenePos(
                    btn=btn,
                )
                # This is the final position in the drag
                if ev.isFinish():
-                    # import pdbp; pdbp.set_trace()
+                    self.select_box.mouse_drag_released(down_pos, pos)
                    # NOTE: think of this as a `.mouse_drag_release()`
                    # (bc HINT that's what i called the shit ass
                    # method that wrapped this call [yes, as a single
                    # fucking call] originally.. you bish, guille)
                    # Bo.. oraleeee
                    self.select_box.set_scen_pos(
                        # down_pos,
                        # pos,
                        scen_down_pos,
                        scen_pos,
                    )
                    # this is the zoom transform cmd
                    ax = QtCore.QRectF(down_pos, pos)
                    ax = self.childGroup.mapRectFromParent(ax)
-                    # self.showAxRect(ax)
+
                    # this is the zoom transform cmd
                    self.showAxRect(ax)
                    # axis history tracking
                    self.axHistoryPointer += 1
                    self.axHistory = self.axHistory[
                        :self.axHistoryPointer] + [ax]
                else:
-                    self.select_box.set_scen_pos(
+                    print('drag finish?')
-                        # down_pos,
+                    self.select_box.set_pos(down_pos, pos)
                        # pos,
                        scen_down_pos,
                        scen_pos,
                    )
                    # update shape of scale box
                    # self.updateScaleBox(ev.buttonDownPos(), ev.pos())
-                    # breakpoint()
+                    self.updateScaleBox(
-                    # self.updateScaleBox(
+                        down_pos,
-                    #     down_pos,
+                        ev.pos(),
-                    #     ev.pos(),
+                    )
                    # )
            # PANNING MODE
            else:
@ -890,7 +716,7 @@ class ChartView(ViewBox):
                # ev.accept()
        # WEIRD "RIGHT-CLICK CENTER ZOOM" MODE
-        elif btn & QtCore.Qt.RightButton:
+        elif button & QtCore.Qt.RightButton:
            if self.state['aspectLocked'] is not False:
                mask[0] = 0
--- a/Show More
+++ b/Show More