Add `.ui` and `.dev` deps groups via `poetry` Bo

Since we eventually want to allow users to minimally deploy `pikerd`
service-tree (aka distributed cross host) installs, we need to offer
a "headless" deps group. Really this is just the core dep set minus Qt
and some aux search related libs (for now).

The new `.dev` group is for adding hacking and testing tools including
`xonsh` since that will eventually be our REPL of choice more then
likely B)

Oh, and fix the namespace path (was a typo) for the `ledger` CLI and
of course bump the lock file.

README.rst

@@ -1,161 +1,235 @@
 piker
 -----
-trading gear for hackers
+trading gear for hackers.
 
 |gh_actions|
 
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
     :target: https://actions-badge.atrox.dev/piker/pikers/goto
 
-``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
-real-time computational trading targeted at `hardcore Linux users
-<comp_trader>`_ .
+``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
+computational trading targeted at `hardcore Linux users <comp_trader>`_ .
 
-we use much bleeding edge tech including (but not limited to):
+we use as much bleeding edge tech as possible including (but not limited to):
 
 - latest python for glue_
-- uv_ for packaging and distribution
-- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
-- Qt_ for pristine low latency UIs
-- pyqtgraph_ (which we've extended) for real-time charting and graphics
-- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
-- `apache arrow and parquet`_ for time-series storage
+- trio_ for `structured concurrency`_
+- tractor_ for distributed, multi-core, real-time streaming
+- marketstore_ for historical and real-time tick data persistence and sharing
+- techtonicdb_ for L2 book storage
+- Qt_ for pristine high performance UIs
+- pyqtgraph_ for real-time charting
+- ``numpy`` and ``numba`` for `fast numerics`_
 
-potential projects we might integrate with soon,
-
-- (already prototyped in ) techtonicdb_ for L2 book storage
-
-.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
-.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
-.. _uv: https://docs.astral.sh/uv/
+.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
+    :target: https://travis-ci.org/pikers/piker
 .. _trio: https://github.com/python-trio/trio
 .. _tractor: https://github.com/goodboy/tractor
 .. _structured concurrency: https://trio.discourse.group/
+.. _marketstore: https://github.com/alpacahq/marketstore
+.. _techtonicdb: https://github.com/0b01/tectonicdb
 .. _Qt: https://www.qt.io/
 .. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
-.. _apache arrow and parquet: https://arrow.apache.org/faq/
+.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
-.. _techtonicdb: https://github.com/0b01/tectonicdb
+.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
 
 
-focus and feats:
+focus and features:
+*******************
+- 100% federated: your code, your hardware, your data feeds, your broker fills.
+- zero web: low latency, native software that doesn't try to re-invent the OS
+- maximal **privacy**: prevent brokers and mms from knowing your
+  planz; smack their spreads with dark volume.
+- zero clutter: modal, context oriented UIs that echew minimalism, reduce
+  thought noise and encourage un-emotion.
+- first class parallelism: built from the ground up on next-gen structured concurrency
+  primitives.
+- traders first: broker/exchange/asset-class agnostic
+- systems grounded: real-time financial signal processing that will
+  make any queuing or DSP eng juice their shorts.
+- non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
+- data collaboration: every process and protocol is multi-host scalable.
+- fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
+
+fitting with these tenets, we're always open to new framework suggestions and ideas.
+
+building the best looking, most reliable, keyboard friendly trading
+platform is the dream; join the cause.
+
+
+install
+*******
+``piker`` is currently under heavy pre-alpha development and as such
+should be cloned from this repo and hacked on directly.
+
+for a development install::
+
+    git clone git@github.com:pikers/piker.git
+    cd piker
+    virtualenv env
+    source ./env/bin/activate
+    pip install -r requirements.txt -e .
+
+install for nixos
+*****************
+for users of `NixOS` we offer a development shell envoirment that can be
+loaded with::
+
+    nix-shell develop.nix
+
+this will setup the required python environment to run piker, make sure to
+run::
+
+    pip install -r requirements.txt -e .
+
+once after loading the shell
+
+install for tinas
+*****************
+for windows peeps you can start by installing all the prerequisite software:
+
+- install git with all default settings - https://git-scm.com/download/win
+- install anaconda all default settings - https://www.anaconda.com/products/individual
+- install microsoft build tools (check the box for Desktop development for C++, you might be able to uncheck some optional downloads)  - https://visualstudio.microsoft.com/visual-cpp-build-tools/
+- install visual studio code default settings - https://code.visualstudio.com/download
+
+
+then, `crack a conda shell`_ and run the following commands::
+
+    mkdir code # create code directory
+    cd code # change directory to code
+    git clone https://github.com/pikers/piker.git # downloads piker installation package from github
+    cd piker # change directory to piker
+    
+    conda create -n pikonda # creates conda environment named pikonda
+    conda activate pikonda # activates pikonda
+    
+    conda install -c conda-forge python-levenshtein # in case it is not already installed
+    conda install pip # may already be installed
+    pip # will show if pip is installed
+    
+    pip install -e . -r requirements.txt # install piker in editable mode
+
+test Piker to see if it is working::
+
+    piker -b binance chart btcusdt.binance # formatting for loading a chart
+    piker -b kraken -b binance chart xbtusdt.kraken
+    piker -b kraken -b binance -b ib chart qqq.nasdaq.ib
+    piker -b ib chart tsla.nasdaq.ib
+
+potential error::
+    
+    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Roaming\\piker\\brokers.toml'
+    
+solution:
+
+- navigate to file directory above (may be different on your machine, location should be listed in the error code)
+- copy and paste file from 'C:\\Users\\user\\code\\data/brokers.toml' or create a blank file using notepad at the location above
+
+Visual Studio Code setup:
+
+- now that piker is installed we can set up vscode as the default terminal for running piker and editing the code
+- open Visual Studio Code
+- file --> Add Folder to Workspace --> C:\Users\user\code\piker (adds piker directory where all piker files are located)
+- file --> Save Workspace As --> save it wherever you want and call it whatever you want, this is going to be your default workspace for running and editing piker code
+- ctrl + shift + p --> start typing Python: Select Interpetter --> when the option comes up select it --> Select at the workspace level --> select the one that shows ('pikonda')
+- change the default terminal to cmd.exe instead of powershell (default)
+- now when you create a new terminal VScode should automatically activate you conda env so that piker can be run as the first command after a new terminal is created
+
+also, try out fancyzones as part of powertoyz for a decent tiling windows manager to manage all the cool new software you are going to be running.
+
+.. _conda installed: https://
+.. _C++ build toolz: https://
+.. _crack a conda shell: https://
+.. _vscode: https://
+
+.. link to the tina guide
+.. _setup a coolio tiled wm console: https://
+
+provider support
 ****************
-fitting with these tenets, we're always open to new
-framework/lib/service interop suggestions and ideas!
+for live data feeds the in-progress set of supported brokers is:
 
-- **100% federated**:
-  your code, your hardware, your data feeds, your broker fills.
+- IB_ via ``ib_insync``, also see our `container docs`_
+- binance_ and kraken_ for crypto over their public websocket API
+- questrade_ (ish) which comes with effectively free L1
 
-- **zero web**:
-  low latency as a prime objective, native UIs and modern IPC
-  protocols without trying to re-invent the "OS-as-an-app"..
+coming soon...
 
-- **maximal privacy**:
-  prevent brokers and mms from knowing your planz; smack their
-  spreads with dark volume from a VPN tunnel.
+- webull_ via the reverse engineered public API
+- yahoo via yliveticker_
 
-- **zero clutter**:
-  modal, context oriented UIs that echew minimalism, reduce thought
-  noise and encourage un-emotion.
+if you want your broker supported and they have an API let us know.
 
-- **first class parallelism**:
-  built from the ground up on a next-gen structured concurrency
-  supervision sys.
-
-- **traders first**:
-  broker/exchange/venue/asset-class/money-sys agnostic
-
-- **systems grounded**:
-  real-time financial signal processing (fsp) that will make any
-  queuing or DSP eng juice their shorts.
-
-- **non-tina UX**:
-  sleek, powerful keyboard driven interaction with expected use in
-  tiling wms (or maybe even a DDE).
-
-- **data collab at scale**:
-  every actor-process and protocol is multi-host aware.
-
-- **fight club ready**:
-  zero interest in adoption by suits; no corporate friendly license,
-  ever.
-
-building the hottest looking, fastest, most reliable, keyboard
-friendly FOSS trading platform is the dream; join the cause.
+.. _IB: https://interactivebrokers.github.io/tws-api/index.html
+.. _container docs: https://github.com/pikers/piker/tree/master/dockering/ib
+.. _questrade: https://www.questrade.com/api/documentation
+.. _kraken: https://www.kraken.com/features/api#public-market-data
+.. _binance: https://github.com/pikers/piker/pull/182
+.. _webull: https://github.com/tedchou12/webull
+.. _yliveticker: https://github.com/yahoofinancelive/yliveticker
+.. _coinbase: https://docs.pro.coinbase.com/#websocket-feed
 
 
-a sane install with `uv`
-************************
-bc why install with `python` when you can faster with `rust` ::
+check out our charts
+********************
+bet you weren't expecting this from the foss::
 
-    uv lock
+    piker -l info -b kraken -b binance chart btcusdt.binance --pdb
 
 
-hacky install on nixos
-**********************
-``NixOS`` is our core devs' distro of choice for which we offer
-a stringently defined development shell envoirment that can be loaded with::
+this runs the main chart (currently with 1m sampled OHLC) in in debug
+mode and you can practice paper trading using the following
+micro-manual:
 
-    nix-shell default.nix
+``order_mode`` (
+    edge triggered activation by any of the following keys,
+    ``mouse-click`` on y-level to submit at that price
+    ):
+
+    - ``f``/ ``ctl-f`` to stage buy
+    - ``d``/ ``ctl-d`` to stage sell
+    - ``a`` to stage alert
 
 
-start a chart
-*************
-run a realtime OHLCV chart stand-alone::
+``search_mode`` (
+    ``ctl-l`` or ``ctl-space`` to open,
+    ``ctl-c`` or ``ctl-space`` to close
+    ) :
 
-    piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
-
-this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
-overlayed on the same graph. Use of `piker` without first starting
-a daemon (`pikerd` - see below) means there is an implicit spawning of the
-multi-actor-runtime (implemented as a `tractor` app).
-
-For additional subsystem feats available through our chart UI see the
-various sub-readmes:
-
-- order control using a mouse-n-keyboard UX B)
-- cross venue market-pair (what most call "symbol") search, select, overlay Bo
-- financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
-- src-asset derivatives scan for anal, like the infamous "max pain" XO
+    - begin typing to have symbol search automatically lookup
+      symbols from all loaded backend (broker) providers
+    - arrow keys and mouse click to navigate selection
+    - vi-like ``ctl-[hjkl]`` for navigation
 
 
-spawn a daemon standalone
-*************************
-we call the root actor-process the ``pikerd``. it can be (and is
-recommended normally to be) started separately from the ``piker
-chart`` program::
+you can also configure your position allocation limits from the
+sidepane.
+
+
+run in distributed mode
+***********************
+start the service manager and data feed daemon in the background and
+connect to it::
 
     pikerd -l info --pdb
 
-the daemon does nothing until a ``piker``-client (like ``piker
-chart``) connects and requests some particular sub-system. for
-a connecting chart ``pikerd`` will spawn and manage at least,
 
-- a data-feed daemon: ``datad`` which does all the work of comms with
-  the backend provider (in this case the ``binance`` cex).
-- a paper-trading engine instance, ``paperboi.binance``, (if no live
-  account has been configured) which allows for auto/manual order
-  control against the live quote stream.
+connect your chart::
 
-*using* an actor-service (aka micro-daemon) manager which dynamically
-supervises various sub-subsystems-as-services throughout the ``piker``
-runtime-stack.
+    piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
 
-now you can (implicitly) connect your chart::
 
-    piker chart btcusdt.spot.binance
-
-since ``pikerd`` was started separately you can now enjoy a persistent
-real-time data stream tied to the daemon-tree's lifetime. i.e. the next
-time you spawn a chart it will obviously not only load much faster
-(since the underlying ``datad.binance`` is left running with its
-in-memory IPC data structures) but also the data-feed and any order
-mgmt states should be persistent until you finally cancel ``pikerd``.
+enjoy persistent real-time data feeds tied to daemon lifetime. the next
+time you spawn a chart it will load much faster since the data feed has
+been cached and is now always running live in the background until you
+kill ``pikerd``.
 
 
 if anyone asks you what this project is about
 *********************************************
-you don't talk about it; just use it.
+you don't talk about it.
 
 
 how do i get involved?
@@ -165,15 +239,6 @@ enter the matrix.
 
 how come there ain't that many docs
 ***********************************
-i mean we want/need them but building the core right has been higher
-prio then marketting (and likely will stay that way Bp).
-
-soo, suck it up bc,
-
-- no one is trying to sell you on anything
-- learning the code base is prolly way more valuable
-- the UI/UXs are intended to be "intuitive" for any hacker..
-
-we obviously need tonz help so if you want to start somewhere and
-can't necessarily write "advanced" concurrent python/rust code, this
-helping document literally anything might be the place for you!
+suck it up, learn the code; no one is trying to sell you on anything.
+also, we need lotsa help so if you want to start somewhere and can't
+necessarily write serious code, this might be the place for you!

config/brokers.toml

@@ -1,18 +1,10 @@
-################
-# ---- CEXY ----
-################
-[binance]
-accounts.paper = 'paper'
-
-accounts.usdtm = 'futes'
-futes.use_testnet = false
-futes.api_key = ''
-futes.api_secret = ''
-
-accounts.spot = 'spot'
-spot.use_testnet = false
-spot.api_key = ''
-spot.api_secret = ''
+[questrade]
+refresh_token = ''
+access_token = ''
+api_server = 'https://api06.iq.questrade.com/'
+expires_in = 1800
+token_type = 'Bearer'
+expires_at = 1616095326.355846
 
 
 [deribit]
@@ -32,18 +24,6 @@ key_secret = ''
 key_passphrase = ''
 
 
-################
-# -- BROKERZ ---
-################
-[questrade]
-refresh_token = ''
-access_token = ''
-api_server = 'https://api06.iq.questrade.com/'
-expires_in = 1800
-token_type = 'Bearer'
-expires_at = 1616095326.355846
-
-
 [ib]
 hosts = [
     '127.0.0.1',

default.nix

@@ -1,134 +0,0 @@
-with (import <nixpkgs> {});
-let
-  glibStorePath = lib.getLib glib;
-  zlibStorePath = lib.getLib zlib;
-  zstdStorePath = lib.getLib zstd;
-  dbusStorePath = lib.getLib dbus;
-  libGLStorePath = lib.getLib libGL;
-  freetypeStorePath = lib.getLib freetype;
-  qt6baseStorePath = lib.getLib qt6.qtbase;
-  fontconfigStorePath = lib.getLib fontconfig;
-  libxkbcommonStorePath = lib.getLib libxkbcommon;
-  xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
-
-  qtpyStorePath = lib.getLib python312Packages.qtpy;
-  pyqt6StorePath = lib.getLib python312Packages.pyqt6;
-  pyqt6SipStorePath = lib.getLib python312Packages.pyqt6-sip;
-  rapidfuzzStorePath = lib.getLib python312Packages.rapidfuzz;
-  qdarkstyleStorePath = lib.getLib python312Packages.qdarkstyle;
-
-  xorgLibX11StorePath = lib.getLib xorg.libX11;
-  xorgLibxcbStorePath = lib.getLib xorg.libxcb;
-  xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
-  xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
-  xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
-  xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
-  xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
-in
-stdenv.mkDerivation {
-  name = "piker-qt6-uv";
-  buildInputs = [
-    # System requirements.
-    glib
-    zlib
-    dbus
-    zstd
-    libGL
-    freetype
-    qt6.qtbase
-    libgcc.lib
-    fontconfig
-    libxkbcommon
-
-    # Xorg requirements
-    xcb-util-cursor
-    xorg.libxcb
-    xorg.libX11
-    xorg.xcbutilwm
-    xorg.xcbutilimage
-    xorg.xcbutilerrors
-    xorg.xcbutilkeysyms
-    xorg.xcbutilrenderutil
-
-    # Python requirements.
-    python312Full
-    python312Packages.uv
-    python312Packages.qdarkstyle
-    python312Packages.rapidfuzz
-    python312Packages.pyqt6
-    python312Packages.qtpy
-  ];
-  src = null;
-  shellHook = ''
-    set -e
-
-    # Set the Qt plugin path
-    # export QT_DEBUG_PLUGINS=1
-
-    QTBASE_PATH="${qt6baseStorePath}/lib"
-    QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
-    QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
-
-    LIB_GCC_PATH="${libgcc.lib}/lib"
-    GLIB_PATH="${glibStorePath}/lib"
-    ZSTD_PATH="${zstdStorePath}/lib"
-    ZLIB_PATH="${zlibStorePath}/lib"
-    DBUS_PATH="${dbusStorePath}/lib"
-    LIBGL_PATH="${libGLStorePath}/lib"
-    FREETYPE_PATH="${freetypeStorePath}/lib"
-    FONTCONFIG_PATH="${fontconfigStorePath}/lib"
-    LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
-
-    XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
-    XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
-    XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
-    XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
-    XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
-    XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
-    XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
-    XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
-
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
-
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
-
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
-
-    export LD_LIBRARY_PATH
-
-    RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.12/site-packages"
-    QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.12/site-packages"
-    QTPY_PATH="${qtpyStorePath}/lib/python3.12/site-packages"
-    PYQT6_PATH="${pyqt6StorePath}/lib/python3.12/site-packages"
-    PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.12/site-packages"
-
-    PATCH="$PATCH:$RPDFUZZ_PATH"
-    PATCH="$PATCH:$QDRKSTYLE_PATH"
-    PATCH="$PATCH:$QTPY_PATH"
-    PATCH="$PATCH:$PYQT6_PATH"
-    PATCH="$PATCH:$PYQT6_SIP_PATH"
-
-    export PATCH
-
-    # Install deps
-    uv lock
-
-  '';
-}

develop.nix

@@ -1,34 +1,18 @@
 with (import <nixpkgs> {});
-
+with python310Packages;
 stdenv.mkDerivation {
-  name = "poetry-env";
+  name = "pip-env";
   buildInputs = [
     # System requirements.
     readline
 
-    # TODO: hacky non-poetry install stuff we need to get rid of!!
-    poetry
-    # virtualenv
-    # setuptools
-    # pip
-
     # Python requirements (enough to get a virtualenv going).
-    python311Full
-
-    # obviously, and see below for hacked linking
-    python311Packages.pyqt5
-    python311Packages.pyqt5_sip
-    # python311Packages.qtpy
-
-    # numerics deps
-    python311Packages.levenshtein
-    python311Packages.fastparquet
-    python311Packages.polars
-
+    python310Full
+    virtualenv
+    setuptools
+    pyqt5
+    pip
   ];
-  # environment.sessionVariables = {
-  #   LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
-  # };
   src = null;
   shellHook = ''
     # Allow the use of wheels.
@@ -36,12 +20,13 @@ stdenv.mkDerivation {
 
     # Augment the dynamic linker path
     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
+
     export QT_QPA_PLATFORM_PLUGIN_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.version}/plugins";
 
-    if [ ! -d ".venv" ]; then
-        poetry install --with uis
+    if [ ! -d "venv" ]; then
+        virtualenv venv
     fi
 
-    poetry shell
+    source venv/bin/activate
   '';
 }

dockering/ib/docker-compose.yml

@@ -19,9 +19,8 @@ services:
 
     # other image tags available:
     # https://github.com/waytrade/ib-gateway-docker#supported-tags
-    # image: waytrade/ib-gateway:1012.2i
-    image: ghcr.io/gnzsnz/ib-gateway:latest
-
+    # image: waytrade/ib-gateway:981.3j
+    image: waytrade/ib-gateway:1012.2i
     restart: 'no'  # restart on boot whenev there's a crash or user clicsk
     network_mode: 'host'
 

dockering/ib/ibc.ini

@@ -117,57 +117,9 @@ SecondFactorDevice=
 
 # If you use the IBKR Mobile app for second factor authentication,
 # and you fail to complete the process before the time limit imposed
-# by IBKR, this setting tells IBC whether to automatically restart
-# the login sequence, giving you another opportunity to complete
-# second factor authentication. 
-#
-# Permitted values are 'yes' and 'no'.
-#
-# If this setting is not present or has no value, then the value
-# of the deprecated ExitAfterSecondFactorAuthenticationTimeout is
-# used instead. If this also has no value, then this setting defaults
-# to 'no'.
-#
-# NB: you must be using IBC v3.14.0 or later to use this setting:
-# earlier versions ignore it.
-
-ReloginAfterSecondFactorAuthenticationTimeout=
-
-
-# This setting is only relevant if
-# ReloginAfterSecondFactorAuthenticationTimeout is set to 'yes',
-# or if ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
-#
-# It controls how long (in seconds) IBC waits for login to complete
-# after the user acknowledges the second factor authentication
-# alert at the IBKR Mobile app. If login has not completed after
-# this time, IBC terminates.
-# The default value is 60.
-
-SecondFactorAuthenticationExitInterval=
-
-
-# This setting specifies the timeout for second factor authentication
-# imposed by IB. The value is in seconds. You should not change this
-# setting unless you have reason to believe that IB has changed the
-# timeout. The default value is 180.
-
-SecondFactorAuthenticationTimeout=180
-
-
-# DEPRECATED SETTING
-# ------------------
-#
-# ExitAfterSecondFactorAuthenticationTimeout - THIS SETTING WILL BE
-# REMOVED IN A FUTURE RELEASE. For IBC version 3.14.0 and later, see
-# the notes for ReloginAfterSecondFactorAuthenticationTimeout above.
-#
-# For IBC versions earlier than 3.14.0: If you use the IBKR Mobile
-# app for second factor authentication, and you fail to complete the
-# process before the time limit imposed by IBKR, you can use this
-# setting to tell IBC to exit: arrangements can then be made to
-# automatically restart IBC in order to initiate the login sequence
-# afresh. Otherwise, manual intervention at TWS's
+# by IBKR, you can use this setting to tell IBC to exit: arrangements
+# can then be made to automatically restart IBC in order to initiate
+# the login sequence afresh. Otherwise, manual intervention at TWS's
 # Second Factor Authentication dialog is needed to complete the
 # login.
 #
@@ -180,18 +132,29 @@ SecondFactorAuthenticationTimeout=180
 ExitAfterSecondFactorAuthenticationTimeout=no
 
 
+# This setting is only relevant if
+# ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
+#
+# It controls how long (in seconds) IBC waits for login to complete
+# after the user acknowledges the second factor authentication
+# alert at the IBKR Mobile app. If login has not completed after
+# this time, IBC terminates.
+# The default value is 40.
+
+SecondFactorAuthenticationExitInterval=
+
+
 # Trading Mode
 # ------------
 #
-# This indicates whether the live account or the paper trading
-# account corresponding to the supplied credentials is to be used.
-# The allowed values are 'live' (the default) and 'paper'.
-#
-# If this is set to 'live', then the credentials for the live
-# account must be supplied. If it is set to 'paper', then either
-# the live or the paper-trading credentials may be supplied.
+# TWS 955 introduced a new Trading Mode combo box on its login
+# dialog. This indicates whether the live account or the paper
+# trading account corresponding to the supplied credentials is
+# to be used. The allowed values are 'live' (the default) and
+# 'paper'. For earlier versions of TWS this setting has no
+# effect.
 
-TradingMode=paper
+TradingMode=
 
 
 # Paper-trading Account Warning
@@ -225,7 +188,7 @@ AcceptNonBrokerageAccountWarning=yes
 #
 # The default value is 60.
 
-LoginDialogDisplayTimeout=60
+LoginDialogDisplayTimeout=20
 
 
 
@@ -254,15 +217,7 @@ LoginDialogDisplayTimeout=60
 # but they are acceptable.
 #
 # The default is the current working directory when IBC is
-# started, unless the TWS_SETTINGS_PATH setting in the relevant
-# start script is set.
-#
-# If both this setting and TWS_SETTINGS_PATH are set, then this
-# setting takes priority. Note that if they have different values,
-# auto-restart will not work.
-#
-# NB: this setting is now DEPRECATED. You should use the
-# TWS_SETTINGS_PATH setting in the relevant start script.
+# started.
 
 IbDir=/root/Jts
 
@@ -329,32 +284,15 @@ ExistingSessionDetectedAction=primary
 # Override TWS API Port Number
 # ----------------------------
 #
-# If OverrideTwsApiPort is set to an integer, IBC changes the 
-# 'Socket port' in TWS's API configuration to that number shortly 
-# after startup (but note that for the FIX Gateway, this setting is
-# actually stored in jts.ini rather than the Gateway's settings
-# file). Leaving the setting blank will make no change to 
-# the current setting. This setting is only intended for use in 
-# certain specialized situations where the port number needs to 
-# be set dynamically at run-time, and for the FIX Gateway: most
-# non-FIX users will never need it, so don't use it unless you know
-# you need it.
-
-OverrideTwsApiPort=4000
-
-
-# Override TWS Master Client ID
-# -----------------------------
-#
-# If OverrideTwsMasterClientID is set to an integer, IBC changes the
-# 'Master Client ID' value in TWS's API configuration to that 
-# value shortly after startup. Leaving the setting blank will make
-# no change to the current setting. This setting is only intended 
-# for use in certain specialized situations where the value needs to
+# If OverrideTwsApiPort is set to an integer, IBC changes the
+# 'Socket port' in TWS's API configuration to that number shortly
+# after startup. Leaving the setting blank will make no change to
+# the current setting. This setting is only intended for use in
+# certain specialized situations where the port number needs to
 # be set dynamically at run-time: most users will never need it,
 # so don't use it unless you know you need it.
 
-OverrideTwsMasterClientID=
+; OverrideTwsApiPort=4002
 
 
 # Read-only Login
@@ -364,13 +302,11 @@ OverrideTwsMasterClientID=
 # account security programme, the user will not be asked to perform
 # the second factor authentication action, and login to TWS will
 # occur automatically in read-only mode: in this mode, placing or
-# managing orders is not allowed. 
-#
-# If set to 'no', and the user is enrolled in IB's account security
-# programme, the second factor authentication process is handled
-# according to the Second Factor Authentication Settings described
-# elsewhere in this file.
-#
+# managing orders is not allowed. If set to 'no', and the user is
+# enrolled in IB's account security programme, the user must perform
+# the relevant second factor authentication action to complete the
+# login. 
+
 # If the user is not enrolled in IB's account security programme,
 # this setting is ignored. The default is 'no'.
 
@@ -390,44 +326,7 @@ ReadOnlyLogin=no
 # set the relevant checkbox (this only needs to be done once) and
 # not provide a value for this setting.
 
-ReadOnlyApi=
-
-
-# API Precautions
-# ---------------
-# 
-# These settings relate to the corresponding 'Precautions' checkboxes in the
-# API section of the Global Configuration dialog.
-#
-# For all of these, the accepted values are:
-# - 'yes' sets the checkbox
-# - 'no' clears the checkbox
-# - if not set, the existing TWS/Gateway configuration is unchanged
-#
-# NB: thess settings are really only supplied for the benefit of new TWS
-# or Gateway instances that are being automatically installed and
-# started without user intervention, or where user settings are not preserved
-# between sessions (eg some Docker containers). Where a user is involved, they
-# should use the Global Configuration to set the relevant checkboxes and not
-# provide values for these settings.
-
-BypassOrderPrecautions=
-
-BypassBondWarning=
-
-BypassNegativeYieldToWorstConfirmation=
-
-BypassCalledBondWarning=
-
-BypassSameActionPairTradeWarning=
-
-BypassPriceBasedVolatilityRiskWarning=
-
-BypassUSStocksMarketDataInSharesWarning=
-
-BypassRedirectOrderWarning=
-
-BypassNoOverfillProtectionPrecaution=
+ReadOnlyApi=no
 
 
 # Market data size for US stocks - lots or shares
@@ -482,145 +381,54 @@ AcceptBidAskLastSizeDisplayUpdateNotification=accept
 SendMarketDataInLotsForUSstocks=
 
 
-# Trusted API Client IPs
-# ----------------------
-#
-# NB: THIS SETTING IS ONLY RELEVANT FOR THE GATEWAY, AND ONLY WHEN FIX=yes.
-# In all other cases it is ignored.
-#
-# This is a list of IP addresses separated by commas. API clients with IP
-# addresses in this list are able to connect to the API without Gateway
-# generating the 'Incoming connection' popup.
-#
-# Note that 127.0.0.1 is always permitted to connect, so do not include it
-# in this setting.
-
-TrustedTwsApiClientIPs=
-
-
-# Reset Order ID Sequence
-# -----------------------
-#
-# The setting resets the order id sequence for orders submitted via the API, so
-# that the next invocation of the `NextValidId` API callback will return the
-# value 1. The reset occurs when TWS starts.
-#
-# Note that order ids are reset for all API clients, except those that have
-# outstanding (ie incomplete) orders: their order id sequence carries on as
-# before.
-#
-# Valid values are 'yes', 'true', 'false' and 'no'. The default is 'no'.
-
-ResetOrderIdsAtStart=
-
-
-# This setting specifies IBC's action when TWS displays the dialog asking for
-# confirmation of a request to reset the API order id sequence.
-#
-# Note that the Gateway never displays this dialog, so this setting is ignored
-# for a Gateway session.
-#
-# Valid values consist of two strings separated by a solidus '/'. The first
-# value specifies the action to take when the order id reset request resulted
-# from setting ResetOrderIdsAtStart=yes. The second specifies the action to
-# take when the order id reset request is a result of the user clicking the
-# 'Reset API order ID sequence' button in the API configuration. Each value
-# must be one of the following:
-#
-#    'confirm' 
-#        order ids will be reset
-#
-#    'reject' 
-#        order ids will not be reset
-#
-#    'ignore' 
-#        IBC will ignore the dialog. The user must take action.
-#
-#    The default setting is ignore/ignore
-
-# Examples:
-#
-#    'confirm/reject' - confirm order id reset only if ResetOrderIdsAtStart=yes
-#                       and reject any user-initiated requests
-#
-#    'ignore/confirm' - user must decide what to do if ResetOrderIdsAtStart=yes
-#                       and confirm user-initiated requests
-#
-#    'reject/ignore'  - reject order id reset if  ResetOrderIdsAtStart=yes but
-#                       allow user to handle user-initiated requests 
-
-ConfirmOrderIdReset=
-
-
 
 # =============================================================================
-# 4.   TWS Auto-Logoff and Auto-Restart
+# 4.   TWS Auto-Closedown
 # =============================================================================
 #
-# TWS and Gateway insist on being restarted every day. Two alternative
-# automatic options are offered: 
+# IMPORTANT NOTE: Starting with TWS 974, this setting no longer
+# works properly, because IB have changed the way TWS handles its
+# autologoff mechanism. 
 #
-#    - Auto-Logoff: at a specified time, TWS shuts down tidily, without
-#      restarting.
+# You should now configure the TWS autologoff time to something
+# convenient for you, and restart IBC each day.
 #
-#    - Auto-Restart: at a specified time, TWS shuts down and then restarts
-#      without the user having to re-autheticate.
-#
-# The normal way to configure the time at which this happens is via the Lock
-# and Exit section of the Configuration dialog. Once this time has been
-# configured in this way, the setting persists until the user changes it again.
-#
-# However, there are situations where there is no user available to do this
-# configuration, or where there is no persistent storage (for example some
-# Docker images). In such cases, the auto-restart or auto-logoff time can be
-# set whenever IBC starts with the settings below.
-#
-# The value, if specified, must be a time in HH:MM AM/PM format, for example
-# 08:00 AM or 10:00 PM. Note that there must be a single space between the
-# two parts of this value; also that midnight is "12:00 AM" and midday is
-# "12:00 PM".
-#
-# If no value is specified for either setting, the currently configured
-# settings will apply. If a value is supplied for one setting, the other
-# setting is cleared. If values are supplied for both settings, only the
-# auto-restart time is set, and the auto-logoff time is cleared.
-#
-# Note that for a normal TWS/Gateway installation with persistent storage
-# (for example on a desktop computer) the value will be persisted as if the
-# user had set it via the configuration dialog.
-#
-# If you choose to auto-restart, you should take note of the considerations
-# described at the link below. Note that where this information mentions
-# 'manual authentication', restarting IBC will do the job (IBKR does not
-# recognise the existence of IBC in its docuemntation).
-#
-#  https://www.interactivebrokers.com/en/software/tws/twsguide.htm#usersguidebook/configuretws/auto_restart_info.htm
-#
-# If you use the "RESTART" command via the IBC command server, and IBC is
-# running any version of the Gateway (or a version of TWS earlier than 1018),
-# note that this will set the Auto-Restart time in Gateway/TWS's configuration
-# dialog to the time at which the restart actually happens (which may be up to
-# a minute after the RESTART command is issued). To prevent future auto-
-# restarts at this time, you must make sure you have set AutoLogoffTime or
-# AutoRestartTime to your desired value before running IBC. NB: this does not
-# apply to TWS from version 1018 onwards.
+# Alternatively, discontinue use of IBC and use the auto-relogin
+# mechanism within TWS 974 and later versions (note that the 
+# auto-relogin mechanism provided by IB is not available if you
+# use IBC).
 
-AutoLogoffTime=
+# Set to yes or no (lower case).
+#
+#   yes   means allow TWS to shut down automatically at its
+# 	  specified shutdown time, which is set via the TWS
+#	  configuration menu.
+#
+#   no    means TWS never shuts down automatically.
+#
+# NB: IB recommends that you do not keep TWS running
+# continuously. If you set this setting to 'no', you may
+# experience incorrect TWS operation.
+#
+# NB: the default for this setting is 'no'. Since this will
+# only work properly with TWS versions earlier than 974, you
+# should explicitly set this to 'yes' for version 974 and later.
+
+IbAutoClosedown=yes
 
-AutoRestartTime=
 
 
 # =============================================================================
 # 5.   TWS Tidy Closedown Time
 # =============================================================================
 #
-# Specifies a time at which TWS will close down tidily, with no restart.
+# NB: starting with TWS 974 this is no longer a useful option
+# because both TWS and Gateway now have the same auto-logoff
+# mechanism, and IBC can no longer avoid this.
 #
-# There is little reason to use this setting. It is similar to AutoLogoffTime,
-# but can include a day-of-the-week, whereas AutoLogoffTime and AutoRestartTime
-# apply every day. So for example you could use ClosedownAt in conjunction with
-# AutoRestartTime to shut down TWS on Friday evenings after the markets
-# close, without it running on Saturday as well.
+# Note that giving this setting a value does not change TWS's
+# auto-logoff in any way: any setting will be additional to the
+# TWS auto-logoff.
 #
 # To tell IBC to tidily close TWS at a specified time every
 # day, set this value to <hh:mm>, for example:
@@ -679,7 +487,7 @@ AcceptIncomingConnectionAction=reject
 #   no    means the dialog remains on display and must be
 #         handled by the user.
 
-AllowBlindTrading=no
+AllowBlindTrading=yes
 
 
 # Save Settings on a Schedule
@@ -722,26 +530,6 @@ AllowBlindTrading=no
 SaveTwsSettingsAt=
 
 
-# Confirm Crypto Currency Orders Automatically
-# --------------------------------------------
-#
-# When you place an order for a cryptocurrency contract, a dialog is displayed
-# asking you to confirm that you want to place the order, and notifying you
-# that you are placing an order to trade cryptocurrency with Paxos, a New York
-# limited trust company, and not at Interactive Brokers.
-#
-#   transmit    means that the order will be placed automatically, and the
-#               dialog will then be closed
-#
-#   cancel      means that the order will not be placed, and the dialog will
-#               then be closed
-#
-#   manual      means that IBC will take no action and the user must deal
-#               with the dialog
-
-ConfirmCryptoCurrencyOrders=transmit
-
-
 
 # =============================================================================
 # 7.   Settings Specific to Indian Versions of TWS
@@ -778,17 +566,13 @@ DismissNSEComplianceNotice=yes
 #
 # The port number that IBC listens on for commands
 # such as "STOP". DO NOT set this to the port number
-# used for TWS API connections.
-#
-# The convention is to use 7462 for this port,
-# but it must be set to a different value from any other
-# IBC instance that might run at the same time.
-#
-# The default value is 0, which tells IBC not to start
-# the command server
+# used for TWS API connections. There is no good reason
+# to change this setting unless the port is used by
+# some other application (typically another instance of
+# IBC). The default value is 0, which tells IBC not to
+# start the command server
 
 #CommandServerPort=7462
-CommandServerPort=0
 
 
 # Permitted Command Sources
@@ -799,19 +583,19 @@ CommandServerPort=0
 # IBC.  Commands can always be sent from the
 # same host as IBC is running on.
 
-ControlFrom=
+ControlFrom=127.0.0.1
 
 
 # Address for Receiving Commands
 # ------------------------------
 #
 # Specifies the IP address on which the Command Server
-# is to listen. For a multi-homed host, this can be used
+# is so listen. For a multi-homed host, this can be used
 # to specify that connection requests are only to be
 # accepted on the specified address. The default is to
 # accept connection requests on all local addresses.
 
-BindAddress=
+BindAddress=127.0.0.1
 
 
 # Command Prompt
@@ -837,7 +621,7 @@ CommandPrompt=
 # information is sent. The default is that such information
 # is not sent.
 
-SuppressInfoMessages=yes
+SuppressInfoMessages=no
 
 
 
@@ -867,10 +651,10 @@ SuppressInfoMessages=yes
 # The LogStructureScope setting indicates which windows are
 # eligible for structure logging:
 #
-#    - (default value) if set to 'known', only windows that
-#      IBC recognizes are eligible - these are windows that
-#      IBC has some interest in monitoring, usually to take
-#      some action on the user's behalf;
+#    - if set to 'known', only windows that IBC recognizes
+#      are eligible - these are windows that IBC has some
+#      interest in monitoring, usually to take some action
+#      on the user's behalf;
 #
 #    - if set to 'unknown', only windows that IBC does not
 #      recognize are eligible. Most windows displayed by
@@ -883,8 +667,9 @@ SuppressInfoMessages=yes
 #    - if set to 'all', then every window displayed by TWS
 #      is eligible.
 #
+# The default value is 'known'.
 
-LogStructureScope=known
+LogStructureScope=all
 
 
 # When to Log Window Structure
@@ -897,15 +682,13 @@ LogStructureScope=known
 #       structure of an eligible window the first time it
 #       is encountered;
 #
-#     - if set to 'openclose', the structure is logged every
-#       time an eligible window is opened or closed;
-#
 #    - if set to 'activate', the structure is logged every
 #      time an eligible window is made active;
 #
-#    - (default value) if set to 'never' or 'no' or 'false',
-#      structure information is never logged.
+#    - if set to 'never' or 'no' or 'false', structure
+#      information is never logged.
 #
+# The default value is 'never'.
 
 LogStructureWhen=never
 
@@ -925,3 +708,4 @@ LogStructureWhen=never
 #LogComponents=
 
 
+

docs/tina_readme.rst

@@ -1,91 +0,0 @@
-### NOTE this is likely out of date given it was written some
-(years) time ago by a user that has since not really partaken in
-contributing since.
-
-install for tinas
-*****************
-for windows peeps you can start by installing all the prerequisite software:
-
-- install git with all default settings - https://git-scm.com/download/win
-- install anaconda all default settings - https://www.anaconda.com/products/individual
-- install microsoft build tools (check the box for Desktop development for C++, you might be able to uncheck some optional downloads)  - https://visualstudio.microsoft.com/visual-cpp-build-tools/
-- install visual studio code default settings - https://code.visualstudio.com/download
-
-
-then, `crack a conda shell`_ and run the following commands::
-
-    mkdir code # create code directory
-    cd code # change directory to code
-    git clone https://github.com/pikers/piker.git # downloads piker installation package from github
-    cd piker # change directory to piker
-    
-    conda create -n pikonda # creates conda environment named pikonda
-    conda activate pikonda # activates pikonda
-    
-    conda install -c conda-forge python-levenshtein # in case it is not already installed
-    conda install pip # may already be installed
-    pip # will show if pip is installed
-    
-    pip install -e . -r requirements.txt # install piker in editable mode
-
-test Piker to see if it is working::
-
-    piker -b binance chart btcusdt.binance # formatting for loading a chart
-    piker -b kraken -b binance chart xbtusdt.kraken
-    piker -b kraken -b binance -b ib chart qqq.nasdaq.ib
-    piker -b ib chart tsla.nasdaq.ib
-
-potential error::
-    
-    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Roaming\\piker\\brokers.toml'
-    
-solution:
-
-- navigate to file directory above (may be different on your machine, location should be listed in the error code)
-- copy and paste file from 'C:\\Users\\user\\code\\data/brokers.toml' or create a blank file using notepad at the location above
-
-Visual Studio Code setup:
-
-- now that piker is installed we can set up vscode as the default terminal for running piker and editing the code
-- open Visual Studio Code
-- file --> Add Folder to Workspace --> C:\Users\user\code\piker (adds piker directory where all piker files are located)
-- file --> Save Workspace As --> save it wherever you want and call it whatever you want, this is going to be your default workspace for running and editing piker code
-- ctrl + shift + p --> start typing Python: Select Interpetter --> when the option comes up select it --> Select at the workspace level --> select the one that shows ('pikonda')
-- change the default terminal to cmd.exe instead of powershell (default)
-- now when you create a new terminal VScode should automatically activate you conda env so that piker can be run as the first command after a new terminal is created
-
-also, try out fancyzones as part of powertoyz for a decent tiling windows manager to manage all the cool new software you are going to be running.
-
-.. _conda installed: https://
-.. _C++ build toolz: https://
-.. _crack a conda shell: https://
-.. _vscode: https://
-
-.. link to the tina guide
-.. _setup a coolio tiled wm console: https://
-
-provider support
-****************
-for live data feeds the in-progress set of supported brokers is:
-
-- IB_ via ``ib_insync``, also see our `container docs`_
-- binance_ and kraken_ for crypto over their public websocket API
-- questrade_ (ish) which comes with effectively free L1
-
-coming soon...
-
-- webull_ via the reverse engineered public API
-- yahoo via yliveticker_
-
-if you want your broker supported and they have an API let us know.
-
-.. _IB: https://interactivebrokers.github.io/tws-api/index.html
-.. _container docs: https://github.com/pikers/piker/tree/master/dockering/ib
-.. _questrade: https://www.questrade.com/api/documentation
-.. _kraken: https://www.kraken.com/features/api#public-market-data
-.. _binance: https://github.com/pikers/piker/pull/182
-.. _webull: https://github.com/tedchou12/webull
-.. _yliveticker: https://github.com/yahoofinancelive/yliveticker
-.. _coinbase: https://docs.pro.coinbase.com/#websocket-feed
-
-

examples/basic_order_bot.py

@@ -1,263 +0,0 @@
-# from pprint import pformat
-from functools import partial
-from decimal import Decimal
-from typing import Callable
-
-import tractor
-import trio
-from uuid import uuid4
-
-from piker.service import maybe_open_pikerd
-from piker.accounting import dec_digits
-from piker.clearing import (
-    open_ems,
-    OrderClient,
-)
-# TODO: we should probably expose these top level in this subsys?
-from piker.clearing._messages import (
-    Order,
-    Status,
-    BrokerdPosition,
-)
-from piker.data import (
-    iterticks,
-    Flume,
-    open_feed,
-    Feed,
-    # ShmArray,
-)
-
-
-# TODO: handle other statuses:
-# - fills, errors, and position tracking
-async def wait_for_order_status(
-    trades_stream: tractor.MsgStream,
-    oid: str,
-    expect_status: str,
-
-) -> tuple[
-    list[Status],
-    list[BrokerdPosition],
-]:
-    '''
-    Wait for a specific order status for a given dialog, return msg flow
-    up to that msg and any position update msgs in a tuple.
-
-    '''
-    # Wait for position message before moving on to verify flow(s)
-    # for the multi-order position entry/exit.
-    status_msgs: list[Status] = []
-    pp_msgs: list[BrokerdPosition] = []
-
-    async for msg in trades_stream:
-        match msg:
-            case {'name': 'position'}:
-                ppmsg = BrokerdPosition(**msg)
-                pp_msgs.append(ppmsg)
-
-            case {
-                'name': 'status',
-            }:
-                msg = Status(**msg)
-                status_msgs.append(msg)
-
-                # if we get the status we expect then return all
-                # collected msgs from the brokerd dialog up to the
-                # exected msg B)
-                if (
-                     msg.resp == expect_status
-                     and msg.oid == oid
-                ):
-                    return status_msgs, pp_msgs
-
-
-async def bot_main():
-    '''
-    Boot the piker runtime, open an ems connection, submit
-    and process orders statuses in real-time.
-
-    '''
-    ll: str = 'info'
-
-    # open an order ctl client, live data feed, trio nursery for
-    # spawning an order trailer task
-    client: OrderClient
-    trades_stream: tractor.MsgStream
-    feed: Feed
-    accounts: list[str]
-
-    fqme: str = 'btcusdt.usdtm.perp.binance'
-
-    async with (
-
-        # TODO: do this implicitly inside `open_ems()` ep below?
-        # init and sync actor-service runtime
-        maybe_open_pikerd(
-            loglevel=ll,
-            debug_mode=True,
-
-        ),
-        open_ems(
-            fqme,
-            mode='paper',  # {'live', 'paper'}
-            # mode='live',  # for real-brokerd submissions
-            loglevel=ll,
-
-        ) as (
-            client,  # OrderClient
-            trades_stream,  # tractor.MsgStream startup_pps,
-            _,  # positions
-            accounts,
-            _,  # dialogs
-        ),
-
-        open_feed(
-            fqmes=[fqme],
-            loglevel=ll,
-
-            # TODO: if you want to throttle via downsampling
-            # how many tick updates your feed received on
-            # quote streams B)
-            # tick_throttle=10,
-        ) as feed,
-
-        trio.open_nursery() as tn,
-    ):
-        assert accounts
-        print(f'Loaded binance accounts: {accounts}')
-
-        flume: Flume = feed.flumes[fqme]
-        min_tick = Decimal(flume.mkt.price_tick)
-        min_tick_digits: int = dec_digits(min_tick)
-        price_round: Callable = partial(
-            round,
-            ndigits=min_tick_digits,
-        )
-
-        quote_stream: trio.abc.ReceiveChannel = feed.streams['binance']
-
-
-        # always keep live limit 0.003% below last
-        # clearing price
-        clear_margin: float = 0.9997
-
-        async def trailer(
-            order: Order,
-        ):
-            # ref shm OHLCV array history, if you want
-            # s_shm: ShmArray = flume.rt_shm
-            # m_shm: ShmArray = flume.hist_shm
-
-            # NOTE: if you wanted to frame ticks by type like the
-            # the quote throttler does.. and this is probably
-            # faster in terms of getting the latest tick type
-            # embedded value of interest?
-            # from piker.data._sampling import frame_ticks
-
-            async for quotes in quote_stream:
-                for fqme, quote in quotes.items():
-                    # print(
-                    #     f'{quote["symbol"]} -> {quote["ticks"]}\n'
-                    #     f'last 1s OHLC:\n{s_shm.array[-1]}\n'
-                    #     f'last 1m OHLC:\n{m_shm.array[-1]}\n'
-                    # )
-
-                    for tick in iterticks(
-                        quote,
-                        reverse=True,
-                        # types=('trade', 'dark_trade'), # defaults
-                    ):
-
-                        await client.update(
-                            uuid=order.oid,
-                            price=price_round(
-                                clear_margin
-                                *
-                                tick['price']
-                            ),
-                        )
-                        msgs, pps = await wait_for_order_status(
-                            trades_stream,
-                            order.oid,
-                            'open'
-                        )
-                        # if multiple clears per quote just
-                        # skip to the next quote?
-                        break
-
-
-        # get first live quote to be sure we submit the initial
-        # live buy limit low enough that it doesn't clear due to
-        # a stale initial price from the data feed layer!
-        first_ask_price: float | None = None
-        async for quotes in quote_stream:
-            for fqme, quote in quotes.items():
-                # print(quote['symbol'])
-                for tick in iterticks(quote, types=('ask')):
-                    first_ask_price: float = tick['price']
-                    break
-
-            if first_ask_price:
-                break
-
-        # setup order dialog via first msg
-        price: float = price_round(
-                clear_margin
-                *
-                first_ask_price,
-        )
-
-        # compute a 1k USD sized pos
-        size: float = round(1e3/price, ndigits=3)
-
-        order = Order(
-
-            # docs on how this all works, bc even i'm not entirely
-            # clear XD. also we probably want to figure  out how to
-            # offer both the paper engine running and the brokerd
-            # order ctl tasks with the ems choosing which stream to
-            # route msgs on given the account value!
-            account='paper',  # use built-in paper clearing engine and .accounting
-            # account='binance.usdtm',  # for live binance futes
-
-            oid=str(uuid4()),
-            exec_mode='live',  # {'dark', 'live', 'alert'}
-
-            action='buy',  # TODO: remove this from our schema?
-
-            size=size,
-            symbol=fqme,
-            price=price,
-            brokers=['binance'],
-        )
-        await client.send(order)
-
-        msgs, pps = await wait_for_order_status(
-            trades_stream,
-            order.oid,
-            'open',
-        )
-
-        assert not pps
-        assert msgs[-1].oid == order.oid
-
-        # start "trailer task" which tracks rt quote stream
-        tn.start_soon(trailer, order)
-
-        try:
-            # wait for ctl-c from user..
-            await trio.sleep_forever()
-        except KeyboardInterrupt:
-            # cancel the open order
-            await client.cancel(order.oid)
-
-            msgs, pps = await wait_for_order_status(
-                trades_stream,
-                order.oid,
-                'canceled'
-            )
-            raise
-
-
-if __name__ == '__main__':
-    trio.run(bot_main)

flake.nix

@@ -6,11 +6,6 @@
 # - then manually ensuring all deps are converted over:
 # - add this file to the repo and commit it
 # - 
-
-# GROKin tips:
-# - CLI eps are (ostensibly) added via an `entry_points.txt`:
-#   - https://packaging.python.org/en/latest/specifications/entry-points/#file-format
-#   - https://github.com/nix-community/poetry2nix/blob/master/editable.nix#L49
 {
   description = "piker: trading gear for hackers (pkged with poetry2nix)";
 
@@ -106,7 +101,7 @@
             # won't be needed - thanks @k900:
             # https://github.com/nix-community/poetry2nix/pull/1257
             pyqt5 = prev.pyqt5.override {
-              # withWebkit = false;
+              withWebkit = false;
               preferWheel = true;
             };
 
@@ -129,52 +124,55 @@
 
       # WHY!? -> output-attrs that `nix develop` scans for:
       # https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html#flake-output-attributes
-      in
-      rec {
-        packages = {
-          # piker = poetry2nix.legacyPackages.x86_64-linux.mkPoetryEditablePackage {
-          #   editablePackageSources = { piker = ./piker; };
+      in {
+          packages = {
+            # piker = poetry2nix.legacyPackages.x86_64-linux.mkPoetryEditablePackage {
+            #   editablePackageSources = { piker = ./piker; };
 
-          piker = p2npkgs.mkPoetryApplication {
-            projectDir = projectDir;
+            piker = p2npkgs.mkPoetryApplication {
+              projectDir = projectDir;
 
-            # SEE ABOVE for auto-genned input set, override
-            # buncha deps with extras.. like `setuptools` mostly.
-            # TODO: maybe propose a patch to p2n to show that you
-            # can even do this in the edgecases docs?
-            overrides = ahot_overrides;
+              # SEE ABOVE for auto-genned input set, override
+              # buncha deps with extras.. like `setuptools` mostly.
+              # TODO: maybe propose a patch to p2n to show that you
+              # can even do this in the edgecases docs?
+              overrides = ahot_overrides;
 
-            # XXX: won't work on llvmlite..
-            # preferWheels = true;
+              # XXX: won't work on llvmlite..
+              # preferWheels = true;
+            };
           };
-        };
 
-        # devShells.default = pkgs.mkShell {
-        #   projectDir = projectDir;
-        #   python = "python3.10";
-        #   overrides = ahot_overrides;
-        #   inputsFrom = [ self.packages.x86_64-linux.piker ];
-        #   packages = packages;
-        #   # packages = [ poetry2nix.packages.${system}.poetry ];
-        # };
+          devShells.default = pkgs.mkShell {
+            # packages = [ poetry2nix.packages.${system}.poetry ];
+            packages = [ poetry2nix.packages.x86_64-linux.poetry ];
+            inputsFrom = [ self.packages.x86_64-linux.piker ];
 
-        # TODO: grok the difference here..
-        # - avoid re-cloning git repos on every develop entry..
-        # - ideally allow hacking on the src code of some deps
-        #   (tractor, pyqtgraph, tomlkit, etc.) WITHOUT having to
-        #   re-install them every time a change is made.
-        # - boot a usable xonsh inside the poetry virtualenv when
-        #   defined via a custom entry point?
-        devShells.default = p2npkgs.mkPoetryEnv {
-        # env = p2npkgs.mkPoetryEnv {
-            projectDir = projectDir;
-            python = pkgs.python310;
-            overrides = ahot_overrides;
-            editablePackageSources = packages;
-              # piker = "./";
-              # tractor = "../tractor/";
-            # };  # wut?
-        };
-      }
+            # TODO: boot xonsh inside the poetry virtualenv when
+            # defined via a custom entry point?
+            # NOTE XXX: apparently DON'T do these..?
+            # shellHook = "poetry run xonsh";
+            # shellHook = "poetry shell";
+          };
+
+
+          # TODO: grok the difference here..
+          # - avoid re-cloning git repos on every develop entry..
+          # - ideally allow hacking on the src code of some deps
+          #   (tractor, pyqtgraph, tomlkit, etc.) WITHOUT having to
+          #   re-install them every time a change is made.
+
+          # devShells.default = (p2npkgs.mkPoetryEnv {
+          # # let {
+          # #   devEnv = p2npkgs.mkPoetryEnv {
+          #     projectDir = projectDir;
+          #     overrides = ahot_overrides;
+          #     inputsFrom = [ self.packages.x86_64-linux.piker ];
+          #   }).env.overrideAttrs (old: {
+          #       buildInputs = [ packages.piker ];
+          #     }
+          #   );
+
+        }
     );  # end of .outputs scope
 }

piker/_cacheables.py

@@ -20,6 +20,9 @@ Cacheing apis and toolz.
 '''
 
 from collections import OrderedDict
+from contextlib import (
+    asynccontextmanager as acm,
+)
 from typing import (
     Awaitable,
     Callable,
@@ -27,8 +30,12 @@ from typing import (
     TypeVar,
 )
 
+from tractor.trionics import maybe_open_context
+
+from .brokers import get_brokermod
 from .log import get_logger
 
+
 log = get_logger(__name__)
 
 T = TypeVar("T")
@@ -97,3 +104,21 @@ def async_lifo_cache(
         return decorated
 
     return decorator
+
+
+# TODO: move this to `.brokers.utils`..
+@acm
+async def open_cached_client(
+    brokername: str,
+) -> 'Client':  # noqa
+    '''
+    Get a cached broker client from the current actor's local vars.
+
+    If one has not been setup do it and cache it.
+
+    '''
+    brokermod = get_brokermod(brokername)
+    async with maybe_open_context(
+        acm_func=brokermod.get_client,
+    ) as (cache_hit, client):
+        yield client

piker/accounting/README.rst

@@ -1,16 +0,0 @@
-.accounting
------------
-A subsystem for transaction processing, storage and historical
-measurement.
-
-
-.pnl
-----
-BEP, the break even price: the price at which liquidating
-a remaining position results in a zero PnL since the position was
-"opened" in the destination asset.
-
-PPU: price-per-unit: the "average cost" (in cumulative mean terms)
-of the "entry" transactions which "make a position larger"; taking
-a profit relative to this price means that you will "make more
-profit then made prior" since the position was opened.

piker/accounting/__init__.py

@@ -21,21 +21,17 @@ for tendiez.
 '''
 from ..log import get_logger
 
-from .calc import (
-    iter_by_dt,
-)
 from ._ledger import (
+    iter_by_dt,
     Transaction,
     TransactionLedger,
     open_trade_ledger,
 )
 from ._pos import (
-    Account,
-    load_account,
-    load_account_from_ledger,
+    load_pps_from_ledger,
     open_pps,
-    open_account,
     Position,
+    PpTable,
 )
 from ._mktinfo import (
     Asset,
@@ -44,36 +40,31 @@ from ._mktinfo import (
     MktPair,
     Symbol,
     unpack_fqme,
-    _derivs as DerivTypes,
 )
 from ._allocate import (
     mk_allocator,
     Allocator,
 )
 
-
 log = get_logger(__name__)
 
 __all__ = [
-    'Account',
     'Allocator',
     'Asset',
     'MktPair',
     'Position',
+    'PpTable',
     'Symbol',
     'Transaction',
     'TransactionLedger',
     'dec_digits',
     'digits_to_dec',
     'iter_by_dt',
-    'load_account',
-    'load_account_from_ledger',
+    'load_pps_from_ledger',
     'mk_allocator',
-    'open_account',
     'open_pps',
     'open_trade_ledger',
     'unpack_fqme',
-    'DerivTypes',
 ]
 
 
@@ -89,7 +80,7 @@ def get_likely_pair(
 
     '''
     try:
-        src_name_start: str = bs_mktid.rindex(src)
+        src_name_start = bs_mktid.rindex(src)
     except (
         ValueError,   # substr not found
     ):
@@ -100,8 +91,25 @@ def get_likely_pair(
         # log.warning(
         #     f'No src fiat {src} found in {bs_mktid}?'
         # )
-        return None
+        return
 
-    likely_dst: str = bs_mktid[:src_name_start]
+    likely_dst = bs_mktid[:src_name_start]
     if likely_dst == dst:
         return bs_mktid
+
+
+if __name__ == '__main__':
+    import sys
+    from pprint import pformat
+
+    args = sys.argv
+    assert len(args) > 1, 'Specifiy account(s) from `brokers.toml`'
+    args = args[1:]
+    for acctid in args:
+        broker, name = acctid.split('.')
+        trans, updated_pps = load_pps_from_ledger(broker, name)
+        print(
+            f'Processing transactions into pps for {broker}:{acctid}\n'
+            f'{pformat(trans)}\n\n'
+            f'{pformat(updated_pps)}'
+        )

piker/accounting/_allocate.py

@@ -25,7 +25,7 @@ from bidict import bidict
 
 from ._pos import Position
 from . import MktPair
-from piker.types import Struct
+from ..data.types import Struct
 
 
 _size_units = bidict({
@@ -118,9 +118,9 @@ class Allocator(Struct):
         ld: int = mkt.size_tick_digits
 
         size_unit = self.size_unit
-        live_size = live_pp.cumsize
+        live_size = live_pp.size
         abs_live_size = abs(live_size)
-        abs_startup_size = abs(startup_pp.cumsize)
+        abs_startup_size = abs(startup_pp.size)
 
         u_per_slot, currency_per_slot = self.step_sizes()
 
@@ -213,6 +213,8 @@ class Allocator(Struct):
             slots_used = self.slots_used(
                 Position(
                     mkt=mkt,
+                    size=order_size,
+                    ppu=price,
                     bs_mktid=mkt.bs_mktid,
                 )
             )
@@ -239,7 +241,7 @@ class Allocator(Struct):
         Calc and return the number of slots used by this ``Position``.
 
         '''
-        abs_pp_size = abs(pp.cumsize)
+        abs_pp_size = abs(pp.size)
 
         if self.size_unit == 'currency':
             # live_currency_size = size or (abs_pp_size * pp.ppu)

piker/accounting/_ledger.py

@@ -21,77 +21,65 @@ Trade and transaction ledger processing.
 from __future__ import annotations
 from collections import UserDict
 from contextlib import contextmanager as cm
-from functools import partial
 from pathlib import Path
-from pprint import pformat
-from types import ModuleType
 from typing import (
     Any,
     Callable,
-    Generator,
-    Literal,
-    TYPE_CHECKING,
+    Iterator,
+    Union,
+    Generator
 )
 
 from pendulum import (
+    datetime,
     DateTime,
+    from_timestamp,
+    parse,
 )
 import tomli_w  # for fast ledger writing
 
-from piker.types import Struct
-from piker import config
+from .. import config
+from ..data.types import Struct
 from ..log import get_logger
-from .calc import (
-    iter_by_dt,
+from ._mktinfo import (
+    Symbol,  # legacy
+    MktPair,
+    Asset,
 )
 
-if TYPE_CHECKING:
-    from ..data._symcache import (
-        SymbologyCache,
-    )
-
 log = get_logger(__name__)
 
 
-TxnType = Literal[
-    'clear',
-    'transfer',
-
-    # TODO: see https://github.com/pikers/piker/issues/510
-    # 'split',
-    # 'rename',
-    # 'resize',
-    # 'removal',
-]
-
-
 class Transaction(Struct, frozen=True):
 
-    # NOTE: this is a unified acronym also used in our `MktPair`
-    # and can stand for any of a
-    # "fully qualified <blank> endpoint":
-    # - "market" in the case of financial trades
-    #   (btcusdt.spot.binance).
-    # - "merkel (tree)" aka a blockchain system "wallet tranfers"
-    #   (btc.blockchain)
-    # - "money" for tradtitional (digital databases)
-    #   *bank accounts* (usd.swift, eur.sepa)
+    # TODO: unify this with the `MktPair`,
+    # once we have that as a required field,
+    # we don't really need the fqme any more..
     fqme: str
 
-    tid: str | int  # unique transaction id
+    tid: Union[str, int]  # unique transaction id
     size: float
     price: float
     cost: float  # commisions or other additional costs
-    dt: DateTime
-
-    # the "event type" in terms of "market events" see above and
-    # https://github.com/pikers/piker/issues/510
-    etype: TxnType = 'clear'
+    dt: datetime
 
     # TODO: we can drop this right since we
     # can instead expect the backend to provide this
     # via the `MktPair`?
-    expiry: DateTime | None = None
+    expiry: datetime | None = None
+
+    # TODO: drop the Symbol type, construct using
+    # t.sys (the transaction system)
+
+    # the underlying "transaction system", normally one of a ``MktPair``
+    # (a description of a tradable double auction) or a ledger-recorded
+    # ("ledger" in any sense as long as you can record transfers) of any
+    # sort) ``Asset``.
+    sym: MktPair | Asset | Symbol | None = None
+
+    @property
+    def sys(self) -> Symbol:
+        return self.sym
 
     # (optional) key-id defined by the broker-service backend which
     # ensures the instrument-symbol market key for this record is unique
@@ -100,16 +88,15 @@ class Transaction(Struct, frozen=True):
     # service.
     bs_mktid: str | int | None = None
 
-    def to_dict(
-        self,
-        **kwargs,
-    ) -> dict:
-        dct: dict[str, Any] = super().to_dict(**kwargs)
+    def to_dict(self) -> dict:
+        dct = super().to_dict()
+
+        # TODO: switch to sys!
+        dct.pop('sym')
 
         # ensure we use a pendulum formatted
         # ISO style str here!@
         dct['dt'] = str(self.dt)
-
         return dct
 
 
@@ -121,63 +108,30 @@ class TransactionLedger(UserDict):
     outside.
 
     '''
-    # NOTE: see `open_trade_ledger()` for defaults, this should
-    # never be constructed manually!
     def __init__(
         self,
         ledger_dict: dict,
         file_path: Path,
-        account: str,
-        mod: ModuleType,  # broker mod
         tx_sort: Callable,
-        symcache: SymbologyCache,
 
     ) -> None:
-        self.account: str = account
-        self.file_path: Path = file_path
-        self.mod: ModuleType = mod
-        self.tx_sort: Callable = tx_sort
-
-        self._symcache: SymbologyCache = symcache
-
-        # any added txns we keep in that form for meta-data
-        # gathering purposes
-        self._txns: dict[str, Transaction] = {}
-
+        self.file_path = file_path
+        self.tx_sort = tx_sort
         super().__init__(ledger_dict)
 
-    def __repr__(self) -> str:
-        return (
-            f'TransactionLedger: {len(self)}\n'
-            f'{pformat(list(self.data))}'
-        )
-
-    @property
-    def symcache(self) -> SymbologyCache:
-        '''
-        Read-only ref to backend's ``SymbologyCache``.
-
-        '''
-        return self._symcache
-
     def update_from_t(
         self,
         t: Transaction,
     ) -> None:
-        '''
-        Given an input `Transaction`, cast to `dict` and update
-        from it's transaction id.
-
-        '''
         self.data[t.tid] = t.to_dict()
-        self._txns[t.tid] = t
 
-    def iter_txns(
+    def iter_trans(
         self,
-        symcache: SymbologyCache | None = None,
+        mkt_by_fqme: dict[str, MktPair],
+        broker: str = 'paper',
 
     ) -> Generator[
-        Transaction,
+        tuple[str, Transaction],
         None,
         None,
     ]:
@@ -186,162 +140,123 @@ class TransactionLedger(UserDict):
         form via generator.
 
         '''
-        symcache = symcache or self._symcache
+        if broker != 'paper':
+            raise NotImplementedError('Per broker support not dun yet!')
 
-        if self.account == 'paper':
-            from piker.clearing import _paper_engine
-            norm_trade: Callable = partial(
-                _paper_engine.norm_trade,
-                brokermod=self.mod,
+            # TODO: lookup some standard normalizer
+            # func in the backend?
+            # from ..brokers import get_brokermod
+            # mod = get_brokermod(broker)
+            # trans_dict = mod.norm_trade_records(self.data)
+
+            # NOTE: instead i propose the normalizer is
+            # a one shot routine (that can be lru cached)
+            # and instead call it for each entry incrementally:
+            # normer = mod.norm_trade_record(txdict)
+
+        # TODO: use tx_sort here yah?
+        for tid, txdict in self.data.items():
+            # special field handling for datetimes
+            # to ensure pendulum is used!
+            fqme = txdict.get('fqme') or txdict['fqsn']
+            dt = parse(txdict['dt'])
+            expiry = txdict.get('expiry')
+
+            mkt = mkt_by_fqme.get(fqme)
+            if not mkt:
+                # we can't build a trans if we don't have
+                # the ``.sys: MktPair`` info, so skip.
+                continue
+
+            tx = Transaction(
+                fqme=fqme,
+                tid=txdict['tid'],
+                dt=dt,
+                price=txdict['price'],
+                size=txdict['size'],
+                cost=txdict.get('cost', 0),
+                bs_mktid=txdict['bs_mktid'],
+
+                # TODO: change to .sys!
+                sym=mkt,
+                expiry=parse(expiry) if expiry else None,
             )
+            yield tid, tx
 
-        else:
-            norm_trade: Callable = self.mod.norm_trade
-
-        # datetime-sort and pack into txs
-        for tid, txdict in self.tx_sort(self.data.items()):
-            txn: Transaction = norm_trade(
-                tid,
-                txdict,
-                pairs=symcache.pairs,
-                symcache=symcache,
-            )
-            yield txn
-
-    def to_txns(
+    def to_trans(
         self,
-        symcache: SymbologyCache | None = None,
+        **kwargs,
 
     ) -> dict[str, Transaction]:
         '''
-        Return entire output from ``.iter_txns()`` in a ``dict``.
+        Return entire output from ``.iter_trans()`` in a ``dict``.
 
         '''
-        txns: dict[str, Transaction] = {}
-        for t in self.iter_txns(symcache=symcache):
+        return dict(self.iter_trans(**kwargs))
 
-            if not t:
-                log.warning(f'{self.mod.name}:{self.account} TXN is -> {t}')
-                continue
+    def write_config(
+        self,
 
-            txns[t.tid] = t
-
-        return txns
-
-    def write_config(self) -> None:
+    ) -> None:
         '''
-        Render the self.data ledger dict to its TOML file form.
-
-        ALWAYS order datetime sorted!
+        Render the self.data ledger dict to it's TOML file form.
 
         '''
-        is_paper: bool = self.account == 'paper'
-
-        symcache: SymbologyCache = self._symcache
+        cpy = self.data.copy()
         towrite: dict[str, Any] = {}
-        for tid, txdict in self.tx_sort(self.data.copy()):
-            # write blank-str expiry for non-expiring assets
+        for tid, trans in cpy.items():
+
+            # drop key for non-expiring assets
+            txdict = towrite[tid] = self.data[tid]
             if (
                 'expiry' in txdict
                 and txdict['expiry'] is None
             ):
-                txdict['expiry'] = ''
+                txdict.pop('expiry')
 
-            # (maybe) re-write old acro-key
-            if (
-                is_paper
-                # if symcache is empty/not supported (yet), don't
-                # bother xD
-                and symcache.mktmaps
-            ):
-                fqme: str = txdict.pop('fqsn', None) or txdict['fqme']
-                bs_mktid: str | None = txdict.get('bs_mktid')
-
-                if (
-
-                    fqme not in symcache.mktmaps
-                    or (
-                        # also try to see if this is maybe a paper
-                        # engine ledger in which case the bs_mktid
-                        # should be the fqme as well!
-                        bs_mktid
-                        and fqme != bs_mktid
-                    )
-                ):
-                    # always take any (paper) bs_mktid if defined and
-                    # in the backend's cache key set.
-                    if bs_mktid in symcache.mktmaps:
-                        fqme: str = bs_mktid
-                    else:
-                        best_fqme: str = list(symcache.search(fqme))[0]
-                        log.warning(
-                            f'Could not find FQME: {fqme} in qualified set?\n'
-                            f'Qualifying and expanding {fqme} -> {best_fqme}'
-                        )
-                        fqme = best_fqme
-
-                if (
-                    bs_mktid
-                    and bs_mktid != fqme
-                ):
-                    # in paper account case always make sure both the
-                    # fqme and bs_mktid are fully qualified..
-                    txdict['bs_mktid'] = fqme
-
-                # in paper ledgers always write the latest
-                # symbology key field: an FQME.
+            # re-write old acro-key
+            fqme = txdict.get('fqsn')
+            if fqme:
                 txdict['fqme'] = fqme
 
-            towrite[tid] = txdict
-
         with self.file_path.open(mode='wb') as fp:
             tomli_w.dump(towrite, fp)
 
 
-def load_ledger(
-    brokername: str,
-    acctid: str,
+def iter_by_dt(
+    records: dict[str, Any],
 
-    # for testing or manual load from file
-    dirpath: Path | None = None,
+    # NOTE: parsers are looked up in the insert order
+    # so if you know that the record stats show some field
+    # is more common then others, stick it at the top B)
+    parsers: dict[tuple[str], Callable] = {
+        'dt': None,  # parity case
+        'datetime': parse,  # datetime-str
+        'time': from_timestamp,  # float epoch
+    },
+    key: Callable | None = None,
 
-) -> tuple[dict, Path]:
+) -> Iterator[tuple[str, dict]]:
     '''
-    Load a ledger (TOML) file from user's config directory:
-    $CONFIG_DIR/accounting/ledgers/trades_<brokername>_<acctid>.toml
-
-    Return its `dict`-content and file path.
+    Iterate entries of a ``records: dict`` table sorted by entry recorded
+    datetime presumably set at the ``'dt'`` field in each entry.
 
     '''
-    import time
-    try:
-        import tomllib
-    except ModuleNotFoundError:
-        import tomli as tomllib
-
-    ldir: Path = (
-        dirpath
-        or
-        config._config_dir / 'accounting' / 'ledgers'
-    )
-    if not ldir.is_dir():
-        ldir.mkdir()
-
-    fname = f'trades_{brokername}_{acctid}.toml'
-    fpath: Path = ldir / fname
-
-    if not fpath.is_file():
-        log.info(
-            f'Creating new local trades ledger: {fpath}'
+    def dyn_parse_to_dt(
+        pair: tuple[str, dict],
+    ) -> DateTime:
+        _, txdict = pair
+        k, v, parser = next(
+            (k, txdict[k], parsers[k]) for k in parsers if k in txdict
         )
-        fpath.touch()
 
-    with fpath.open(mode='rb') as cf:
-        start = time.time()
-        ledger_dict = tomllib.load(cf)
-        log.debug(f'Ledger load took {time.time() - start}s')
+        return parser(v) if parser else v
 
-    return ledger_dict, fpath
+    for tid, data in sorted(
+        records.items(),
+        key=key or dyn_parse_to_dt,
+    ):
+        yield tid, data
 
 
 @cm
@@ -349,17 +264,10 @@ def open_trade_ledger(
     broker: str,
     account: str,
 
-    allow_from_sync_code: bool = False,
-    symcache: SymbologyCache | None = None,
-
     # default is to sort by detected datetime-ish field
     tx_sort: Callable = iter_by_dt,
-    rewrite: bool = False,
 
-    # for testing or manual load from file
-    _fp: Path | None = None,
-
-) -> Generator[TransactionLedger, None, None]:
+) -> Generator[dict, None, None]:
     '''
     Indempotently create and read in a trade log file from the
     ``<configuration_dir>/ledgers/`` directory.
@@ -369,52 +277,18 @@ def open_trade_ledger(
     name as defined in the user's ``brokers.toml`` config.
 
     '''
-    from ..brokers import get_brokermod
-    mod: ModuleType = get_brokermod(broker)
-
-    ledger_dict, fpath = load_ledger(
-        broker,
-        account,
-        dirpath=_fp,
-    )
+    ledger_dict, fpath = config.load_ledger(broker, account)
     cpy = ledger_dict.copy()
-
-    # XXX NOTE: if not provided presume we are being called from
-    # sync code and need to maybe run `trio` to generate..
-    if symcache is None:
-
-        # XXX: be mega pendantic and ensure the caller knows what
-        # they're doing!
-        if not allow_from_sync_code:
-            raise RuntimeError(
-                'You MUST set `allow_from_sync_code=True` when '
-                'calling `open_trade_ledger()` from sync code! '
-                'If you are calling from async code you MUST '
-                'instead pass a `symcache: SymbologyCache`!'
-            )
-
-        from ..data._symcache import (
-            get_symcache,
-        )
-        symcache: SymbologyCache = get_symcache(broker)
-
-    assert symcache
-
     ledger = TransactionLedger(
         ledger_dict=cpy,
         file_path=fpath,
-        account=account,
-        mod=mod,
-        symcache=symcache,
-        tx_sort=getattr(mod, 'tx_sort', tx_sort),
+        tx_sort=tx_sort,
     )
     try:
         yield ledger
     finally:
-        if (
-            ledger.data != ledger_dict
-            or rewrite
-        ):
+        if ledger.data != ledger_dict:
+
             # TODO: show diff output?
             # https://stackoverflow.com/questions/12956957/print-diff-of-python-dictionaries
             log.info(f'Updating ledger for {fpath}:\n')

piker/accounting/_mktinfo.py

@@ -36,10 +36,9 @@ from typing import (
     Literal,
 )
 
-from piker.types import Struct
+from ..data.types import Struct
 
 
-# TODO: make these literals..
 _underlyings: list[str] = [
     'stock',
     'bond',
@@ -48,10 +47,6 @@ _underlyings: list[str] = [
     'commodity',
 ]
 
-_crypto_derivs: list[str] = [
-    'perpetual_future',
-    'crypto_future',
-]
 
 _derivs: list[str] = [
     'swap',
@@ -71,8 +66,6 @@ AssetTypeName: Literal[
     _underlyings
     +
     _derivs
-    +
-    _crypto_derivs
 ]
 
 # egs. stock, futer, option, bond etc.
@@ -128,31 +121,10 @@ class Asset(Struct, frozen=True):
 
     # NOTE: additional info optionally packed in by the backend, but
     # should not be explicitly required in our generic API.
-    info: dict | None = None
+    info: dict = {}  # make it frozen?
 
-    # `None` is not toml-compat so drop info
-    # if no extra data added..
-    def to_dict(
-        self,
-        **kwargs,
-    ) -> dict:
-        dct = super().to_dict(**kwargs)
-        if (info := dct.pop('info', None)):
-            dct['info'] = info
-
-        assert dct['tx_tick']
-        return dct
-
-    @classmethod
-    def from_msg(
-        cls,
-        msg: dict[str, Any],
-    ) -> Asset:
-        return cls(
-            tx_tick=Decimal(str(msg.pop('tx_tick'))),
-            info=msg.pop('info', None),
-            **msg,
-        )
+    # TODO?
+    # _to_dict_skip = {'info'}
 
     def __str__(self) -> str:
         return self.name
@@ -235,33 +207,6 @@ class MktPair(Struct, frozen=True):
     <dst>/<src>.<expiry>.<con_info_1>.<con_info_2>. -> .<venue>.<broker>
           ^ -- optional tokens ------------------------------- ^
 
-
-    Notes:
-    ------
-
-    Some venues provide a different semantic (which we frankly find
-    confusing and non-general) such as "base" and "quote" asset.
-    For example this is how `binance` defines the terms:
-
-    https://binance-docs.github.io/apidocs/websocket_api/en/#public-api-definitions
-    https://binance-docs.github.io/apidocs/futures/en/#public-endpoints-info
-
-    - *base* asset refers to the asset that is the *quantity* of a symbol.
-    - *quote* asset refers to the asset that is the *price* of a symbol.
-
-    In other words the "quote" asset is the asset that the market
-    is pricing "buys" *in*, and the *base* asset it the one that the market
-    allows you to "buy" an *amount of*. Put more simply the *quote*
-    asset is our "source" asset and the *base* asset is our "destination"
-    asset.
-
-    This defintion can be further understood reading our
-    `.brokers.binance.api.Pair` type wherein the
-    `Pair.[quote/base]AssetPrecision` field determines the (transfer)
-    transaction precision available per asset; i.e. the satoshis
-    unit in bitcoin for representing the minimum size of a
-    transaction that can take place on the blockchain.
-
     '''
     dst: str | Asset
     # "destination asset" (name) used to buy *to*
@@ -309,40 +254,12 @@ class MktPair(Struct, frozen=True):
     # strike price, call or put, swap type, exercise model, etc.
     contract_info: list[str] | None = None
 
-    # TODO: rename to sectype since all of these can
-    # be considered "securities"?
     _atype: str = ''
 
-    # allow explicit disable of the src part of the market
-    # pair name -> useful for legacy markets like qqq.nasdaq.ib
-    _fqme_without_src: bool = False
-
     # NOTE: when cast to `str` return fqme
     def __str__(self) -> str:
         return self.fqme
 
-    def to_dict(
-        self,
-        **kwargs,
-    ) -> dict:
-        d = super().to_dict(**kwargs)
-        d['src'] = self.src.to_dict(**kwargs)
-
-        if not isinstance(self.dst, str):
-            d['dst'] = self.dst.to_dict(**kwargs)
-        else:
-            d['dst'] = str(self.dst)
-
-        d['price_tick'] = str(self.price_tick)
-        d['size_tick'] = str(self.size_tick)
-
-        if self.contract_info is None:
-            d.pop('contract_info')
-
-        # d.pop('_fqme_without_src')
-
-        return d
-
     @classmethod
     def from_msg(
         cls,
@@ -353,32 +270,36 @@ class MktPair(Struct, frozen=True):
         Constructor for a received msg-dict normally received over IPC.
 
         '''
-        if not isinstance(
-            dst_asset_msg := msg.pop('dst'),
-            str,
-        ):
-            dst: Asset = Asset.from_msg(dst_asset_msg)  # .copy()
+        dst_asset_msg = msg.pop('dst')
+        src_asset_msg = msg.pop('src')
+
+        if isinstance(dst_asset_msg, str):
+            src: str = str(src_asset_msg)
+            assert isinstance(src, str)
+            return cls.from_fqme(
+                dst_asset_msg,
+                src=src,
+                **msg,
+            )
+
         else:
-            dst: str = dst_asset_msg
+            # NOTE: we call `.copy()` here to ensure
+            # type casting!
+            dst = Asset(**dst_asset_msg).copy()
+            if not isinstance(src_asset_msg, str):
+                src = Asset(**src_asset_msg).copy()
+            else:
+                src = str(src_asset_msg)
 
-        src_asset_msg: dict = msg.pop('src')
-        src: Asset = Asset.from_msg(src_asset_msg)  # .copy()
-
-        # XXX NOTE: ``msgspec`` can encode `Decimal` but it doesn't
-        # decide to it by default since we aren't spec-cing these
-        # msgs as structs proper to get them to decode implictily
-        # (yet) as per,
-        # - https://github.com/pikers/piker/pull/354
-        # - https://github.com/goodboy/tractor/pull/311
-        # SO we have to ensure we do a struct type
-        # case (which `.copy()` does) to ensure we get the right
-        # type!
         return cls(
             dst=dst,
             src=src,
-            price_tick=Decimal(msg.pop('price_tick')),
-            size_tick=Decimal(msg.pop('size_tick')),
             **msg,
+        # XXX NOTE: ``msgspec`` can encode `Decimal`
+        # but it doesn't decide to it by default since
+        # we aren't spec-cing these msgs as structs, SO
+        # we have to ensure we do a struct type case (which `.copy()`
+        # does) to ensure we get the right type!
         ).copy()
 
     @property
@@ -406,20 +327,7 @@ class MktPair(Struct, frozen=True):
         ):
             _fqme = f'{fqme}.{broker}'
 
-        broker, mkt_ep_key, venue, expiry = unpack_fqme(_fqme)
-
-        kven: str = kwargs.pop('venue', venue)
-        if venue:
-            assert venue == kven
-        else:
-            venue = kven
-
-        exp: str = kwargs.pop('expiry', expiry)
-        if expiry:
-            assert exp == expiry
-        else:
-            expiry = exp
-
+        broker, mkt_ep_key, venue, suffix = unpack_fqme(_fqme)
         dst: Asset = Asset.guess_from_mkt_ep_key(
             mkt_ep_key,
             atype=kwargs.get('_atype'),
@@ -431,15 +339,14 @@ class MktPair(Struct, frozen=True):
         # which we expect to be filled in by some
         # backend client with access to that data-info.
         return cls(
-            dst=dst,
             # XXX: not resolved to ``Asset`` :(
-            #src=src,
+            dst=dst,
 
             broker=broker,
             venue=venue,
             # XXX NOTE: we presume this token
             # if the expiry for now!
-            expiry=expiry,
+            expiry=suffix,
 
             price_tick=price_tick,
             size_tick=size_tick,
@@ -545,7 +452,7 @@ class MktPair(Struct, frozen=True):
         '''
         key: str = (
             self.pair(delim_char=delim_char)
-            if not (without_src or self._fqme_without_src)
+            if not without_src
             else str(self.dst)
         )
 
@@ -606,15 +513,10 @@ class MktPair(Struct, frozen=True):
     # TODO: BACKWARD COMPAT, TO REMOVE?
     @property
     def type_key(self) -> str:
-
-        # if set explicitly then use it!
-        if self._atype:
-            return self._atype
-
         if isinstance(self.dst, Asset):
             return str(self.dst.atype)
 
-        return 'UNKNOWN'
+        return self._atype
 
     @property
     def price_tick_digits(self) -> int:

piker/accounting/calc.py

@@ -1,698 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Calculation routines for balance and position tracking such that
-you know when you're losing money (if possible) XD
-
-'''
-from __future__ import annotations
-from collections.abc import ValuesView
-from contextlib import contextmanager as cm
-from math import copysign
-from typing import (
-    Any,
-    Callable,
-    Iterator,
-    TYPE_CHECKING,
-)
-
-import polars as pl
-from pendulum import (
-    DateTime,
-    from_timestamp,
-    parse,
-)
-
-if TYPE_CHECKING:
-    from ._ledger import (
-        Transaction,
-        TransactionLedger,
-    )
-
-
-def ppu(
-    clears: Iterator[Transaction],
-
-    # include transaction cost in breakeven price
-    # and presume the worst case of the same cost
-    # to exit this transaction (even though in reality
-    # it will be dynamic based on exit stratetgy).
-    cost_scalar: float = 2,
-
-    # return the ledger of clears as a (now dt sorted) dict with
-    # new position fields inserted alongside each entry.
-    as_ledger: bool = False,
-
-) -> float | list[(str, dict)]:
-    '''
-    Compute the "price-per-unit" price for the given non-zero sized
-    rolling position.
-
-    The recurrence relation which computes this (exponential) mean
-    per new clear which **increases** the accumulative postiion size
-    is:
-
-    ppu[-1] = (
-        ppu[-2] * accum_size[-2]
-        +
-        ppu[-1] * size
-    ) / accum_size[-1]
-
-    where `cost_basis` for the current step is simply the price
-    * size of the most recent clearing transaction.
-
-    -----
-    TODO: get the BEP computed and working similarly!
-    -----
-    the equivalent "break even price" or bep at each new clear
-    event step conversely only changes when an "position exiting
-    clear" which **decreases** the cumulative dst asset size:
-
-    bep[-1] = ppu[-1] - (cum_pnl[-1] / cumsize[-1])
-
-    '''
-    asize_h: list[float] = []  # historical accumulative size
-    ppu_h: list[float] = []  # historical price-per-unit
-    # ledger: dict[str, dict] = {}
-    ledger: list[dict] = []
-
-    t: Transaction
-    for t in clears:
-        clear_size: float = t.size
-        clear_price: str | float = t.price
-        is_clear: bool = not isinstance(clear_price, str)
-
-        last_accum_size = asize_h[-1] if asize_h else 0
-        accum_size: float = last_accum_size + clear_size
-        accum_sign = copysign(1, accum_size)
-        sign_change: bool = False
-
-        # on transfers we normally write some non-valid
-        # price since withdrawal to another account/wallet
-        # has nothing to do with inter-asset-market prices.
-        # TODO: this should be better handled via a `type: 'tx'`
-        # field as per existing issue surrounding all this:
-        # https://github.com/pikers/piker/issues/510
-        if isinstance(clear_price, str):
-            # TODO: we can't necessarily have this commit to
-            # the overall pos size since we also need to
-            # include other positions contributions to this
-            # balance or we might end up with a -ve balance for
-            # the position..
-            continue
-
-        # test if the pp somehow went "passed" a net zero size state
-        # resulting in a change of the "sign" of the size (+ve for
-        # long, -ve for short).
-        sign_change = (
-            copysign(1, last_accum_size) + accum_sign == 0
-            and last_accum_size != 0
-        )
-
-        # since we passed the net-zero-size state the new size
-        # after sum should be the remaining size the new
-        # "direction" (aka, long vs. short) for this clear.
-        if sign_change:
-            clear_size: float = accum_size
-            abs_diff: float = abs(accum_size)
-            asize_h.append(0)
-            ppu_h.append(0)
-
-        else:
-            # old size minus the new size gives us size diff with
-            # +ve -> increase in pp size
-            # -ve -> decrease in pp size
-            abs_diff = abs(accum_size) - abs(last_accum_size)
-
-        # XXX: LIFO breakeven price update. only an increaze in size
-        # of the position contributes the breakeven price,
-        # a decrease does not (i.e. the position is being made
-        # smaller).
-        # abs_clear_size = abs(clear_size)
-        abs_new_size: float | int = abs(accum_size)
-
-        if (
-            abs_diff > 0
-            and is_clear
-        ):
-            cost_basis = (
-                # cost basis for this clear
-                clear_price * abs(clear_size)
-                +
-                # transaction cost
-                accum_sign * cost_scalar * t.cost
-            )
-
-            if asize_h:
-                size_last: float = abs(asize_h[-1])
-                cb_last: float = ppu_h[-1] * size_last
-                ppu: float = (cost_basis + cb_last) / abs_new_size
-
-            else:
-                ppu: float = cost_basis / abs_new_size
-
-        else:
-            # TODO: for PPU we should probably handle txs out
-            # (aka withdrawals) similarly by simply not having
-            # them contrib to the running PPU calc and only
-            # when the next entry clear comes in (which will
-            # then have a higher weighting on the PPU).
-
-            # on "exit" clears from a given direction,
-            # only the size changes not the price-per-unit
-            # need to be updated since the ppu remains constant
-            # and gets weighted by the new size.
-            ppu: float = ppu_h[-1] if ppu_h else 0  # set to previous value
-
-        # extend with new rolling metric for this step
-        ppu_h.append(ppu)
-        asize_h.append(accum_size)
-
-        # ledger[t.tid] = {
-            # 'txn': t,
-        # ledger[t.tid] = t.to_dict() | {
-        ledger.append((
-            t.tid,
-            t.to_dict() | {
-                'ppu': ppu,
-                'cumsize': accum_size,
-                'sign_change': sign_change,
-
-                # TODO: cum_pnl, bep
-            }
-        ))
-
-    final_ppu = ppu_h[-1] if ppu_h else 0
-    # TODO: once we have etypes in all ledger entries..
-    # handle any split info entered (for now) manually by user
-    # if self.split_ratio is not None:
-    #     final_ppu /= self.split_ratio
-
-    if as_ledger:
-        return ledger
-
-    else:
-        return final_ppu
-
-
-def iter_by_dt(
-    records: (
-        dict[str, dict[str, Any]]
-        | ValuesView[dict]  # eg. `Position._events.values()`
-        | list[dict]
-        | list[Transaction]  # XXX preferred!
-    ),
-
-    # NOTE: parsers are looked up in the insert order
-    # so if you know that the record stats show some field
-    # is more common then others, stick it at the top B)
-    parsers: dict[str, Callable | None] = {
-        'dt': parse,  # parity case
-        'datetime': parse,  # datetime-str
-        'time': from_timestamp,  # float epoch
-    },
-    key: Callable | None = None,
-
-) -> Iterator[tuple[str, dict]]:
-    '''
-    Iterate entries of a transaction table sorted by entry recorded
-    datetime presumably set at the ``'dt'`` field in each entry.
-
-    '''
-    if isinstance(records, dict):
-        records: list[tuple[str, dict]] = list(records.items())
-
-    def dyn_parse_to_dt(
-        tx: tuple[str, dict[str, Any]] | Transaction,
-    ) -> DateTime:
-
-        # handle `.items()` inputs
-        if isinstance(tx, tuple):
-            tx = tx[1]
-
-        # dict or tx object?
-        isdict: bool = isinstance(tx, dict)
-
-        # get best parser for this record..
-        for k in parsers:
-            if (
-                isdict and k in tx
-                 or getattr(tx, k, None)
-            ):
-                v = tx[k] if isdict else tx.dt
-                assert v is not None, f'No valid value for `{k}`!?'
-
-                # only call parser on the value if not None from
-                # the `parsers` table above (when NOT using
-                # `.get()`), otherwise pass through the value and
-                # sort on it directly
-                if (
-                    not isinstance(v, DateTime)
-                    and (parser := parsers.get(k))
-                ):
-                    return parser(v)
-                else:
-                    return v
-
-        else:
-            # XXX: should never get here..
-            breakpoint()
-
-    entry: tuple[str, dict] | Transaction
-    for entry in sorted(
-        records,
-        key=key or dyn_parse_to_dt,
-    ):
-        # NOTE the type sig above; either pairs or txns B)
-        yield entry
-
-
-# TODO: probably just move this into the test suite or
-# keep it here for use from as such?
-# def ensure_state(self) -> None:
-#     '''
-#     Audit either the `.cumsize` and `.ppu` local instance vars against
-#     the clears table calculations and return the calc-ed values if
-#     they differ and log warnings to console.
-
-#     '''
-#     # clears: list[dict] = self._clears
-
-#     # self.first_clear_dt = min(clears, key=lambda e: e['dt'])['dt']
-#     last_clear: dict = clears[-1]
-#     csize: float = self.calc_size()
-#     accum: float = last_clear['accum_size']
-
-#     if not self.expired():
-#         if (
-#             csize != accum
-#             and csize != round(accum * (self.split_ratio or 1))
-#         ):
-#             raise ValueError(f'Size mismatch: {csize}')
-#     else:
-#         assert csize == 0, 'Contract is expired but non-zero size?'
-
-#     if self.cumsize != csize:
-#         log.warning(
-#             'Position state mismatch:\n'
-#             f'{self.cumsize} => {csize}'
-#         )
-#         self.cumsize = csize
-
-#     cppu: float = self.calc_ppu()
-#     ppu: float = last_clear['ppu']
-#     if (
-#         cppu != ppu
-#         and self.split_ratio is not None
-
-#         # handle any split info entered (for now) manually by user
-#         and cppu != (ppu / self.split_ratio)
-#     ):
-#         raise ValueError(f'PPU mismatch: {cppu}')
-
-#     if self.ppu != cppu:
-#         log.warning(
-#             'Position state mismatch:\n'
-#             f'{self.ppu} => {cppu}'
-#         )
-#         self.ppu = cppu
-
-
-@cm
-def open_ledger_dfs(
-
-    brokername: str,
-    acctname: str,
-
-    ledger: TransactionLedger | None = None,
-
-    **kwargs,
-
-) -> tuple[
-    dict[str, pl.DataFrame],
-    TransactionLedger,
-]:
-    '''
-    Open a ledger of trade records (presumably from some broker
-    backend), normalize the records into `Transactions` via the
-    backend's declared endpoint, cast to a `polars.DataFrame` which
-    can update the ledger on exit.
-
-    '''
-    from piker.toolz import open_crash_handler
-    with open_crash_handler():
-        if not ledger:
-            import time
-            from ._ledger import open_trade_ledger
-
-            now = time.time()
-
-            with open_trade_ledger(
-                    brokername,
-                    acctname,
-                    rewrite=True,
-                    allow_from_sync_code=True,
-
-                    # proxied through from caller
-                    **kwargs,
-
-            ) as ledger:
-                if not ledger:
-                    raise ValueError(f'No ledger for {acctname}@{brokername} exists?')
-
-                print(f'LEDGER LOAD TIME: {time.time() - now}')
-
-        yield ledger_to_dfs(ledger), ledger
-
-
-def ledger_to_dfs(
-    ledger: TransactionLedger,
-
-) -> dict[str, pl.DataFrame]:
-
-    txns: dict[str, Transaction] = ledger.to_txns()
-
-    # ldf = pl.DataFrame(
-    #     list(txn.to_dict() for txn in txns.values()),
-    ldf = pl.from_dicts(
-        list(txn.to_dict() for txn in txns.values()),
-
-        # only for ordering the cols
-        schema=[
-            ('fqme', str),
-            ('tid', str),
-            ('bs_mktid', str),
-            ('expiry', str),
-            ('etype', str),
-            ('dt', str),
-            ('size', pl.Float64),
-            ('price', pl.Float64),
-            ('cost', pl.Float64),
-        ],
-    ).sort(  # chronological order
-        'dt'
-    ).with_columns([
-        pl.col('dt').str.to_datetime(),
-        # pl.col('expiry').str.to_datetime(),
-        # pl.col('expiry').dt.date(),
-    ])
-
-    # filter out to the columns matching values filter passed
-    # as input.
-    # if filter_by_ids:
-    #     for col, vals in filter_by_ids.items():
-    #         str_vals = set(map(str, vals))
-    #         pred: pl.Expr = pl.col(col).eq(str_vals.pop())
-    #         for val in str_vals:
-    #             pred |= pl.col(col).eq(val)
-
-    #     fdf = df.filter(pred)
-
-    # TODO: originally i had tried just using a plain ol' groupby
-    # + agg here but the issue was re-inserting to the src frame.
-    # however, learning more about `polars` seems like maybe we can
-    # use `.over()`?
-    # https://pola-rs.github.io/polars/py-polars/html/reference/expressions/api/polars.Expr.over.html#polars.Expr.over
-    # => CURRENTLY we break up into a frame per mkt / fqme
-    dfs: dict[str, pl.DataFrame] = ldf.partition_by(
-        'bs_mktid',
-        as_dict=True,
-    )
-
-    # TODO: not sure if this is even possible but..
-    # - it'd be more ideal to use `ppt = df.groupby('fqme').agg([`
-    # - ppu and bep calcs!
-    for key in dfs:
-
-        # covert to lazy form (since apparently we might need it
-        # eventually ...)
-        df: pl.DataFrame = dfs[key]
-
-        ldf: pl.LazyFrame = df.lazy()
-
-        df = dfs[key] = ldf.with_columns([
-
-            pl.cumsum('size').alias('cumsize'),
-
-            # amount of source asset "sent" (via buy txns in
-            # the market) to acquire the dst asset, PER txn.
-            # when this value is -ve (i.e. a sell operation) then
-            # the amount sent is actually "returned".
-            (
-                (pl.col('price') * pl.col('size'))
-                +
-                (pl.col('cost')) # * pl.col('size').sign())
-            ).alias('dst_bot'),
-
-        ]).with_columns([
-
-            # rolling balance in src asset units
-            (pl.col('dst_bot').cumsum() * -1).alias('src_balance'),
-
-            # "position operation type" in terms of increasing the
-            # amount in the dst asset (entering) or decreasing the
-            # amount in the dst asset (exiting).
-            pl.when(
-                pl.col('size').sign() == pl.col('cumsize').sign()
-
-            ).then(
-                pl.lit('enter')  # see above, but is just price * size per txn
-
-            ).otherwise(
-                pl.when(pl.col('cumsize') == 0)
-                .then(pl.lit('exit_to_zero'))
-                .otherwise(pl.lit('exit'))
-            ).alias('descr'),
-
-            (pl.col('cumsize').sign() == pl.col('size').sign())
-            .alias('is_enter'),
-
-        ]).with_columns([
-
-            # pl.lit(0, dtype=pl.Utf8).alias('virt_cost'),
-            pl.lit(0, dtype=pl.Float64).alias('applied_cost'),
-            pl.lit(0, dtype=pl.Float64).alias('pos_ppu'),
-            pl.lit(0, dtype=pl.Float64).alias('per_txn_pnl'),
-            pl.lit(0, dtype=pl.Float64).alias('cum_pos_pnl'),
-            pl.lit(0, dtype=pl.Float64).alias('pos_bep'),
-            pl.lit(0, dtype=pl.Float64).alias('cum_ledger_pnl'),
-            pl.lit(None, dtype=pl.Float64).alias('ledger_bep'),
-
-            # TODO: instead of the iterative loop below i guess we
-            # could try using embedded lists to track which txns
-            # are part of which ppu / bep calcs? Not sure this will
-            # look any better nor be any more performant though xD
-            # pl.lit([[0]], dtype=pl.List(pl.Float64)).alias('list'),
-
-        # choose fields to emit for accounting puposes
-        ]).select([
-            pl.exclude([
-                'tid',
-                # 'dt',
-                'expiry',
-                'bs_mktid',
-                'etype',
-                # 'is_enter',
-            ]),
-        ]).collect()
-
-        # compute recurrence relations for ppu and bep
-        last_ppu: float = 0
-        last_cumsize: float = 0
-        last_ledger_pnl: float = 0
-        last_pos_pnl: float = 0
-        virt_costs: list[float, float] = [0., 0.]
-
-        # imperatively compute the PPU (price per unit) and BEP
-        # (break even price) iteratively over the ledger, oriented
-        # around each position state: a state of split balances in
-        # > 1 asset.
-        for i, row in enumerate(df.iter_rows(named=True)):
-
-            cumsize: float = row['cumsize']
-            is_enter: bool = row['is_enter']
-            price: float = row['price']
-            size: float = row['size']
-
-            # the profit is ALWAYS decreased, aka made a "loss"
-            # by the constant fee charged by the txn provider!
-            # see below in final PnL calculation and row element
-            # set.
-            txn_cost: float = row['cost']
-            pnl: float = 0
-
-            # ALWAYS reset per-position cum PnL
-            if last_cumsize == 0:
-                last_pos_pnl: float = 0
-
-            # a "position size INCREASING" or ENTER transaction
-            # which "makes larger", in src asset unit terms, the
-            # trade's side-size of the destination asset:
-            # - "buying" (more) units of the dst asset
-            # - "selling" (more short) units of the dst asset
-            if is_enter:
-
-                # Naively include transaction cost in breakeven
-                # price and presume the worst case of the
-                # exact-same-cost-to-exit this transaction's worth
-                # of size even though in reality it will be dynamic
-                # based on exit strategy, price, liquidity, etc..
-                virt_cost: float = txn_cost
-
-                # cpu: float = cost / size
-                # cummean of the cost-per-unit used for modelling
-                # a projected future exit cost which we immediately
-                # include in the costs incorporated to BEP on enters
-                last_cum_costs_size, last_cpu = virt_costs
-                cum_costs_size: float = last_cum_costs_size + abs(size)
-                cumcpu = (
-                    (last_cpu * last_cum_costs_size)
-                    +
-                    txn_cost
-                ) / cum_costs_size
-                virt_costs = [cum_costs_size, cumcpu]
-
-                txn_cost = txn_cost + virt_cost
-                # df[i, 'virt_cost'] = f'{-virt_cost} FROM {cumcpu}@{cum_costs_size}'
-
-                # a cumulative mean of the price-per-unit acquired
-                # in the destination asset:
-                # https://en.wikipedia.org/wiki/Moving_average#Cumulative_average
-                # You could also think of this measure more
-                # generally as an exponential mean with `alpha
-                # = 1/N` where `N` is the current number of txns
-                # included in the "position" defining set:
-                # https://en.wikipedia.org/wiki/Exponential_smoothing
-                ppu: float = (
-                    (
-                        (last_ppu * last_cumsize)
-                        +
-                        (price * size)
-                    ) /
-                    cumsize
-                )
-
-            # a "position size DECREASING" or EXIT transaction
-            # which "makes smaller" the trade's side-size of the
-            # destination asset:
-            # - selling previously bought units of the dst asset
-            #   (aka 'closing' a long position).
-            # - buying previously borrowed and sold (short) units
-            #   of the dst asset (aka 'covering'/'closing' a short
-            #   position).
-            else:
-                # only changes on position size increasing txns
-                ppu: float = last_ppu
-
-                # UNWIND IMPLIED COSTS FROM ENTRIES
-                # => Reverse the virtual/modelled (2x predicted) txn
-                # cost that was included in the least-recently
-                # entered txn that is still part of the current CSi
-                # set.
-                # => we look up the cost-per-unit cumsum and apply
-                # if over the current txn size (by multiplication)
-                # and then reverse that previusly applied cost on
-                # the txn_cost for this record.
-                #
-                # NOTE: current "model" is just to previously assumed 2x
-                # the txn cost for a matching enter-txn's
-                # cost-per-unit; we then immediately reverse this
-                # prediction and apply the real cost received here.
-                last_cum_costs_size, last_cpu = virt_costs
-                prev_virt_cost: float = last_cpu * abs(size)
-                txn_cost: float = txn_cost - prev_virt_cost  # +ve thus a "reversal"
-                cum_costs_size: float = last_cum_costs_size - abs(size)
-                virt_costs = [cum_costs_size, last_cpu]
-
-                # df[i, 'virt_cost'] = (
-                #     f'{-prev_virt_cost} FROM {last_cpu}@{cum_costs_size}'
-                # )
-
-                # the per-txn profit or loss (PnL) given we are
-                # (partially) "closing"/"exiting" the position via
-                # this txn.
-                pnl: float = (last_ppu - price) * size
-
-            # always subtract txn cost from total txn pnl
-            txn_pnl: float = pnl - txn_cost
-
-            # cumulative PnLs per txn
-            last_ledger_pnl = (
-                last_ledger_pnl + txn_pnl
-            )
-            last_pos_pnl = df[i, 'cum_pos_pnl'] = (
-                last_pos_pnl + txn_pnl
-            )
-
-            if cumsize == 0:
-                last_ppu = ppu = 0
-
-            # compute the BEP: "break even price", a value that
-            # determines at what price the remaining cumsize can be
-            # liquidated such that the net-PnL on the current
-            # position will result in ZERO gain or loss from open
-            # to close including all txn costs B)
-            if (
-                abs(cumsize) > 0  # non-exit-to-zero position txn
-            ):
-                cumsize_sign: float = copysign(1, cumsize)
-                ledger_bep: float = (
-                    (
-                        (ppu * cumsize)
-                        -
-                        (last_ledger_pnl * cumsize_sign)
-                    ) / cumsize
-                )
-
-                # NOTE: when we "enter more" dst asset units (aka
-                # increase position state) AFTER having exited some
-                # units (aka decreasing the pos size some) the bep
-                # needs to be RECOMPUTED based on new ppu such that
-                # liquidation of the cumsize at the bep price
-                # results in a zero-pnl for the existing position
-                # (since the last one).
-                # for position lifetime BEP we never can have
-                # a valid value once the position is "closed"
-                # / full exitted Bo
-                pos_bep: float = (
-                    (
-                        (ppu * cumsize)
-                        -
-                        (last_pos_pnl * cumsize_sign)
-                    ) / cumsize
-                )
-
-            # inject DF row with all values
-            df[i, 'pos_ppu'] = ppu
-            df[i, 'per_txn_pnl'] = txn_pnl
-            df[i, 'applied_cost'] = -txn_cost
-            df[i, 'cum_pos_pnl'] = last_pos_pnl
-            df[i, 'pos_bep'] = pos_bep
-            df[i, 'cum_ledger_pnl'] = last_ledger_pnl
-            df[i, 'ledger_bep'] = ledger_bep
-
-            # keep backrefs to suffice reccurence relation
-            last_ppu: float = ppu
-            last_cumsize: float = cumsize
-
-    # TODO?: pass back the current `Position` object loaded from
-    # the account as well? Would provide incentive to do all
-    # this ledger loading inside a new async open_account().
-    # bs_mktid: str = df[0]['bs_mktid']
-    # pos: Position = acnt.pps[bs_mktid]
-
-    return dfs

piker/accounting/cli.py

@@ -18,13 +18,12 @@
 CLI front end for trades ledger and position tracking management.
 
 '''
-from __future__ import annotations
-from pprint import pformat
-
+from typing import (
+    Any,
+)
 
 from rich.console import Console
 from rich.markdown import Markdown
-import polars as pl
 import tractor
 import trio
 import typer
@@ -35,38 +34,63 @@ from ..service import (
 )
 from ..clearing._messages import BrokerdPosition
 from ..calc import humanize
-from ..brokers._daemon import broker_init
-from ._ledger import (
-    load_ledger,
-    TransactionLedger,
-    # open_trade_ledger,
-)
-from .calc import (
-    open_ledger_dfs,
-)
 
 
 ledger = typer.Typer()
 
 
-def unpack_fqan(
-    fully_qualified_account_name: str,
-    console: Console | None = None,
-) -> tuple | bool:
-    try:
-        brokername, account = fully_qualified_account_name.split('.')
-        return brokername, account
-    except ValueError:
-        if console is not None:
-            md = Markdown(
-                f'=> `{fully_qualified_account_name}` <=\n\n'
-                'is not a valid '
-                '__fully qualified account name?__\n\n'
-                'Your account name needs to be of the form '
-                '`<brokername>.<account_name>`\n'
-            )
-            console.print(md)
-        return False
+def broker_init(
+    brokername: str,
+    loglevel: str | None = None,
+
+    **start_actor_kwargs,
+
+) -> dict:
+    '''
+    Given an input broker name, load all named arguments
+    which can be passed to a daemon + context spawn for
+    the relevant `brokerd` service endpoint.
+
+    '''
+    from ..brokers import get_brokermod
+    brokermod = get_brokermod(brokername)
+    modpath = brokermod.__name__
+
+    start_actor_kwargs['name'] = f'brokerd.{brokername}'
+    start_actor_kwargs.update(
+        getattr(
+            brokermod,
+            '_spawn_kwargs',
+            {},
+        )
+    )
+
+    # lookup actor-enabled modules declared by the backend offering the
+    # `brokerd` endpoint(s).
+    enabled = start_actor_kwargs['enable_modules'] = [modpath]
+    for submodname in getattr(
+        brokermod,
+        '__enable_modules__',
+        [],
+    ):
+        subpath = f'{modpath}.{submodname}'
+        enabled.append(subpath)
+
+        # TODO XXX: DO WE NEED THIS?
+        # enabled.append('piker.data.feed')
+
+    # non-blocking setup of brokerd service nursery
+    from ..brokers._daemon import _setup_persistent_brokerd
+
+    return (
+        start_actor_kwargs,  # to `ActorNursery.start_actor()`
+        _setup_persistent_brokerd,  # service task ep
+        getattr(  # trades endpoint
+            brokermod,
+            'trades_dialogue',
+            None,
+        ),
+    )
 
 
 @ledger.command()
@@ -82,20 +106,23 @@ def sync(
     log = get_logger(loglevel)
     console = Console()
 
-    pair: tuple[str, str]
-    if not (pair := unpack_fqan(
-        fully_qualified_account_name,
-        console,
-    )):
+    try:
+        brokername, account = fully_qualified_account_name.split('.')
+    except ValueError:
+        md = Markdown(
+            f'=> `{fully_qualified_account_name}` <=\n\n'
+            'is not a valid '
+            '__fully qualified account name?__\n\n'
+            'Your account name needs to be of the form '
+            '`<brokername>.<account_name>`\n'
+        )
+        console.print(md)
         return
 
-    brokername, account = pair
-
-    brokermod, start_kwargs, deamon_ep = broker_init(
+    start_kwargs, _, trades_ep = broker_init(
         brokername,
         loglevel=loglevel,
     )
-    brokername: str = brokermod.name
 
     async def main():
 
@@ -109,203 +136,99 @@ def sync(
 
             tractor.open_nursery() as an,
         ):
-            try:
-                log.info(
-                    f'Piker runtime up as {actor.uid}@{sockaddr}'
-                )
+            log.info(
+                f'Piker runtime up as {actor.uid}@{sockaddr}'
+            )
 
-                portal = await an.start_actor(
+            portal = await an.start_actor(
+                loglevel=loglevel,
+                debug_mode=pdb,
+                **start_kwargs,
+            )
+
+            if (
+                brokername == 'paper'
+                or trades_ep is None
+            ):
+                from ..clearing import _paper_engine as paper
+                open_trades_endpoint = paper.open_paperboi(
+                    fqme=None,  # tell paper to not start clearing loop
+                    broker=brokername,
+                    loglevel=loglevel,
+                )
+            else:
+                # open live brokerd trades endpoint
+                open_trades_endpoint = portal.open_context(
+                    trades_ep,
                     loglevel=loglevel,
-                    debug_mode=pdb,
-                    **start_kwargs,
                 )
 
-                from ..clearing import (
-                    open_brokerd_dialog,
+            positions: dict[str, Any]
+            accounts: list[str]
+            async with (
+                open_trades_endpoint as (
+                    brokerd_ctx,
+                    (positions, accounts),
+                ),
+            ):
+                assert len(accounts) == 1
+                summary: str = (
+                    '[dim underline]Piker Position Summary[/] '
+                    f'[dim blue underline]{brokername}[/]'
+                    '[dim].[/]'
+                    f'[blue underline]{account}[/]'
+                    f'[dim underline] -> total pps: [/]'
+                    f'[green]{len(positions)}[/]\n'
                 )
-                brokerd_stream: tractor.MsgStream
-
-                async with (
-                    # engage the brokerd daemon context
-                    portal.open_context(
-                        deamon_ep,
-                        brokername=brokername,
-                        loglevel=loglevel,
-                    ),
-
-                    # manually open the brokerd trade dialog EP
-                    # (what the EMS normally does internall) B)
-                    open_brokerd_dialog(
-                        brokermod,
-                        portal,
-                        exec_mode=(
-                            'paper'
-                            if account == 'paper'
-                            else 'live'
-                        ),
-                        loglevel=loglevel,
-                    ) as (
-                        brokerd_stream,
-                        pp_msg_table,
-                        accounts,
-                    ),
-                ):
-                    try:
-                        assert len(accounts) == 1
-                        if not pp_msg_table:
-                            ld, fpath = load_ledger(brokername, account)
-                            assert not ld, f'WTF did we fail to parse ledger:\n{ld}'
-
-                            console.print(
-                                '[yellow]'
-                                'No pps found for '
-                                f'`{brokername}.{account}` '
-                                'account!\n\n'
-                                '[/][underline]'
-                                'None of the following ledger files exist:\n\n[/]'
-                                f'{fpath.as_uri()}\n'
-                            )
-                            return
-
-                        pps_by_symbol: dict[str, BrokerdPosition] = pp_msg_table[
-                            brokername,
-                            account,
-                        ]
-
-                        summary: str = (
-                            '[dim underline]Piker Position Summary[/] '
-                            f'[dim blue underline]{brokername}[/]'
-                            '[dim].[/]'
-                            f'[blue underline]{account}[/]'
-                            f'[dim underline] -> total pps: [/]'
-                            f'[green]{len(pps_by_symbol)}[/]\n'
+                for ppdict in positions:
+                    ppmsg = BrokerdPosition(**ppdict)
+                    size = ppmsg.size
+                    if size:
+                        ppu: float = round(
+                            ppmsg.avg_price,
+                            ndigits=2,
                         )
-                        # for ppdict in positions:
-                        for fqme, ppmsg in pps_by_symbol.items():
-                            # ppmsg = BrokerdPosition(**ppdict)
-                            size = ppmsg.size
-                            if size:
-                                ppu: float = round(
-                                    ppmsg.avg_price,
-                                    ndigits=2,
-                                )
-                                cost_basis: str = humanize(size * ppu)
-                                h_size: str = humanize(size)
+                        cost_basis: str = humanize(size * ppu)
+                        h_size: str = humanize(size)
 
-                                if size < 0:
-                                    pcolor = 'red'
-                                else:
-                                    pcolor = 'green'
+                        if size < 0:
+                            pcolor = 'red'
+                        else:
+                            pcolor = 'green'
 
-                                # sematic-highlight of fqme
-                                fqme = ppmsg.symbol
-                                tokens = fqme.split('.')
-                                styled_fqme = f'[blue underline]{tokens[0]}[/]'
-                                for tok in tokens[1:]:
-                                    styled_fqme += '[dim].[/]'
-                                    styled_fqme += f'[dim blue underline]{tok}[/]'
+                        # sematic-highlight of fqme
+                        fqme = ppmsg.symbol
+                        tokens = fqme.split('.')
+                        styled_fqme = f'[blue underline]{tokens[0]}[/]'
+                        for tok in tokens[1:]:
+                            styled_fqme += '[dim].[/]'
+                            styled_fqme += f'[dim blue underline]{tok}[/]'
 
-                                # TODO: instead display in a ``rich.Table``?
-                                summary += (
-                                    styled_fqme +
-                                    '[dim]: [/]'
-                                    f'[{pcolor}]{h_size}[/]'
-                                    '[dim blue]u @[/]'
-                                    f'[{pcolor}]{ppu}[/]'
-                                    '[dim blue] = [/]'
-                                    f'[{pcolor}]$ {cost_basis}\n[/]'
-                                )
+                        # TODO: instead display in a ``rich.Table``?
+                        summary += (
+                            styled_fqme +
+                            '[dim]: [/]'
+                            f'[{pcolor}]{h_size}[/]'
+                            '[dim blue]u @[/]'
+                            f'[{pcolor}]{ppu}[/]'
+                            '[dim blue] = [/]'
+                            f'[{pcolor}]$ {cost_basis}\n[/]'
+                        )
 
-                        console.print(summary)
+                console.print(summary)
 
-                    finally:
-                        # exit via ctx cancellation.
-                        brokerd_ctx: tractor.Context = brokerd_stream._ctx
-                        await brokerd_ctx.cancel(timeout=1)
+                # exit via ctx cancellation.
+                await brokerd_ctx.cancel(timeout=1)
+                # TODO: once ported to newer tractor branch we should
+                # be able to do a loop like this:
+                # while brokerd_ctx.cancel_called_remote is None:
+                #     await trio.sleep(0.01)
+                #     await brokerd_ctx.cancel()
 
-                    # TODO: once ported to newer tractor branch we should
-                    # be able to do a loop like this:
-                    # while brokerd_ctx.cancel_called_remote is None:
-                    #     await trio.sleep(0.01)
-                    #     await brokerd_ctx.cancel()
-
-            finally:
-                await portal.cancel_actor()
+            await portal.cancel_actor()
 
     trio.run(main)
 
 
-@ledger.command()
-def disect(
-    # "fully_qualified_account_name"
-    fqan: str,
-    fqme: str,  # for ib
-
-    # TODO: in tractor we should really have
-    # a debug_mode ctx for wrapping any kind of code no?
-    pdb: bool = False,
-    bs_mktid: str = typer.Option(
-        None,
-        "-bid",
-    ),
-    loglevel: str = typer.Option(
-        'error',
-        "-l",
-    ),
-):
-    from piker.log import get_console_log
-    from piker.toolz import open_crash_handler
-    get_console_log(loglevel)
-
-    pair: tuple[str, str]
-    if not (pair := unpack_fqan(fqan)):
-        raise ValueError('{fqan} malformed!?')
-
-    brokername, account = pair
-
-    # ledger dfs groupby-partitioned by fqme
-    dfs: dict[str, pl.DataFrame]
-    # actual ledger instance
-    ldgr: TransactionLedger
-
-    pl.Config.set_tbl_cols(-1)
-    pl.Config.set_tbl_rows(-1)
-    with (
-        open_crash_handler(),
-        open_ledger_dfs(
-            brokername,
-            account,
-        ) as (dfs, ldgr),
-    ):
-
-        # look up specific frame for fqme-selected asset
-        if (df := dfs.get(fqme)) is None:
-            mktids2fqmes: dict[str, list[str]] = {}
-            for bs_mktid in dfs:
-                df: pl.DataFrame = dfs[bs_mktid]
-                fqmes: pl.Series[str] = df['fqme']
-                uniques: list[str] = fqmes.unique()
-                mktids2fqmes[bs_mktid] = set(uniques)
-                if fqme in uniques:
-                    break
-            print(
-                f'No specific ledger for fqme={fqme} could be found in\n'
-                f'{pformat(mktids2fqmes)}?\n'
-                f'Maybe the `{brokername}` backend uses something '
-                'else for its `bs_mktid` then the `fqme`?\n'
-                'Scanning for matches in unique fqmes per frame..\n'
-            )
-
-        # :pray:
-        assert not df.is_empty()
-
-        # muck around in pdbp REPL
-        breakpoint()
-
-        # TODO: we REALLY need a better console REPL for this
-        # kinda thing..
-        # - `xonsh` is an obvious option (and it looks amazin) but
-        # we need to figure out how to embed it better then just:
-        # from xonsh.main import main
-        # main(argv=[])
-        # which will not actually inject the `df` to globals?
+if __name__ == "__main__":
+    ledger()  # this is called from ``>> ledger <accountname>``

piker/brokers/__init__.py

@@ -17,40 +17,14 @@
 """
 Broker clients, daemons and general back end machinery.
 """
-from contextlib import (
-    asynccontextmanager as acm,
-)
 from importlib import import_module
 from types import ModuleType
 
-from tractor.trionics import maybe_open_context
-
-from ._util import (
-    log,
-    BrokerError,
-    SymbolNotFound,
-    NoData,
-    DataUnavailable,
-    DataThrottle,
-    resproc,
-    get_logger,
-)
-
-__all__: list[str] = [
-    'BrokerError',
-    'SymbolNotFound',
-    'NoData',
-    'DataUnavailable',
-    'DataThrottle',
-    'resproc',
-    'get_logger',
-]
-
-__brokers__: list[str] = [
+__brokers__ = [
     'binance',
     'ib',
     'kraken',
-    'kucoin',
+    'kucoin'
 
     # broken but used to work
     # 'questrade',
@@ -71,7 +45,7 @@ def get_brokermod(brokername: str) -> ModuleType:
     Return the imported broker module by name.
 
     '''
-    module: ModuleType = import_module('.' + brokername, 'piker.brokers')
+    module = import_module('.' + brokername, 'piker.brokers')
     # we only allow monkeying because it's for internal keying
     module.name = module.__name__.split('.')[-1]
     return module
@@ -84,28 +58,3 @@ def iter_brokermods():
     '''
     for name in __brokers__:
         yield get_brokermod(name)
-
-
-@acm
-async def open_cached_client(
-    brokername: str,
-    **kwargs,
-
-) -> 'Client':  # noqa
-    '''
-    Get a cached broker client from the current actor's local vars.
-
-    If one has not been setup do it and cache it.
-
-    '''
-    brokermod = get_brokermod(brokername)
-    async with maybe_open_context(
-        acm_func=brokermod.get_client,
-        kwargs=kwargs,
-
-    ) as (cache_hit, client):
-
-        if cache_hit:
-            log.runtime(f'Reusing existing {client}')
-
-        yield client

piker/brokers/_daemon.py

@@ -19,16 +19,9 @@ Broker-daemon-actor "endpoint-hooks": the service task entry points for
 ``brokerd``.
 
 '''
-from __future__ import annotations
 from contextlib import (
     asynccontextmanager as acm,
 )
-from types import ModuleType
-from typing import (
-    TYPE_CHECKING,
-    AsyncContextManager,
-)
-import exceptiongroup as eg
 
 import tractor
 import trio
@@ -36,14 +29,11 @@ import trio
 from . import _util
 from . import get_brokermod
 
-if TYPE_CHECKING:
-    from ..data import _FeedsBus
-
 # `brokerd` enabled modules
 # TODO: move this def to the `.data` subpkg..
 # NOTE: keeping this list as small as possible is part of our caps-sec
 # model and should be treated with utmost care!
-_data_mods: str = [
+_data_mods = [
     'piker.brokers.core',
     'piker.brokers.data',
     'piker.brokers._daemon',
@@ -68,125 +58,31 @@ async def _setup_persistent_brokerd(
     the broker backend as needed.
 
     '''
-    # NOTE: we only need to setup logging once (and only) here
-    # since all hosted daemon tasks will reference this same
-    # log instance's (actor local) state and thus don't require
-    # any further (level) configuration on their own B)
     log = _util.get_console_log(
         loglevel or tractor.current_actor().loglevel,
         name=f'{_util.subsys}.{brokername}',
     )
-
     # set global for this actor to this new process-wide instance B)
     _util.log = log
 
-    # further, set the log level on any broker broker specific
-    # logger instance.
-
-    from piker.data import feed
-    assert not feed._bus
-
-    # allocate a nursery to the bus for spawning background
-    # tasks to service client IPC requests, normally
-    # `tractor.Context` connections to explicitly required
-    # `brokerd` endpoints such as:
-    # - `stream_quotes()`,
-    # - `manage_history()`,
-    # - `allocate_persistent_feed()`,
-    # - `open_symbol_search()`
-    # NOTE: see ep invocation details inside `.data.feed`.
-    try:
-        async with trio.open_nursery() as service_nursery:
-            bus: _FeedsBus = feed.get_feed_bus(
-                brokername,
-                service_nursery,
-            )
-            assert bus is feed._bus
-
-            # unblock caller
-            await ctx.started()
-
-            # we pin this task to keep the feeds manager active until the
-            # parent actor decides to tear it down
-            await trio.sleep_forever()
-
-    except eg.ExceptionGroup:
-        # TODO: likely some underlying `brokerd` IPC connection
-        # broke so here we handle a respawn and re-connect attempt!
-        # This likely should pair with development of the OCO task
-        # nusery in dev over @ `tractor` B)
-        # https://github.com/goodboy/tractor/pull/363
-        raise
-
-
-def broker_init(
-    brokername: str,
-    loglevel: str | None = None,
-
-    **start_actor_kwargs,
-
-) -> tuple[
-    ModuleType,
-    dict,
-    AsyncContextManager,
-]:
-    '''
-    Given an input broker name, load all named arguments
-    which can be passed for daemon endpoint + context spawn
-    as required in every `brokerd` (actor) service.
-
-    This includes:
-    - load the appropriate <brokername>.py pkg module,
-    - reads any declared `__enable_modules__: listr[str]` which will be
-      passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
-      at actor start time,
-    - deliver a references to the daemon lifetime fixture, which
-      for now is always the `_setup_persistent_brokerd()` context defined
-      above.
-
-    '''
-    from ..brokers import get_brokermod
-    brokermod = get_brokermod(brokername)
-    modpath: str = brokermod.__name__
-
-    start_actor_kwargs['name'] = f'brokerd.{brokername}'
-    start_actor_kwargs.update(
-        getattr(
-            brokermod,
-            '_spawn_kwargs',
-            {},
-        )
+    from piker.data.feed import (
+        _bus,
+        get_feed_bus,
     )
+    global _bus
+    assert not _bus
 
-    # XXX TODO: make this not so hacky/monkeypatched..
-    # -> we need a sane way to configure the logging level for all
-    # code running in brokerd.
-    # if utilmod := getattr(brokermod, '_util', False):
-    #     utilmod.log.setLevel(loglevel.upper())
+    async with trio.open_nursery() as service_nursery:
+        # assign a nursery to the feeds bus for spawning
+        # background tasks from clients
+        get_feed_bus(brokername, service_nursery)
 
-    # lookup actor-enabled modules declared by the backend offering the
-    # `brokerd` endpoint(s).
-    enabled: list[str]
-    enabled = start_actor_kwargs['enable_modules'] = [
-        __name__,  # so that eps from THIS mod can be invoked
-        modpath,
-    ]
-    for submodname in getattr(
-        brokermod,
-        '__enable_modules__',
-        [],
-    ):
-        subpath: str = f'{modpath}.{submodname}'
-        enabled.append(subpath)
+        # unblock caller
+        await ctx.started()
 
-    return (
-        brokermod,
-        start_actor_kwargs,  # to `ActorNursery.start_actor()`
-
-        # XXX see impl above; contains all (actor global)
-        # setup/teardown expected in all `brokerd` actor instances.
-        _setup_persistent_brokerd,
-    )
+        # we pin this task to keep the feeds manager active until the
+        # parent actor decides to tear it down
+        await trio.sleep_forever()
 
 
 async def spawn_brokerd(
@@ -198,44 +94,44 @@ async def spawn_brokerd(
 
 ) -> bool:
 
+    from piker.service import Services
     from piker.service._util import log  # use service mngr log
+
     log.info(f'Spawning {brokername} broker daemon')
 
-    (
-        brokermode,
-        tractor_kwargs,
-        daemon_fixture_ep,
-    ) = broker_init(
-        brokername,
-        loglevel,
-        **tractor_kwargs,
-    )
-
     brokermod = get_brokermod(brokername)
+    dname = f'brokerd.{brokername}'
+
     extra_tractor_kwargs = getattr(brokermod, '_spawn_kwargs', {})
     tractor_kwargs.update(extra_tractor_kwargs)
 
     # ask `pikerd` to spawn a new sub-actor and manage it under its
     # actor nursery
-    from piker.service import Services
+    modpath = brokermod.__name__
+    broker_enable = [modpath]
+    for submodname in getattr(
+        brokermod,
+        '__enable_modules__',
+        [],
+    ):
+        subpath = f'{modpath}.{submodname}'
+        broker_enable.append(subpath)
 
-    dname: str = tractor_kwargs.pop('name')  # f'brokerd.{brokername}'
     portal = await Services.actor_n.start_actor(
         dname,
-        enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
+        enable_modules=_data_mods + broker_enable,
+        loglevel=loglevel,
         debug_mode=Services.debug_mode,
         **tractor_kwargs
     )
 
-    # NOTE: the service mngr expects an already spawned actor + its
-    # portal ref in order to do non-blocking setup of brokerd
-    # service nursery.
+    # non-blocking setup of brokerd service nursery
     await Services.start_service_task(
         dname,
         portal,
 
         # signature of target root-task endpoint
-        daemon_fixture_ep,
+        _setup_persistent_brokerd,
         brokername=brokername,
         loglevel=loglevel,
     )
@@ -252,11 +148,8 @@ async def maybe_spawn_brokerd(
 
 ) -> tractor.Portal:
     '''
-    Helper to spawn a brokerd service *from* a client who wishes to
-    use the sub-actor-daemon but is fine with re-using any existing
-    and contactable `brokerd`.
-
-    Mas o menos, acts as a cached-actor-getter factory.
+    Helper to spawn a brokerd service *from* a client
+    who wishes to use the sub-actor-daemon.
 
     '''
     from piker.service import maybe_spawn_daemon

piker/brokers/_util.py

@@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -18,11 +18,10 @@
 Handy cross-broker utils.
 
 """
-from __future__ import annotations
 from functools import partial
 
 import json
-import httpx
+import asks
 import logging
 
 from ..log import (
@@ -33,8 +32,6 @@ from ..log import (
 subsys: str = 'piker.brokers'
 
 # NOTE: level should be reset by any actor that is spawned
-# as well as given a (more) explicit name/key such
-# as `piker.brokers.binance` matching the subpkg.
 log = get_logger(subsys)
 
 get_console_log = partial(
@@ -51,7 +48,6 @@ class SymbolNotFound(BrokerError):
     "Symbol not found by broker search"
 
 
-# TODO: these should probably be moved to `.tsp/.data`?
 class NoData(BrokerError):
     '''
     Symbol data not permitted or no data
@@ -61,15 +57,14 @@ class NoData(BrokerError):
     def __init__(
         self,
         *args,
-        info: dict|None = None,
+        frame_size: int = 1000,
 
     ) -> None:
         super().__init__(*args)
-        self.info: dict|None = info
 
         # when raised, machinery can check if the backend
         # set a "frame size" for doing datetime calcs.
-        # self.frame_size: int = 1000
+        self.frame_size: int = 1000
 
 
 class DataUnavailable(BrokerError):
@@ -91,18 +86,16 @@ class DataThrottle(BrokerError):
 
 
 def resproc(
-    resp: httpx.Response,
+    resp: asks.response_objects.Response,
     log: logging.Logger,
     return_json: bool = True,
     log_resp: bool = False,
 
-) -> httpx.Response:
-    '''
-    Process response and return its json content.
+) -> asks.response_objects.Response:
+    """Process response and return its json content.
 
     Raise the appropriate error on non-200 OK responses.
-
-    '''
+    """
     if not resp.status_code == 200:
         raise BrokerError(resp.body)
     try:

piker/brokers/binance.py

@@ -0,0 +1,684 @@
+# piker: trading gear for hackers
+# Copyright (C)
+#   Guillermo Rodriguez
+#   Tyler Goodlet
+#   (in stewardship for pikers)
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+"""
+Binance backend
+
+"""
+from contextlib import (
+    asynccontextmanager as acm,
+    aclosing,
+)
+from datetime import datetime
+from decimal import Decimal
+import itertools
+from typing import (
+    Any, Union, Optional,
+    AsyncGenerator, Callable,
+)
+import time
+
+import trio
+from trio_typing import TaskStatus
+import pendulum
+import asks
+from fuzzywuzzy import process as fuzzy
+import numpy as np
+import tractor
+
+from .._cacheables import async_lifo_cache
+from ..accounting._mktinfo import (
+    Asset,
+    MktPair,
+    digits_to_dec,
+)
+from .._cacheables import open_cached_client
+from ._util import (
+    resproc,
+    SymbolNotFound,
+    DataUnavailable,
+)
+from ._util import (
+    log,
+    get_console_log,
+)
+from ..data.types import Struct
+from ..data.validate import FeedInit
+from ..data._web_bs import (
+    open_autorecon_ws,
+    NoBsWs,
+)
+
+
+_url = 'https://api.binance.com'
+
+
+# Broker specific ohlc schema (rest)
+_ohlc_dtype = [
+    ('index', int),
+    ('time', int),
+    ('open', float),
+    ('high', float),
+    ('low', float),
+    ('close', float),
+    ('volume', float),
+    ('bar_wap', float),  # will be zeroed by sampler if not filled
+
+    # XXX: some additional fields are defined in the docs:
+    # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
+
+    # ('close_time', int),
+    # ('quote_vol', float),
+    # ('num_trades', int),
+    # ('buy_base_vol', float),
+    # ('buy_quote_vol', float),
+    # ('ignore', float),
+]
+
+# UI components allow this to be declared such that additional
+# (historical) fields can be exposed.
+ohlc_dtype = np.dtype(_ohlc_dtype)
+
+_show_wap_in_history = False
+
+
+# https://binance-docs.github.io/apidocs/spot/en/#exchange-information
+
+# TODO: make this frozen again by pre-processing the
+# filters list to a dict at init time?
+class Pair(Struct, frozen=True):
+    symbol: str
+    status: str
+
+    baseAsset: str
+    baseAssetPrecision: int
+    cancelReplaceAllowed: bool
+    allowTrailingStop: bool
+    quoteAsset: str
+    quotePrecision: int
+    quoteAssetPrecision: int
+
+    baseCommissionPrecision: int
+    quoteCommissionPrecision: int
+
+    orderTypes: list[str]
+
+    icebergAllowed: bool
+    ocoAllowed: bool
+    quoteOrderQtyMarketAllowed: bool
+    isSpotTradingAllowed: bool
+    isMarginTradingAllowed: bool
+
+    defaultSelfTradePreventionMode: str
+    allowedSelfTradePreventionModes: list[str]
+
+    filters: dict[
+        str,
+        Union[str, int, float]
+    ]
+    permissions: list[str]
+
+    @property
+    def price_tick(self) -> Decimal:
+        # XXX: lul, after manually inspecting the response format we
+        # just directly pick out the info we need
+        step_size: str = self.filters['PRICE_FILTER']['tickSize'].rstrip('0')
+        return Decimal(step_size)
+
+    @property
+    def size_tick(self) -> Decimal:
+        step_size: str = self.filters['LOT_SIZE']['stepSize'].rstrip('0')
+        return Decimal(step_size)
+
+
+class OHLC(Struct):
+    '''
+    Description of the flattened OHLC quote format.
+
+    For schema details see:
+    https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-streams
+
+    '''
+    time: int
+
+    open: float
+    high: float
+    low: float
+    close: float
+    volume: float
+
+    close_time: int
+
+    quote_vol: float
+    num_trades: int
+    buy_base_vol: float
+    buy_quote_vol: float
+    ignore: int
+
+    # null the place holder for `bar_wap` until we
+    # figure out what to extract for this.
+    bar_wap: float = 0.0
+
+
+class L1(Struct):
+    # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
+
+    update_id: int
+    sym: str
+
+    bid: float
+    bsize: float
+    ask: float
+    asize: float
+
+
+# convert datetime obj timestamp to unixtime in milliseconds
+def binance_timestamp(
+    when: datetime
+) -> int:
+    return int((when.timestamp() * 1000) + (when.microsecond / 1000))
+
+
+class Client:
+
+    def __init__(self) -> None:
+        self._sesh = asks.Session(connections=4)
+        self._sesh.base_location = _url
+        self._pairs: dict[str, Pair] = {}
+
+    async def _api(
+        self,
+        method: str,
+        params: dict,
+    ) -> dict[str, Any]:
+        resp = await self._sesh.get(
+            path=f'/api/v3/{method}',
+            params=params,
+            timeout=float('inf')
+        )
+        return resproc(resp, log)
+
+    async def exch_info(
+
+        self,
+        sym: str | None = None,
+
+    ) -> dict[str, Pair] | Pair:
+        '''
+        Fresh exchange-pairs info query for symbol ``sym: str``:
+        https://binance-docs.github.io/apidocs/spot/en/#exchange-information
+
+        '''
+        cached_pair = self._pairs.get(sym)
+        if cached_pair:
+            return cached_pair
+
+        # retrieve all symbols by default
+        params = {}
+        if sym is not None:
+            sym = sym.lower()
+            params = {'symbol': sym}
+
+        resp = await self._api('exchangeInfo', params=params)
+        entries = resp['symbols']
+        if not entries:
+            raise SymbolNotFound(f'{sym} not found:\n{resp}')
+
+        # pre-process .filters field into a table
+        pairs = {}
+        for item in entries:
+            symbol = item['symbol']
+            filters = {}
+            filters_ls: list = item.pop('filters')
+            for entry in filters_ls:
+                ftype = entry['filterType']
+                filters[ftype] = entry
+
+            pairs[symbol] = Pair(
+                filters=filters,
+                **item,
+            )
+
+        # pairs = {
+        #     item['symbol']: Pair(**item) for item in entries
+        # }
+        self._pairs.update(pairs)
+
+        if sym is not None:
+            return pairs[sym]
+        else:
+            return self._pairs
+
+    symbol_info = exch_info
+
+    async def search_symbols(
+        self,
+        pattern: str,
+        limit: int = None,
+    ) -> dict[str, Any]:
+        if self._pairs is not None:
+            data = self._pairs
+        else:
+            data = await self.exch_info()
+
+        matches = fuzzy.extractBests(
+            pattern,
+            data,
+            score_cutoff=50,
+        )
+        # repack in dict form
+        return {item[0]['symbol']: item[0]
+                for item in matches}
+
+    async def bars(
+        self,
+        symbol: str,
+        start_dt: Optional[datetime] = None,
+        end_dt: Optional[datetime] = None,
+        limit: int = 1000,  # <- max allowed per query
+        as_np: bool = True,
+
+    ) -> dict:
+
+        if end_dt is None:
+            end_dt = pendulum.now('UTC').add(minutes=1)
+
+        if start_dt is None:
+            start_dt = end_dt.start_of(
+                'minute').subtract(minutes=limit)
+
+        start_time = binance_timestamp(start_dt)
+        end_time = binance_timestamp(end_dt)
+
+        # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
+        bars = await self._api(
+            'klines',
+            params={
+                'symbol': symbol.upper(),
+                'interval': '1m',
+                'startTime': start_time,
+                'endTime': end_time,
+                'limit': limit
+            }
+        )
+
+        # TODO: pack this bars scheme into a ``pydantic`` validator type:
+        # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
+
+        # TODO: we should port this to ``pydantic`` to avoid doing
+        # manual validation ourselves..
+        new_bars = []
+        for i, bar in enumerate(bars):
+
+            bar = OHLC(*bar)
+            bar.typecast()
+
+            row = []
+            for j, (name, ftype) in enumerate(_ohlc_dtype[1:]):
+
+                # TODO: maybe we should go nanoseconds on all
+                # history time stamps?
+                if name == 'time':
+                    # convert to epoch seconds: float
+                    row.append(bar.time / 1000.0)
+
+                else:
+                    row.append(getattr(bar, name))
+
+            new_bars.append((i,) + tuple(row))
+
+        array = np.array(new_bars, dtype=_ohlc_dtype) if as_np else bars
+        return array
+
+
+@acm
+async def get_client() -> Client:
+    client = Client()
+    log.info('Caching exchange infos..')
+    await client.exch_info()
+    yield client
+
+
+# validation type
+class AggTrade(Struct):
+    e: str  # Event type
+    E: int  # Event time
+    s: str  # Symbol
+    a: int  # Aggregate trade ID
+    p: float  # Price
+    q: float  # Quantity
+    f: int  # First trade ID
+    l: int  # Last trade ID
+    T: int  # Trade time
+    m: bool  # Is the buyer the market maker?
+    M: bool  # Ignore
+
+
+async def stream_messages(
+    ws: NoBsWs,
+) -> AsyncGenerator[NoBsWs, dict]:
+
+    # TODO: match syntax here!
+    msg: dict[str, Any]
+    async for msg in ws:
+        match msg:
+            # for l1 streams binance doesn't add an event type field so
+            # identify those messages by matching keys
+            # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
+            case {
+                # NOTE: this is never an old value it seems, so
+                # they are always sending real L1 spread updates.
+                'u': upid,  # update id
+                's': sym,
+                'b': bid,
+                'B': bsize,
+                'a': ask,
+                'A': asize,
+            }:
+                # TODO: it would be super nice to have a `L1` piker type
+                # which "renders" incremental tick updates from a packed
+                # msg-struct:
+                # - backend msgs after packed into the type such that we
+                #   can reduce IPC usage but without each backend having
+                #   to do that incremental update logic manually B)
+                # - would it maybe be more efficient to use this instead?
+                #   https://binance-docs.github.io/apidocs/spot/en/#diff-depth-stream
+                l1 = L1(
+                    update_id=upid,
+                    sym=sym,
+                    bid=bid,
+                    bsize=bsize,
+                    ask=ask,
+                    asize=asize,
+                )
+                l1.typecast()
+
+                # repack into piker's tick-quote format
+                yield 'l1', {
+                    'symbol': l1.sym,
+                    'ticks': [
+                        {
+                            'type': 'bid',
+                            'price': l1.bid,
+                            'size': l1.bsize,
+                        },
+                        {
+                            'type': 'bsize',
+                            'price': l1.bid,
+                            'size': l1.bsize,
+                        },
+                        {
+                            'type': 'ask',
+                            'price': l1.ask,
+                            'size': l1.asize,
+                        },
+                        {
+                            'type': 'asize',
+                            'price': l1.ask,
+                            'size': l1.asize,
+                        }
+                    ]
+                }
+
+            # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
+            case {
+                'e': 'aggTrade',
+            }:
+                # NOTE: this is purely for a definition,
+                # ``msgspec.Struct`` does not runtime-validate until you
+                # decode/encode, see:
+                # https://jcristharif.com/msgspec/structs.html#type-validation
+                msg = AggTrade(**msg)
+                msg.typecast()
+                yield 'trade', {
+                    'symbol': msg.s,
+                    'last': msg.p,
+                    'brokerd_ts': time.time(),
+                    'ticks': [{
+                        'type': 'trade',
+                        'price': msg.p,
+                        'size': msg.q,
+                        'broker_ts': msg.T,
+                    }],
+                }
+
+
+def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
+    """Create a request subscription packet dict.
+
+    https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
+    """
+    return {
+        'method': 'SUBSCRIBE',
+        'params': [
+            f'{pair.lower()}@{sub_name}'
+            for pair in pairs
+        ],
+        'id': uid
+    }
+
+
+@acm
+async def open_history_client(
+    mkt: MktPair,
+
+) -> tuple[Callable, int]:
+
+    symbol: str = mkt.bs_fqme
+
+    # TODO implement history getter for the new storage layer.
+    async with open_cached_client('binance') as client:
+
+        async def get_ohlc(
+            timeframe: float,
+            end_dt: datetime | None = None,
+            start_dt: datetime | None = None,
+
+        ) -> tuple[
+            np.ndarray,
+            datetime,  # start
+            datetime,  # end
+        ]:
+            if timeframe != 60:
+                raise DataUnavailable('Only 1m bars are supported')
+
+            array = await client.bars(
+                symbol,
+                start_dt=start_dt,
+                end_dt=end_dt,
+            )
+            times = array['time']
+            if (
+                end_dt is None
+            ):
+                inow = round(time.time())
+                if (inow - times[-1]) > 60:
+                    await tractor.breakpoint()
+
+            start_dt = pendulum.from_timestamp(times[0])
+            end_dt = pendulum.from_timestamp(times[-1])
+
+            return array, start_dt, end_dt
+
+        yield get_ohlc, {'erlangs': 3, 'rate': 3}
+
+
+@async_lifo_cache()
+async def get_mkt_info(
+    fqme: str,
+
+) -> tuple[MktPair, Pair]:
+
+    async with open_cached_client('binance') as client:
+
+        pair: Pair = await client.exch_info(fqme.upper())
+        mkt = MktPair(
+            dst=Asset(
+                name=pair.baseAsset,
+                atype='crypto',
+                tx_tick=digits_to_dec(pair.baseAssetPrecision),
+            ),
+            src=Asset(
+                name=pair.quoteAsset,
+                atype='crypto',
+                tx_tick=digits_to_dec(pair.quoteAssetPrecision),
+            ),
+            price_tick=pair.price_tick,
+            size_tick=pair.size_tick,
+            bs_mktid=pair.symbol,
+            broker='binance',
+        )
+        both = mkt, pair
+        return both
+
+
+async def stream_quotes(
+
+    send_chan: trio.abc.SendChannel,
+    symbols: list[str],
+    feed_is_live: trio.Event,
+    loglevel: str = None,
+
+    # startup sync
+    task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
+
+) -> None:
+    # XXX: required to propagate ``tractor`` loglevel to piker logging
+    get_console_log(loglevel or tractor.current_actor().loglevel)
+
+    async with (
+        send_chan as send_chan,
+    ):
+        init_msgs: list[FeedInit] = []
+        for sym in symbols:
+            mkt, pair = await get_mkt_info(sym)
+
+            # build out init msgs according to latest spec
+            init_msgs.append(
+                FeedInit(mkt_info=mkt)
+            )
+
+        iter_subids = itertools.count()
+
+        @acm
+        async def subscribe(ws: NoBsWs):
+            # setup subs
+
+            subid: int = next(iter_subids)
+
+            # trade data (aka L1)
+            # https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
+            l1_sub = make_sub(symbols, 'bookTicker', subid)
+            await ws.send_msg(l1_sub)
+
+            # aggregate (each order clear by taker **not** by maker)
+            # trades data:
+            # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
+            agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
+            await ws.send_msg(agg_trades_sub)
+
+            # might get ack from ws server, or maybe some
+            # other msg still in transit..
+            res = await ws.recv_msg()
+            subid: str | None = res.get('id')
+            if subid:
+                assert res['id'] == subid
+
+            yield
+
+            subs = []
+            for sym in symbols:
+                subs.append("{sym}@aggTrade")
+                subs.append("{sym}@bookTicker")
+
+            # unsub from all pairs on teardown
+            if ws.connected():
+                await ws.send_msg({
+                    "method": "UNSUBSCRIBE",
+                    "params": subs,
+                    "id": subid,
+                })
+
+                # XXX: do we need to ack the unsub?
+                # await ws.recv_msg()
+
+        async with (
+            open_autorecon_ws(
+                # XXX: see api docs which show diff addr?
+                # https://developers.binance.com/docs/binance-trading-api/websocket_api#general-api-information
+                # 'wss://ws-api.binance.com:443/ws-api/v3',
+                'wss://stream.binance.com/ws',
+                fixture=subscribe,
+            ) as ws,
+
+            # avoid stream-gen closure from breaking trio..
+            aclosing(stream_messages(ws)) as msg_gen,
+        ):
+            typ, quote = await anext(msg_gen)
+
+            # pull a first quote and deliver
+            while typ != 'trade':
+                typ, quote = await anext(msg_gen)
+
+            task_status.started((init_msgs, quote))
+
+            # signal to caller feed is ready for consumption
+            feed_is_live.set()
+
+            # import time
+            # last = time.time()
+
+            # start streaming
+            async for typ, msg in msg_gen:
+
+                # period = time.time() - last
+                # hz = 1/period if period else float('inf')
+                # if hz > 60:
+                #     log.info(f'Binance quotez : {hz}')
+
+                topic = msg['symbol'].lower()
+                await send_chan.send({topic: msg})
+                # last = time.time()
+
+
+@tractor.context
+async def open_symbol_search(
+    ctx: tractor.Context,
+) -> Client:
+    async with open_cached_client('binance') as client:
+
+        # load all symbols locally for fast search
+        cache = await client.exch_info()
+        await ctx.started()
+
+        async with ctx.open_stream() as stream:
+
+            async for pattern in stream:
+                # results = await client.exch_info(sym=pattern.upper())
+
+                matches = fuzzy.extractBests(
+                    pattern,
+                    cache,
+                    score_cutoff=50,
+                )
+                # repack in dict form
+                await stream.send({
+                    item[0].symbol: item[0]
+                    for item in matches
+                })

piker/brokers/binance/__init__.py

@@ -1,60 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C)
-#   Guillermo Rodriguez (aka ze jefe)
-#   Tyler Goodlet
-#   (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-binancial secs on the floor, in the office, behind the dumpster.
-
-"""
-from .api import (
-    get_client,
-)
-from .feed import (
-    get_mkt_info,
-    open_history_client,
-    open_symbol_search,
-    stream_quotes,
-)
-from .broker import (
-    open_trade_dialog,
-    get_cost,
-)
-from .venues import (
-    SpotPair,
-    FutesPair,
-)
-
-__all__ = [
-    'get_client',
-    'get_mkt_info',
-    'get_cost',
-    'SpotPair',
-    'FutesPair',
-    'open_trade_dialog',
-    'open_history_client',
-    'open_symbol_search',
-    'stream_quotes',
-]
-
-
-# `brokerd` modules
-__enable_modules__: list[str] = [
-    'api',
-    'feed',
-    'broker',
-]

piker/brokers/binance/broker.py

@@ -1,710 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C)
-#   Guillermo Rodriguez (aka ze jefe)
-#   Tyler Goodlet
-#   (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Live order control B)
-
-'''
-from __future__ import annotations
-from pprint import pformat
-from typing import (
-    Any,
-    AsyncIterator,
-)
-import time
-from time import time_ns
-
-from bidict import bidict
-import tractor
-import trio
-
-from piker.accounting import (
-    Asset,
-)
-from piker.brokers._util import (
-    get_logger,
-)
-from piker.data._web_bs import (
-    open_autorecon_ws,
-    NoBsWs,
-)
-from piker.brokers import (
-    open_cached_client,
-    BrokerError,
-)
-from piker.clearing import (
-    OrderDialogs,
-)
-from piker.clearing._messages import (
-    BrokerdOrder,
-    BrokerdOrderAck,
-    BrokerdStatus,
-    BrokerdPosition,
-    BrokerdFill,
-    BrokerdCancel,
-    BrokerdError,
-    Status,
-    Order,
-)
-from .venues import (
-    Pair,
-    _futes_ws,
-    _testnet_futes_ws,
-)
-from .api import Client
-
-log = get_logger('piker.brokers.binance')
-
-
-# Fee schedule template, mostly for paper engine fees modelling.
-# https://www.binance.com/en/support/faq/what-are-market-makers-and-takers-360007720071
-def get_cost(
-    price: float,
-    size: float,
-    is_taker: bool = False,
-
-) -> float:
-
-    # https://www.binance.com/en/fee/trading
-    cb: float = price * size
-    match is_taker:
-        case True:
-            return cb * 0.001000
-
-        case False if cb < 1e6:
-            return cb * 0.001000
-
-        case False if 1e6 >= cb < 5e6:
-            return cb * 0.000900
-
-        # NOTE: there's more but are you really going
-        # to have a cb bigger then this per trade?
-        case False if cb >= 5e6:
-            return cb * 0.000800
-
-
-async def handle_order_requests(
-    ems_order_stream: tractor.MsgStream,
-    client: Client,
-    dids: bidict[str, str],
-    dialogs: OrderDialogs,
-
-) -> None:
-    '''
-    Receive order requests from `emsd`, translate tramsit API calls and transmit.
-
-    '''
-    msg: dict | BrokerdOrder | BrokerdCancel
-    async for msg in ems_order_stream:
-        log.info(f'Rx order request:\n{pformat(msg)}')
-        match msg:
-            case {
-                'action': 'cancel',
-            }:
-                cancel = BrokerdCancel(**msg)
-                existing: BrokerdOrder | None = dialogs.get(cancel.oid)
-                if not existing:
-                    log.error(
-                        f'NO Existing order-dialog for {cancel.oid}!?'
-                    )
-                    await ems_order_stream.send(BrokerdError(
-                        oid=cancel.oid,
-
-                        # TODO: do we need the symbol?
-                        # https://github.com/pikers/piker/issues/514
-                        symbol='unknown',
-
-                        reason=(
-                            'Invalid `binance` order request dialog oid',
-                        )
-                    ))
-                    continue
-
-                else:
-                    symbol: str = existing['symbol']
-                    try:
-                        await client.submit_cancel(
-                            symbol,
-                            cancel.oid,
-                        )
-                    except BrokerError as be:
-                        await ems_order_stream.send(
-                            BrokerdError(
-                                oid=msg['oid'],
-                                symbol=symbol,
-                                reason=(
-                                    '`binance` CANCEL failed:\n'
-                                    f'{be}'
-                                ))
-                        )
-                        continue
-
-            case {
-                'account': ('binance.usdtm' | 'binance.spot') as account,
-                'action': action,
-            } if action in {'buy', 'sell'}:
-
-                # validate
-                order = BrokerdOrder(**msg)
-                oid: str = order.oid  # emsd order id
-                modify: bool = False
-
-                # NOTE: check and report edits
-                if existing := dialogs.get(order.oid):
-                    log.info(
-                        f'Existing order for {oid} updated:\n'
-                        f'{pformat(existing.maps[-1])} -> {pformat(msg)}'
-                    )
-                    modify = True
-
-                    # only add new msg AFTER the existing check
-                    dialogs.add_msg(oid, msg)
-
-                else:
-                    # XXX NOTE: update before the ack!
-                    # track latest request state such that map
-                    # lookups start at the most recent msg and then
-                    # scan reverse-chronologically.
-                    dialogs.add_msg(oid, msg)
-
-                    # XXX: ACK the request **immediately** before sending
-                    # the api side request to ensure the ems maps the oid ->
-                    # reqid correctly!
-                    resp = BrokerdOrderAck(
-                        oid=oid,  # ems order request id
-                        reqid=oid,  # our custom int mapping
-                        account='binance',  # piker account
-                    )
-                    await ems_order_stream.send(resp)
-
-                # call our client api to submit the order
-                # NOTE: modifies only require diff key for user oid:
-                # https://binance-docs.github.io/apidocs/futures/en/#modify-order-trade
-                try:
-                    reqid = await client.submit_limit(
-                        symbol=order.symbol,
-                        side=order.action,
-                        quantity=order.size,
-                        price=order.price,
-                        oid=oid,
-                        modify=modify,
-                    )
-
-                    # SMH they do gen their own order id: ints.. 
-                    # assert reqid == order.oid
-                    dids[order.oid] = reqid
-
-                except BrokerError as be:
-                    await ems_order_stream.send(
-                        BrokerdError(
-                            oid=msg['oid'],
-                            symbol=msg['symbol'],
-                            reason=(
-                                '`binance` request failed:\n'
-                                f'{be}'
-                            ))
-                    )
-                    continue
-
-            case _:
-                account = msg.get('account')
-                if account not in {'binance.spot', 'binance.futes'}:
-                    log.error(
-                        'Order request does not have a valid binance account name?\n'
-                        'Only one of\n'
-                        '- `binance.spot` or,\n'
-                        '- `binance.usdtm`\n'
-                        'is currently valid!'
-                    )
-                await ems_order_stream.send(
-                    BrokerdError(
-                        oid=msg['oid'],
-                        symbol=msg['symbol'],
-                        reason=(
-                            f'Invalid `binance` broker request msg:\n{msg}'
-                        ))
-                )
-
-
-@tractor.context
-async def open_trade_dialog(
-    ctx: tractor.Context,
-
-) -> AsyncIterator[dict[str, Any]]:
-
-    # TODO: how do we set this from the EMS such that
-    # positions are loaded from the correct venue on the user
-    # stream at startup? (that is in an attempt to support both
-    # spot and futes markets?)
-    # - I guess we just want to instead start 2 separate user
-    #   stream tasks right? unless we want another actor pool?
-    #   XXX: see issue: <urlhere>
-    venue_name: str = 'futes'
-    venue_mode: str = 'usdtm_futes'
-    account_name: str = 'usdtm'
-    use_testnet: bool = False
-
-    # TODO: if/when we add .accounting support we need to
-    # do a open_symcache() call.. though maybe we can hide
-    # this in a new async version of open_account()?
-    async with open_cached_client('binance') as client:
-        subconf: dict|None = client.conf.get(venue_name)
-
-        # XXX: if no futes.api_key or spot.api_key has been set we
-        # always fall back to the paper engine!
-        if (
-            not subconf
-            or
-            not subconf.get('api_key')
-        ):
-            await ctx.started('paper')
-            return
-
-        use_testnet: bool = subconf.get('use_testnet', False)
-
-    async with (
-        open_cached_client('binance') as client,
-    ):
-        client.mkt_mode: str = venue_mode
-
-        # TODO: map these wss urls depending on spot or futes
-        # setting passed when this task is spawned?
-        wss_url: str = _futes_ws if not use_testnet else _testnet_futes_ws
-
-        wss: NoBsWs
-        async with (
-            client.manage_listen_key() as listen_key,
-            open_autorecon_ws(f'{wss_url}/?listenKey={listen_key}') as wss,
-        ):
-            nsid: int = time_ns()
-            await wss.send_msg({
-                # "method": "SUBSCRIBE",
-                "method": "REQUEST",
-                "params":
-                [
-                    f"{listen_key}@account",
-                    f"{listen_key}@balance",
-                    f"{listen_key}@position",
-
-                    # TODO: does this even work!? seems to cause
-                    # a hang on the first msg..? lelelel.
-                    # f"{listen_key}@order",
-                ],
-                "id": nsid
-            })
-
-            with trio.fail_after(6):
-                msg = await wss.recv_msg()
-                assert msg['id'] == nsid
-
-            # TODO: load other market wide data / statistics:
-            # - OI: https://binance-docs.github.io/apidocs/futures/en/#open-interest
-            # - OI stats: https://binance-docs.github.io/apidocs/futures/en/#open-interest-statistics
-            accounts: bidict[str, str] = bidict({'binance.usdtm': None})
-            balances: dict[Asset, float] = {}
-            positions: list[BrokerdPosition] = []
-
-            for resp_dict in msg['result']:
-                resp: dict = resp_dict['res']
-                req: str = resp_dict['req']
-
-                # @account response should be something like:
-                # {'accountAlias': 'sRFzFzAuuXsR',
-                #  'canDeposit': True,
-                #  'canTrade': True,
-                #  'canWithdraw': True,
-                #  'feeTier': 0}
-                if 'account' in req:
-                    # NOTE: fill in the hash-like key/alias binance
-                    # provides for the account.
-                    alias: str = resp['accountAlias']
-                    accounts['binance.usdtm'] = alias
-
-                # @balance response:
-                # {'accountAlias': 'sRFzFzAuuXsR',
-                #      'balances': [{'asset': 'BTC',
-                #                    'availableBalance': '0.00000000',
-                #                    'balance': '0.00000000',
-                #                    'crossUnPnl': '0.00000000',
-                #                    'crossWalletBalance': '0.00000000',
-                #                    'maxWithdrawAmount': '0.00000000',
-                #                    'updateTime': 0}]
-                #                     ...
-                # }
-                elif 'balance' in req:
-                    for entry in resp['balances']:
-                        name: str = entry['asset']
-                        balance: float = float(entry['balance'])
-                        last_update_t: int = entry['updateTime']
-
-                        spot_asset: Asset = client._venue2assets['spot'][name]
-
-                        if balance > 0:
-                            balances[spot_asset] = (balance, last_update_t)
-                            # await tractor.pause()
-
-                # @position response:
-                # {'positions': [{'entryPrice': '0.0',
-                #                    'isAutoAddMargin': False,
-                #                    'isolatedMargin': '0',
-                #                    'leverage': 20,
-                #                    'liquidationPrice': '0',
-                #                    'marginType': 'CROSSED',
-                #                    'markPrice': '0.60289650',
-                #                    'markPrice': '0.00000000',
-                #                    'maxNotionalValue': '25000',
-                #                    'notional': '0',
-                #                    'positionAmt': '0',
-                #                    'positionSide': 'BOTH',
-                #                    'symbol': 'ETHUSDT_230630',
-                #                    'unRealizedProfit': '0.00000000',
-                #                    'updateTime': 1672741444894}
-                #                    ...
-                # }
-                elif 'position' in req:
-                    for entry in resp['positions']:
-                        bs_mktid: str = entry['symbol']
-                        entry_size: float = float(entry['positionAmt'])
-
-                        pair: Pair | None = client._venue2pairs[
-                            venue_mode
-                        ].get(bs_mktid)
-                        if (
-                            pair
-                            and entry_size > 0
-                        ):
-                            entry_price: float = float(entry['entryPrice'])
-
-                            ppmsg = BrokerdPosition(
-                                broker='binance',
-                                account=f'binance.{account_name}',
-
-                                # TODO: maybe we should be passing back
-                                # a `MktPair` here?
-                                symbol=pair.bs_fqme.lower() + '.binance',
-
-                                size=entry_size,
-                                avg_price=entry_price,
-                            )
-                            positions.append(ppmsg)
-
-                        if pair is None:
-                            log.warning(
-                                f'`{bs_mktid}` Position entry but no market pair?\n'
-                                f'{pformat(entry)}\n'
-                            )
-
-            await ctx.started((
-                positions,
-                list(accounts)
-            ))
-
-            # TODO: package more state tracking into the dialogs API?
-            # - hmm maybe we could include `OrderDialogs.dids:
-            #   bidict` as part of the interface and then ask for
-            #   a reqid field to be passed at init?
-            #   |-> `OrderDialog(reqid_field='orderId')` kinda thing?
-            # - also maybe bundle in some kind of dialog to account
-            #   table?
-            dialogs = OrderDialogs()
-            dids: dict[str, int] = bidict()
-
-            # TODO: further init setup things to get full EMS and
-            # .accounting support B)
-            # - live order loading via user stream subscription and
-            #   update to the order dialog table.
-            #   - MAKE SURE we add live orders loaded during init
-            #   into the dialogs table to ensure they can be
-            #   cancelled, meaning we can do a symbol lookup.
-            # - position loading using `piker.accounting` subsys
-            #   and comparison with binance's own position calcs.
-            # - load pps and accounts using accounting apis, write
-            #   the ledger and account files
-            #   - table: Account
-            #   - ledger: TransactionLedger
-
-            async with (
-                trio.open_nursery() as tn,
-                ctx.open_stream() as ems_stream,
-            ):
-                # deliver all pre-exist open orders to EMS thus syncing
-                # state with existing live limits reported by them.
-                order: Order
-                for order in await client.get_open_orders():
-                    status_msg = Status(
-                        time_ns=time.time_ns(),
-                        resp='open',
-                        oid=order.oid,
-                        reqid=order.oid,
-
-                        # embedded order info
-                        req=order,
-                        src='binance',
-                    )
-                    dialogs.add_msg(order.oid, order.to_dict())
-                    await ems_stream.send(status_msg)
-
-                tn.start_soon(
-                    handle_order_requests,
-                    ems_stream,
-                    client,
-                    dids,
-                    dialogs,
-                )
-                tn.start_soon(
-                    handle_order_updates,
-                    venue_mode,
-                    account_name,
-                    client,
-                    ems_stream,
-                    wss,
-                    dialogs,
-
-                )
-
-                await trio.sleep_forever()
-
-
-async def handle_order_updates(
-    venue: str,
-    account_name: str,
-    client: Client,
-    ems_stream: tractor.MsgStream,
-    wss: NoBsWs,
-    dialogs: OrderDialogs,
-
-) -> None:
-    '''
-    Main msg handling loop for all things order management.
-
-    This code is broken out to make the context explicit and state
-    variables defined in the signature clear to the reader.
-
-    '''
-    async for msg in wss:
-        log.info(f'Rx USERSTREAM msg:\n{pformat(msg)}')
-        match msg:
-
-            # ORDER update
-            # spot: https://binance-docs.github.io/apidocs/spot/en/#payload-balance-update
-            # futes: https://binance-docs.github.io/apidocs/futures/en/#event-order-update
-            # futes: https://binance-docs.github.io/apidocs/futures/en/#event-balance-and-position-update
-            # {'o': {
-            #    'L': '0',
-            #    'N': 'USDT',
-            #    'R': False,
-            #    'S': 'BUY',
-            #    'T': 1687028772484,
-            #    'X': 'NEW',
-            #    'a': '0',
-            #    'ap': '0',
-            #    'b': '7012.06520',
-            #    'c': '518d4122-8d3e-49b0-9a1e-1fabe6f62e4c',
-            #    'cp': False,
-            #    'f': 'GTC',
-            #    'i': 3376956924,
-            #    'l': '0',
-            #    'm': False,
-            #    'n': '0',
-            #    'o': 'LIMIT',
-            #    'ot': 'LIMIT',
-            #    'p': '21136.80',
-            #    'pP': False,
-            #    'ps': 'BOTH',
-            #    'q': '0.047',
-            #    'rp': '0',
-            #    's': 'BTCUSDT',
-            #    'si': 0,
-            #    'sp': '0',
-            #    'ss': 0,
-            #    't': 0,
-            #    'wt': 'CONTRACT_PRICE',
-            #    'x': 'NEW',
-            #    'z': '0'}
-            # }
-            case {
-                # 'e': 'executionReport',
-                'e': 'ORDER_TRADE_UPDATE',
-                'T': int(epoch_ms),
-                'o': {
-                    's': bs_mktid,
-
-                    # XXX NOTE XXX see special ids for market
-                    # events or margin calls:
-                    # // special client order id:
-                    # // starts with "autoclose-": liquidation order
-                    # // "adl_autoclose": ADL auto close order
-                    # // "settlement_autoclose-": settlement order
-                    #     for delisting or delivery
-                    'c': oid,
-                    # 'i': reqid,  # binance internal int id
-
-                    # prices
-                    'a': submit_price,
-                    'ap': avg_price,
-                    'L': fill_price,
-
-                    # sizing
-                    'q': req_size,
-                    'l': clear_size_filled,  # this event
-                    'z': accum_size_filled,  # accum
-
-                    # commissions
-                    'n': cost,
-                    'N': cost_asset,
-
-                    # state
-                    'S': side,
-                    'X': status,
-                },
-            } as order_msg:
-                log.info(
-                    f'{status} for {side} ORDER oid: {oid}\n'
-                    f'bs_mktid: {bs_mktid}\n\n'
-
-                    f'order size: {req_size}\n'
-                    f'cleared size: {clear_size_filled}\n'
-                    f'accum filled size: {accum_size_filled}\n\n'
-
-                    f'submit price: {submit_price}\n'
-                    f'fill_price: {fill_price}\n'
-                    f'avg clearing price: {avg_price}\n\n'
-
-                    f'cost: {cost}@{cost_asset}\n'
-                )
-
-                # status remap from binance to piker's
-                # status set:
-                # - NEW
-                # - PARTIALLY_FILLED
-                # - FILLED
-                # - CANCELED
-                # - EXPIRED
-                # https://binance-docs.github.io/apidocs/futures/en/#event-order-update
-
-                req_size: float = float(req_size)
-                accum_size_filled: float = float(accum_size_filled)
-                fill_price: float = float(fill_price)
-
-                match status:
-                    case 'PARTIALLY_FILLED' | 'FILLED':
-                        status = 'fill'
-
-                        fill_msg = BrokerdFill(
-                            time_ns=time_ns(),
-                            # reqid=reqid,
-                            reqid=oid,
-
-                            # just use size value for now?
-                            # action=action,
-                            size=clear_size_filled,
-                            price=fill_price,
-
-                            # TODO: maybe capture more msg data
-                            # i.e fees?
-                            broker_details={'name': 'broker'} | order_msg,
-                            broker_time=time.time(),
-                        )
-                        await ems_stream.send(fill_msg)
-
-                        if accum_size_filled == req_size:
-                            status = 'closed'
-                            dialogs.pop(oid)
-
-                    case 'NEW':
-                        status = 'open'
-
-                    case 'EXPIRED':
-                        status = 'canceled'
-                        dialogs.pop(oid)
-
-                    case _:
-                        status = status.lower()
-
-                resp = BrokerdStatus(
-                    time_ns=time_ns(),
-                    # reqid=reqid,
-                    reqid=oid,
-
-                    # TODO: i feel like we don't need to make the
-                    # ems and upstream clients aware of this?
-                    # account='binance.usdtm',
-
-                    status=status,
-
-                    filled=accum_size_filled,
-                    remaining=req_size - accum_size_filled,
-                    broker_details={
-                        'name': 'binance',
-                        'broker_time': epoch_ms / 1000.
-                    }
-                )
-                await ems_stream.send(resp)
-
-            # ACCOUNT and POSITION update B)
-            # {
-            #  'E': 1687036749218,
-            #  'e': 'ACCOUNT_UPDATE'
-            #  'T': 1687036749215,
-            #  'a': {'B': [{'a': 'USDT',
-            #               'bc': '0',
-            #               'cw': '1267.48920735',
-            #               'wb': '1410.90245576'}],
-            #        'P': [{'cr': '-3292.10973007',
-            #               'ep': '26349.90000',
-            #               'iw': '143.41324841',
-            #               'ma': 'USDT',
-            #               'mt': 'isolated',
-            #               'pa': '0.038',
-            #               'ps': 'BOTH',
-            #               's': 'BTCUSDT',
-            #               'up': '5.17555453'}],
-            #        'm': 'ORDER'},
-            # }
-            case {
-                'T': int(epoch_ms),
-                'e': 'ACCOUNT_UPDATE',
-                'a': {
-                    'P': [{
-                        's': bs_mktid,
-                        'pa': pos_amount,
-                        'ep': entry_price,
-                    }],
-                },
-            }:
-                # real-time relay position updates back to EMS
-                pair: Pair | None = client._venue2pairs[venue].get(bs_mktid)
-                ppmsg = BrokerdPosition(
-                    broker='binance',
-                    account=f'binance.{account_name}',
-
-                    # TODO: maybe we should be passing back
-                    # a `MktPair` here?
-                    symbol=pair.bs_fqme.lower() + '.binance',
-
-                    size=float(pos_amount),
-                    avg_price=float(entry_price),
-                )
-                await ems_stream.send(ppmsg)
-
-            case _:
-                log.warning(
-                    'Unhandled event:\n'
-                    f'{pformat(msg)}'
-                )

piker/brokers/binance/feed.py

@@ -1,557 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Real-time and historical data feed endpoints.
-
-'''
-from __future__ import annotations
-from contextlib import (
-    asynccontextmanager as acm,
-    aclosing,
-)
-from datetime import datetime
-from functools import (
-    partial,
-)
-import itertools
-from pprint import pformat
-from typing import (
-    Any,
-    AsyncGenerator,
-    Callable,
-    Generator,
-)
-import time
-
-import trio
-from trio_typing import TaskStatus
-from pendulum import (
-    from_timestamp,
-)
-import numpy as np
-import tractor
-
-from piker.brokers import (
-    open_cached_client,
-    NoData,
-)
-from piker._cacheables import (
-    async_lifo_cache,
-)
-from piker.accounting import (
-    Asset,
-    DerivTypes,
-    MktPair,
-    unpack_fqme,
-)
-from piker.types import Struct
-from piker.data.validate import FeedInit
-from piker.data._web_bs import (
-    open_autorecon_ws,
-    NoBsWs,
-)
-from piker.brokers._util import (
-    DataUnavailable,
-    get_logger,
-)
-
-from .api import (
-    Client,
-)
-from .venues import (
-    Pair,
-    FutesPair,
-    get_api_eps,
-)
-
-log = get_logger('piker.brokers.binance')
-
-
-class L1(Struct):
-    # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
-
-    update_id: int
-    sym: str
-
-    bid: float
-    bsize: float
-    ask: float
-    asize: float
-
-
-# validation type
-class AggTrade(Struct, frozen=True):
-    e: str  # Event type
-    E: int  # Event time
-    s: str  # Symbol
-    a: int  # Aggregate trade ID
-    p: float  # Price
-    q: float  # Quantity
-    f: int  # First trade ID
-    l: int  # noqa Last trade ID
-    T: int  # Trade time
-    m: bool  # Is the buyer the market maker?
-    M: bool | None = None  # Ignore
-
-
-async def stream_messages(
-    ws: NoBsWs,
-
-) -> AsyncGenerator[NoBsWs, dict]:
-
-    # TODO: match syntax here!
-    msg: dict[str, Any]
-    async for msg in ws:
-        match msg:
-            # for l1 streams binance doesn't add an event type field so
-            # identify those messages by matching keys
-            # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
-            case {
-                # NOTE: this is never an old value it seems, so
-                # they are always sending real L1 spread updates.
-                'u': upid,  # update id
-                's': sym,
-                'b': bid,
-                'B': bsize,
-                'a': ask,
-                'A': asize,
-            }:
-                # TODO: it would be super nice to have a `L1` piker type
-                # which "renders" incremental tick updates from a packed
-                # msg-struct:
-                # - backend msgs after packed into the type such that we
-                #   can reduce IPC usage but without each backend having
-                #   to do that incremental update logic manually B)
-                # - would it maybe be more efficient to use this instead?
-                #   https://binance-docs.github.io/apidocs/spot/en/#diff-depth-stream
-                l1 = L1(
-                    update_id=upid,
-                    sym=sym,
-                    bid=bid,
-                    bsize=bsize,
-                    ask=ask,
-                    asize=asize,
-                )
-                # for speed probably better to only specifically
-                # cast fields we need in numerical form?
-                # l1.typecast()
-
-                # repack into piker's tick-quote format
-                yield 'l1', {
-                    'symbol': l1.sym,
-                    'ticks': [
-                        {
-                            'type': 'bid',
-                            'price': float(l1.bid),
-                            'size': float(l1.bsize),
-                        },
-                        {
-                            'type': 'bsize',
-                            'price': float(l1.bid),
-                            'size': float(l1.bsize),
-                        },
-                        {
-                            'type': 'ask',
-                            'price': float(l1.ask),
-                            'size': float(l1.asize),
-                        },
-                        {
-                            'type': 'asize',
-                            'price': float(l1.ask),
-                            'size': float(l1.asize),
-                        }
-                    ]
-                }
-
-            # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
-            case {
-                'e': 'aggTrade',
-            }:
-                # NOTE: this is purely for a definition,
-                # ``msgspec.Struct`` does not runtime-validate until you
-                # decode/encode, see:
-                # https://jcristharif.com/msgspec/structs.html#type-validation
-                msg = AggTrade(**msg)  # TODO: should we .copy() ?
-                piker_quote: dict = {
-                    'symbol': msg.s,
-                    'last': float(msg.p),
-                    'brokerd_ts': time.time(),
-                    'ticks': [{
-                        'type': 'trade',
-                        'price': float(msg.p),
-                        'size': float(msg.q),
-                        'broker_ts': msg.T,
-                    }],
-                }
-                yield 'trade', piker_quote
-
-
-def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
-    '''
-    Create a request subscription packet dict.
-
-    - spot:
-      https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
-
-    - futes:
-      https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
-
-    '''
-    return {
-        'method': 'SUBSCRIBE',
-        'params': [
-            f'{pair.lower()}@{sub_name}'
-            for pair in pairs
-        ],
-        'id': uid
-    }
-
-
-# TODO, why aren't frame resp `log.info()`s showing in upstream
-# code?!
-@acm
-async def open_history_client(
-    mkt: MktPair,
-
-) -> tuple[Callable, int]:
-
-    # TODO implement history getter for the new storage layer.
-    async with open_cached_client('binance') as client:
-
-        async def get_ohlc(
-            timeframe: float,
-            end_dt: datetime | None = None,
-            start_dt: datetime | None = None,
-
-        ) -> tuple[
-            np.ndarray,
-            datetime,  # start
-            datetime,  # end
-        ]:
-            if timeframe != 60:
-                raise DataUnavailable('Only 1m bars are supported')
-
-            # TODO: better wrapping for venue / mode?
-            # - eventually logic for usd vs. coin settled futes
-            #   based on `MktPair.src` type/value?
-            # - maybe something like `async with
-            # Client.use_venue('usdtm_futes')`
-            if mkt.type_key in DerivTypes:
-                client.mkt_mode = 'usdtm_futes'
-            else:
-                client.mkt_mode = 'spot'
-
-            array: np.ndarray = await client.bars(
-                mkt=mkt,
-                start_dt=start_dt,
-                end_dt=end_dt,
-            )
-            if array.size == 0:
-                raise NoData(
-                    f'No frame for {start_dt} -> {end_dt}\n'
-                )
-
-            times = array['time']
-            if not times.any():
-                raise ValueError(
-                    'Bad frame with null-times?\n\n'
-                    f'{times}'
-                )
-
-            if end_dt is None:
-                inow: int = round(time.time())
-                if (inow - times[-1]) > 60:
-                    await tractor.pause()
-
-            start_dt = from_timestamp(times[0])
-            end_dt = from_timestamp(times[-1])
-            return array, start_dt, end_dt
-
-        yield get_ohlc, {'erlangs': 3, 'rate': 3}
-
-
-@async_lifo_cache()
-async def get_mkt_info(
-    fqme: str,
-
-) -> tuple[MktPair, Pair] | None:
-
-    # uppercase since kraken bs_mktid is always upper
-    if 'binance' not in fqme.lower():
-        fqme += '.binance'
-
-    mkt_mode: str = ''
-    broker, mkt_ep, venue, expiry = unpack_fqme(fqme)
-
-    # NOTE: we always upper case all tokens to be consistent with
-    # binance's symbology style for pairs, like `BTCUSDT`, but in
-    # theory we could also just keep things lower case; as long as
-    # we're consistent and the symcache matches whatever this func
-    # returns, always!
-    expiry: str = expiry.upper()
-    venue: str = venue.upper()
-    venue_lower: str = venue.lower()
-
-    # XXX TODO: we should change the usdtm_futes name to just
-    # usdm_futes (dropping the tether part) since it turns out that
-    # there are indeed USD-tokens OTHER THEN tether being used as
-    # the margin assets.. it's going to require a wholesale
-    # (variable/key) rename as well as file name adjustments to any
-    # existing tsdb set..
-    if 'usd' in venue_lower:
-        mkt_mode: str = 'usdtm_futes'
-
-    # NO IDEA what these contracts (some kinda DEX-ish futes?) are
-    # but we're masking them for now..
-    elif (
-        'defi' in venue_lower
-
-        # TODO: handle coinm futes which have a margin asset that
-        # is some crypto token!
-        # https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
-        or 'btc' in venue_lower
-    ):
-        return None
-
-    else:
-        # NOTE: see the `FutesPair.bs_fqme: str` implementation
-        # to understand the reverse market info lookup below.
-        mkt_mode = venue_lower or 'spot'
-
-    if (
-        venue
-        and 'spot' not in venue_lower
-
-        # XXX: catch all in case user doesn't know which
-        # venue they want (usdtm vs. coinm) and we can choose
-        # a default (via config?) once we support coin-m APIs.
-        or 'perp' in venue_lower
-    ):
-        if not mkt_mode:
-            mkt_mode: str = f'{venue_lower}_futes'
-
-    async with open_cached_client(
-        'binance',
-    ) as client:
-
-        assets: dict[str, Asset] = await client.get_assets()
-        pair_str: str = mkt_ep.upper()
-
-        # switch venue-mode depending on input pattern parsing
-        # since we want to use a particular endpoint (set) for
-        # pair info lookup!
-        client.mkt_mode = mkt_mode
-
-        pair: Pair = await client.exch_info(
-            pair_str,
-            venue=mkt_mode,  # explicit
-            expiry=expiry,
-        )
-
-        if 'futes' in mkt_mode:
-            assert isinstance(pair, FutesPair)
-
-        dst: Asset | None = assets.get(pair.bs_dst_asset)
-        if (
-            not dst
-            # TODO: a known asset DNE list?
-            # and pair.baseAsset == 'DEFI'
-        ):
-            log.warning(
-                f'UNKNOWN {venue} asset {pair.baseAsset} from,\n'
-                f'{pformat(pair.to_dict())}'
-            )
-
-            # XXX UNKNOWN missing "asset", though no idea why?
-            # maybe it's only avail in the margin venue(s): /dapi/ ?
-            return None
-
-        mkt = MktPair(
-            dst=dst,
-            src=assets[pair.bs_src_asset],
-            price_tick=pair.price_tick,
-            size_tick=pair.size_tick,
-            bs_mktid=pair.symbol,
-            expiry=expiry,
-            venue=venue,
-            broker='binance',
-
-            # NOTE: sectype is always taken from dst, see
-            # `MktPair.type_key` and `Client._cache_pairs()`
-            # _atype=sectype,
-        )
-        return mkt, pair
-
-
-@acm
-async def subscribe(
-    ws: NoBsWs,
-    symbols: list[str],
-
-    # defined once at import time to keep a global state B)
-    iter_subids: Generator[int, None, None] = itertools.count(),
-
-):
-    # setup subs
-
-    subid: int = next(iter_subids)
-
-    # trade data (aka L1)
-    # https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
-    l1_sub = make_sub(symbols, 'bookTicker', subid)
-    await ws.send_msg(l1_sub)
-
-    # aggregate (each order clear by taker **not** by maker)
-    # trades data:
-    # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
-    agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
-    await ws.send_msg(agg_trades_sub)
-
-    # might get ack from ws server, or maybe some
-    # other msg still in transit..
-    res = await ws.recv_msg()
-    subid: str | None = res.get('id')
-    if subid:
-        assert res['id'] == subid
-
-    yield
-
-    subs = []
-    for sym in symbols:
-        subs.append("{sym}@aggTrade")
-        subs.append("{sym}@bookTicker")
-
-    # unsub from all pairs on teardown
-    if ws.connected():
-        await ws.send_msg({
-            "method": "UNSUBSCRIBE",
-            "params": subs,
-            "id": subid,
-        })
-
-        # XXX: do we need to ack the unsub?
-        # await ws.recv_msg()
-
-
-async def stream_quotes(
-
-    send_chan: trio.abc.SendChannel,
-    symbols: list[str],
-    feed_is_live: trio.Event,
-    loglevel: str = None,
-
-    # startup sync
-    task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
-
-) -> None:
-
-    async with (
-        send_chan as send_chan,
-        open_cached_client('binance') as client,
-    ):
-        init_msgs: list[FeedInit] = []
-        for sym in symbols:
-            mkt: MktPair
-            pair: Pair
-            mkt, pair = await get_mkt_info(sym)
-
-            # build out init msgs according to latest spec
-            init_msgs.append(
-                FeedInit(mkt_info=mkt)
-            )
-
-        wss_url: str = get_api_eps(client.mkt_mode)[1]  # 2nd elem is wss url
-
-        # TODO: for sanity, but remove eventually Xp
-        if 'future' in mkt.type_key:
-            assert 'fstream' in wss_url
-
-        async with (
-            open_autorecon_ws(
-                url=wss_url,
-                fixture=partial(
-                    subscribe,
-                    symbols=[mkt.bs_mktid],
-                ),
-            ) as ws,
-
-            # avoid stream-gen closure from breaking trio..
-            aclosing(stream_messages(ws)) as msg_gen,
-        ):
-            # log.info('WAITING ON FIRST LIVE QUOTE..')
-            typ, quote = await anext(msg_gen)
-
-            # pull a first quote and deliver
-            while typ != 'trade':
-                typ, quote = await anext(msg_gen)
-
-            task_status.started((init_msgs, quote))
-
-            # signal to caller feed is ready for consumption
-            feed_is_live.set()
-
-            # import time
-            # last = time.time()
-
-            # XXX NOTE: can't include the `.binance` suffix
-            # or the sampling loop will not broadcast correctly
-            # since `bus._subscribers.setdefault(bs_fqme, set())`
-            # is used inside `.data.open_feed_bus()` !!!
-            topic: str = mkt.bs_fqme
-
-            # start streaming
-            async for typ, quote in msg_gen:
-                # period = time.time() - last
-                # hz = 1/period if period else float('inf')
-                # if hz > 60:
-                #     log.info(f'Binance quotez : {hz}')
-                await send_chan.send({topic: quote})
-                # last = time.time()
-
-
-@tractor.context
-async def open_symbol_search(
-    ctx: tractor.Context,
-) -> Client:
-
-    # NOTE: symbology tables are loaded as part of client
-    # startup in ``.api.get_client()`` and in this case
-    # are stored as `Client._pairs`.
-    async with open_cached_client('binance') as client:
-
-        # TODO: maybe we should deliver the cache
-        # so that client's can always do a local-lookup-first
-        # style try and then update async as (new) match results
-        # are delivered from here?
-        await ctx.started()
-
-        async with ctx.open_stream() as stream:
-
-            pattern: str
-            async for pattern in stream:
-                # NOTE: pattern fuzzy-matching is done within
-                # the methd impl.
-                pairs: dict[str, Pair] = await client.search_symbols(
-                    pattern,
-                )
-
-                # repack in fqme-keyed table
-                byfqme: dict[str, Pair] = {}
-                for pair in pairs.values():
-                    byfqme[pair.bs_fqme] = pair
-
-                await stream.send(byfqme)

piker/brokers/binance/venues.py

@@ -1,303 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-Per market data-type definitions and schemas types.
-
-"""
-from __future__ import annotations
-from typing import (
-    Literal,
-)
-from decimal import Decimal
-
-from msgspec import field
-
-from piker.types import Struct
-
-
-# API endpoint paths by venue / sub-API
-_domain: str = 'binance.com'
-_spot_url = f'https://api.{_domain}'
-_futes_url = f'https://fapi.{_domain}'
-
-# WEBsocketz
-# NOTE XXX: see api docs which show diff addr?
-# https://developers.binance.com/docs/binance-trading-api/websocket_api#general-api-information
-_spot_ws: str = 'wss://stream.binance.com/ws'
-# or this one? ..
-# 'wss://ws-api.binance.com:443/ws-api/v3',
-
-# https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
-_futes_ws: str = f'wss://fstream.{_domain}/ws'
-_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
-
-# test nets
-# NOTE: spot test network only allows certain ep sets:
-# https://testnet.binance.vision/
-# https://www.binance.com/en/support/faq/how-to-test-my-functions-on-binance-testnet-ab78f9a1b8824cf0a106b4229c76496d
-_testnet_spot_url: str = 'https://testnet.binance.vision/api'
-_testnet_spot_ws: str = 'wss://testnet.binance.vision/ws'
-# or this one? ..
-# 'wss://testnet.binance.vision/ws-api/v3'
-
-_testnet_futes_url: str = 'https://testnet.binancefuture.com'
-_testnet_futes_ws: str = 'wss://stream.binancefuture.com/ws'
-
-
-MarketType = Literal[
-    'spot',
-    # 'margin',
-    'usdtm_futes',
-    # 'coinm_futes',
-]
-
-
-def get_api_eps(venue: MarketType) -> tuple[str, str]:
-    '''
-    Return API ep root paths per venue.
-
-    '''
-    return {
-        'spot': (
-            _spot_url,
-            _spot_ws,
-        ),
-        'usdtm_futes': (
-            _futes_url,
-            _futes_ws,
-        ),
-    }[venue]
-
-
-class Pair(Struct, frozen=True, kw_only=True):
-
-    symbol: str
-    status: str
-    orderTypes: list[str]
-
-    # src
-    quoteAsset: str
-    quotePrecision: int
-
-    # dst
-    baseAsset: str
-    baseAssetPrecision: int
-
-    filters: dict[
-        str,
-        str | int | float,
-    ] = field(default_factory=dict)
-
-    @property
-    def price_tick(self) -> Decimal:
-        # XXX: lul, after manually inspecting the response format we
-        # just directly pick out the info we need
-        step_size: str = self.filters['PRICE_FILTER']['tickSize'].rstrip('0')
-        return Decimal(step_size)
-
-    @property
-    def size_tick(self) -> Decimal:
-        step_size: str = self.filters['LOT_SIZE']['stepSize'].rstrip('0')
-        return Decimal(step_size)
-
-    @property
-    def bs_fqme(self) -> str:
-        return self.symbol
-
-    @property
-    def bs_mktid(self) -> str:
-        return f'{self.symbol}.{self.venue}'
-
-
-class SpotPair(Pair, frozen=True):
-
-    cancelReplaceAllowed: bool
-    allowTrailingStop: bool
-    quoteAssetPrecision: int
-
-    baseCommissionPrecision: int
-    quoteCommissionPrecision: int
-
-    icebergAllowed: bool
-    ocoAllowed: bool
-    quoteOrderQtyMarketAllowed: bool
-    isSpotTradingAllowed: bool
-    isMarginTradingAllowed: bool
-    otoAllowed: bool
-
-    defaultSelfTradePreventionMode: str
-    allowedSelfTradePreventionModes: list[str]
-    permissions: list[str]
-    permissionSets: list[list[str]]
-
-    # NOTE: see `.data._symcache.SymbologyCache.load()` for why
-    ns_path: str = 'piker.brokers.binance:SpotPair'
-
-    @property
-    def venue(self) -> str:
-        return 'SPOT'
-
-    @property
-    def bs_fqme(self) -> str:
-        return f'{self.symbol}.SPOT'
-
-    @property
-    def bs_src_asset(self) -> str:
-        return f'{self.quoteAsset}'
-
-    @property
-    def bs_dst_asset(self) -> str:
-        return f'{self.baseAsset}'
-
-
-class FutesPair(Pair):
-    symbol: str  # 'BTCUSDT',
-    pair: str  # 'BTCUSDT',
-    baseAssetPrecision: int # 8,
-    contractType: str  # 'PERPETUAL',
-    deliveryDate: int   # 4133404800000,
-    liquidationFee: float  # '0.012500',
-    maintMarginPercent: float  # '2.5000',
-    marginAsset: str  # 'USDT',
-    marketTakeBound: float  # '0.05',
-    maxMoveOrderLimit: int  # 10000,
-    onboardDate: int  # 1569398400000,
-    pricePrecision: int  # 2,
-    quantityPrecision: int  # 3,
-    quoteAsset: str  # 'USDT',
-    quotePrecision: int  # 8,
-    requiredMarginPercent: float  # '5.0000',
-    timeInForce: list[str]  # ['GTC', 'IOC', 'FOK', 'GTX'],
-    triggerProtect: float  # '0.0500',
-    underlyingSubType: list[str]  # ['PoW'],
-    underlyingType: str  # 'COIN'
-
-    # NOTE: see `.data._symcache.SymbologyCache.load()` for why
-    ns_path: str = 'piker.brokers.binance:FutesPair'
-
-    # NOTE: for compat with spot pairs and `MktPair.src: Asset`
-    # processing..
-    @property
-    def quoteAssetPrecision(self) -> int:
-        return self.quotePrecision
-
-    @property
-    def expiry(self) -> str:
-        symbol: str = self.symbol
-        contype: str = self.contractType
-        match contype:
-            case (
-                'CURRENT_QUARTER'
-                | 'CURRENT_QUARTER DELIVERING'
-                | 'NEXT_QUARTER'  # su madre binance..
-            ):
-                pair, _, expiry = symbol.partition('_')
-                assert pair == self.pair  # sanity
-                return f'{expiry}'
-
-            case 'PERPETUAL':
-                return 'PERP'
-
-            case '':
-                subtype: list[str] = self.underlyingSubType
-                if not subtype:
-                    if self.status == 'PENDING_TRADING':
-                        return 'PENDING'
-
-                match subtype:
-                    case ['DEFI']:
-                        return 'PERP'
-
-        # wow, just wow you binance guys suck..
-        if self.status == 'PENDING_TRADING':
-            return 'PENDING'
-
-        # XXX: yeah no clue then..
-        raise ValueError(
-            f'Bad .expiry token match: {contype} for {symbol}'
-        )
-
-    @property
-    def venue(self) -> str:
-        symbol: str = self.symbol
-        ctype: str = self.contractType
-        margin: str = self.marginAsset
-
-        match ctype:
-            case 'PERPETUAL':
-                return f'{margin}M'
-
-            case (
-                'CURRENT_QUARTER'
-                | 'CURRENT_QUARTER DELIVERING'
-                | 'NEXT_QUARTER'  # su madre binance..
-            ):
-                _, _, expiry = symbol.partition('_')
-                return f'{margin}M'
-
-            case '':
-                subtype: list[str] = self.underlyingSubType
-                if not subtype:
-                    if self.status == 'PENDING_TRADING':
-                        return f'{margin}M'
-
-                match subtype:
-                    case (
-                        ['DEFI']
-                        | ['USDC']
-                    ):
-                        return f'{subtype[0]}'
-
-        # XXX: yeah no clue then..
-        raise ValueError(
-            f'Bad .venue token match: {ctype}'
-        )
-
-    @property
-    def bs_fqme(self) -> str:
-        symbol: str = self.symbol
-        ctype: str = self.contractType
-        venue: str = self.venue
-        pair: str = self.pair
-
-        match ctype:
-            case (
-                'CURRENT_QUARTER'
-                | 'NEXT_QUARTER'  # su madre binance..
-            ):
-                pair, _, expiry = symbol.partition('_')
-                assert pair == self.pair
-
-        return f'{pair}.{venue}.{self.expiry}'
-
-    @property
-    def bs_src_asset(self) -> str:
-        return f'{self.quoteAsset}'
-
-    @property
-    def bs_dst_asset(self) -> str:
-        return f'{self.baseAsset}.{self.venue}'
-
-
-PAIRTYPES: dict[MarketType, Pair] = {
-    'spot': SpotPair,
-    'usdtm_futes': FutesPair,
-
-    # TODO: support coin-margined venue:
-    # https://binance-docs.github.io/apidocs/delivery/en/#change-log
-    # 'coinm_futes': CoinFutesPair,
-}

piker/brokers/cli.py

@@ -21,7 +21,6 @@ import os
 from functools import partial
 from operator import attrgetter
 from operator import itemgetter
-from types import ModuleType
 
 import click
 import trio
@@ -195,7 +194,7 @@ def brokercheck(config, broker):
 
 @cli.command()
 @click.option('--keys', '-k', multiple=True,
-            help='Return results only for these keys')
+              help='Return results only for these keys')
 @click.argument('meth', nargs=1)
 @click.argument('kwargs', nargs=-1)
 @click.pass_obj
@@ -242,7 +241,7 @@ def quote(config, tickers):
 
     '''
     # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
 
     quotes = trio.run(partial(core.stocks_quote, brokermod, tickers))
     if not quotes:
@@ -269,7 +268,7 @@ def bars(config, symbol, count):
 
     '''
     # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
 
     # broker backend should return at the least a
     # list of candle dictionaries
@@ -304,7 +303,7 @@ def record(config, rate, name, dhost, filename):
 
     '''
     # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
     loglevel = config['loglevel']
     log = config['log']
 
@@ -369,7 +368,7 @@ def optsquote(config, symbol, date):
 
     '''
     # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
 
     quotes = trio.run(
         partial(
@@ -386,151 +385,58 @@ def optsquote(config, symbol, date):
 @cli.command()
 @click.argument('tickers', nargs=-1, required=True)
 @click.pass_obj
-def mkt_info(
-    config: dict,
-    tickers: list[str],
-):
+def symbol_info(config, tickers):
     '''
     Print symbol quotes to the console
 
     '''
-    from msgspec.json import encode, decode
-    from ..accounting import MktPair
-    from ..service import (
-        open_piker_runtime,
-    )
-
     # global opts
-    brokermods: dict[str, ModuleType] = config['brokermods']
+    brokermod = config['brokermods'][0]
 
-    mkts: list[MktPair] = []
-    async def main():
-
-        async with open_piker_runtime(
-            name='mkt_info_query',
-            # loglevel=loglevel,
-            debug_mode=True,
-
-        ) as (_, _):
-            for fqme in tickers:
-                bs_fqme, _, broker = fqme.rpartition('.')
-                brokermod: ModuleType = brokermods[broker]
-                mkt, bs_pair = await core.mkt_info(
-                    brokermod,
-                    bs_fqme,
-                )
-                mkts.append((mkt, bs_pair))
-
-    trio.run(main)
-
-    if not mkts:
-        log.error(
-            f'No market info could be found for {tickers}'
-        )
+    quotes = trio.run(partial(core.symbol_info, brokermod, tickers))
+    if not quotes:
+        log.error(f"No quotes could be found for {tickers}?")
         return
 
-    if len(mkts) < len(tickers):
-        syms = tuple(map(itemgetter('fqme'), mkts))
+    if len(quotes) < len(tickers):
+        syms = tuple(map(itemgetter('symbol'), quotes))
         for ticker in tickers:
             if ticker not in syms:
-                log.warn(f"Could not find symbol {ticker}?")
+                brokermod.log.warn(f"Could not find symbol {ticker}?")
 
-
-    # TODO: use ``rich.Table`` intead here!
-    for mkt, bs_pair in mkts:
-        click.echo(
-            '\n'
-            '----------------------------------------------------\n'
-            f'{type(bs_pair)}\n'
-            '----------------------------------------------------\n'
-            f'{colorize_json(bs_pair.to_dict())}\n'
-            '----------------------------------------------------\n'
-            f'as piker `MktPair` with fqme: {mkt.fqme}\n'
-            '----------------------------------------------------\n'
-            # NOTE: roundtrip to json codec for console print
-            f'{colorize_json(decode(encode(mkt)))}'
-        )
+    click.echo(colorize_json(quotes))
 
 
 @cli.command()
 @click.argument('pattern', required=True)
-# TODO: move this to top level click/typer context for all subs
-@click.option(
-    '--pdb',
-    is_flag=True,
-    help='Enable tractor debug mode',
-)
 @click.pass_obj
-def search(
-    config: dict,
-    pattern: str,
-    pdb: bool,
-):
+def search(config, pattern):
     '''
     Search for symbols from broker backend(s).
 
     '''
     # global opts
-    brokermods = list(config['brokermods'].values())
+    brokermods = config['brokermods']
 
     # define tractor entrypoint
     async def main(func):
 
         async with maybe_open_pikerd(
             loglevel=config['loglevel'],
-            debug_mode=pdb,
         ):
             return await func()
 
-    from piker.toolz import open_crash_handler
-    with open_crash_handler():
-        quotes = trio.run(
-            main,
-            partial(
-                core.symbol_search,
-                brokermods,
-                pattern,
-            ),
-        )
+    quotes = trio.run(
+        main,
+        partial(
+            core.symbol_search,
+            brokermods,
+            pattern,
+        ),
+    )
 
-        if not quotes:
-            log.error(f"No matches could be found for {pattern}?")
-            return
+    if not quotes:
+        log.error(f"No matches could be found for {pattern}?")
+        return
 
-        click.echo(colorize_json(quotes))
-
-
-@cli.command()
-@click.argument('section', required=False)
-@click.argument('value', required=False)
-@click.option('--delete', '-d', flag_value=True, help='Delete section')
-@click.pass_obj
-def brokercfg(config, section, value, delete):
-    '''
-    If invoked with no arguments, open an editor to edit broker
-    configs file or get / update an individual section.
-
-    '''
-    from .. import config
-
-    if section:
-        conf, path = config.load()
-
-        if not delete:
-            if value:
-                config.set_value(conf, section, value)
-
-            click.echo(
-                colorize_json(
-                    config.get_value(conf, section))
-            )
-        else:
-            config.del_value(conf, section)
-
-        config.write(config=conf)
-
-    else:
-        conf, path = config.load(raw=True)
-        config.write(
-            raw=click.edit(text=conf)
-        )
+    click.echo(colorize_json(quotes))

piker/brokers/core.py

@@ -29,8 +29,7 @@ import trio
 from ._util import log
 from . import get_brokermod
 from ..service import maybe_spawn_brokerd
-from . import open_cached_client
-from ..accounting import MktPair
+from .._cacheables import open_cached_client
 
 
 async def api(brokername: str, methname: str, **kwargs) -> dict:
@@ -95,15 +94,15 @@ async def option_chain(
             return await client.option_chains(contracts)
 
 
-# async def contracts(
-#     brokermod: ModuleType,
-#     symbol: str,
-# ) -> Dict[str, Dict[str, Dict[str, Any]]]:
-#     """Return option contracts (all expiries) for ``symbol``.
-#     """
-#     async with brokermod.get_client() as client:
-#         # return await client.get_all_contracts([symbol])
-#         return await client.get_all_contracts([symbol])
+async def contracts(
+    brokermod: ModuleType,
+    symbol: str,
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
+    """Return option contracts (all expiries) for ``symbol``.
+    """
+    async with brokermod.get_client() as client:
+        # return await client.get_all_contracts([symbol])
+        return await client.get_all_contracts([symbol])
 
 
 async def bars(
@@ -117,6 +116,17 @@ async def bars(
         return await client.bars(symbol, **kwargs)
 
 
+async def symbol_info(
+    brokermod: ModuleType,
+    symbol: str,
+    **kwargs,
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
+    """Return symbol info from broker.
+    """
+    async with brokermod.get_client() as client:
+        return await client.symbol_info(symbol, **kwargs)
+
+
 async def search_w_brokerd(name: str, pattern: str) -> dict:
 
     async with open_cached_client(name) as client:
@@ -145,11 +155,7 @@ async def symbol_search(
 
         async with maybe_spawn_brokerd(
             mod.name,
-            infect_asyncio=getattr(
-                mod,
-                '_infect_asyncio',
-                False,
-            ),
+            infect_asyncio=getattr(mod, '_infect_asyncio', False),
         ) as portal:
 
             results.append((
@@ -167,20 +173,3 @@ async def symbol_search(
             n.start_soon(search_backend, mod.name)
 
     return results
-
-
-async def mkt_info(
-    brokermod: ModuleType,
-    fqme: str,
-    **kwargs,
-
-) -> MktPair:
-    '''
-    Return MktPair info from broker including src and dst assets.
-
-    '''
-    async with open_cached_client(brokermod.name) as client:
-        assert client
-        return await brokermod.get_mkt_info(
-            fqme.replace(brokermod.name, '')
-        )

piker/brokers/deribit/__init__.py

@@ -21,6 +21,8 @@ Deribit backend.
 
 from piker.log import get_logger
 
+log = get_logger(__name__)
+
 from .api import (
     get_client,
 )
@@ -28,15 +30,13 @@ from .feed import (
     open_history_client,
     open_symbol_search,
     stream_quotes,
-    # backfill_bars,
+    backfill_bars
 )
 # from .broker import (
-    # open_trade_dialog,
+    # trades_dialogue,
     # norm_trade_records,
 # )
 
-log = get_logger(__name__)
-
 __all__ = [
     'get_client',
 #    'trades_dialogue',

piker/brokers/deribit/api.py

@@ -18,33 +18,43 @@
 Deribit backend.
 
 '''
-import asyncio
-from contextlib import (
-    asynccontextmanager as acm,
-)
-from datetime import datetime
-from functools import partial
+import json
 import time
-from typing import (
-    Any,
-    Optional,
-    Callable,
-)
+import asyncio
+
+from contextlib import asynccontextmanager as acm, AsyncExitStack
+from functools import partial
+from datetime import datetime
+from typing import Any, Optional, Iterable, Callable
 
 import pendulum
+import asks
 import trio
-from trio_typing import TaskStatus
-from rapidfuzz import process as fuzzy
+from trio_typing import Nursery, TaskStatus
+from fuzzywuzzy import process as fuzzy
 import numpy as np
+
+from piker.data.types import Struct
+from piker.data._web_bs import (
+    NoBsWs,
+    open_autorecon_ws,
+    open_jsonrpc_session
+)
+
+from .._util import resproc
+
+from piker import config
+from piker.log import get_logger
+
 from tractor.trionics import (
     broadcast_receiver,
+    BroadcastReceiver,
     maybe_open_context
 )
 from tractor import to_asyncio
-# XXX WOOPS XD
-# yeah you'll need to install it since it was removed in #489 by
-# accident; well i thought we had removed all usage..
+
 from cryptofeed import FeedHandler
+
 from cryptofeed.defines import (
     DERIBIT,
     L1_BOOK, TRADES,
@@ -52,20 +62,6 @@ from cryptofeed.defines import (
 )
 from cryptofeed.symbols import Symbol
 
-from piker.data import (
-    def_iohlcv_fields,
-    match_from_pairs,
-    Struct,
-)
-from piker.data._web_bs import (
-    open_jsonrpc_session
-)
-
-
-from piker import config
-from piker.log import get_logger
-
-
 log = get_logger(__name__)
 
 
@@ -79,13 +75,26 @@ _ws_url = 'wss://www.deribit.com/ws/api/v2'
 _testnet_ws_url = 'wss://test.deribit.com/ws/api/v2'
 
 
+# Broker specific ohlc schema (rest)
+_ohlc_dtype = [
+    ('index', int),
+    ('time', int),
+    ('open', float),
+    ('high', float),
+    ('low', float),
+    ('close', float),
+    ('volume', float),
+    ('bar_wap', float),  # will be zeroed by sampler if not filled
+]
+
+
 class JSONRPCResult(Struct):
     jsonrpc: str = '2.0'
     id: int
-    result: Optional[list[dict]] = None
+    result: Optional[dict] = None
     error: Optional[dict] = None
-    usIn: int
-    usOut: int
+    usIn: int 
+    usOut: int 
     usDiff: int
     testnet: bool
 
@@ -292,29 +301,24 @@ class Client:
         currency: str = 'btc',  # BTC, ETH, SOL, USDC
         kind: str = 'option',
         expired: bool = False
+    ) -> dict[str, Any]:
+        """Get symbol info for the exchange.
 
-    ) -> dict[str, dict]:
-        '''
-        Get symbol infos.
-
-        '''
+        """
         if self._pairs:
             return self._pairs
 
         # will retrieve all symbols by default
-        params: dict[str, str] = {
+        params = {
             'currency': currency.upper(),
             'kind': kind,
             'expired': str(expired).lower()
         }
 
-        resp: JSONRPCResult = await self.json_rpc(
-            'public/get_instruments',
-            params,
-        )
-        # convert to symbol-keyed table
-        results: list[dict] | None = resp.result
-        instruments: dict[str, dict] = {
+        resp = await self.json_rpc('public/get_instruments', params)
+        results = resp.result
+
+        instruments = {
             item['instrument_name'].lower(): item
             for item in results
         }
@@ -327,7 +331,6 @@ class Client:
     async def cache_symbols(
         self,
     ) -> dict:
-
         if not self._pairs:
             self._pairs = await self.symbol_info()
 
@@ -338,23 +341,17 @@ class Client:
         pattern: str,
         limit: int = 30,
     ) -> dict[str, Any]:
-        '''
-        Fuzzy search symbology set for pairs matching `pattern`.
+        data = await self.symbol_info()
 
-        '''
-        pairs: dict[str, Any] = await self.symbol_info()
-        matches: dict[str, Pair] = match_from_pairs(
-            pairs=pairs,
-            query=pattern.upper(),
+        matches = fuzzy.extractBests(
+            pattern,
+            data,
             score_cutoff=35,
             limit=limit
         )
-
-       # repack in name-keyed table
-        return {
-            pair['instrument_name'].lower(): pair
-            for pair in matches.values()
-        }
+        # repack in dict form
+        return {item[0]['instrument_name'].lower(): item[0]
+                for item in matches}
 
     async def bars(
         self,
@@ -408,7 +405,7 @@ class Client:
 
             new_bars.append((i,) + tuple(row))
 
-        array = np.array(new_bars, dtype=def_iohlcv_fields) if as_np else klines
+        array = np.array(new_bars, dtype=_ohlc_dtype) if as_np else klines
         return array
 
     async def last_trades(

piker/brokers/deribit/feed.py

@@ -26,11 +26,11 @@ import time
 import trio
 from trio_typing import TaskStatus
 import pendulum
-from rapidfuzz import process as fuzzy
+from fuzzywuzzy import process as fuzzy
 import numpy as np
 import tractor
 
-from piker.brokers import open_cached_client
+from piker._cacheables import open_cached_client
 from piker.log import get_logger, get_console_log
 from piker.data import ShmArray
 from piker.brokers._util import (
@@ -39,6 +39,7 @@ from piker.brokers._util import (
 )
 
 from cryptofeed import FeedHandler
+
 from cryptofeed.defines import (
     DERIBIT, L1_BOOK, TRADES, OPTION, CALL, PUT
 )

piker/brokers/ib/__init__.py

@@ -30,33 +30,19 @@ from .api import (
 )
 from .feed import (
     open_history_client,
+    open_symbol_search,
     stream_quotes,
 )
 from .broker import (
-    open_trade_dialog,
-)
-from .ledger import (
-    norm_trade,
-    norm_trade_records,
-    tx_sort,
-)
-from .symbols import (
-    get_mkt_info,
-    open_symbol_search,
-    _search_conf,
+    trades_dialogue,
 )
 
 __all__ = [
     'get_client',
-    'get_mkt_info',
-    'norm_trade',
-    'norm_trade_records',
-    'open_trade_dialog',
+    'trades_dialogue',
     'open_history_client',
     'open_symbol_search',
     'stream_quotes',
-    '_search_conf',
-    'tx_sort',
 ]
 
 _brokerd_mods: list[str] = [
@@ -66,7 +52,6 @@ _brokerd_mods: list[str] = [
 
 _datad_mods: list[str] = [
     'feed',
-    'symbols',
 ]
 
 
@@ -86,8 +71,3 @@ _spawn_kwargs = {
 # know if ``brokerd`` should be spawned with
 # ``tractor``'s aio mode.
 _infect_asyncio: bool = True
-
-# XXX NOTE: for now we disable symcache with this backend since
-# there is no clearly simple nor practical way to download "all
-# symbology info" for all supported venues..
-_no_symcache: bool = True

piker/brokers/ib/_flex_reports.py

@@ -159,11 +159,7 @@ def load_flex_trades(
     for acctid in trades_by_account:
         trades_by_id = trades_by_account[acctid]
 
-        with open_trade_ledger(
-            'ib',
-            acctid,
-            allow_from_sync_code=True,
-        ) as ledger_dict:
+        with open_trade_ledger('ib', acctid) as ledger_dict:
             tid_delta = set(trades_by_id) - set(ledger_dict)
             log.info(
                 'New trades detected\n'

piker/brokers/ib/_util.py

@@ -29,13 +29,14 @@ import subprocess
 
 import tractor
 
-from piker.brokers._util import get_logger
+from .._util import log
 
 if TYPE_CHECKING:
-    from .api import Client
-    from ib_insync import IB
+    from .api import (
+        MethodProxy,
+        ib_Client
+    )
 
-log = get_logger('piker.brokers.ib')
 
 _reset_tech: Literal[
     'vnc',
@@ -49,8 +50,7 @@ _reset_tech: Literal[
 
 
 async def data_reset_hack(
-    # vnc_host: str,
-    client: Client,
+    vnc_host: str,
     reset_type: Literal['data', 'connection'],
 
 ) -> None:
@@ -81,32 +81,12 @@ async def data_reset_hack(
           that need to be wrangle.
 
     '''
-    ib_client: IB = client.ib
-
-    # look up any user defined vnc socket address mapped from
-    # a particular API socket port.
-    api_port: str = str(ib_client.client.port)
-    vnc_host: str
-    vnc_port: int
-    vnc_sockaddr: tuple[str] | None = client.conf.get('vnc_addrs')
 
     no_setup_msg:str = (
-        f'No data reset hack test setup for {vnc_sockaddr}!\n'
-        'See config setup tips @\n'
+        'No data reset hack test setup for {vnc_host}!\n'
+        'See setup @\n'
         'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
     )
-
-    if not vnc_sockaddr:
-        log.warning(
-            no_setup_msg
-            +
-            'REQUIRES A `vnc_addrs: array` ENTRY'
-        )
-
-    vnc_host, vnc_port = vnc_sockaddr.get(
-        api_port,
-        ('localhost', 3003)
-    )
     global _reset_tech
 
     match _reset_tech:
@@ -116,7 +96,6 @@ async def data_reset_hack(
                     partial(
                         vnc_click_hack,
                         host=vnc_host,
-                        port=vnc_port,
                     )
                 )
             except OSError:
@@ -125,7 +104,7 @@ async def data_reset_hack(
                     return False
 
                 try:
-                    import i3ipc  # noqa  (since a deps dynamic check)
+                    import i3ipc
                 except ModuleNotFoundError:
                     log.warning(no_setup_msg)
                     return False
@@ -149,8 +128,7 @@ async def data_reset_hack(
 
 
 async def vnc_click_hack(
-    host: str,
-    port: int,
+    host: str = 'localhost',
     reset_type: str = 'data'
 ) -> None:
     '''
@@ -176,12 +154,8 @@ async def vnc_click_hack(
 
     async with asyncvnc.connect(
         host,
-        port=port,
-
-        # TODO: doesn't work see:
-        # https://github.com/barneygale/asyncvnc/issues/7
+        port=3003,
         # password='ibcansmbz',
-
     ) as client:
 
         # move to middle of screen
@@ -195,11 +169,6 @@ async def vnc_click_hack(
 
 
 def i3ipc_xdotool_manual_click_hack() -> None:
-    '''
-    Do the data reset hack but expecting a local X-window using `xdotool`.
-
-    '''
-    import i3ipc
     i3 = i3ipc.Connection()
 
     # TODO: might be worth offering some kinda api for grabbing
@@ -259,7 +228,7 @@ def i3ipc_xdotool_manual_click_hack() -> None:
                         timeout=timeout,
                     )
 
-        # re-activate and focus original window
+    # re-activate and focus original window
         subprocess.call([
             'xdotool',
             'windowactivate', '--sync', str(orig_win_id),

piker/brokers/ib/feed.py

@@ -21,47 +21,39 @@ from __future__ import annotations
 import asyncio
 from contextlib import (
     asynccontextmanager as acm,
+    nullcontext,
 )
+from decimal import Decimal
 from dataclasses import asdict
 from datetime import datetime
 from functools import partial
-from pprint import pformat
 from math import isnan
 import time
 from typing import (
-    Any,
     Callable,
-    TYPE_CHECKING,
+    Optional,
+    Awaitable,
 )
 
 from async_generator import aclosing
-import ib_insync as ibis
+from fuzzywuzzy import process as fuzzy
 import numpy as np
-from pendulum import (
-    now,
-    from_timestamp,
-    # DateTime,
-    Duration,
-    duration as mk_duration,
-)
+import pendulum
 import tractor
 import trio
 from trio_typing import TaskStatus
 
-from piker.accounting import (
-    MktPair,
-)
-from piker.data.validate import FeedInit
-from piker.brokers._util import (
+from .._util import (
     NoData,
     DataUnavailable,
+    SymbolNotFound,
 )
 from .api import (
     # _adhoc_futes_set,
-    Client,
     con2fqme,
     log,
     load_aio_clients,
+    ibis,
     MethodProxy,
     open_client_proxies,
     get_preferred_data_client,
@@ -70,13 +62,16 @@ from .api import (
     RequestError,
 )
 from ._util import data_reset_hack
-from .symbols import get_mkt_info
-
-if TYPE_CHECKING:
-    from trio._core._run import Task
+from piker._cacheables import (
+    async_lifo_cache,
+)
+from piker.accounting import (
+    Asset,
+    MktPair,
+)
+from piker.data.validate import FeedInit
 
 
-# XXX NOTE: See available types table docs:
 # https://interactivebrokers.github.io/tws-api/tick_types.html
 tick_types = {
     77: 'trade',
@@ -96,9 +91,9 @@ tick_types = {
 
     # ``ib_insync`` already packs these into
     # quotes under the following fields.
-    55: 'trades_per_min',  # `'tradeRate'`
-    56: 'vlm_per_min',  # `'volumeRate'`
-    89: 'shortable_units',  # `'shortableShares'`
+    # 55: 'trades_per_min',  # `'tradeRate'`
+    # 56: 'vlm_per_min',  # `'volumeRate'`
+    # 89: 'shortable',  # `'shortableShares'`
 }
 
 
@@ -143,21 +138,32 @@ async def open_history_client(
     #   memory.
 
     # IB's internal symbology does not expect the "source asset" in
-    # the "symbol name", what we call the "market pair name". This is
+    # the "symbol name", what we call the "market name". This is
     # common in most legacy market brokers since it's presumed that
     # given a certain stock exchange, listed assets are traded
-    # "from" a particular source fiat, normally something like USD
-    # on the given venue-provider, eg. nasdaq, nyse, etc.
-    fqme_kwargs: dict[str, Any] = {}
-    if mkt.dst.atype != 'fiat':
-        fqme_kwargs  = {
-            'without_src': True,  # default is True
-            'delim_char': '',  # bc they would normally use a frickin `.` smh
-        }
+    # "from" a particular source fiat, normally something like USD.
+    if (
+        mkt.src
+        and mkt.src.atype == 'fiat'
+    ):
+        fqme_kwargs: dict[str, Any] = {}
 
-    fqme: str = mkt.get_bs_fqme(**(fqme_kwargs))
+        if mkt.dst.atype == 'forex':
+
+            # XXX: for now we do need the src token kept in since
+            fqme_kwargs  = {
+                'without_src': False,  # default is True
+                'delim_char': '',  # bc they would normally use a frickin `.` smh
+            }
+
+        fqme: str = mkt.get_bs_fqme(**(fqme_kwargs))
+
+    else:
+        fqme = mkt.bs_fqme
 
     async with open_data_client() as proxy:
+
+
         max_timeout: float = 2.
         mean: float = 0
         count: int = 0
@@ -165,36 +171,27 @@ async def open_history_client(
         head_dt: None | datetime = None
         if (
             # fx cons seem to not provide this endpoint?
-            # TODO: guard against all contract types which don't
-            # support it?
             'idealpro' not in fqme
         ):
-            head_dt: datetime | None = await proxy.maybe_get_head_time(
-                fqme=fqme
-            )
+            try:
+                head_dt = await proxy.get_head_time(fqme=fqme)
+            except RequestError:
+                head_dt = None
 
         async def get_hist(
             timeframe: float,
-            end_dt: datetime | None = None,
-            start_dt: datetime | None = None,
+            end_dt: Optional[datetime] = None,
+            start_dt: Optional[datetime] = None,
 
         ) -> tuple[np.ndarray, str]:
-
             nonlocal max_timeout, mean, count
 
-            if (
-                start_dt
-                and start_dt.timestamp() == 0
-            ):
-                await tractor.pause()
-
             query_start = time.time()
             out, timedout = await get_bars(
                 proxy,
                 fqme,
                 timeframe,
                 end_dt=end_dt,
-                start_dt=start_dt,
             )
             latency = time.time() - query_start
             if (
@@ -208,48 +205,24 @@ async def open_history_client(
                     f'mean: {mean}'
                 )
 
-            # could be trying to retreive bars over weekend
-            if out is None:
+            if (
+                out is None
+            ):
+                # could be trying to retreive bars over weekend
                 log.error(f"Can't grab bars starting at {end_dt}!?!?")
-                if (
-                    end_dt
-                    and head_dt
-                    and end_dt <= head_dt
-                ):
-                    raise DataUnavailable(
-                        f'First timestamp is {head_dt}\n'
-                        f'But {end_dt} was requested..'
-                    )
+                raise NoData(
+                    f'{end_dt}',
+                    # frame_size=2000,
+                )
 
-                else:
-                    raise NoData(
-                        info={
-                            'fqme': fqme,
-                            'head_dt': head_dt,
-                            'start_dt': start_dt,
-                            'end_dt': end_dt,
-                            'timedout': timedout,
-                        },
-                    )
+            if (
+                end_dt
+                and head_dt
+                and end_dt <= head_dt
+            ):
+                raise DataUnavailable(f'First timestamp is {head_dt}')
 
-            # also see return type for `get_bars()`
-            bars: ibis.objects.BarDataList
-            bars_array: np.ndarray
-            first_dt: datetime
-            last_dt: datetime
-            (
-                bars,
-                bars_array,
-                first_dt,
-                last_dt,
-            ) = out
-
-            # TODO: audit the sampling period here as well?
-            # timestep should always be at least as large as the
-            # period step.
-            # tdiff: np.ndarray = np.diff(bars_array['time'])
-            # if (tdiff < timeframe).any():
-            #     await tractor.pause()
+            bars, bars_array, first_dt, last_dt = out
 
             # volume cleaning since there's -ve entries,
             # wood luv to know what crookery that is..
@@ -263,18 +236,7 @@ async def open_history_client(
         # quite sure why.. needs some tinkering and probably
         # a lookthrough of the ``ib_insync`` machinery, for eg. maybe
         # we have to do the batch queries on the `asyncio` side?
-        yield (
-            get_hist,
-            {
-                'erlangs': 1,  # max conc reqs
-                'rate': 3,  # max req rate
-                'frame_types': {  # expected frame sizes
-                    1: mk_duration(seconds=2e3),
-                    60: mk_duration(days=2),
-                }
-
-            },
-        )
+        yield get_hist, {'erlangs': 1, 'rate': 3}
 
 
 _pacing: str = (
@@ -313,7 +275,7 @@ async def wait_on_data_reset(
     # )
     # try to wait on the reset event(s) to arrive, a timeout
     # will trigger a retry up to 6 times (for now).
-    client: Client = proxy._aio_ns
+    client = proxy._aio_ns.ib.client
 
     done = trio.Event()
     with trio.move_on_after(timeout) as cs:
@@ -322,10 +284,10 @@ async def wait_on_data_reset(
 
         log.warning(
             'Sending DATA RESET request:\n'
-            f'{client.ib.client}'
+            f'{client}'
         )
         res = await data_reset_hack(
-            client=client,
+            vnc_host=client.host,
             reset_type=reset_type,
         )
 
@@ -360,10 +322,9 @@ async def wait_on_data_reset(
     return False
 
 
-_data_resetter_task: Task | None = None
+_data_resetter_task: trio.Task | None = None
 _failed_resets: int = 0
 
-
 async def get_bars(
 
     proxy: MethodProxy,
@@ -372,7 +333,6 @@ async def get_bars(
 
     # blank to start which tells ib to look up the latest datum
     end_dt: str = '',
-    start_dt: str | None = '',
 
     # TODO: make this more dynamic based on measured frame rx latency?
     # how long before we trigger a feed reset (seconds)
@@ -386,15 +346,7 @@ async def get_bars(
 
     task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
 
-) -> tuple[
-    tuple[  # result tuple
-        ibis.objects.BarDataList,
-        np.ndarray,
-        datetime,
-        datetime,
-    ] | None,
-    bool,  # timed out hint
-]:
+) -> (dict, np.ndarray):
     '''
     Retrieve historical data from a ``trio``-side task using
     a ``MethoProxy``.
@@ -419,11 +371,7 @@ async def get_bars(
 
         while _failed_resets < max_failed_resets:
             try:
-                (
-                    bars,
-                    bars_array,
-                    dt_duration,
-                ) = await proxy.bars(
+                out = await proxy.bars(
                     fqme=fqme,
                     end_dt=end_dt,
                     sample_period_s=timeframe,
@@ -434,58 +382,28 @@ async def get_bars(
                     # current impl) to detect a cancel case.
                     # timeout=timeout,
                 )
-                # usually either a request during a venue closure
-                # or into a large (weekend) closure gap.
-                if not bars:
-                    # no data returned?
+                if out is None:
+                    raise NoData(f'{end_dt}')
+
+                bars, bars_array, dt_duration = out
+
+                if (
+                    not bars
+                    and end_dt
+                ):
                     log.warning(
-                        'History frame is blank?\n'
-                        f'start_dt: {start_dt}\n'
-                        f'end_dt: {end_dt}\n'
-                        f'duration: {dt_duration}\n'
+                        f'History is blank for {dt_duration} from {end_dt}'
                     )
-                    # NOTE: REQUIRED to pass back value..
-                    result = None
-                    return None
+                    end_dt -= dt_duration
+                    continue
 
-                # not enough bars signal, likely due to venue
-                # operational gaps.
-                if end_dt:
-                    dur_s: float = len(bars) * timeframe
-                    bars_dur = Duration(seconds=dur_s)
-                    dt_dur_s: float = dt_duration.in_seconds()
-                    if dur_s < dt_dur_s:
-                        log.warning(
-                            'History frame is shorter then expected?\n'
-                            f'start_dt: {start_dt}\n'
-                            f'end_dt: {end_dt}\n'
-                            f'duration: {dt_dur_s}\n'
-                            f'frame duration seconds: {dur_s}\n'
-                            f'dur diff: {dt_duration - bars_dur}\n'
-                        )
-                        # NOTE: we used to try to get a minimal
-                        # set of bars by recursing but this ran
-                        # into possible infinite query loops
-                        # when logic in the `Client.bars()` dt
-                        # diffing went bad. So instead for now
-                        # we just return the
-                        # shorter-then-expected history with
-                        # a warning.
-                        # TODO: in the future it prolly makes
-                        # the most send to do venue operating
-                        # hours lookup and
-                        # timestamp-in-operating-range set
-                        # checking to know for sure if we can
-                        # safely and quickly ignore non-uniform history
-                        # frame timestamp gaps..
-                        # end_dt -= dt_duration
-                        # continue
-                        # await tractor.pause()
+                if bars_array is None:
+                    raise SymbolNotFound(fqme)
 
-                first_dt = from_timestamp(
+                first_dt = pendulum.from_timestamp(
                     bars[0].date.timestamp())
 
-                last_dt = from_timestamp(
+                last_dt = pendulum.from_timestamp(
                     bars[-1].date.timestamp())
 
                 time = bars_array['time']
@@ -498,22 +416,15 @@ async def get_bars(
                 if data_cs:
                     data_cs.cancel()
 
-                # NOTE: setting this is critical!
-                result = (
-                    bars,  # ib native
-                    bars_array,  # numpy
-                    first_dt,
-                    last_dt,
-                )
+                result = (bars, bars_array, first_dt, last_dt)
 
                 # signal data reset loop parent task
                 result_ready.set()
 
-                # NOTE: this isn't getting collected anywhere!
                 return result
 
             except RequestError as err:
-                msg: str = err.message
+                msg = err.message
 
                 if 'No market data permissions for' in msg:
                     # TODO: signalling for no permissions searches
@@ -535,7 +446,7 @@ async def get_bars(
                     if end_dt is not None:
                         end_dt = end_dt.subtract(days=1)
                     elif end_dt is None:
-                        end_dt = now().subtract(days=1)
+                        end_dt = pendulum.now().subtract(days=1)
 
                     log.warning(
                         f'NO DATA found ending @ {end_dt}\n'
@@ -551,29 +462,21 @@ async def get_bars(
                     nodatas_count += 1
                     continue
 
-                elif (
-                    'API historical data query cancelled'
-                    in
-                    err.message
-                ):
+                elif 'API historical data query cancelled' in err.message:
                     log.warning(
                         'Query cancelled by IB (:eyeroll:):\n'
                         f'{err.message}'
                     )
                     continue
-
                 elif (
                     'Trading TWS session is connected from a different IP'
-                    in
-                    err.message
+                        in err.message
                 ):
                     log.warning("ignoring ip address warning")
                     continue
 
                 # XXX: more or less same as above timeout case
-                elif (
-                    _pacing in msg
-                ):
+                elif _pacing in msg:
                     log.warning(
                         'History throttle rate reached!\n'
                         'Resetting farms with `ctrl-alt-f` hack\n'
@@ -625,10 +528,9 @@ async def get_bars(
                 # don't double invoke the reset hack if another
                 # requester task already has it covered.
                 continue
-
             else:
                 _data_resetter_task = trio.lowlevel.current_task()
-                unset_resetter: bool = True
+                unset_resetter = True
 
             # spawn new data reset task
             data_cs, reset_done = await nurse.start(
@@ -641,16 +543,30 @@ async def get_bars(
             # sync wait on reset to complete
             await reset_done.wait()
 
-    _data_resetter_task = (
-        None
-        if unset_resetter
-        else _data_resetter_task
-    )
-    assert result
-    return (
-        result,
-        data_cs is not None,
-    )
+    _data_resetter_task = None if unset_resetter else _data_resetter_task
+    return result, data_cs is not None
+
+
+# re-mapping to piker asset type names
+# https://github.com/erdewit/ib_insync/blob/master/ib_insync/contract.py#L113
+_asset_type_map = {
+    'STK': 'stock',
+    'OPT': 'option',
+    'FUT': 'future',
+    'CONTFUT': 'continuous_future',
+    'CASH': 'forex',
+    'IND': 'index',
+    'CFD': 'cfd',
+    'BOND': 'bond',
+    'CMDTY': 'commodity',
+    'FOP': 'futures_option',
+    'FUND': 'mutual_fund',
+    'WAR': 'warrant',
+    'IOPT': 'warran',
+    'BAG': 'bag',
+    'CRYPTO': 'crypto',  # bc it's diff then fiat?
+    # 'NEWS': 'news',
+}
 
 
 _quote_streams: dict[str, trio.abc.ReceiveStream] = {}
@@ -671,8 +587,8 @@ async def _setup_quote_stream(
         # making them mostly useless and explains why the scanner
         # is always slow XD
         # '293',  # Trade count for day
-        # '294',  # Trade rate / minute
-        # '295',  # Vlm rate / minute
+        '294',  # Trade rate / minute
+        '295',  # Vlm rate / minute
     ),
     contract: Contract | None = None,
 
@@ -686,12 +602,13 @@ async def _setup_quote_stream(
     '''
     global _quote_streams
 
+    to_trio.send_nowait(None)
+
     async with load_aio_clients(
         disconnect_on_exit=False,
     ) as accts2clients:
         caccount_name, client = get_preferred_data_client(accts2clients)
         contract = contract or (await client.find_contract(symbol))
-        to_trio.send_nowait(contract)  # cuz why not
         ticker: Ticker = client.ib.reqMktData(contract, ','.join(opts))
 
         # NOTE: it's batch-wise and slow af but I guess could
@@ -783,9 +700,7 @@ async def open_aio_quote_stream(
         symbol=symbol,
         contract=contract,
 
-    ) as (contract, from_aio):
-
-        assert contract
+    ) as (first, from_aio):
 
         # cache feed for later consumers
         _quote_streams[symbol] = from_aio
@@ -856,6 +771,98 @@ def normalize(
     return data
 
 
+@async_lifo_cache()
+async def get_mkt_info(
+    fqme: str,
+
+    proxy: MethodProxy | None = None,
+
+) -> tuple[MktPair, ibis.ContractDetails]:
+
+    # XXX: we don't need to split off any fqme broker part?
+    # bs_fqme, _, broker = fqme.partition('.')
+
+    proxy: MethodProxy
+    get_details: bool = False
+    if proxy is not None:
+        client_ctx = nullcontext(proxy)
+    else:
+        client_ctx = open_data_client
+
+    async with client_ctx as proxy:
+        try:
+            (
+                con,  # Contract
+                details,  # ContractDetails
+            ) = await proxy.get_sym_details(symbol=fqme)
+        except ConnectionError:
+            log.exception(f'Proxy is ded {proxy._aio_ns}')
+            raise
+
+    # TODO: more consistent field translation
+    init_info: dict = {}
+    atype = _asset_type_map[con.secType]
+
+    if atype == 'commodity':
+        venue: str = 'cmdty'
+    else:
+        venue = con.primaryExchange or con.exchange
+
+    price_tick: Decimal = Decimal(str(details.minTick))
+
+    if atype == 'stock':
+        # XXX: GRRRR they don't support fractional share sizes for
+        # stocks from the API?!
+        # if con.secType == 'STK':
+        size_tick = Decimal('1')
+    else:
+        size_tick: Decimal = Decimal(str(details.minSize).rstrip('0'))
+        # |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
+
+    # NOTE: this is duplicate from the .broker.norm_trade_records()
+    # routine, we should factor all this parsing somewhere..
+    expiry_str = str(con.lastTradeDateOrContractMonth)
+    # if expiry:
+    #     expiry_str: str = str(pendulum.parse(
+    #         str(expiry).strip(' ')
+    #     ))
+
+    # TODO: currently we can't pass the fiat src asset because
+    # then we'll get a `MNQUSD` request for history data..
+    # we need to figure out how we're going to handle this (later?)
+    # but likely we want all backends to eventually handle
+    # ``dst/src.venue.`` style !?
+    src: str | Asset = ''
+    if atype == 'forex':
+        src = Asset(
+            name=str(con.currency),
+            atype='fiat',
+            tx_tick=Decimal('0.01'),  # right?
+        )
+
+    mkt = MktPair(
+        dst=Asset(
+            name=con.symbol.lower(),
+            atype=atype,
+            tx_tick=size_tick,
+        ),
+        src=src,
+
+        price_tick=price_tick,
+        size_tick=size_tick,
+
+        bs_mktid=str(con.conId),
+        venue=str(venue),
+        expiry=expiry_str,
+        broker='ib',
+
+        # TODO: options contract info as str?
+        # contract_info=<optionsdetails>
+    )
+
+    return mkt, details
+
+
 async def stream_quotes(
 
     send_chan: trio.abc.SendChannel,
@@ -883,10 +890,7 @@ async def stream_quotes(
     proxy: MethodProxy
     mkt: MktPair
     details: ibis.ContractDetails
-    async with (
-        open_data_client() as proxy,
-        # trio.open_nursery() as tn,
-    ):
+    async with open_data_client() as proxy:
         mkt, details = await get_mkt_info(
             sym,
             proxy=proxy,  # passed to avoid implicit client load
@@ -895,7 +899,7 @@ async def stream_quotes(
         init_msg = FeedInit(mkt_info=mkt)
 
         if mkt.dst.atype in {
-            'fiat',
+            'forex',
             'index',
             'commodity',
         }:
@@ -906,50 +910,26 @@ async def stream_quotes(
         init_msgs.append(init_msg)
 
         con: Contract = details.contract
-        first_ticker: Ticker | None = None
+        first_ticker: Ticker = await proxy.get_quote(contract=con)
+        first_quote: dict = normalize(first_ticker)
+        log.runtime(f'FIRST QUOTE: {first_quote}')
+
+        # TODO: we should instead spawn a task that waits on a feed to start
+        # and let it wait indefinitely..instead of this hard coded stuff.
         with trio.move_on_after(1):
-            first_ticker: Ticker = await proxy.get_quote(
-                contract=con,
-                raise_on_timeout=False,
-            )
+            first_ticker = await proxy.get_quote(contract=con)
 
-        if first_ticker:
-            first_quote: dict = normalize(first_ticker)
-
-            # TODO: we need a stack-oriented log levels filters for
-            # this!
-            # log.info(message, filter={'stack': 'live_feed'}) ?
-            log.runtime(
-                'Rxed init quote:\n\n'
-                f'{pformat(first_quote)}\n'
-            )
-
-        # NOTE: it might be outside regular trading hours for
-        # assets with "standard venue operating hours" so we
-        # only "pretend the feed is live" when the dst asset
-        # type is NOT within the NON-NORMAL-venue set: aka not
-        # commodities, forex or crypto currencies which CAN
-        # always return a NaN on a snap quote request during
-        # normal venue hours. In the case of a closed venue
-        # (equitiies, futes, bonds etc.) we at least try to
-        # grab the OHLC history.
+        # it might be outside regular trading hours so see if we can at
+        # least grab history.
         if (
-            first_ticker
-            and
-            isnan(first_ticker.last)
-            # SO, if the last quote price value is NaN we ONLY
-            # "pretend to do" `feed_is_live.set()` if it's a known
-            # dst asset venue with a lot of closed operating hours.
+            isnan(first_ticker.last)  # last quote price value is nan
             and mkt.dst.atype not in {
                 'commodity',
-                'fiat',
+                'forex',
                 'crypto',
             }
         ):
-            task_status.started((
-                init_msgs,
-                first_quote,
-            ))
+            task_status.started((init_msgs, first_quote))
 
             # it's not really live but this will unblock
             # the brokerd feed task to tell the ui to update?
@@ -959,28 +939,6 @@ async def stream_quotes(
             await trio.sleep_forever()
             return  # we never expect feed to come up?
 
-        # TODO: we should instead spawn a task that waits on a feed
-        # to start and let it wait indefinitely..instead of this
-        # hard coded stuff.
-        # async def wait_for_first_quote():
-        #     with trio.CancelScope() as cs:
-
-        # XXX: MUST acquire a ticker + first quote before starting
-        # the live quotes loop!
-        # with trio.move_on_after(1):
-        first_ticker = await proxy.get_quote(
-            contract=con,
-            raise_on_timeout=True,
-        )
-        first_quote: dict = normalize(first_ticker)
-
-        # TODO: we need a stack-oriented log levels filters for
-        # this!
-        # log.info(message, filter={'stack': 'live_feed'}) ?
-        log.runtime(
-            'Rxed init quote:\n'
-            f'{pformat(first_quote)}'
-        )
         cs: trio.CancelScope | None = None
         startup: bool = True
         while (
@@ -1001,11 +959,8 @@ async def stream_quotes(
 
                     # only on first entry at feed boot up
                     if startup:
-                        startup: bool = False
-                        task_status.started((
-                            init_msgs,
-                            first_quote,
-                        ))
+                        startup = False
+                        task_status.started((init_msgs, first_quote))
 
                     # start a stream restarter task which monitors the
                     # data feed event.
@@ -1029,7 +984,7 @@ async def stream_quotes(
 
                             # generally speaking these feeds don't
                             # include vlm data.
-                            atype: str = mkt.dst.atype
+                            atype = mkt.dst.atype
                             log.info(
                                 f'No-vlm {mkt.fqme}@{atype}, skipping quote poll'
                             )
@@ -1065,16 +1020,154 @@ async def stream_quotes(
                             quote = normalize(ticker)
                             log.debug(f"First ticker received {quote}")
 
-                        # tell data-layer spawner-caller that live
-                        # quotes are now streaming.
+                        # tell caller quotes are now coming in live
                         feed_is_live.set()
 
                         # last = time.time()
                         async for ticker in stream:
                             quote = normalize(ticker)
                             fqme = quote['fqme']
+                            # print(f'sending {fqme}:\n{quote}')
                             await send_chan.send({fqme: quote})
 
                             # ugh, clear ticks since we've consumed them
                             ticker.ticks = []
                             # last = time.time()
+
+
+@tractor.context
+async def open_symbol_search(
+    ctx: tractor.Context,
+
+) -> None:
+
+    # TODO: load user defined symbol set locally for fast search?
+    await ctx.started({})
+
+    async with (
+        open_client_proxies() as (proxies, clients),
+        open_data_client() as data_proxy,
+    ):
+        async with ctx.open_stream() as stream:
+
+            # select a non-history client for symbol search to lighten
+            # the load in the main data node.
+            proxy = data_proxy
+            for name, proxy in proxies.items():
+                if proxy is data_proxy:
+                    continue
+                break
+
+            ib_client = proxy._aio_ns.ib
+            log.info(f'Using {ib_client} for symbol search')
+
+            last = time.time()
+            async for pattern in stream:
+                log.info(f'received {pattern}')
+                now = time.time()
+
+                # this causes tractor hang...
+                # assert 0
+
+                assert pattern, 'IB can not accept blank search pattern'
+
+                # throttle search requests to no faster then 1Hz
+                diff = now - last
+                if diff < 1.0:
+                    log.debug('throttle sleeping')
+                    await trio.sleep(diff)
+                    try:
+                        pattern = stream.receive_nowait()
+                    except trio.WouldBlock:
+                        pass
+
+                if (
+                    not pattern
+                    or pattern.isspace()
+
+                    # XXX: not sure if this is a bad assumption but it
+                    # seems to make search snappier?
+                    or len(pattern) < 1
+                ):
+                    log.warning('empty pattern received, skipping..')
+
+                    # TODO: *BUG* if nothing is returned here the client
+                    # side will cache a null set result and not showing
+                    # anything to the use on re-searches when this query
+                    # timed out. We probably need a special "timeout" msg
+                    # or something...
+
+                    # XXX: this unblocks the far end search task which may
+                    # hold up a multi-search nursery block
+                    await stream.send({})
+
+                    continue
+
+                log.info(f'searching for {pattern}')
+
+                last = time.time()
+
+                # async batch search using api stocks endpoint and module
+                # defined adhoc symbol set.
+                stock_results = []
+
+                async def stash_results(target: Awaitable[list]):
+                    try:
+                        results = await target
+                    except tractor.trionics.Lagged:
+                        print("IB SYM-SEARCH OVERRUN?!?")
+                        return
+
+                    stock_results.extend(results)
+
+                for i in range(10):
+                    with trio.move_on_after(3) as cs:
+                        async with trio.open_nursery() as sn:
+                            sn.start_soon(
+                                stash_results,
+                                proxy.search_symbols(
+                                    pattern=pattern,
+                                    upto=5,
+                                ),
+                            )
+
+                            # trigger async request
+                            await trio.sleep(0)
+
+                    if cs.cancelled_caught:
+                        log.warning(
+                            f'Search timeout? {proxy._aio_ns.ib.client}'
+                        )
+                        continue
+                    else:
+                        break
+
+                    # # match against our ad-hoc set immediately
+                    # adhoc_matches = fuzzy.extractBests(
+                    #     pattern,
+                    #     list(_adhoc_futes_set),
+                    #     score_cutoff=90,
+                    # )
+                    # log.info(f'fuzzy matched adhocs: {adhoc_matches}')
+                    # adhoc_match_results = {}
+                    # if adhoc_matches:
+                    #     # TODO: do we need to pull contract details?
+                    #     adhoc_match_results = {i[0]: {} for i in
+                    #     adhoc_matches}
+
+                log.debug(f'fuzzy matching stocks {stock_results}')
+                stock_matches = fuzzy.extractBests(
+                    pattern,
+                    stock_results,
+                    score_cutoff=50,
+                )
+
+                # matches = adhoc_match_results | {
+                matches = {
+                    item[0]: {} for item in stock_matches
+                }
+                # TODO: we used to deliver contract details
+                # {item[2]: item[0] for item in stock_matches}
+
+                log.debug(f"sending matches: {matches.keys()}")
+                await stream.send(matches)

piker/brokers/ib/ledger.py

@@ -1,529 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Trade transaction accounting and normalization.
-
-'''
-from __future__ import annotations
-from bisect import insort
-from dataclasses import asdict
-from decimal import Decimal
-from functools import partial
-from pprint import pformat
-from typing import (
-    Any,
-    Callable,
-    TYPE_CHECKING,
-)
-
-from bidict import bidict
-from pendulum import (
-    DateTime,
-    parse,
-    from_timestamp,
-)
-from ib_insync import (
-    Contract,
-    Commodity,
-    Fill,
-    Execution,
-    CommissionReport,
-)
-
-from piker.types import Struct
-from piker.data import (
-    SymbologyCache,
-)
-from piker.accounting import (
-    Asset,
-    dec_digits,
-    digits_to_dec,
-    Transaction,
-    MktPair,
-    iter_by_dt,
-)
-from ._flex_reports import parse_flex_dt
-from ._util import log
-
-if TYPE_CHECKING:
-    from .api import (
-        Client,
-        MethodProxy,
-    )
-
-
-tx_sort: Callable = partial(
-    iter_by_dt,
-    parsers={
-        'dateTime': parse_flex_dt,
-        'datetime': parse,
-
-        # XXX: for some some fucking 2022 and
-        # back options records.. f@#$ me..
-        'date': parse,
-    }
-)
-
-
-def norm_trade(
-    tid: str,
-    record: dict[str, Any],
-
-    # this is the dict that was returned from
-    # `Client.get_mkt_pairs()` and when running offline ledger
-    # processing from `.accounting`, this will be the table loaded
-    # into `SymbologyCache.pairs`.
-    pairs: dict[str, Struct],
-    symcache: SymbologyCache | None = None,
-
-) -> Transaction | None:
-
-    conid: int = str(record.get('conId') or record['conid'])
-    bs_mktid: str = str(conid)
-
-    # NOTE: sometimes weird records (like BTTX?)
-    # have no field for this?
-    comms: float = -1 * (
-        record.get('commission')
-        or record.get('ibCommission')
-        or 0
-    )
-    if not comms:
-        log.warning(
-            'No commissions found for record?\n'
-            f'{pformat(record)}\n'
-        )
-
-    price: float = (
-        record.get('price')
-        or record.get('tradePrice')
-    )
-    if price is None:
-        log.warning(
-            'No `price` field found in record?\n'
-            'Skipping normalization..\n'
-            f'{pformat(record)}\n'
-        )
-        return None
-
-    # the api doesn't do the -/+ on the quantity for you but flex
-    # records do.. are you fucking serious ib...!?
-    size: float|int = (
-        record.get('quantity')
-        or record['shares']
-    ) * {
-        'BOT': 1,
-        'SLD': -1,
-    }[record['side']]
-
-    symbol: str = record['symbol']
-    exch: str = (
-        record.get('listingExchange')
-        or record.get('primaryExchange')
-        or record['exchange']
-    )
-
-    # NOTE: remove null values since `tomlkit` can't serialize
-    # them to file.
-    if dnc := record.pop('deltaNeutralContract', None):
-        record['deltaNeutralContract'] = dnc
-
-    # likely an opts contract record from a flex report..
-    # TODO: no idea how to parse ^ the strike part from flex..
-    # (00010000 any, or 00007500 tsla, ..)
-    # we probably must do the contract lookup for this?
-    if (
-        '   ' in symbol
-        or '--' in exch
-    ):
-        underlying, _, tail = symbol.partition('   ')
-        exch: str = 'opt'
-        expiry: str = tail[:6]
-        # otype = tail[6]
-        # strike = tail[7:]
-
-        log.warning(
-            f'Skipping option contract -> NO SUPPORT YET!\n'
-            f'{symbol}\n'
-        )
-        return None
-
-    # timestamping is way different in API records
-    dtstr: str = record.get('datetime')
-    date: str = record.get('date')
-    flex_dtstr: str = record.get('dateTime')
-
-    if dtstr or date:
-        dt: DateTime = parse(dtstr or date)
-
-    elif flex_dtstr:
-        # probably a flex record with a wonky non-std timestamp..
-        dt: DateTime = parse_flex_dt(record['dateTime'])
-
-    # special handling of symbol extraction from
-    # flex records using some ad-hoc schema parsing.
-    asset_type: str = (
-        record.get('assetCategory')
-        or record.get('secType')
-        or 'STK'
-    )
-
-    if (expiry := (
-            record.get('lastTradeDateOrContractMonth')
-            or record.get('expiry')
-        )
-    ):
-        expiry: str = str(expiry).strip(' ')
-        # NOTE: we directly use the (simple and usually short)
-        # date-string expiry token when packing the `MktPair`
-        # since we want the fqme to contain *that* token.
-        # It might make sense later to instead parse and then
-        # render different output str format(s) for this same
-        # purpose depending on asset-type-market down the road.
-        # Eg. for derivs we use the short token only for fqme
-        # but use the isoformat('T') for transactions and
-        # account file position entries?
-        # dt_str: str = pendulum.parse(expiry).isoformat('T')
-
-    # XXX: pretty much all legacy market assets have a fiat
-    # currency (denomination) determined by their venue.
-    currency: str = record['currency']
-    src = Asset(
-        name=currency.lower(),
-        atype='fiat',
-        tx_tick=Decimal('0.01'),
-    )
-
-    match asset_type:
-        case 'FUT':
-            # XXX (flex) ledger entries don't necessarily have any
-            # simple 3-char key.. sometimes the .symbol is some
-            # weird internal key that we probably don't want in the
-            # .fqme => we should probably just wrap `Contract` to
-            # this like we do other crypto$ backends XD
-
-            # NOTE: at least older FLEX records should have
-            # this field.. no idea about API entries..
-            local_symbol: str | None = record.get('localSymbol')
-            underlying_key: str = record.get('underlyingSymbol')
-            descr: str | None = record.get('description')
-
-            if (
-                not (
-                    local_symbol
-                    and symbol in local_symbol
-                )
-                and (
-                        descr
-                        and symbol not in descr
-                    )
-            ):
-                con_key, exp_str = descr.split(' ')
-                symbol: str = underlying_key or con_key
-
-            dst = Asset(
-                name=symbol.lower(),
-                atype='future',
-                tx_tick=Decimal('1'),
-            )
-
-        case 'STK':
-            dst = Asset(
-                name=symbol.lower(),
-                atype='stock',
-                tx_tick=Decimal('1'),
-            )
-
-        case 'CASH':
-            if currency not in symbol:
-                # likely a dict-casted `Forex` contract which
-                # has .symbol as the dst and .currency as the
-                # src.
-                name: str = symbol.lower()
-            else:
-                # likely a flex-report record which puts
-                # EUR.USD as the symbol field and just USD in
-                # the currency field.
-                name: str = symbol.lower().replace(f'.{src.name}', '')
-
-            dst = Asset(
-                name=name,
-                atype='fiat',
-                tx_tick=Decimal('0.01'),
-            )
-
-        case 'OPT':
-            dst = Asset(
-                name=symbol.lower(),
-                atype='option',
-                tx_tick=Decimal('1'),
-
-                # TODO: we should probably always cast to the
-                # `Contract` instance then dict-serialize that for
-                # the `.info` field!
-                # info=asdict(Option()),
-            )
-
-        case 'CMDTY':
-            from .symbols import _adhoc_symbol_map
-            con_kwargs, _ = _adhoc_symbol_map[symbol.upper()]
-            dst = Asset(
-                name=symbol.lower(),
-                atype='commodity',
-                tx_tick=Decimal('1'),
-                info=asdict(Commodity(**con_kwargs)),
-            )
-
-    # try to build out piker fqme from record.
-    # src: str = record['currency']
-    price_tick: Decimal = digits_to_dec(dec_digits(price))
-
-    # NOTE: can't serlialize `tomlkit.String` so cast to native
-    atype: str = str(dst.atype)
-
-    # if not (mkt := symcache.mktmaps.get(bs_mktid)):
-    mkt = MktPair(
-        bs_mktid=bs_mktid,
-        dst=dst,
-
-        price_tick=price_tick,
-        # NOTE: for "legacy" assets, volume is normally discreet, not
-        # a float, but we keep a digit in case the suitz decide
-        # to get crazy and change it; we'll be kinda ready
-        # schema-wise..
-        size_tick=Decimal('1'),
-
-        src=src,  # XXX: normally always a fiat
-
-        _atype=atype,
-
-        venue=exch,
-        expiry=expiry,
-        broker='ib',
-
-        _fqme_without_src=(atype != 'fiat'),
-    )
-
-    fqme: str = mkt.fqme
-
-    # XXX: if passed in, we fill out the symcache ad-hoc in order
-    # to make downstream accounting work..
-    if symcache is not None:
-        orig_mkt: MktPair | None  = symcache.mktmaps.get(bs_mktid)
-        if (
-            orig_mkt
-            and orig_mkt.fqme != mkt.fqme
-        ):
-            log.warning(
-            # print(
-                f'Contracts with common `conId`: {bs_mktid} mismatch..\n'
-                f'{orig_mkt.fqme} -> {mkt.fqme}\n'
-                # 'with DIFF:\n'
-                # f'{mkt - orig_mkt}'
-            )
-
-        symcache.mktmaps[bs_mktid] = mkt
-        symcache.mktmaps[fqme] = mkt
-        symcache.assets[src.name] = src
-        symcache.assets[dst.name] = dst
-
-    # NOTE: for flex records the normal fields for defining an fqme
-    # sometimes won't be available so we rely on two approaches for
-    # the "reverse lookup" of piker style fqme keys:
-    # - when dealing with API trade records received from
-    #   `IB.trades()` we do a contract lookup at he time of processing
-    # - when dealing with flex records, it is assumed the record
-    #   is at least a day old and thus the TWS position reporting system
-    #   should already have entries if the pps are still open, in
-    #   which case, we can pull the fqme from that table (see
-    #   `trades_dialogue()` above).
-    return Transaction(
-        fqme=fqme,
-        tid=tid,
-        size=size,
-        price=price,
-        cost=comms,
-        dt=dt,
-        expiry=expiry,
-        bs_mktid=str(conid),
-    )
-
-
-
-def norm_trade_records(
-    ledger: dict[str, Any],
-    symcache: SymbologyCache | None = None,
-
-) -> dict[str, Transaction]:
-    '''
-    Normalize (xml) flex-report or (recent) API trade records into
-    our ledger format with parsing for `MktPair` and `Asset`
-    extraction to fill in the `Transaction.sys: MktPair` field.
-
-    '''
-    records: list[Transaction] = []
-    for tid, record in ledger.items():
-
-        txn = norm_trade(
-            tid,
-            record,
-
-            # NOTE: currently no symcache support
-            pairs={},
-            symcache=symcache,
-        )
-
-        if txn is None:
-            continue
-
-        # inject txns sorted by datetime
-        insort(
-            records,
-            txn,
-            key=lambda t: t.dt
-        )
-
-    return {r.tid: r for r in records}
-
-
-def api_trades_to_ledger_entries(
-    accounts: bidict[str, str],
-    fills: list[Fill],
-
-) -> dict[str, dict]:
-    '''
-    Convert API execution objects entry objects into
-    flattened-``dict`` form, pretty much straight up without
-    modification except add a `pydatetime` field from the parsed
-    timestamp so that on write
-
-    '''
-    trades_by_account: dict[str, dict] = {}
-    for fill in fills:
-
-        # NOTE: for the schema, see the defn for `Fill` which is
-        # a `NamedTuple` subtype
-        fdict: dict = fill._asdict()
-
-        # flatten all (sub-)objects and convert to dicts.
-        # with values packed into one top level entry.
-        val: CommissionReport | Execution | Contract
-        txn_dict: dict[str, Any] = {}
-        for attr_name, val in fdict.items():
-            match attr_name:
-                # value is a `@dataclass` subtype
-                case 'contract' | 'execution' | 'commissionReport':
-                    txn_dict.update(asdict(val))
-
-                case 'time':
-                    # ib has wack ns timestamps, or is that us?
-                    continue
-
-                # TODO: we can remove this case right since there's
-                # only 4 fields on a `Fill`?
-                case _:
-                    txn_dict[attr_name] = val
-
-        tid = str(txn_dict['execId'])
-        dt = from_timestamp(txn_dict['time'])
-        txn_dict['datetime'] = str(dt)
-        acctid = accounts[txn_dict['acctNumber']]
-
-        # NOTE: only inserted (then later popped) for sorting below!
-        txn_dict['pydatetime'] = dt
-
-        if not tid:
-            # this is likely some kind of internal adjustment
-            # transaction, likely one of the following:
-            # - an expiry event that will show a "book trade" indicating
-            #   some adjustment to cash balances: zeroing or itm settle.
-            # - a manual cash balance position adjustment likely done by
-            #   the user from the accounts window in TWS where they can
-            #   manually set the avg price and size:
-            #   https://api.ibkr.com/lib/cstools/faq/web1/index.html#/tag/DTWS_ADJ_AVG_COST
-            log.warning(
-                'Skipping ID-less ledger txn_dict:\n'
-                f'{pformat(txn_dict)}'
-            )
-            continue
-
-        trades_by_account.setdefault(
-            acctid, {}
-        )[tid] = txn_dict
-
-    # TODO: maybe we should just bisect.insort() into a list of
-    # tuples and then return a dict of that?
-    # sort entries in output by python based datetime
-    for acctid in trades_by_account:
-        trades_by_account[acctid] = dict(sorted(
-            trades_by_account[acctid].items(),
-            key=lambda entry: entry[1].pop('pydatetime'),
-        ))
-
-    return trades_by_account
-
-
-async def update_ledger_from_api_trades(
-    fills: list[Fill],
-    client: Client | MethodProxy,
-    accounts_def_inv: bidict[str, str],
-
-    # NOTE: provided for ad-hoc insertions "as transactions are
-    # processed" -> see `norm_trade()` signature requirements.
-    symcache: SymbologyCache | None = None,
-
-) -> tuple[
-    dict[str, Transaction],
-    dict[str, dict],
-]:
-    # XXX; ERRGGG..
-    # pack in the "primary/listing exchange" value from a
-    # contract lookup since it seems this isn't available by
-    # default from the `.fills()` method endpoint...
-    fill: Fill
-    for fill in fills:
-        con: Contract = fill.contract
-        conid: str = con.conId
-        pexch: str | None = con.primaryExchange
-
-        if not pexch:
-            cons = await client.get_con(conid=conid)
-            if cons:
-                con = cons[0]
-                pexch = con.primaryExchange or con.exchange
-            else:
-                # for futes it seems like the primary is always empty?
-                pexch: str = con.exchange
-
-        # pack in the ``Contract.secType``
-        # entry['asset_type'] = condict['secType']
-
-    entries: dict[str, dict] = api_trades_to_ledger_entries(
-        accounts_def_inv,
-        fills,
-    )
-    # normalize recent session's trades to the `Transaction` type
-    trans_by_acct: dict[str, dict[str, Transaction]] = {}
-
-    for acctid, trades_by_id in entries.items():
-        # normalize to transaction form
-        trans_by_acct[acctid] = norm_trade_records(
-            trades_by_id,
-            symcache=symcache,
-        )
-
-    return trans_by_acct, entries

piker/brokers/ib/symbols.py

@@ -1,615 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Symbology search and normalization.
-
-'''
-from __future__ import annotations
-from contextlib import (
-    nullcontext,
-)
-from decimal import Decimal
-import time
-from typing import (
-    Awaitable,
-    TYPE_CHECKING,
-)
-
-from rapidfuzz import process as fuzzy
-import ib_insync as ibis
-import tractor
-import trio
-
-from piker.accounting import (
-    Asset,
-    MktPair,
-    unpack_fqme,
-)
-from piker._cacheables import (
-    async_lifo_cache,
-)
-
-from ._util import (
-    log,
-)
-
-if TYPE_CHECKING:
-    from .api import (
-        MethodProxy,
-        Client,
-    )
-
-_futes_venues = (
-    'GLOBEX',
-    'NYMEX',
-    'CME',
-    'CMECRYPTO',
-    'COMEX',
-    # 'CMDTY',  # special name case..
-    'CBOT',  # (treasury) yield futures
-)
-
-_adhoc_cmdty_set = {
-    # metals
-    # https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=69067924
-    'xauusd.cmdty',  # london gold spot ^
-    'xagusd.cmdty',  # silver spot
-}
-
-# NOTE: if you aren't seeing one of these symbol's futues contracts
-# show up, it's likely the `.<venue>` part is wrong!
-_adhoc_futes_set = {
-
-    # equities
-    'nq.cme',
-    'mnq.cme',  # micro
-
-    'es.cme',
-    'mes.cme',  # micro
-
-    # cypto$
-    'brr.cme',
-    'mbt.cme',  # micro
-    'ethusdrr.cme',
-
-    # agriculture
-    'he.comex',  # lean hogs
-    'le.comex',  # live cattle (geezers)
-    'gf.comex',  # feeder cattle (younguns)
-
-    # raw
-    'lb.comex',  # random len lumber
-
-    'gc.comex',
-    'mgc.comex',  # micro
-
-    # oil & gas
-    'cl.nymex',
-
-    'ni.comex',  # silver futes
-    'qi.comex',  # mini-silver futes
-
-    # treasury yields
-    # etfs by duration:
-    # SHY -> IEI -> IEF -> TLT
-    'zt.cbot',  # 2y
-    'z3n.cbot',  # 3y
-    'zf.cbot',  # 5y
-    'zn.cbot',  # 10y
-    'zb.cbot',  # 30y
-
-    # (micros of above)
-    '2yy.cbot',
-    '5yy.cbot',
-    '10y.cbot',
-    '30y.cbot',
-}
-
-
-# taken from list here:
-# https://www.interactivebrokers.com/en/trading/products-spot-currencies.php
-_adhoc_fiat_set = set((
-    'USD, AED, AUD, CAD,'
-    'CHF, CNH, CZK, DKK,'
-    'EUR, GBP, HKD, HUF,'
-    'ILS, JPY, MXN, NOK,'
-    'NZD, PLN, RUB, SAR,'
-    'SEK, SGD, TRY, ZAR'
-    ).split(' ,')
-)
-
-# manually discovered tick discrepancies,
-# onl god knows how or why they'd cuck these up..
-_adhoc_mkt_infos: dict[int | str, dict] = {
-    'vtgn.nasdaq': {'price_tick': Decimal('0.01')},
-}
-
-
-# map of symbols to contract ids
-_adhoc_symbol_map = {
-    # https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=69067924
-
-    # NOTE: some cmdtys/metals don't have trade data like gold/usd:
-    # https://groups.io/g/twsapi/message/44174
-    'XAUUSD': ({'conId': 69067924}, {'whatToShow': 'MIDPOINT'}),
-}
-for qsn in _adhoc_futes_set:
-    sym, venue = qsn.split('.')
-    assert venue.upper() in _futes_venues, f'{venue}'
-    _adhoc_symbol_map[sym.upper()] = (
-        {'exchange': venue},
-        {},
-    )
-
-
-# exchanges we don't support at the moment due to not knowing
-# how to do symbol-contract lookup correctly likely due
-# to not having the data feeds subscribed.
-_exch_skip_list = {
-
-    'ASX',  # aussie stocks
-    'MEXI',  # mexican stocks
-
-    # no idea
-    'NSE',
-    'VALUE',
-    'FUNDSERV',
-    'SWB2',
-    'PSE',
-    'PHLX',
-}
-
-# optional search config the backend can register for
-# it's symbol search handling (in this case we avoid
-# accepting patterns before the kb has settled more then
-# a quarter second).
-_search_conf = {
-    'pause_period': 6 / 16,
-}
-
-
-@tractor.context
-async def open_symbol_search(ctx: tractor.Context) -> None:
-    '''
-    Symbology search brokerd-endpoint.
-
-    '''
-    from .api import open_client_proxies
-    from .feed import open_data_client
-
-    # TODO: load user defined symbol set locally for fast search?
-    await ctx.started({})
-
-    async with (
-        open_client_proxies() as (proxies, _),
-        open_data_client() as data_proxy,
-    ):
-        async with ctx.open_stream() as stream:
-
-            # select a non-history client for symbol search to lighten
-            # the load in the main data node.
-            proxy = data_proxy
-            for name, proxy in proxies.items():
-                if proxy is data_proxy:
-                    continue
-                break
-
-            ib_client = proxy._aio_ns.ib
-            log.info(
-                f'Using API client for symbol-search\n'
-                f'{ib_client}\n'
-            )
-
-            last = time.time()
-            async for pattern in stream:
-                log.info(f'received {pattern}')
-                now: float = time.time()
-
-                # this causes tractor hang...
-                # assert 0
-
-                assert pattern, 'IB can not accept blank search pattern'
-
-                # throttle search requests to no faster then 1Hz
-                diff = now - last
-                if diff < 1.0:
-                    log.debug('throttle sleeping')
-                    await trio.sleep(diff)
-                    try:
-                        pattern = stream.receive_nowait()
-                    except trio.WouldBlock:
-                        pass
-
-                if (
-                    not pattern
-                    or pattern.isspace()
-
-                    # XXX: not sure if this is a bad assumption but it
-                    # seems to make search snappier?
-                    or len(pattern) < 1
-                ):
-                    log.warning('empty pattern received, skipping..')
-
-                    # TODO: *BUG* if nothing is returned here the client
-                    # side will cache a null set result and not showing
-                    # anything to the use on re-searches when this query
-                    # timed out. We probably need a special "timeout" msg
-                    # or something...
-
-                    # XXX: this unblocks the far end search task which may
-                    # hold up a multi-search nursery block
-                    await stream.send({})
-
-                    continue
-
-                log.info(f'searching for {pattern}')
-
-                last = time.time()
-
-                # async batch search using api stocks endpoint and module
-                # defined adhoc symbol set.
-                stock_results = []
-
-                async def extend_results(
-                    target: Awaitable[list]
-                ) -> None:
-                    try:
-                        results = await target
-                    except tractor.trionics.Lagged:
-                        print("IB SYM-SEARCH OVERRUN?!?")
-                        return
-
-                    stock_results.extend(results)
-
-                for _ in range(10):
-                    with trio.move_on_after(3) as cs:
-                        async with trio.open_nursery() as sn:
-                            sn.start_soon(
-                                extend_results,
-                                proxy.search_symbols(
-                                    pattern=pattern,
-                                    upto=5,
-                                ),
-                            )
-
-                            # trigger async request
-                            await trio.sleep(0)
-
-                    if cs.cancelled_caught:
-                        log.warning(
-                            f'Search timeout? {proxy._aio_ns.ib.client}'
-                        )
-                        continue
-                    elif stock_results:
-                        break
-                    # else:
-                    # await tractor.pause()
-
-                    # # match against our ad-hoc set immediately
-                    # adhoc_matches = fuzzy.extract(
-                    #     pattern,
-                    #     list(_adhoc_futes_set),
-                    #     score_cutoff=90,
-                    # )
-                    # log.info(f'fuzzy matched adhocs: {adhoc_matches}')
-                    # adhoc_match_results = {}
-                    # if adhoc_matches:
-                    #     # TODO: do we need to pull contract details?
-                    #     adhoc_match_results = {i[0]: {} for i in
-                    #     adhoc_matches}
-
-                log.debug(f'fuzzy matching stocks {stock_results}')
-                stock_matches = fuzzy.extract(
-                    pattern,
-                    stock_results,
-                    score_cutoff=50,
-                )
-
-                # matches = adhoc_match_results | {
-                matches = {
-                    item[0]: {} for item in stock_matches
-                }
-                # TODO: we used to deliver contract details
-                # {item[2]: item[0] for item in stock_matches}
-
-                log.debug(f"sending matches: {matches.keys()}")
-                await stream.send(matches)
-
-
-# re-mapping to piker asset type names
-# https://github.com/erdewit/ib_insync/blob/master/ib_insync/contract.py#L113
-_asset_type_map = {
-    'STK': 'stock',
-    'OPT': 'option',
-    'FUT': 'future',
-    'CONTFUT': 'continuous_future',
-    'CASH': 'fiat',
-    'IND': 'index',
-    'CFD': 'cfd',
-    'BOND': 'bond',
-    'CMDTY': 'commodity',
-    'FOP': 'futures_option',
-    'FUND': 'mutual_fund',
-    'WAR': 'warrant',
-    'IOPT': 'warran',
-    'BAG': 'bag',
-    'CRYPTO': 'crypto',  # bc it's diff then fiat?
-    # 'NEWS': 'news',
-}
-
-
-def parse_patt2fqme(
-    # client: Client,
-    pattern: str,
-
-) -> tuple[str, str, str, str]:
-
-    # TODO: we can't use this currently because
-    # ``wrapper.starTicker()`` currently cashes ticker instances
-    # which means getting a singel quote will potentially look up
-    # a quote for a ticker that it already streaming and thus run
-    # into state clobbering (eg. list: Ticker.ticks). It probably
-    # makes sense to try this once we get the pub-sub working on
-    # individual symbols...
-
-    # XXX UPDATE: we can probably do the tick/trades scraping
-    # inside our eventkit handler instead to bypass this entirely?
-
-    currency = ''
-
-    # fqme parsing stage
-    # ------------------
-    if '.ib' in pattern:
-        _, symbol, venue, expiry = unpack_fqme(pattern)
-
-    else:
-        symbol = pattern
-        expiry = ''
-
-        # # another hack for forex pairs lul.
-        # if (
-        #     '.idealpro' in symbol
-        #     # or '/' in symbol
-        # ):
-        #     exch: str = 'IDEALPRO'
-        #     symbol = symbol.removesuffix('.idealpro')
-        #     if '/' in symbol:
-        #         symbol, currency = symbol.split('/')
-
-        # else:
-        # TODO: yes, a cache..
-        # try:
-        #     # give the cache a go
-        #     return client._contracts[symbol]
-        # except KeyError:
-        #     log.debug(f'Looking up contract for {symbol}')
-        expiry: str = ''
-        if symbol.count('.') > 1:
-            symbol, _, expiry = symbol.rpartition('.')
-
-        # use heuristics to figure out contract "type"
-        symbol, venue = symbol.upper().rsplit('.', maxsplit=1)
-
-    return symbol, currency, venue, expiry
-
-
-def con2fqme(
-    con: ibis.Contract,
-    _cache: dict[int, (str, bool)] = {}
-
-) -> tuple[str, bool]:
-    '''
-    Convert contracts to fqme-style strings to be used both in
-    symbol-search matching and as feed tokens passed to the front
-    end data deed layer.
-
-    Previously seen contracts are cached by id.
-
-    '''
-    # should be real volume for this contract by default
-    calc_price: bool = False
-    if con.conId:
-        try:
-            # TODO: LOL so apparently IB just changes the contract
-            # ID (int) on a whim.. so we probably need to use an
-            # FQME style key after all...
-            return _cache[con.conId]
-        except KeyError:
-            pass
-
-    suffix: str = con.primaryExchange or con.exchange
-    symbol: str = con.symbol
-    expiry: str = con.lastTradeDateOrContractMonth or ''
-
-    match con:
-        case ibis.Option():
-            # TODO: option symbol parsing and sane display:
-            symbol = con.localSymbol.replace(' ', '')
-
-        case (
-            ibis.Commodity()
-            # search API endpoint returns std con box..
-            | ibis.Contract(secType='CMDTY')
-        ):
-            # commodities and forex don't have an exchange name and
-            # no real volume so we have to calculate the price
-            suffix = con.secType
-
-            # no real volume on this tract
-            calc_price = True
-
-        case ibis.Forex() | ibis.Contract(secType='CASH'):
-            dst, src = con.localSymbol.split('.')
-            symbol = ''.join([dst, src])
-            suffix = con.exchange or 'idealpro'
-
-            # no real volume on forex feeds..
-            calc_price = True
-
-    if not suffix:
-        entry = _adhoc_symbol_map.get(
-            con.symbol or con.localSymbol
-        )
-        if entry:
-            meta, kwargs = entry
-            cid = meta.get('conId')
-            if cid:
-                assert con.conId == meta['conId']
-            suffix = meta['exchange']
-
-    # append a `.<suffix>` to the returned symbol
-    # key for derivatives that normally is the expiry
-    # date key.
-    if expiry:
-        suffix += f'.{expiry}'
-
-    fqme_key = symbol.lower()
-    if suffix:
-        fqme_key = '.'.join((fqme_key, suffix)).lower()
-
-    _cache[con.conId] = fqme_key, calc_price
-    return fqme_key, calc_price
-
-
-@async_lifo_cache()
-async def get_mkt_info(
-    fqme: str,
-
-    proxy: MethodProxy | None = None,
-
-) -> tuple[MktPair, ibis.ContractDetails]:
-
-    if '.ib' not in fqme:
-        fqme += '.ib'
-    broker, pair, venue, expiry = unpack_fqme(fqme)
-
-    proxy: MethodProxy
-    if proxy is not None:
-        client_ctx = nullcontext(proxy)
-    else:
-        from .feed import (
-            open_data_client,
-        )
-        client_ctx = open_data_client
-
-    async with client_ctx as proxy:
-        try:
-            (
-                con,  # Contract
-                details,  # ContractDetails
-            ) = await proxy.get_sym_details(fqme=fqme)
-        except ConnectionError:
-            log.exception(f'Proxy is ded {proxy._aio_ns}')
-            raise
-
-    # TODO: more consistent field translation
-    atype = _asset_type_map[con.secType]
-
-    if atype == 'commodity':
-        venue: str = 'cmdty'
-    else:
-        venue = con.primaryExchange or con.exchange
-
-    price_tick: Decimal = Decimal(str(details.minTick))
-    ib_min_tick_gt_2: Decimal = Decimal('0.01')
-    if (
-        price_tick < ib_min_tick_gt_2
-    ):
-        # TODO: we need to add some kinda dynamic rounding sys
-        # to our MktPair i guess?
-        # not sure where the logic should sit, but likely inside
-        # the `.clearing._ems` i suppose...
-        log.warning(
-            'IB seems to disallow a min price tick < 0.01 '
-            'when the price is > 2.0..?\n'
-            f'Decreasing min tick precision for {fqme} to 0.01'
-        )
-        # price_tick = ib_min_tick
-        # await tractor.pause()
-
-    if atype == 'stock':
-        # XXX: GRRRR they don't support fractional share sizes for
-        # stocks from the API?!
-        # if con.secType == 'STK':
-        size_tick = Decimal('1')
-    else:
-        size_tick: Decimal = Decimal(
-            str(details.minSize).rstrip('0')
-        )
-        # |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
-
-    # NOTE: this is duplicate from the .broker.norm_trade_records()
-    # routine, we should factor all this parsing somewhere..
-    expiry_str = str(con.lastTradeDateOrContractMonth)
-    # if expiry:
-    #     expiry_str: str = str(pendulum.parse(
-    #         str(expiry).strip(' ')
-    #     ))
-
-    # TODO: currently we can't pass the fiat src asset because
-    # then we'll get a `MNQUSD` request for history data..
-    # we need to figure out how we're going to handle this (later?)
-    # but likely we want all backends to eventually handle
-    # ``dst/src.venue.`` style !?
-    src = Asset(
-        name=str(con.currency).lower(),
-        atype='fiat',
-        tx_tick=Decimal('0.01'),  # right?
-    )
-    dst = Asset(
-        name=con.symbol.lower(),
-        atype=atype,
-        tx_tick=size_tick,
-    )
-
-    mkt = MktPair(
-        src=src,
-        dst=dst,
-
-        price_tick=price_tick,
-        size_tick=size_tick,
-
-        bs_mktid=str(con.conId),
-        venue=str(venue),
-        expiry=expiry_str,
-        broker='ib',
-
-        # TODO: options contract info as str?
-        # contract_info=<optionsdetails>
-        _fqme_without_src=(atype != 'fiat'),
-    )
-
-    # just.. wow.
-    if entry := _adhoc_mkt_infos.get(mkt.bs_fqme):
-        log.warning(f'Frickin {mkt.fqme} has an adhoc {entry}..')
-        new = mkt.to_dict()
-        new['price_tick'] = entry['price_tick']
-        new['src'] = src
-        new['dst'] = dst
-        mkt = MktPair(**new)
-
-    # if possible register the bs_mktid to the just-built
-    # mkt so that it can be retreived by order mode tasks later.
-    # TODO NOTE: this is going to be problematic if/when we split
-    # out the datatd vs. brokerd actors since the mktmap lookup
-    # table will now be inaccessible..
-    if proxy is not None:
-        client: Client = proxy._aio_ns
-        client._contracts[mkt.bs_fqme] = con
-        client._cons2mkts[con] = mkt
-
-    return mkt, details

piker/brokers/kraken/__init__.py

@@ -19,57 +19,44 @@ Kraken backend.
 
 Sub-modules within break into the core functionalities:
 
-- .api: for the core API machinery which generally
-        a ``asks``/``trio-websocket`` implemented ``Client``.
-- .broker: part for orders / trading endpoints.
-- .feed: for real-time and historical data query endpoints.
-- .ledger: for transaction processing as it pertains to accounting.
-- .symbols: for market (name) search and symbology meta-defs.
+- ``broker.py`` part for orders / trading endpoints
+- ``feed.py`` for real-time data feed endpoints
+- ``api.py`` for the core API machinery which is ``trio``-ized
+  wrapping around ``ib_insync``.
 
 '''
-from .symbols import (
-    Pair,  # for symcache
-    open_symbol_search,
-    # required by `.accounting`, `.data`
-    get_mkt_info,
-)
-# required by `.brokers`
+
+from piker.log import get_logger
+
+log = get_logger(__name__)
+
 from .api import (
     get_client,
 )
 from .feed import (
-    # required by `.data`
-    stream_quotes,
+    get_mkt_info,
     open_history_client,
+    open_symbol_search,
+    stream_quotes,
 )
 from .broker import (
-    # required by `.clearing`
-    open_trade_dialog,
-)
-from .ledger import (
-    # required by `.accounting`
-    norm_trade,
+    trades_dialogue,
     norm_trade_records,
 )
 
-
 __all__ = [
     'get_client',
-    'get_mkt_info',
-    'Pair',
-    'open_trade_dialog',
+    'trades_dialogue',
     'open_history_client',
     'open_symbol_search',
     'stream_quotes',
     'norm_trade_records',
-    'norm_trade',
 ]
 
 
 # tractor RPC enable arg
 __enable_modules__: list[str] = [
     'api',
-    'broker',
     'feed',
-    'symbols',
+    'broker',
 ]

piker/brokers/kraken/api.py

@@ -15,11 +15,12 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-Core (web) API client
+Kraken web API wrapping.
 
 '''
 from contextlib import asynccontextmanager as acm
 from datetime import datetime
+from decimal import Decimal
 import itertools
 from typing import (
     Any,
@@ -27,8 +28,10 @@ from typing import (
 )
 import time
 
-import httpx
+from bidict import bidict
 import pendulum
+import asks
+from fuzzywuzzy import process as fuzzy
 import numpy as np
 import urllib.parse
 import hashlib
@@ -37,14 +40,10 @@ import base64
 import trio
 
 from piker import config
-from piker.data import (
-    def_iohlcv_fields,
-    match_from_pairs,
-)
+from piker.data.types import Struct
 from piker.accounting._mktinfo import (
     Asset,
     digits_to_dec,
-    dec_digits,
 )
 from piker.brokers._util import (
     resproc,
@@ -53,21 +52,29 @@ from piker.brokers._util import (
     DataThrottle,
 )
 from piker.accounting import Transaction
-from piker.log import get_logger
-from .symbols import Pair
-
-log = get_logger('piker.brokers.kraken')
+from . import log
 
 # <uri>/<version>/
 _url = 'https://api.kraken.com/0'
 
-_headers: dict[str, str] = {
-    'User-Agent': 'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
-}
 
-# TODO: this is the only backend providing this right?
-# in which case we should drop it from the defaults and
-# instead make a custom fields descr in this module!
+# Broker specific ohlc schema which includes a vwap field
+_ohlc_dtype = [
+    ('index', int),
+    ('time', int),
+    ('open', float),
+    ('high', float),
+    ('low', float),
+    ('close', float),
+    ('volume', float),
+    ('count', int),
+    ('bar_wap', float),
+]
+
+# UI components allow this to be declared such that additional
+# (historical) fields can be exposed.
+ohlc_dtype = np.dtype(_ohlc_dtype)
+
 _show_wap_in_history = True
 _symbol_info_translation: dict[str, str] = {
     'tick_decimals': 'pair_decimals',
@@ -75,18 +82,12 @@ _symbol_info_translation: dict[str, str] = {
 
 
 def get_config() -> dict[str, Any]:
-    '''
-    Load our section from `piker/brokers.toml`.
 
-    '''
-    conf, path = config.load(
-        conf_name='brokers',
-        touch_if_dne=True,
-    )
-    if (section := conf.get('kraken')) is None:
-        log.warning(
-            f'No config section found for kraken in {path}'
-        )
+    conf, path = config.load()
+    section = conf.get('kraken')
+
+    if section is None:
+        log.warning(f'No config section found for kraken in {path}')
         return {}
 
     return section
@@ -117,51 +118,92 @@ class InvalidKey(ValueError):
     '''
 
 
+# https://www.kraken.com/features/api#get-tradable-pairs
+class Pair(Struct):
+    altname: str  # alternate pair name
+    wsname: str  # WebSocket pair name (if available)
+    aclass_base: str  # asset class of base component
+    base: str  # asset id of base component
+    aclass_quote: str  # asset class of quote component
+    quote: str  # asset id of quote component
+    lot: str  # volume lot size
+
+    cost_decimals: int
+    costmin: float
+    pair_decimals: int  # scaling decimal places for pair
+    lot_decimals: int  # scaling decimal places for volume
+
+    # amount to multiply lot volume by to get currency volume
+    lot_multiplier: float
+
+    # array of leverage amounts available when buying
+    leverage_buy: list[int]
+    # array of leverage amounts available when selling
+    leverage_sell: list[int]
+
+    # fee schedule array in [volume, percent fee] tuples
+    fees: list[tuple[int, float]]
+
+    # maker fee schedule array in [volume, percent fee] tuples (if on
+    # maker/taker)
+    fees_maker: list[tuple[int, float]]
+
+    fee_volume_currency: str  # volume discount currency
+    margin_call: str  # margin call level
+    margin_stop: str  # stop-out/liquidation margin level
+    ordermin: float  # minimum order volume for pair
+    tick_size: float  # min price step size
+    status: str
+
+    short_position_limit: float = 0
+    long_position_limit: float = float('inf')
+
+    @property
+    def price_tick(self) -> Decimal:
+        return digits_to_dec(self.pair_decimals)
+
+    @property
+    def size_tick(self) -> Decimal:
+        return digits_to_dec(self.lot_decimals)
+
+
 class Client:
 
-    # assets and mkt pairs are key-ed by kraken's ReST response
-    # symbol-bs_mktids (we call them "X-keys" like fricking
-    # "XXMRZEUR"). these keys used directly since ledger endpoints
-    # return transaction sets keyed with the same set!
-    _Assets: dict[str, Asset] = {}
-    _AssetPairs: dict[str, Pair] = {}
+    # symbol mapping from all names to the altname
+    _ntable: dict[str, str] = {}
 
-    # offer lookup tables for all .altname and .wsname
-    # to the equivalent .xname so that various symbol-schemas
-    # can be mapped to `Pair`s in the tables above.
-    _altnames: dict[str, str] = {}
-    _wsnames: dict[str, str] = {}
+    # 2-way map of symbol names to their "alt names" ffs XD
+    _altnames: bidict[str, str] = bidict()
 
-    # key-ed by `Pair.bs_fqme: str`, and thus used for search
-    # allowing for lookup using piker's own FQME symbology sys.
     _pairs: dict[str, Pair] = {}
-    _assets: dict[str, Asset] = {}
 
     def __init__(
         self,
         config: dict[str, str],
-        httpx_client: httpx.AsyncClient,
-
         name: str = '',
         api_key: str = '',
         secret: str = ''
     ) -> None:
-
-        self._sesh: httpx.AsyncClient = httpx_client
-
+        self._sesh = asks.Session(connections=4)
+        self._sesh.base_location = _url
+        self._sesh.headers.update({
+            'User-Agent':
+                'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
+        })
         self._name = name
         self._api_key = api_key
         self._secret = secret
 
         self.conf: dict[str, str] = config
+        self.assets: dict[str, Asset] = {}
 
     @property
     def pairs(self) -> dict[str, Pair]:
-
         if self._pairs is None:
             raise RuntimeError(
-                "Client didn't run `.get_mkt_pairs()` on startup?!"
+                "Make sure to run `cache_symbols()` on startup!"
             )
+            # retreive and cache all symbols
 
         return self._pairs
 
@@ -170,9 +212,10 @@ class Client:
         method: str,
         data: dict,
     ) -> dict[str, Any]:
-        resp: httpx.Response = await self._sesh.post(
-            url=f'/public/{method}',
+        resp = await self._sesh.post(
+            path=f'/public/{method}',
             json=data,
+            timeout=float('inf')
         )
         return resproc(resp, log)
 
@@ -183,18 +226,18 @@ class Client:
         uri_path: str
     ) -> dict[str, Any]:
         headers = {
-            'Content-Type': 'application/x-www-form-urlencoded',
-            'API-Key': self._api_key,
-            'API-Sign': get_kraken_signature(
-                uri_path,
-                data,
-                self._secret,
-            ),
+            'Content-Type':
+                'application/x-www-form-urlencoded',
+            'API-Key':
+                self._api_key,
+            'API-Sign':
+                get_kraken_signature(uri_path, data, self._secret)
         }
-        resp: httpx.Response = await self._sesh.post(
-            url=f'/private/{method}',
+        resp = await self._sesh.post(
+            path=f'/private/{method}',
             data=data,
             headers=headers,
+            timeout=float('inf')
         )
         return resproc(resp, log)
 
@@ -220,29 +263,17 @@ class Client:
             'Balance',
             {},
         )
-        by_bsmktid: dict[str, dict] = resp['result']
+        by_bsmktid = resp['result']
 
-        balances: dict = {}
-        for xname, bal in by_bsmktid.items():
-            asset: Asset = self._Assets[xname]
+        # TODO: we need to pull out the "asset" decimals
+        # data and return a `decimal.Decimal` instead here!
+        # using the underlying Asset
+        return {
+            self._altnames[sym].lower(): float(bal)
+            for sym, bal in by_bsmktid.items()
+        }
 
-            # TODO: which KEY should we use? it's used to index
-            # the `Account.pps: dict` ..
-            key: str = asset.name.lower()
-            # TODO: should we just return a `Decimal` here
-            # or is the rounded version ok?
-            balances[key] = round(
-                float(bal),
-                ndigits=dec_digits(asset.tx_tick)
-            )
-
-        return balances
-
-    async def get_assets(
-        self,
-        reload: bool = False,
-
-    ) -> dict[str, Asset]:
+    async def get_assets(self) -> dict[str, Asset]:
         '''
         Load and cache all asset infos and pack into
         our native ``Asset`` struct.
@@ -260,37 +291,21 @@ class Client:
             }
 
         '''
-        if (
-            not self._assets
-            or reload
-        ):
-            resp = await self._public('Assets', {})
-            assets: dict[str, dict] = resp['result']
+        resp = await self._public('Assets', {})
+        assets = resp['result']
 
-            for bs_mktid, info in assets.items():
+        for bs_mktid, info in assets.items():
+            altname = self._altnames[bs_mktid] = info['altname']
+            aclass: str = info['aclass']
 
-                altname: str = info['altname']
-                aclass: str = info['aclass']
-                asset = Asset(
-                    name=altname,
-                    atype=f'crypto_{aclass}',
-                    tx_tick=digits_to_dec(info['decimals']),
-                    info=info,
-                )
-                # NOTE: yes we keep 2 sets since kraken insists on
-                # keeping 3 frickin sets bc apparently they have
-                # no sane data engineers whol all like different
-                # keys for their fricking symbology sets..
-                self._Assets[bs_mktid] = asset
-                self._assets[altname.lower()] = asset
-                self._assets[altname] = asset
+            self.assets[bs_mktid] = Asset(
+                name=altname.lower(),
+                atype=f'crypto_{aclass}',
+                tx_tick=digits_to_dec(info['decimals']),
+                info=info,
+            )
 
-        # we return the "most native" set merged with our preferred
-        # naming (which i guess is the "altname" one) since that's
-        # what the symcache loader will be storing, and we need the
-        # keys that are easiest to match against in any trade
-        # records.
-        return self._Assets | self._assets
+        return self.assets
 
     async def get_trades(
         self,
@@ -371,25 +386,23 @@ class Client:
         #     'amount': '0.00300726', 'fee': '0.00001000', 'time':
         #     1658347714, 'status': 'Success'}]}
 
-        if xfers:
-            import tractor
-            await tractor.pp()
-
         trans: dict[str, Transaction] = {}
         for entry in xfers:
+
             # look up the normalized name and asset info
-            asset_key: str = entry['asset']
-            asset: Asset = self._Assets[asset_key]
-            asset_key: str = asset.name.lower()
+            asset_key = entry['asset']
+            asset = self.assets[asset_key]
+            asset_key = self._altnames[asset_key].lower()
 
             # XXX: this is in the asset units (likely) so it isn't
             # quite the same as a commisions cost necessarily..)
-            # TODO: also round this based on `Pair` cost precision info?
             cost = float(entry['fee'])
-            # fqme: str = asset_key + '.kraken'
+
+            fqme = asset_key + '.kraken'
 
             tx = Transaction(
-                fqme=asset_key,  # this must map to an entry in .assets!
+                fqme=fqme,
+                sym=asset,
                 tid=entry['txid'],
                 dt=pendulum.from_timestamp(entry['time']),
                 bs_mktid=f'{asset_key}{src_asset}',
@@ -404,11 +417,6 @@ class Client:
 
                 # XXX: see note above
                 cost=cost,
-
-                # not a trade but a withdrawal or deposit on the
-                # asset (chain) system.
-                etype='transfer',
-
             )
             trans[tx.tid] = tx
 
@@ -459,7 +467,7 @@ class Client:
         # txid is a transaction id given by kraken
         return await self.endpoint('CancelOrder', {"txid": reqid})
 
-    async def asset_pairs(
+    async def pair_info(
         self,
         pair_patt: str | None = None,
 
@@ -471,76 +479,64 @@ class Client:
         https://docs.kraken.com/rest/#tag/Market-Data/operation/getTradableAssetPairs
 
         '''
-        if not self._AssetPairs:
-            # get all pairs by default, or filter
-            # to whatever pattern is provided as input.
-            req_pairs: dict[str, str] | None = None
-            if pair_patt is not None:
-                req_pairs = {'pair': pair_patt}
+        # get all pairs by default, or filter
+        # to whatever pattern is provided as input.
+        pairs: dict[str, str] | None = None
+        if pair_patt is not None:
+            pairs = {'pair': pair_patt}
 
-            resp = await self._public(
-                'AssetPairs',
-                req_pairs,
-            )
-            err = resp['error']
-            if err:
-                raise SymbolNotFound(pair_patt)
+        resp = await self._public(
+            'AssetPairs',
+            pairs,
+        )
+        err = resp['error']
+        if err:
+            raise SymbolNotFound(pair_patt)
 
-            # NOTE: we try to key pairs by our custom defined
-            # `.bs_fqme` field since we want to offer search over
-            # this pattern set, callers should fill out lookup
-            # tables for kraken's bs_mktid keys to map to these
-            # keys!
-            # XXX: FURTHER kraken's data eng team decided to offer
-            # 3 frickin market-pair-symbol key sets depending on
-            # which frickin API is being used.
-            # Example for the trading pair 'LTC<EUR'
-            # - the "X-key" from rest eps 'XLTCZEUR'
-            # - the "websocket key" from ws msgs is 'LTC/EUR'
-            # - the "altname key" also delivered in pair info is 'LTCEUR'
-            for xkey, data in resp['result'].items():
+        pairs: dict[str, Pair] = {
 
-                # NOTE: always cache in pairs tables for faster lookup
-                pair = Pair(xname=xkey, **data)
-
-                # register the above `Pair` structs for all
-                # key-sets/monikers: a set of 4 (frickin) tables
-                # acting as a combined surjection of all possible
-                # (and stupid) kraken names to their `Pair` obj.
-                self._AssetPairs[xkey] = pair
-                self._pairs[pair.bs_fqme] = pair
-                self._altnames[pair.altname] = pair
-                self._wsnames[pair.wsname] = pair
+            key: Pair(**data)
+            for key, data in resp['result'].items()
+        }
+        # always cache so we can possibly do faster lookup
+        self._pairs.update(pairs)
 
         if pair_patt is not None:
-            return next(iter(self._pairs.items()))[1]
+            return next(iter(pairs.items()))[1]
 
-        return self._AssetPairs
+        return pairs
 
-    async def get_mkt_pairs(
-        self,
-        reload: bool = False,
-    ) -> dict:
+    async def cache_symbols(self) -> dict:
         '''
-        Load all market pair info build and cache it for downstream
-        use.
+        Load all market pair info build and cache it for downstream use.
 
-        Multiple pair info lookup tables (like ``._altnames:
-        dict[str, str]``) are created for looking up the
-        piker-native `Pair`-struct from any input of the three
-        (yes, it's that idiotic..) available symbol/pair-key-sets
-        that kraken frickin offers depending on the API including
-        the .altname, .wsname and the weird ass default set they
-        return in ReST responses .xname..
+        A ``._ntable: dict[str, str]`` is available for mapping the
+        websocket pair name-keys and their http endpoint API (smh)
+        equivalents to the "alternative name" which is generally the one
+        we actually want to use XD
 
         '''
-        if (
-            not self._pairs
-            or reload
-        ):
-            await self.asset_pairs()
+        if not self._pairs:
+            pairs = await self.pair_info()
+            assert self._pairs == pairs
 
-        return self._AssetPairs
+            # table of all ws and rest keys to their alt-name values.
+            ntable: dict[str, str] = {}
+
+            for rest_key in list(pairs.keys()):
+
+                pair: Pair = pairs[rest_key]
+                altname = pair.altname
+                wsname = pair.wsname
+                ntable[altname] = ntable[rest_key] = ntable[wsname] = altname
+
+                # register the pair under all monikers, a giant flat
+                # surjection of all possible names to each info obj.
+                self._pairs[altname] = self._pairs[wsname] = pair
+
+            self._ntable.update(ntable)
+
+        return self._pairs
 
     async def search_symbols(
         self,
@@ -556,20 +552,16 @@ class Client:
 
         '''
         if not len(self._pairs):
-            await self.get_mkt_pairs()
-            assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
+            await self.cache_symbols()
+            assert self._pairs, '`Client.cache_symbols()` was never called!?'
 
-        matches: dict[str, Pair] = match_from_pairs(
-            pairs=self._pairs,
-            query=pattern.upper(),
+        matches = fuzzy.extractBests(
+            pattern,
+            self._pairs,
             score_cutoff=50,
         )
-
-        # repack in .altname-keyed output table
-        return {
-            pair.altname: pair
-            for pair in matches.values()
-        }
+        # repack in dict form
+        return {item[0].altname: item[0] for item in matches}
 
     async def bars(
         self,
@@ -630,11 +622,11 @@ class Client:
                 new_bars.append(
                     (i,) + tuple(
                         ftype(bar[j]) for j, (name, ftype) in enumerate(
-                            def_iohlcv_fields[1:]
+                            _ohlc_dtype[1:]
                         )
                     )
                 )
-            array = np.array(new_bars, dtype=def_iohlcv_fields) if as_np else bars
+            array = np.array(new_bars, dtype=_ohlc_dtype) if as_np else bars
             return array
         except KeyError:
             errmsg = json['error'][0]
@@ -649,55 +641,37 @@ class Client:
                 raise BrokerError(errmsg)
 
     @classmethod
-    def to_bs_fqme(
+    def normalize_symbol(
         cls,
-        pair_str: str
-    ) -> str:
+        ticker: str
+    ) -> tuple[str, Pair]:
         '''
         Normalize symbol names to to a 3x3 pair from the global
         definition map which we build out from the data retreived from
         the 'AssetPairs' endpoint, see methods above.
 
         '''
-        try:
-            return cls._altnames[pair_str.upper()].bs_fqme
-        except KeyError as ke:
-            raise SymbolNotFound(f'kraken has no {ke.args[0]}')
+        return cls._ntable[ticker].lower()
 
 
 @acm
 async def get_client() -> Client:
 
-    conf: dict[str, Any] = get_config()
-    async with httpx.AsyncClient(
-        base_url=_url,
-        headers=_headers,
+    conf = get_config()
+    if conf:
+        client = Client(
+            conf,
+            name=conf['key_descr'],
+            api_key=conf['api_key'],
+            secret=conf['secret']
+        )
+    else:
+        client = Client({})
 
-        # TODO: is there a way to numerate this?
-        # https://www.python-httpx.org/advanced/clients/#why-use-a-client
-        # connections=4
-    ) as trio_client:
-        if conf:
-            client = Client(
-                conf,
-                httpx_client=trio_client,
+    # at startup, load all symbols, and asset info in
+    # batch requests.
+    async with trio.open_nursery() as nurse:
+        nurse.start_soon(client.get_assets)
+        await client.cache_symbols()
 
-                # TODO: don't break these up and just do internal
-                # conf lookups instead..
-                name=conf['key_descr'],
-                api_key=conf['api_key'],
-                secret=conf['secret']
-            )
-        else:
-            client = Client(
-                conf={},
-                httpx_client=trio_client,
-            )
-
-        # at startup, load all symbols, and asset info in
-        # batch requests.
-        async with trio.open_nursery() as nurse:
-            nurse.start_soon(client.get_assets)
-            await client.get_mkt_pairs()
-
-        yield client
+    yield client

piker/brokers/kraken/broker.py

@@ -18,12 +18,14 @@
 Order api and machinery
 
 '''
+from collections import ChainMap, defaultdict
 from contextlib import (
     asynccontextmanager as acm,
     aclosing,
 )
 from functools import partial
 from itertools import count
+import math
 from pprint import pformat
 import time
 from typing import (
@@ -34,19 +36,21 @@ from typing import (
 )
 
 from bidict import bidict
+import pendulum
 import trio
 import tractor
 
 from piker.accounting import (
     Position,
-    Account,
+    PpTable,
     Transaction,
     TransactionLedger,
     open_trade_ledger,
-    open_account,
+    open_pps,
+    get_likely_pair,
 )
-from piker.clearing import(
-    OrderDialogs,
+from piker.accounting._mktinfo import (
+    MktPair,
 )
 from piker.clearing._messages import (
     Order,
@@ -59,24 +63,18 @@ from piker.clearing._messages import (
     BrokerdPosition,
     BrokerdStatus,
 )
-from piker.brokers import (
-    open_cached_client,
-)
-from piker.data import open_symcache
+from . import log
 from .api import (
-    log,
     Client,
     BrokerError,
+    get_client,
 )
 from .feed import (
+    get_mkt_info,
     open_autorecon_ws,
     NoBsWs,
     stream_messages,
 )
-from .ledger import (
-    norm_trade_records,
-    verify_balances,
-)
 
 MsgUnion = Union[
     BrokerdCancel,
@@ -126,7 +124,7 @@ async def handle_order_requests(
     client: Client,
     ems_order_stream: tractor.MsgStream,
     token: str,
-    apiflows: OrderDialogs,
+    apiflows: dict[int, ChainMap[dict[str, dict]]],
     ids: bidict[str, int],
     reqids2txids: dict[int, str],
 
@@ -136,8 +134,10 @@ async def handle_order_requests(
     and deliver acks or errors.
 
     '''
-    # XXX: UGH, let's unify this.. with ``msgspec``!!!
-    msg: dict | Order
+    # XXX: UGH, let's unify this.. with ``msgspec``.
+    msg: dict[str, Any]
+    order: BrokerdOrder
+
     async for msg in ems_order_stream:
         log.info(f'Rx order msg:\n{pformat(msg)}')
         match msg:
@@ -183,12 +183,11 @@ async def handle_order_requests(
 
                 # logic from old `Client.submit_limit()`
                 if order.oid in ids:
-                    ep: str = 'editOrder'
-                    reqid: int = ids[order.oid]  # integer not txid
+                    ep = 'editOrder'
+                    reqid = ids[order.oid]  # integer not txid
                     try:
-                        txid: str = reqids2txids[reqid]
+                        txid = reqids2txids[reqid]
                     except KeyError:
-
                         # XXX: not sure if this block ever gets hit now?
                         log.error('TOO FAST EDIT')
                         reqids2txids[reqid] = TooFastEdit(reqid)
@@ -209,7 +208,7 @@ async def handle_order_requests(
                         }
 
                 else:
-                    ep: str = 'addOrder'
+                    ep = 'addOrder'
 
                     reqid = BrokerClient.new_reqid()
                     ids[order.oid] = reqid
@@ -222,12 +221,8 @@ async def handle_order_requests(
                         'type': order.action,
                     }
 
-                    # XXX strip any .<venue> token which should
-                    # ONLY ever be '.spot' rn, until we support
-                    # futes.
-                    bs_fqme: str = order.symbol.replace('.spot', '')
-                    psym: str = bs_fqme.upper()
-                    pair: str = f'{psym[:3]}/{psym[3:]}'
+                    psym = order.symbol.upper()
+                    pair = f'{psym[:3]}/{psym[3:]}'
 
                 # XXX: ACK the request **immediately** before sending
                 # the api side request to ensure the ems maps the oid ->
@@ -265,7 +260,7 @@ async def handle_order_requests(
                 await ws.send_msg(req)
 
                 # placehold for sanity checking in relay loop
-                apiflows.add_msg(reqid, msg)
+                apiflows[reqid].maps.append(msg)
 
             case _:
                 account = msg.get('account')
@@ -371,23 +366,24 @@ async def subscribe(
 
 
 def trades2pps(
-    acnt: Account,
-    ledger: TransactionLedger,
+    table: PpTable,
     acctid: str,
     new_trans: dict[str, Transaction] = {},
 
     write_storage: bool = True,
 
-) -> list[BrokerdPosition]:
+) -> tuple[
+    list[BrokerdPosition],
+    list[Transaction],
+]:
     if new_trans:
-        updated = acnt.update_from_ledger(
+        updated = table.update_from_trans(
             new_trans,
-            symcache=ledger.symcache,
         )
         log.info(f'Updated pps:\n{pformat(updated)}')
 
-    pp_entries, closed_pp_objs = acnt.dump_active()
-    pp_objs: dict[Union[str, int], Position] = acnt.pps
+    pp_entries, closed_pp_objs = table.dump_active()
+    pp_objs: dict[Union[str, int], Position] = table.pps
 
     pps: dict[int, Position]
     position_msgs: list[dict] = []
@@ -401,13 +397,13 @@ def trades2pps(
                 # backend suffix prefixed but when
                 # reading accounts from ledgers we
                 # don't need it and/or it's prefixed
-                # in the section acnt.. we should
+                # in the section table.. we should
                 # just strip this from the message
                 # right since `.broker` is already
                 # included?
                 account='kraken.' + acctid,
                 symbol=p.mkt.fqme,
-                size=p.cumsize,
+                size=p.size,
                 avg_price=p.ppu,
                 currency='',
             )
@@ -418,29 +414,30 @@ def trades2pps(
         # as little as possible. we need to either do
         # these writes in another actor, or try out `trio`'s
         # async file IO api?
-        acnt.write_config()
+        table.write_config()
 
     return position_msgs
 
 
 @tractor.context
-async def open_trade_dialog(
+async def trades_dialogue(
     ctx: tractor.Context,
+    loglevel: str = None,
 
 ) -> AsyncIterator[dict[str, Any]]:
 
-    async with (
-        # TODO: maybe bind these together and deliver
-        # a tuple from `.open_cached_client()`?
-        open_cached_client('kraken') as client,
-        open_symcache('kraken') as symcache,
-    ):
-        # make ems flip to paper mode when no creds setup in
-        # `brokers.toml` B0
+    async with get_client() as client:
+
         if not client._api_key:
             await ctx.started('paper')
             return
 
+        # TODO: make ems flip to paper mode via
+        # some returned signal if the user only wants to use
+        # the data feed or we return this?
+        # else:
+        #     await ctx.started(({}, ['paper']))
+
         # NOTE: currently we expect the user to define a "source fiat"
         # (much like the web UI let's you set an "account currency")
         # such that all positions (nested or flat) will be translated to
@@ -452,7 +449,10 @@ async def open_trade_dialog(
         acc_name = 'kraken.' + acctid
 
         # task local msg dialog tracking
-        apiflows = OrderDialogs()
+        apiflows: defaultdict[
+            int,
+            ChainMap[dict[str, dict]],
+        ] = defaultdict(ChainMap)
 
         # 2way map for ems ids to kraken int reqids..
         ids: bidict[str, int] = bidict()
@@ -464,8 +464,8 @@ async def open_trade_dialog(
         # - delete the *ABSOLUTE LAST* entry from account's corresponding
         #   trade ledgers file (NOTE this MUST be the last record
         #   delivered from the api ledger),
-        # - open you ``account.kraken.spot.toml`` and find that
-        #   same tid and delete it from the pos's clears table,
+        # - open you ``pps.toml`` and find that same tid and delete it
+        #   from the pp's clears table,
         # - set this flag to `True`
         #
         # You should see an update come in after the order mode
@@ -476,85 +476,172 @@ async def open_trade_dialog(
         # update things correctly.
         simulate_pp_update: bool = False
 
-        acnt: Account
+        table: PpTable
         ledger: TransactionLedger
         with (
-            open_account(
+            open_pps(
                 'kraken',
                 acctid,
                 write_on_exit=True,
-            ) as acnt,
+            ) as table,
 
             open_trade_ledger(
                 'kraken',
                 acctid,
-                symcache=symcache,
             ) as ledger,
         ):
-            # TODO: loading ledger entries should all be done
-            # within a newly implemented `async with open_account()
-            # as acnt` where `Account.ledger: TransactionLedger`
-            # can be used to explicitily update and write the
-            # offline TOML files!
-            # ------ - ------
-            # MOL the init sequence is:
-            # - get `Account` (with presumed pre-loaded ledger done
-            #   beind the scenes as part of ctx enter).
-            # - pull new trades from API, update the ledger with
-            #   normalized to `Transaction` entries of those
-            #   records, presumably (and implicitly) update the
-            #   acnt state including expiries, positions,
-            #   transfers..), and finally of course existing
-            #   per-asset balances.
-            # - validate all pos and balances ensuring there's
-            #   no seemingly noticeable discrepancies?
+            # transaction-ify the ledger entries
+            ledger_trans = await norm_trade_records(ledger)
 
-            # LOAD and transaction-ify the EXISTING LEDGER
-            ledger_trans: dict[str, Transaction] = await norm_trade_records(
-                ledger,
-                client,
-                api_name_set='xname',
-            )
-
-            if not acnt.pps:
-                acnt.update_from_ledger(
-                    ledger_trans,
-                    symcache=ledger.symcache,
-                )
-                acnt.write_config()
+            if not table.pps:
+                # NOTE: we can't use this since it first needs
+                # broker: str input support!
+                # table.update_from_trans(ledger.to_trans())
+                table.update_from_trans(ledger_trans)
+                table.write_config()
 
             # TODO: eventually probably only load
             # as far back as it seems is not deliverd in the
             # most recent 50 trades and assume that by ordering we
-            # already have those records in the ledger?
-            tids2trades: dict[str, dict] = await client.get_trades()
+            # already have those records in the ledger.
+            tids2trades = await client.get_trades()
             ledger.update(tids2trades)
             if tids2trades:
                 ledger.write_config()
 
-            api_trans: dict[str, Transaction] = await norm_trade_records(
-                tids2trades,
-                client,
-                api_name_set='xname',
-            )
+            api_trans = await norm_trade_records(tids2trades)
 
             # retrieve kraken reported balances
             # and do diff with ledger to determine
             # what amount of trades-transactions need
             # to be reloaded.
-            balances: dict[str, float] = await client.get_balances()
+            balances = await client.get_balances()
 
-            verify_balances(
-                acnt,
-                src_fiat,
-                balances,
-                client,
-                ledger,
-                ledger_trans,
-                api_trans,
-            )
+            for dst, size in balances.items():
 
-            # XXX NOTE: only for simulate-testing a "new fill" since
+                # we don't care about tracking positions
+                # in the user's source fiat currency.
+                if (
+                    dst == src_fiat
+                    or not any(
+                        dst in bs_mktid for bs_mktid in table.pps
+                    )
+                ):
+                    log.warning(
+                        f'Skipping balance `{dst}`:{size} for position calcs!'
+                    )
+                    continue
+
+                def has_pp(
+                    dst: str,
+                    size: float,
+
+                ) -> Position | None:
+
+                    src2dst: dict[str, str] = {}
+
+                    for bs_mktid in table.pps:
+                        likely_pair = get_likely_pair(
+                            src_fiat,
+                            dst,
+                            bs_mktid,
+                        )
+                        if likely_pair:
+                            src2dst[src_fiat] = dst
+
+                    for src, dst in src2dst.items():
+                        pair = f'{dst}{src_fiat}'
+                        pp = table.pps.get(pair)
+                        if (
+                            pp
+                            and math.isclose(pp.size, size)
+                        ):
+                            return pp
+
+                        elif (
+                            size == 0
+                            and pp.size
+                        ):
+                            log.warning(
+                                f'`kraken` account says you have  a ZERO '
+                                f'balance for {bs_mktid}:{pair}\n'
+                                f'but piker seems to think `{pp.size}`\n'
+                                'This is likely a discrepancy in piker '
+                                'accounting if the above number is'
+                                "large,' though it's likely to due lack"
+                                "f tracking xfers fees.."
+                            )
+                            return pp
+
+                    return None  # signal no entry
+
+                pos = has_pp(dst, size)
+                if not pos:
+
+                    # we have a balance for which there is no pp
+                    # entry? so we have to likely update from the
+                    # ledger.
+                    updated = table.update_from_trans(ledger_trans)
+                    log.info(f'Updated pps from ledger:\n{pformat(updated)}')
+                    pos = has_pp(dst, size)
+
+                    if (
+                        not pos
+                        and not simulate_pp_update
+                    ):
+                        # try reloading from API
+                        table.update_from_trans(api_trans)
+                        pos = has_pp(dst, size)
+                        if not pos:
+
+                            # get transfers to make sense of abs balances.
+                            # NOTE: we do this after ledger and API
+                            # loading since we might not have an entry
+                            # in the ``pps.toml`` for the necessary pair
+                            # yet and thus this likely pair grabber will
+                            # likely fail.
+                            for bs_mktid in table.pps:
+                                likely_pair = get_likely_pair(
+                                    src_fiat,
+                                    dst,
+                                    bs_mktid,
+                                )
+                                if likely_pair:
+                                    break
+                            else:
+                                raise ValueError(
+                                    'Could not find a position pair in '
+                                    'ledger for likely widthdrawal '
+                                    f'candidate: {dst}'
+                                )
+
+                            if likely_pair:
+                                # this was likely pp that had a withdrawal
+                                # from the dst asset out of the account.
+
+                                xfer_trans = await client.get_xfers(
+                                    dst,
+                                    # TODO: not all src assets are
+                                    # 3 chars long...
+                                    src_asset=likely_pair[3:],
+                                )
+                                if xfer_trans:
+                                    updated = table.update_from_trans(
+                                        xfer_trans,
+                                        cost_scalar=1,
+                                    )
+                                    log.info(
+                                        f'Updated {dst} from transfers:\n'
+                                        f'{pformat(updated)}'
+                                    )
+
+                        if has_pp(dst, size):
+                            raise ValueError(
+                                'Could not reproduce balance:\n'
+                                f'dst: {dst}, {size}\n'
+                            )
+
+            # only for simulate-testing a "new fill" since
             # otherwise we have to actually conduct a live clear.
             if simulate_pp_update:
                 tid = list(tids2trades)[0]
@@ -562,28 +649,26 @@ async def open_trade_dialog(
                 # stage a first reqid of `0`
                 reqids2txids[0] = last_trade_dict['ordertxid']
 
-            ppmsgs: list[BrokerdPosition] = trades2pps(
-                acnt,
-                ledger,
+            ppmsgs = trades2pps(
+                table,
                 acctid,
             )
-            # sync with EMS delivering pps and accounts
             await ctx.started((ppmsgs, [acc_name]))
 
             # TODO: ideally this blocks the this task
             # as little as possible. we need to either do
             # these writes in another actor, or try out `trio`'s
             # async file IO api?
-            acnt.write_config()
+            table.write_config()
 
             # Get websocket token for authenticated data stream
             # Assert that a token was actually received.
             resp = await client.endpoint('GetWebSocketsToken', {})
-            if err := resp.get('error'):
+            err = resp.get('error')
+            if err:
                 raise BrokerError(err)
 
-            # resp token for ws init
-            token: str = resp['result']['token']
+            token = resp['result']['token']
 
             ws: NoBsWs
             async with (
@@ -612,35 +697,32 @@ async def open_trade_dialog(
 
                 # enter relay loop
                 await handle_order_updates(
-                    client=client,
-                    ws=ws,
-                    ws_stream=stream,
-                    ems_stream=ems_stream,
-                    apiflows=apiflows,
-                    ids=ids,
-                    reqids2txids=reqids2txids,
-                    acnt=acnt,
-                    ledger=ledger,
-                    acctid=acctid,
-                    acc_name=acc_name,
-                    token=token,
+                    ws,
+                    stream,
+                    ems_stream,
+                    apiflows,
+                    ids,
+                    reqids2txids,
+                    table,
+                    api_trans,
+                    acctid,
+                    acc_name,
+                    token,
                 )
 
 
 async def handle_order_updates(
-    client: Client,  # only for pairs table needed in ledger proc
     ws: NoBsWs,
     ws_stream: AsyncIterator,
     ems_stream: tractor.MsgStream,
-    apiflows: OrderDialogs,
+    apiflows: dict[int, ChainMap[dict[str, dict]]],
     ids: bidict[str, int],
     reqids2txids: bidict[int, str],
-    acnt: Account,
+    table: PpTable,
 
     # transaction records which will be updated
     # on new trade clearing events (aka order "fills")
-    ledger: TransactionLedger,
-    # ledger_trans: dict[str, Transaction],
+    ledger_trans: dict[str, Transaction],
     acctid: str,
     acc_name: str,
     token: str,
@@ -658,7 +740,7 @@ async def handle_order_updates(
 
             # TODO: turns out you get the fill events from the
             # `openOrders` before you get this, so it might be better
-            # to do all fill/status/pos updates in that sub and just use
+            # to do all fill/status/pp updates in that sub and just use
             # this one for ledger syncs?
 
             # For eg. we could take the "last 50 trades" and do a diff
@@ -700,8 +782,7 @@ async def handle_order_updates(
                     # if tid not in ledger_trans
                 }
                 for tid, trade in trades.items():
-                    # assert tid not in ledger_trans
-                    assert tid not in ledger
+                    assert tid not in ledger_trans
                     txid = trade['ordertxid']
                     reqid = trade.get('userref')
 
@@ -744,22 +825,12 @@ async def handle_order_updates(
                     )
                     await ems_stream.send(status_msg)
 
-                new_trans = await norm_trade_records(
-                    trades,
-                    client,
-                    api_name_set='wsname',
+                new_trans = await norm_trade_records(trades)
+                ppmsgs = trades2pps(
+                    table,
+                    acctid,
+                    new_trans,
                 )
-                ppmsgs: list[BrokerdPosition] = trades2pps(
-                    acnt=acnt,
-                    ledger=ledger,
-                    acctid=acctid,
-                    new_trans=new_trans,
-                )
-                # ppmsgs = trades2pps(
-                #     acnt,
-                #     acctid,
-                #     new_trans,
-                # )
                 for pp_msg in ppmsgs:
                     await ems_stream.send(pp_msg)
 
@@ -804,9 +875,8 @@ async def handle_order_updates(
                     # 'vol_exec': exec_vlm}  # 0.0000
                     match update_msg:
 
-                        # EMS-unknown pre-exising-submitted LIVE
-                        # order that needs to be delivered and
-                        # loaded on the client-side.
+                        # EMS-unknown LIVE order that needs to be
+                        # delivered and loaded on the client-side.
                         case {
                             'userref': reqid,
                             'descr': {
@@ -825,7 +895,7 @@ async def handle_order_updates(
                             ids.inverse.get(reqid) is None
                         ):
                             # parse out existing live order
-                            fqme = pair.replace('/', '').lower() + '.spot'
+                            fqme = pair.replace('/', '').lower()
                             price = float(price)
                             size = float(vol)
 
@@ -859,7 +929,7 @@ async def handle_order_updates(
                                 ),
                                 src='kraken',
                             )
-                            apiflows.add_msg(reqid, status_msg.to_dict())
+                            apiflows[reqid].maps.append(status_msg.to_dict())
                             await ems_stream.send(status_msg)
                             continue
 
@@ -995,7 +1065,7 @@ async def handle_order_updates(
                                 ),
                             )
 
-                            apiflows.add_msg(reqid, update_msg)
+                            apiflows[reqid].maps.append(update_msg)
                             await ems_stream.send(resp)
 
                         # fill msg.
@@ -1074,8 +1144,9 @@ async def handle_order_updates(
                     )
                     continue
 
-                # update the msg history
-                apiflows.add_msg(reqid, event)
+                # update the msg chain
+                chain = apiflows[reqid]
+                chain.maps.append(event)
 
                 if status == 'error':
                     # any of ``{'add', 'edit', 'cancel'}``
@@ -1085,16 +1156,11 @@ async def handle_order_updates(
                         f'Failed to {action} order {reqid}:\n'
                         f'{errmsg}'
                     )
-
-                    symbol: str = 'N/A'
-                    if chain := apiflows.get(reqid):
-                        symbol: str = chain.get('symbol', 'N/A')
-
                     await ems_stream.send(BrokerdError(
                         oid=oid,
                         # XXX: use old reqid in case it changed?
                         reqid=reqid,
-                        symbol=symbol,
+                        symbol=chain.get('symbol', 'N/A'),
 
                         reason=f'Failed {action}:\n{errmsg}',
                         broker_details=event
@@ -1119,3 +1185,36 @@ async def handle_order_updates(
                         })
             case _:
                 log.warning(f'Unhandled trades update msg: {msg}')
+
+
+async def norm_trade_records(
+    ledger: dict[str, Any],
+
+) -> dict[str, Transaction]:
+
+    records: dict[str, Transaction] = {}
+
+    for tid, record in ledger.items():
+
+        size = float(record.get('vol')) * {
+            'buy': 1,
+            'sell': -1,
+        }[record['type']]
+
+        # we normalize to kraken's `altname` always..
+        bs_mktid = Client.normalize_symbol(record['pair'])
+        fqme = f'{bs_mktid}.kraken'
+        mkt: MktPair = (await get_mkt_info(fqme))[0]
+
+        records[tid] = Transaction(
+            fqme=fqme,
+            sym=mkt,
+            tid=tid,
+            size=size,
+            price=float(record['price']),
+            cost=float(record['fee']),
+            dt=pendulum.from_timestamp(float(record['time'])),
+            bs_mktid=bs_mktid,
+        )
+
+    return records

piker/brokers/kraken/feed.py

@@ -24,38 +24,44 @@ from contextlib import (
 )
 from datetime import datetime
 from typing import (
+    Any,
     AsyncGenerator,
     Callable,
     Optional,
 )
 import time
 
+from fuzzywuzzy import process as fuzzy
 import numpy as np
 import pendulum
 from trio_typing import TaskStatus
+import tractor
 import trio
 
 from piker.accounting._mktinfo import (
+    Asset,
     MktPair,
 )
-from piker.brokers import (
+from piker._cacheables import (
     open_cached_client,
+    async_lifo_cache,
 )
 from piker.brokers._util import (
     BrokerError,
     DataThrottle,
     DataUnavailable,
 )
-from piker.types import Struct
+from piker.data.types import Struct
 from piker.data.validate import FeedInit
 from piker.data._web_bs import open_autorecon_ws, NoBsWs
+from . import log
 from .api import (
-    log,
+    Client,
+    Pair,
 )
-from .symbols import get_mkt_info
 
 
-class OHLC(Struct, frozen=True):
+class OHLC(Struct):
     '''
     Description of the flattened OHLC quote format.
 
@@ -66,8 +72,6 @@ class OHLC(Struct, frozen=True):
     chan_id: int  # internal kraken id
     chan_name: str  # eg. ohlc-1  (name-interval)
     pair: str  # fx pair
-
-    # unpacked from array
     time: float  # Begin time of interval, in seconds since epoch
     etime: float  # End time of interval, in seconds since epoch
     open: float  # Open price of interval
@@ -77,6 +81,8 @@ class OHLC(Struct, frozen=True):
     vwap: float  # Volume weighted average price within interval
     volume: float  # Accumulated volume **within interval**
     count: int  # Number of trades within interval
+    # (sampled) generated tick data
+    ticks: list[Any] = []
 
 
 async def stream_messages(
@@ -140,15 +146,14 @@ async def process_data_feed_msgs(
                     pair
                 ]:
                     if 'ohlc' in chan_name:
-                        array: list = payload_array[0]
                         ohlc = OHLC(
                             chan_id,
                             chan_name,
                             pair,
-                            *map(float, array[:-1]),
-                            count=array[-1],
+                            *payload_array[0]
                         )
-                        yield 'ohlc', ohlc.copy()
+                        ohlc.typecast()
+                        yield 'ohlc', ohlc
 
                     elif 'spread' in chan_name:
 
@@ -188,18 +193,24 @@ async def process_data_feed_msgs(
                     # yield msg
 
 
-def normalize(ohlc: OHLC) -> dict:
-    '''
-    Norm an `OHLC` msg to piker's minimal (live-)quote schema.
+def normalize(
+    ohlc: OHLC,
 
-    '''
+) -> dict:
     quote = ohlc.to_dict()
     quote['broker_ts'] = quote['time']
     quote['brokerd_ts'] = time.time()
     quote['symbol'] = quote['pair'] = quote['pair'].replace('/', '')
     quote['last'] = quote['close']
     quote['bar_wap'] = ohlc.vwap
-    return quote
+
+    # seriously eh? what's with this non-symmetry everywhere
+    # in subscription systems...
+    # XXX: piker style is always lowercases symbols.
+    topic = quote['pair'].replace('/', '').lower()
+
+    # print(quote)
+    return topic, quote
 
 
 @acm
@@ -208,7 +219,7 @@ async def open_history_client(
 
 ) -> AsyncGenerator[Callable, None]:
 
-    symbol: str = mkt.bs_mktid
+    symbol: str = mkt.bs_fqme
 
     # TODO implement history getter for the new storage layer.
     async with open_cached_client('kraken') as client:
@@ -258,6 +269,44 @@ async def open_history_client(
         yield get_ohlc, {'erlangs': 1, 'rate': 1}
 
 
+@async_lifo_cache()
+async def get_mkt_info(
+    fqme: str,
+
+) -> tuple[MktPair, Pair]:
+    '''
+    Query for and return a `MktPair` and backend-native `Pair` (or
+    wtv else) info.
+
+    If more then one fqme is provided return a ``dict`` of native
+    key-strs to `MktPair`s.
+
+    '''
+    async with open_cached_client('kraken') as client:
+
+        # uppercase since kraken bs_mktid is always upper
+        bs_fqme, _, broker = fqme.partition('.')
+        pair_str: str = bs_fqme.upper()
+        bs_mktid: str = Client.normalize_symbol(pair_str)
+        pair: Pair = await client.pair_info(pair_str)
+
+        assets = client.assets
+        dst_asset: Asset = assets[pair.base]
+        src_asset: Asset = assets[pair.quote]
+
+        mkt = MktPair(
+            dst=dst_asset,
+            src=src_asset,
+
+            price_tick=pair.price_tick,
+            size_tick=pair.size_tick,
+            bs_mktid=bs_mktid,
+
+            broker='kraken',
+        )
+        return mkt, pair
+
+
 async def stream_quotes(
 
     send_chan: trio.abc.SendChannel,
@@ -359,57 +408,76 @@ async def stream_quotes(
         ):
             # pull a first quote and deliver
             typ, ohlc_last = await anext(msg_gen)
-            quote = normalize(ohlc_last)
+            topic, quote = normalize(ohlc_last)
 
             task_status.started((init_msgs,  quote))
             feed_is_live.set()
 
             # keep start of last interval for volume tracking
-            last_interval_start: float = ohlc_last.etime
+            last_interval_start = ohlc_last.etime
 
             # start streaming
-            topic: str = mkt.bs_fqme
-            async for typ, quote in msg_gen:
-                match typ:
+            async for typ, ohlc in msg_gen:
+
+                if typ == 'ohlc':
 
                     # TODO: can get rid of all this by using
-                    # ``trades`` subscription..? Not sure why this
-                    # wasn't used originally? (music queues) zoltannn..
-                    # https://docs.kraken.com/websockets/#message-trade
-                    case 'ohlc':
-                        # generate tick values to match time & sales pane:
-                        # https://trade.kraken.com/charts/KRAKEN:BTC-USD?period=1m
-                        volume = quote.volume
+                    # ``trades`` subscription...
 
-                        # new OHLC sample interval
-                        if quote.etime > last_interval_start:
-                            last_interval_start: float = quote.etime
-                            tick_volume: float = volume
+                    # generate tick values to match time & sales pane:
+                    # https://trade.kraken.com/charts/KRAKEN:BTC-USD?period=1m
+                    volume = ohlc.volume
 
-                        else:
-                            # this is the tick volume *within the interval*
-                            tick_volume: float = volume - ohlc_last.volume
+                    # new OHLC sample interval
+                    if ohlc.etime > last_interval_start:
+                        last_interval_start = ohlc.etime
+                        tick_volume = volume
 
-                        ohlc_last = quote
-                        last = quote.close
+                    else:
+                        # this is the tick volume *within the interval*
+                        tick_volume = volume - ohlc_last.volume
 
-                        quote = normalize(quote)
-                        ticks = quote.setdefault(
-                            'ticks',
-                            [],
-                        )
-                        if tick_volume:
-                            ticks.append({
-                                'type': 'trade',
-                                'price': last,
-                                'size': tick_volume,
-                            })
+                    ohlc_last = ohlc
+                    last = ohlc.close
 
-                    case 'l1':
-                        # passthrough quote msg
-                        pass
+                    if tick_volume:
+                        ohlc.ticks.append({
+                            'type': 'trade',
+                            'price': last,
+                            'size': tick_volume,
+                        })
 
-                    case _:
-                        log.warning(f'Unknown WSS message: {typ}, {quote}')
+                    topic, quote = normalize(ohlc)
+
+                elif typ == 'l1':
+                    quote = ohlc
+                    topic = quote['symbol'].lower()
 
                 await send_chan.send({topic: quote})
+
+
+@tractor.context
+async def open_symbol_search(
+    ctx: tractor.Context,
+
+) -> Client:
+    async with open_cached_client('kraken') as client:
+
+        # load all symbols locally for fast search
+        cache = await client.cache_symbols()
+        await ctx.started(cache)
+
+        async with ctx.open_stream() as stream:
+
+            async for pattern in stream:
+
+                matches = fuzzy.extractBests(
+                    pattern,
+                    cache,
+                    score_cutoff=50,
+                )
+                # repack in dict form
+                await stream.send({
+                    pair[0].altname: pair[0]
+                    for pair in matches
+                })

piker/brokers/kraken/ledger.py

@@ -1,269 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Trade transaction accounting and normalization.
-
-'''
-import math
-from pprint import pformat
-from typing import (
-    Any,
-)
-
-import pendulum
-
-from piker.accounting import (
-    Transaction,
-    Position,
-    Account,
-    get_likely_pair,
-    TransactionLedger,
-    # MktPair,
-)
-from piker.types import Struct
-from piker.data import (
-    SymbologyCache,
-)
-from .api import (
-    log,
-    Client,
-    Pair,
-)
-# from .feed import get_mkt_info
-
-
-def norm_trade(
-    tid: str,
-    record: dict[str, Any],
-
-    # this is the dict that was returned from
-    # `Client.get_mkt_pairs()` and when running offline ledger
-    # processing from `.accounting`, this will be the table loaded
-    # into `SymbologyCache.pairs`.
-    pairs: dict[str, Struct],
-    symcache: SymbologyCache | None = None,
-
-) -> Transaction:
-
-    size: float = float(record.get('vol')) * {
-        'buy': 1,
-        'sell': -1,
-    }[record['type']]
-
-    # NOTE: this value may be either the websocket OR the rest schema
-    # so we need to detect the key format and then choose the
-    # correct symbol lookup table to evetually get a ``Pair``..
-    # See internals of `Client.asset_pairs()` for deats!
-    src_pair_key: str = record['pair']
-
-    # XXX: kraken's data engineering is soo bad they require THREE
-    # different pair schemas (more or less seemingly tied to
-    # transport-APIs)..LITERALLY they return different market id
-    # pairs in the ledger endpoints vs. the websocket event subs..
-    # lookup pair using appropriately provided tabled depending
-    # on API-key-schema..
-    pair: Pair = pairs[src_pair_key]
-    fqme: str = pair.bs_fqme.lower() + '.kraken'
-
-    return Transaction(
-        fqme=fqme,
-        tid=tid,
-        size=size,
-        price=float(record['price']),
-        cost=float(record['fee']),
-        dt=pendulum.from_timestamp(float(record['time'])),
-        bs_mktid=pair.bs_mktid,
-    )
-
-
-async def norm_trade_records(
-    ledger: dict[str, Any],
-    client: Client,
-    api_name_set: str = 'xname',
-
-) -> dict[str, Transaction]:
-    '''
-    Loop through an input ``dict`` of trade records
-    and convert them to ``Transactions``.
-
-    '''
-    records: dict[str, Transaction] = {}
-    for tid, record in ledger.items():
-
-        # manual_fqme: str = f'{bs_mktid.lower()}.kraken'
-        # mkt: MktPair = (await get_mkt_info(manual_fqme))[0]
-        # fqme: str = mkt.fqme
-        # assert fqme == manual_fqme
-        pairs: dict[str, Pair] = {
-            'xname': client._AssetPairs,
-            'wsname': client._wsnames,
-            'altname': client._altnames,
-        }[api_name_set]
-
-        records[tid] = norm_trade(
-            tid,
-            record,
-            pairs=pairs,
-        )
-
-    return records
-
-
-def has_pp(
-    acnt: Account,
-    src_fiat: str,
-    dst: str,
-    size: float,
-
-) -> Position | None:
-
-    src2dst: dict[str, str] = {}
-    for bs_mktid in acnt.pps:
-        likely_pair = get_likely_pair(
-            src_fiat,
-            dst,
-            bs_mktid,
-        )
-        if likely_pair:
-            src2dst[src_fiat] = dst
-
-    for src, dst in src2dst.items():
-        pair: str = f'{dst}{src_fiat}'
-        pos: Position = acnt.pps.get(pair)
-        if (
-            pos
-            and math.isclose(pos.size, size)
-        ):
-            return pos
-
-        elif (
-            size == 0
-            and pos.size
-        ):
-            log.warning(
-                f'`kraken` account says you have  a ZERO '
-                f'balance for {bs_mktid}:{pair}\n'
-                f'but piker seems to think `{pos.size}`\n'
-                'This is likely a discrepancy in piker '
-                'accounting if the above number is'
-                "large,' though it's likely to due lack"
-                "f tracking xfers fees.."
-            )
-            return pos
-
-    return None  # indicate no entry found
-
-
-# TODO: factor most of this "account updating from txns" into the
-# the `Account` impl so has to provide for hiding the mostly
-# cross-provider updates from txn sets
-async def verify_balances(
-    acnt: Account,
-    src_fiat: str,
-    balances: dict[str, float],
-    client: Client,
-    ledger: TransactionLedger,
-    ledger_trans: dict[str, Transaction],  # from toml
-    api_trans: dict[str, Transaction],  # from API
-
-    simulate_pp_update: bool = False,
-
-) -> None:
-    for dst, size in balances.items():
-
-        # we don't care about tracking positions
-        # in the user's source fiat currency.
-        if (
-            dst == src_fiat
-            or not any(
-                dst in bs_mktid for bs_mktid in acnt.pps
-            )
-        ):
-            log.warning(
-                f'Skipping balance `{dst}`:{size} for position calcs!'
-            )
-            continue
-
-        # we have a balance for which there is no pos entry
-        # - we have to likely update from the ledger?
-        if not has_pp(acnt, src_fiat, dst, size):
-            updated = acnt.update_from_ledger(
-                ledger_trans,
-                symcache=ledger.symcache,
-            )
-            log.info(f'Updated pps from ledger:\n{pformat(updated)}')
-
-            # FIRST try reloading from API records
-            if (
-                not has_pp(acnt, src_fiat, dst, size)
-                and not simulate_pp_update
-            ):
-                acnt.update_from_ledger(
-                    api_trans,
-                    symcache=ledger.symcache,
-                )
-
-                # get transfers to make sense of abs
-                # balances.
-                # NOTE: we do this after ledger and API
-                # loading since we might not have an
-                # entry in the
-                # ``account.kraken.spot.toml`` for the
-                # necessary pair yet and thus this
-                # likely pair grabber will likely fail.
-                if not has_pp(acnt, src_fiat, dst, size):
-                    for bs_mktid in acnt.pps:
-                        likely_pair: str | None = get_likely_pair(
-                            src_fiat,
-                            dst,
-                            bs_mktid,
-                        )
-                        if likely_pair:
-                            break
-                    else:
-                        raise ValueError(
-                            'Could not find a position pair in '
-                            'ledger for likely widthdrawal '
-                            f'candidate: {dst}'
-                        )
-
-                    # this was likely pos that had a withdrawal
-                    # from the dst asset out of the account.
-                    if likely_pair:
-                        xfer_trans = await client.get_xfers(
-                            dst,
-
-                            # TODO: not all src assets are
-                            # 3 chars long...
-                            src_asset=likely_pair[3:],
-                        )
-                        if xfer_trans:
-                            updated = acnt.update_from_ledger(
-                                xfer_trans,
-                                cost_scalar=1,
-                                symcache=ledger.symcache,
-                            )
-                            log.info(
-                                f'Updated {dst} from transfers:\n'
-                                f'{pformat(updated)}'
-                            )
-
-                if has_pp(acnt, src_fiat, dst, size):
-                    raise ValueError(
-                        'Could not reproduce balance:\n'
-                        f'dst: {dst}, {size}\n'
-                    )

piker/brokers/kraken/symbols.py

@@ -1,206 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Symbology defs and search.
-
-'''
-from decimal import Decimal
-
-import tractor
-from rapidfuzz import process as fuzzy
-
-from piker._cacheables import (
-    async_lifo_cache,
-)
-from piker.accounting._mktinfo import (
-    digits_to_dec,
-)
-from piker.brokers import (
-    open_cached_client,
-    SymbolNotFound,
-)
-from piker.types import Struct
-from piker.accounting._mktinfo import (
-    Asset,
-    MktPair,
-    unpack_fqme,
-)
-
-
-# https://www.kraken.com/features/api#get-tradable-pairs
-class Pair(Struct):
-    xname: str  # idiotic bs_mktid equiv i guess?
-    altname: str  # alternate pair name
-    wsname: str  # WebSocket pair name (if available)
-    aclass_base: str  # asset class of base component
-    base: str  # asset id of base component
-    aclass_quote: str  # asset class of quote component
-    quote: str  # asset id of quote component
-    lot: str  # volume lot size
-
-    cost_decimals: int
-    costmin: float
-    pair_decimals: int  # scaling decimal places for pair
-    lot_decimals: int  # scaling decimal places for volume
-
-    # amount to multiply lot volume by to get currency volume
-    lot_multiplier: float
-
-    # array of leverage amounts available when buying
-    leverage_buy: list[int]
-    # array of leverage amounts available when selling
-    leverage_sell: list[int]
-
-    # fee schedule array in [volume, percent fee] tuples
-    fees: list[tuple[int, float]]
-
-    # maker fee schedule array in [volume, percent fee] tuples (if on
-    # maker/taker)
-    fees_maker: list[tuple[int, float]]
-
-    fee_volume_currency: str  # volume discount currency
-    margin_call: str  # margin call level
-    margin_stop: str  # stop-out/liquidation margin level
-    ordermin: float  # minimum order volume for pair
-    tick_size: float  # min price step size
-    status: str
-
-    short_position_limit: float = 0
-    long_position_limit: float = float('inf')
-
-    # TODO: should we make this a literal NamespacePath ref?
-    ns_path: str = 'piker.brokers.kraken:Pair'
-
-    @property
-    def bs_mktid(self) -> str:
-        '''
-        Kraken seems to index it's market symbol sets in
-        transaction ledgers using the key returned from rest
-        queries.. so use that since apparently they can't
-        make up their minds on a better key set XD
-
-        '''
-        return self.xname
-
-    @property
-    def price_tick(self) -> Decimal:
-        return digits_to_dec(self.pair_decimals)
-
-    @property
-    def size_tick(self) -> Decimal:
-        return digits_to_dec(self.lot_decimals)
-
-    @property
-    def bs_dst_asset(self) -> str:
-        dst, _ = self.wsname.split('/')
-        return dst
-
-    @property
-    def bs_src_asset(self) -> str:
-        _, src = self.wsname.split('/')
-        return src
-
-    @property
-    def bs_fqme(self) -> str:
-        '''
-        Basically the `.altname` but with special '.' handling and
-        `.SPOT` suffix appending (for future multi-venue support).
-
-        '''
-        dst, src = self.wsname.split('/')
-        # XXX: omg for stupid shite like ETH2.S/ETH..
-        dst = dst.replace('.', '-')
-        return f'{dst}{src}.SPOT'
-
-
-@tractor.context
-async def open_symbol_search(ctx: tractor.Context) -> None:
-    async with open_cached_client('kraken') as client:
-
-        # load all symbols locally for fast search
-        cache = await client.get_mkt_pairs()
-        await ctx.started(cache)
-
-        async with ctx.open_stream() as stream:
-            async for pattern in stream:
-                await stream.send(
-                    await client.search_symbols(pattern)
-                )
-
-
-@async_lifo_cache()
-async def get_mkt_info(
-    fqme: str,
-
-) -> tuple[MktPair, Pair]:
-    '''
-    Query for and return a `MktPair` and backend-native `Pair` (or
-    wtv else) info.
-
-    If more then one fqme is provided return a ``dict`` of native
-    key-strs to `MktPair`s.
-
-    '''
-    venue: str = 'spot'
-    expiry: str = ''
-    if '.kraken' not in fqme:
-        fqme += '.kraken'
-
-    broker, pair, venue, expiry = unpack_fqme(fqme)
-    venue: str = venue or 'spot'
-
-    if venue.lower() != 'spot':
-        raise SymbolNotFound(
-            'kraken only supports spot markets right now!\n'
-            f'{fqme}\n'
-        )
-
-    async with open_cached_client('kraken') as client:
-
-        # uppercase since kraken bs_mktid is always upper
-        # bs_fqme, _, broker = fqme.partition('.')
-        # pair_str: str = bs_fqme.upper()
-        pair_str: str = f'{pair}.{venue}'
-
-        pair: Pair | None = client._pairs.get(pair_str.upper())
-        if not pair:
-            bs_fqme: str = client.to_bs_fqme(pair_str)
-            pair: Pair = client._pairs[bs_fqme]
-
-        if not (assets := client._assets):
-            assets: dict[str, Asset] = await client.get_assets()
-
-        dst_asset: Asset = assets[pair.bs_dst_asset]
-        src_asset: Asset = assets[pair.bs_src_asset]
-
-        mkt = MktPair(
-            dst=dst_asset,
-            src=src_asset,
-
-            price_tick=pair.price_tick,
-            size_tick=pair.size_tick,
-            bs_mktid=pair.bs_mktid,
-
-            expiry=expiry,
-            venue=venue or 'spot',
-
-            # TODO: futes
-            # _atype=_atype,
-
-            broker='kraken',
-        )
-        return mkt, pair

piker/brokers/kucoin.py

@@ -16,9 +16,10 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-Kucoin cex API backend.
+Kucoin broker backend
 
 '''
+
 from contextlib import (
     asynccontextmanager as acm,
     aclosing,
@@ -40,8 +41,9 @@ from typing import (
 import wsproto
 from uuid import uuid4
 
+from fuzzywuzzy import process as fuzzy
 from trio_typing import TaskStatus
-import httpx
+import asks
 from bidict import bidict
 import numpy as np
 import pendulum
@@ -53,20 +55,14 @@ from piker.accounting._mktinfo import (
     digits_to_dec,
     MktPair,
 )
+from piker.data.validate import FeedInit
 from piker import config
-from piker.brokers import (
-    open_cached_client,
-)
 from piker._cacheables import (
+    open_cached_client,
     async_lifo_cache,
 )
 from piker.log import get_logger
-from piker.data.validate import FeedInit
-from piker.types import Struct  # NOTE, this is already a `tractor.msg.Struct`
-from piker.data import (
-    def_iohlcv_fields,
-    match_from_pairs,
-)
+from piker.data.types import Struct
 from piker.data._web_bs import (
     open_autorecon_ws,
     NoBsWs,
@@ -75,7 +71,16 @@ from ._util import DataUnavailable
 
 log = get_logger(__name__)
 
-_no_symcache: bool = True
+_ohlc_dtype = [
+    ('index', int),
+    ('time', int),
+    ('open', float),
+    ('high', float),
+    ('low', float),
+    ('close', float),
+    ('volume', float),
+    ('bar_wap', float),  # will be zeroed by sampler if not filled
+]
 
 
 class KucoinMktPair(Struct, frozen=True):
@@ -89,27 +94,18 @@ class KucoinMktPair(Struct, frozen=True):
 
     @property
     def price_tick(self) -> Decimal:
-        return Decimal(str(self.quoteIncrement))
+        return Decimal(str(self.baseIncrement))
 
     baseMaxSize: float
     baseMinSize: float
 
     @property
     def size_tick(self) -> Decimal:
-        return Decimal(str(self.quoteMinSize))
-
-    callauctionFirstStageStartTime: None|float
-    callauctionIsEnabled: bool
-    callauctionPriceCeiling: float|None
-    callauctionPriceFloor: float|None
-    callauctionSecondStageStartTime: float|None
-    callauctionThirdStageStartTime: float|None
+        return Decimal(str(self.baseMinSize))
 
     enableTrading: bool
-    feeCategory: int
     feeCurrency: str
     isMarginEnabled: bool
-    makerFeeCoefficient: float
     market: str
     minFunds: float
     name: str
@@ -119,10 +115,7 @@ class KucoinMktPair(Struct, frozen=True):
     quoteIncrement: float
     quoteMaxSize: float
     quoteMinSize: float
-    st: bool
     symbol: str  # our bs_mktid, kucoin's internal id
-    takerFeeCoefficient: float
-    tradingStartTime: float|None
 
 
 class AccountTrade(Struct, frozen=True):
@@ -222,13 +215,8 @@ def get_config() -> BrokerConfig | None:
 
 
 class Client:
-
-    def __init__(
-        self,
-        httpx_client: httpx.AsyncClient,
-    ) -> None:
-        self._http: httpx.AsyncClient = httpx_client
-        self._config: BrokerConfig|None = get_config()
+    def __init__(self) -> None:
+        self._config: BrokerConfig | None = get_config()
         self._pairs: dict[str, KucoinMktPair] = {}
         self._fqmes2mktids: bidict[str, str] = bidict()
         self._bars: list[list[float]] = []
@@ -242,24 +230,18 @@ class Client:
 
     ) -> dict[str, str | bytes]:
         '''
-        Generate authenticated request headers:
-
+        Generate authenticated request headers
         https://docs.kucoin.com/#authentication
-        https://www.kucoin.com/docs/basic-info/connection-method/authentication/creating-a-request
-        https://www.kucoin.com/docs/basic-info/connection-method/authentication/signing-a-message
 
         '''
+
         if not self._config:
             raise ValueError(
-                'No config found when trying to send authenticated request'
-            )
+                'No config found when trying to send authenticated request')
 
         str_to_sign = (
             str(int(time.time() * 1000))
-            +
-            action
-            +
-            f'/api/{api}/{endpoint.lstrip("/")}'
+            + action + f'/api/{api}/{endpoint.lstrip("/")}'
         )
 
         signature = base64.b64encode(
@@ -270,7 +252,6 @@ class Client:
             ).digest()
         )
 
-        # TODO: can we cache this between calls?
         passphrase = base64.b64encode(
             hmac.new(
                 self._config.key_secret.encode('utf-8'),
@@ -292,10 +273,8 @@ class Client:
         self,
         action: Literal['POST', 'GET'],
         endpoint: str,
-
         api: str = 'v2',
         headers: dict = {},
-
     ) -> Any:
         '''
         Generic request wrapper for Kucoin API
@@ -308,19 +287,14 @@ class Client:
                 api,
             )
 
-        req_meth: Callable = getattr(
-            self._http,
-            action.lower(),
-        )
-        res = await req_meth(
-            url=f'/{api}/{endpoint}',
-            headers=headers,
-        )
-        json: dict = res.json()
-        if (data := json.get('data')) is not None:
-            return data
+        api_url = f'https://api.kucoin.com/api/{api}/{endpoint}'
+
+        res = await asks.request(action, api_url, headers=headers)
+
+        json = res.json()
+        if 'data' in json:
+            return json['data']
         else:
-            api_url: str = self._http.base_url
             log.error(
                 f'Error making request to {api_url} ->\n'
                 f'{pformat(res)}'
@@ -340,7 +314,7 @@ class Client:
         '''
         token_type = 'private' if private else 'public'
         try:
-            data: dict[str, Any]|None = await self._request(
+            data: dict[str, Any] | None = await self._request(
                 'POST',
                 endpoint=f'bullet-{token_type}',
                 api='v1'
@@ -378,8 +352,8 @@ class Client:
             currencies: dict[str, Currency] = {}
             entries: list[dict] = await self._request(
                 'GET',
-                endpoint='currencies',
                 api='v1',
+                endpoint='currencies',
             )
             for entry in entries:
                 curr = Currency(**entry).copy()
@@ -395,29 +369,20 @@ class Client:
         dict[str, KucoinMktPair],
         bidict[str, KucoinMktPair],
     ]:
-        entries = await self._request(
-            'GET',
-            endpoint='symbols',
-        )
+        entries = await self._request('GET', 'symbols')
         log.info(f' {len(entries)} Kucoin market pairs fetched')
 
         pairs: dict[str, KucoinMktPair] = {}
         fqmes2mktids: bidict[str, str] = bidict()
         for item in entries:
-            try:
-                pair = pairs[item['name']] = KucoinMktPair(**item)
-            except TypeError as te:
-                raise TypeError(
-                    '`KucoinMktPair` and reponse fields do not match ??\n'
-                    f'{KucoinMktPair.fields_diff(item)}\n'
-                ) from te
+            pair = pairs[item['name']] = KucoinMktPair(**item)
             fqmes2mktids[
                 item['name'].lower().replace('-', '')
             ] = pair.name
 
         return pairs, fqmes2mktids
 
-    async def get_mkt_pairs(
+    async def cache_pairs(
         self,
         update: bool = False,
 
@@ -445,27 +410,16 @@ class Client:
 
     ) -> dict[str, KucoinMktPair]:
         '''
-        Use fuzzy search engine to match against pairs, deliver
-        matching ones.
+        Use fuzzy search to match against all market names.
 
         '''
-        if not len(self._pairs):
-            await self.get_mkt_pairs()
-            assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
+        data = await self.cache_pairs()
 
-        matches: dict[str, KucoinMktPair] = match_from_pairs(
-            pairs=self._pairs,
-            # query=pattern.upper(),
-            query=pattern.upper(),
-            score_cutoff=35,
-            limit=limit,
+        matches = fuzzy.extractBests(
+            pattern, data, score_cutoff=35, limit=limit
         )
-
         # repack in dict form
-        return {
-            pair.name: pair
-            for pair in matches.values()
-        }
+        return {item[0].name: item[0] for item in matches}
 
     async def last_trades(self, sym: str) -> list[AccountTrade]:
         trades = await self._request(
@@ -511,27 +465,18 @@ class Client:
             ('low', float),
             ('close', float),
             ('volume', float),
+            ('bar_wap', float),  # will be zeroed by sampler if not filled
         ]
 
         '''
         # Generate generic end and start time if values not passed
         # Currently gives us 12hrs of data
-        if (
-            end_dt is None
-            and start_dt is None
-        ):
+        if end_dt is None:
             end_dt = pendulum.now('UTC').add(minutes=1)
-            start_dt = end_dt.start_of('minute').subtract(minutes=limit)
 
-        if (
-            start_dt
-            and end_dt is None
-        ):
-            # just set end to limit's worth in future
-            end_dt = start_dt.start_of('minute').add(minutes=limit)
-
-        else:
-            start_dt = end_dt.start_of('minute').subtract(minutes=limit)
+        if start_dt is None:
+            start_dt = end_dt.start_of(
+                'minute').subtract(minutes=limit)
 
         start_dt = int(start_dt.timestamp())
         end_dt = int(end_dt.timestamp())
@@ -585,12 +530,12 @@ class Client:
                     # volume
                     float(bar[5]),
                     # bar_wap
-                    # 0.0,
+                    0.0,
                 )
             )
 
         array = np.array(
-            new_bars, dtype=def_iohlcv_fields) if as_np else bars
+            new_bars, dtype=_ohlc_dtype) if as_np else bars
         return array
 
 
@@ -605,21 +550,13 @@ def fqme_to_kucoin_sym(
 
 @acm
 async def get_client() -> AsyncGenerator[Client, None]:
-    '''
-    Load an API `Client` preconfigured from user settings
+    client = Client()
 
-    '''
-    async with (
-        httpx.AsyncClient(
-            base_url='https://api.kucoin.com/api',
-        ) as trio_client,
-    ):
-        client = Client(httpx_client=trio_client)
-        async with trio.open_nursery() as tn:
-            tn.start_soon(client.get_mkt_pairs)
-            await client.get_currencies()
+    async with trio.open_nursery() as n:
+        n.start_soon(client.cache_pairs)
+        await client.get_currencies()
 
-        yield client
+    yield client
 
 
 @tractor.context
@@ -628,7 +565,7 @@ async def open_symbol_search(
 ) -> None:
     async with open_cached_client('kucoin') as client:
         # load all symbols locally for fast search
-        await client.get_mkt_pairs()
+        await client.cache_pairs()
         await ctx.started()
 
         async with ctx.open_stream() as stream:
@@ -655,7 +592,7 @@ async def open_ping_task(
                 await trio.sleep((ping_interval - 1000) / 1000)
                 await ws.send_msg({'id': connect_id, 'type': 'ping'})
 
-        log.warning('Starting ping task for kucoin ws connection')
+        log.info('Starting ping task for kucoin ws connection')
         n.start_soon(ping_server)
 
         yield
@@ -667,21 +604,16 @@ async def open_ping_task(
 async def get_mkt_info(
     fqme: str,
 
-) -> tuple[
-    MktPair,
-    KucoinMktPair,
-]:
+) -> tuple[MktPair, KucoinMktPair]:
     '''
-    Query for and return both a `piker.accounting.MktPair` and
-    `KucoinMktPair` from provided `fqme: str`
-    (fully-qualified-market-endpoint).
+    Query for and return a `MktPair` and `KucoinMktPair`.
 
     '''
     async with open_cached_client('kucoin') as client:
         # split off any fqme broker part
         bs_fqme, _, broker = fqme.partition('.')
 
-        pairs: dict[str, KucoinMktPair] = await client.get_mkt_pairs()
+        pairs: dict[str, KucoinMktPair] = await client.cache_pairs()
 
         try:
             # likely search result key which is already in native mkt symbol form
@@ -749,8 +681,6 @@ async def stream_quotes(
 
         log.info(f'Starting up quote stream(s) for {symbols}')
         for sym_str in symbols:
-            mkt: MktPair
-            pair: KucoinMktPair
             mkt, pair = await get_mkt_info(sym_str)
             init_msgs.append(
                 FeedInit(mkt_info=mkt)
@@ -758,11 +688,7 @@ async def stream_quotes(
 
         ws: NoBsWs
         token, ping_interval = await client._get_ws_token()
-        log.info('API reported ping_interval: {ping_interval}\n')
-
-        connect_id: str = str(uuid4())
-        typ: str
-        quote: dict
+        connect_id = str(uuid4())
         async with (
             open_autorecon_ws(
                 (
@@ -776,37 +702,20 @@ async def stream_quotes(
                 ),
             ) as ws,
             open_ping_task(ws, ping_interval, connect_id),
-            aclosing(
-                iter_normed_quotes(
-                    ws, sym_str
-                )
-            ) as iter_quotes,
+            aclosing(stream_messages(ws, sym_str)) as msg_gen,
         ):
-            typ, quote = await anext(iter_quotes)
+            typ, quote = await anext(msg_gen)
 
-            # take care to not unblock here until we get a real
-            # trade quote?
-            # ^TODO, remove this right?
-            # -[ ] what often blocks chart boot/new-feed switching
-            #   since we'ere waiting for a live quote instead of just
-            #   loading history afap..
-            #  |_ XXX, not sure if we require a bit of rework to core
-            #    feed init logic or if backends justg gotta be
-            #    changed up.. feel like there was some causality
-            #    dilema prolly only seen with IB too..
-            # while typ != 'trade':
-            #     typ, quote = await anext(iter_quotes)
+            while typ != 'trade':
+                # take care to not unblock here until we get a real
+                # trade quote
+                typ, quote = await anext(msg_gen)
 
             task_status.started((init_msgs, quote))
             feed_is_live.set()
 
-            # XXX NOTE, DO NOT include the `.<backend>` suffix!
-            # OW the sampling loop will not broadcast correctly..
-            # since `bus._subscribers.setdefault(bs_fqme, set())`
-            # is used inside `.data.open_feed_bus()` !!!
-            topic: str = mkt.bs_fqme
-            async for typ, quote in iter_quotes:
-                await send_chan.send({topic: quote})
+            async for typ, msg in msg_gen:
+                await send_chan.send({sym_str: msg})
 
 
 @acm
@@ -861,7 +770,7 @@ async def subscribe(
             )
 
 
-async def iter_normed_quotes(
+async def stream_messages(
     ws: NoBsWs,
     sym: str,
 
@@ -892,9 +801,6 @@ async def iter_normed_quotes(
 
                 yield 'trade', {
                     'symbol': sym,
-                    # TODO, is 'last' even used elsewhere/a-good
-                    # semantic? can't we just read the ticks with our
-                    # .data.ticktools.frame_ticks()`/
                     'last': trade_data.price,
                     'brokerd_ts': last_trade_ts,
                     'ticks': [
@@ -987,7 +893,7 @@ async def open_history_client(
             if end_dt is None:
                 inow = round(time.time())
 
-                log.debug(
+                print(
                     f'difference in time between load and processing'
                     f'{inow - times[-1]}'
                 )

piker/brokers/questrade.py

@@ -40,8 +40,7 @@ import wrapt
 import asks
 
 from ..calc import humanize, percent_change
-from . import open_cached_client
-from piker._cacheables import async_lifo_cache
+from .._cacheables import open_cached_client, async_lifo_cache
 from .. import config
 from ._util import resproc, BrokerError, SymbolNotFound
 from ..log import (

piker/clearing/README.rst

@@ -1,49 +0,0 @@
-piker.clearing
-______________
-trade execution-n-control subsys for both live and paper trading as
-well as algo-trading manual override/interaction across any backend
-broker and data provider.
-
-avail UIs
-*********
-
-order ctl
----------
-the `piker.clearing` subsys is exposed mainly though
-the `piker chart` GUI as a "chart trader" style UX and
-is automatically enabled whenever a chart is opened.
-
-.. ^TODO, more prose here!
-
-the "manual" order control features are exposed via the
-`piker.ui.order_mode` API and can pretty much always be
-used (at least) in simulated-trading mode, aka "paper"-mode, and
-the micro-manual is as follows:
-
-``order_mode`` (
-    edge triggered activation by any of the following keys,
-    ``mouse-click`` on y-level to submit at that price
-    ):
-
-    - ``f``/ ``ctl-f`` to stage buy
-    - ``d``/ ``ctl-d`` to stage sell
-    - ``a`` to stage alert
-
-
-``search_mode`` (
-    ``ctl-l`` or ``ctl-space`` to open,
-    ``ctl-c`` or ``ctl-space`` to close
-    ) :
-
-    - begin typing to have symbol search automatically lookup
-      symbols from all loaded backend (broker) providers
-    - arrow keys and mouse click to navigate selection
-    - vi-like ``ctl-[hjkl]`` for navigation
-
-
-position (pp) mgmt
-------------------
-you can also configure your position allocation limits from the
-sidepane.
-
-.. ^TODO, explain and provide tut once more refined!

piker/clearing/__init__.py

@@ -23,32 +23,11 @@ from ._client import (
     open_ems,
     OrderClient,
 )
-from ._ems import (
-    open_brokerd_dialog,
-)
-from ._util import OrderDialogs
-from ._messages import(
-    Order,
-    Status,
-    Cancel,
-
-    # TODO: deprecate these and replace end-2-end with
-    # client-side-dialog set above B)
-    # https://github.com/pikers/piker/issues/514
-    BrokerdPosition
-)
 
 
 __all__ = [
-    'FeeModel',
     'open_ems',
     'OrderClient',
-    'open_brokerd_dialog',
-    'OrderDialogs',
-    'Order',
-    'Status',
-    'Cancel',
-    'BrokerdPosition'
 
 ]
 

piker/clearing/_client.py

@@ -30,13 +30,14 @@ from tractor.trionics import broadcast_receiver
 from ._util import (
     log,  # sub-sys logger
 )
-from piker.types import Struct
+from ..data.types import Struct
 from ..service import maybe_open_emsd
 from ._messages import (
     Order,
     Cancel,
     BrokerdPosition,
 )
+from ..brokers import get_brokermod
 
 if TYPE_CHECKING:
     from ._messages import (
@@ -131,8 +132,6 @@ class OrderClient(Struct):
                 f'Maybe there is a stale entry or line?\n'
                 f'You should report this as a bug!'
             )
-            return
-
         fqme = str(cmd.symbol)
         return Cancel(
             oid=uuid,
@@ -216,8 +215,8 @@ async def open_ems(
     loglevel: str = 'error',
 
 ) -> tuple[
-    OrderClient,  # client
-    tractor.MsgStream,  # order ctl stream
+    OrderClient,
+    tractor.MsgStream,
     dict[
         # brokername, acctid
         tuple[str, str],
@@ -245,6 +244,13 @@ async def open_ems(
         loglevel=loglevel,
     ) as portal:
 
+        mod = get_brokermod(broker)
+        if (
+            not getattr(mod, 'trades_dialogue', None)
+            or mode == 'paper'
+        ):
+            mode = 'paper'
+
         from ._ems import _emsd_main
         async with (
             # connect to emsd

piker/clearing/_ems.py

@@ -24,10 +24,9 @@ from collections import (
     # ChainMap,
 )
 from contextlib import asynccontextmanager as acm
-from decimal import Decimal
 from math import isnan
 from pprint import pformat
-from time import time_ns
+import time
 from types import ModuleType
 from typing import (
     AsyncIterator,
@@ -35,7 +34,6 @@ from typing import (
     Callable,
     Hashable,
     Optional,
-    TYPE_CHECKING,
 )
 
 from bidict import bidict
@@ -47,17 +45,22 @@ from ._util import (
     log,  # sub-sys logger
     get_console_log,
 )
+from ..data._normalize import iterticks
 from ..accounting._mktinfo import (
     unpack_fqme,
-    dec_digits,
+    float_digits,
+)
+from ..data.feed import (
+    Feed,
+    Flume,
+    maybe_open_feed,
 )
-from piker.types import Struct
 from ..ui._notify import notify_from_ems_status_msg
-from ..data import iterticks
+from ..data.types import Struct
+from . import _paper_engine as paper
 from ._messages import (
     Order,
     Status,
-    Error,
     BrokerdCancel,
     BrokerdOrder,
     # BrokerdOrderAck,
@@ -67,12 +70,6 @@ from ._messages import (
     BrokerdPosition,
 )
 
-if TYPE_CHECKING:
-    from ..data import (
-        Feed,
-        Flume,
-    )
-
 
 # TODO: numba all of this
 def mk_check(
@@ -132,16 +129,11 @@ class DarkBook(Struct):
     triggers: dict[
         str,  # symbol
         dict[
-            str,  # uuid for triggerable execution
+            str,  # uuid
             tuple[
                 Callable[[float], bool],  # predicate
-                tuple[str, ...],  # tickfilter
-                dict | Order,  # cmd / msg type
-
-                # live submission constraint parameters
-                float,  # percent_away max price diff
-                float,  # abs_diff_away max price diff
-                int,  # min_tick_digits to round the clearable price
+                str,  # name
+                dict,  # cmd / msg type
             ]
         ]
     ] = {}
@@ -184,8 +176,7 @@ async def clear_dark_triggers(
     async for quotes in quote_stream:
         # start = time.time()
         for sym, quote in quotes.items():
-            # TODO: make this a msg-compat struct
-            execs: tuple = book.triggers.get(sym, {})
+            execs = book.triggers.get(sym, {})
             for tick in iterticks(
                 quote,
                 # dark order price filter(s)
@@ -208,8 +199,7 @@ async def clear_dark_triggers(
                     # TODO: send this msg instead?
                     cmd,
                     percent_away,
-                    abs_diff_away,
-                    price_tick_digits,
+                    abs_diff_away
                 ) in (
                     tuple(execs.items())
                 ):
@@ -242,11 +232,8 @@ async def clear_dark_triggers(
                             size=size,
                         ):
                             bfqme: str = symbol.replace(f'.{broker}', '')
-                            submit_price: float = round(
-                                price + abs_diff_away,
-                                ndigits=price_tick_digits,
-                            )
-                            resp: str = 'triggered'  # hidden on client-side
+                            submit_price = price + abs_diff_away
+                            resp = 'triggered'  # hidden on client-side
 
                             log.info(
                                 f'Dark order triggered for price {price}\n'
@@ -256,7 +243,7 @@ async def clear_dark_triggers(
                                 action=action,
                                 oid=oid,
                                 account=account,
-                                time_ns=time_ns(),
+                                time_ns=time.time_ns(),
                                 symbol=bfqme,
                                 price=submit_price,
                                 size=size,
@@ -269,18 +256,18 @@ async def clear_dark_triggers(
                     # fallthrough logic
                     status = Status(
                         oid=oid,  # ems dialog id
-                        time_ns=time_ns(),
+                        time_ns=time.time_ns(),
                         resp=resp,
                         req=cmd,
                         brokerd_msg=brokerd_msg,
                     )
 
                     # remove exec-condition from set
-                    log.info(f'Removing trigger for {oid}')
-                    trigger: tuple | None = execs.pop(oid, None)
-                    if not trigger:
+                    log.info(f'removing pred for {oid}')
+                    pred = execs.pop(oid, None)
+                    if not pred:
                         log.warning(
-                            f'trigger for {oid} was already removed!?'
+                            f'pred for {oid} was already removed!?'
                         )
 
                     # update actives
@@ -320,177 +307,15 @@ class TradesRelay(Struct):
 
     # map of symbols to dicts of accounts to pp msgs
     positions: dict[
-        # brokername, acctid ->
+        # brokername, acctid
         tuple[str, str],
-        # fqme -> msg
-        dict[str, BrokerdPosition],
+        list[BrokerdPosition],
     ]
 
     # allowed account names
     accounts: tuple[str]
 
 
-@acm
-async def open_brokerd_dialog(
-    brokermod: ModuleType,
-    portal: tractor.Portal,
-    exec_mode: str,
-    fqme: str | None = None,
-    loglevel: str | None = None,
-
-) -> tuple[
-    tractor.MsgStream,
-    # {(brokername, accountname) -> {fqme -> msg}}
-    dict[(str, str), dict[str, BrokerdPosition]],
-    list[str],
-]:
-    '''
-    Open either a live trades control dialog or a dialog with a new
-    paper engine instance depending on live trading support for the
-    broker backend, configuration, or client code usage.
-
-    '''
-    broker: str = brokermod.name
-
-    def mk_paper_ep():
-        from . import _paper_engine as paper_mod
-
-        nonlocal brokermod, exec_mode
-
-        # for logging purposes
-        brokermod = paper_mod
-
-        # for paper mode we need to mock this trades response feed
-        # so we load bidir stream to a new sub-actor running
-        # a paper-simulator clearing engine.
-
-        # load the paper trading engine
-        log.info(f'{broker}: Entering `paper` trading mode')
-
-        # load the paper trading engine as a subactor of this emsd
-        # actor to simulate the real IPC load it'll have when also
-        # pulling data from feeds
-        if not fqme:
-            log.warning(
-                f'Paper engine activate for {broker} but no fqme provided?'
-            )
-
-        return paper_mod.open_paperboi(
-            fqme=fqme,
-            broker=broker,
-            loglevel=loglevel,
-        )
-
-    # take the first supported ep we detect
-    # on the backend mod.
-    trades_endpoint: Callable
-    for ep_name in [
-        'open_trade_dialog',  # probably final name?
-        'trades_dialogue',  # legacy
-    ]:
-        trades_endpoint = getattr(
-            brokermod,
-            ep_name,
-            None,
-        )
-        if trades_endpoint:
-            break
-    else:
-        log.warning(
-            f'No live trading EP found: {brokermod.name}?'
-        )
-        exec_mode: str = 'paper'
-
-    if (
-        trades_endpoint is not None
-        or exec_mode != 'paper'
-    ):
-        # open live brokerd trades endpoint
-        open_trades_endpoint = portal.open_context(
-            trades_endpoint,
-        )
-
-    @acm
-    async def maybe_open_paper_ep():
-        if exec_mode == 'paper':
-            async with mk_paper_ep() as msg:
-                yield msg
-                return
-
-        # open trades-dialog endpoint with backend broker
-        async with open_trades_endpoint as msg:
-            ctx, first = msg
-
-            # runtime indication that the backend can't support live
-            # order ctrl yet, so boot the paperboi B0
-            if first == 'paper':
-                async with mk_paper_ep() as msg:
-                    yield msg
-                    return
-            else:
-                # working live ep case B)
-                yield msg
-                return
-
-    pps_by_broker_account: dict[(str, str), BrokerdPosition] = {}
-
-    async with (
-        maybe_open_paper_ep() as (
-            brokerd_ctx,
-            (position_msgs, accounts),
-        ),
-        brokerd_ctx.open_stream() as brokerd_trades_stream,
-    ):
-        # XXX: really we only want one stream per `emsd`
-        # actor to relay global `brokerd` order events
-        # unless we're going to expect each backend to
-        # relay only orders affiliated with a particular
-        # ``trades_dialogue()`` session (seems annoying
-        # for implementers). So, here we cache the relay
-        # task and instead of running multiple tasks
-        # (which will result in multiples of the same
-        # msg being relayed for each EMS client) we just
-        # register each client stream to this single
-        # relay loop in the dialog table.
-
-        # begin processing order events from the target
-        # brokerd backend by receiving order submission
-        # response messages, normalizing them to EMS
-        # messages and relaying back to the piker order
-        # client set.
-
-        # locally cache and track positions per account with
-        # a nested table of msgs:
-        #  tuple(brokername, acctid) ->
-        #      (fqme: str ->
-        #           `BrokerdPosition`)
-        for msg in position_msgs:
-
-            msg = BrokerdPosition(**msg)
-            log.info(
-                f'loading pp for {brokermod.__name__}:\n'
-                f'{pformat(msg.to_dict())}',
-            )
-
-            # TODO: state any mismatch here?
-            account: str = msg.account
-            assert account in accounts
-
-            pps_by_broker_account.setdefault(
-                (broker, account),
-                {},
-            )[msg.symbol] = msg
-
-        # should be unique entries, verdad!
-        assert len(set(accounts)) == len(accounts)
-
-        yield (
-            brokerd_trades_stream,
-            pps_by_broker_account,
-            accounts,
-        )
-
-
 class Router(Struct):
     '''
     Order router which manages and tracks per-broker dark book,
@@ -522,7 +347,6 @@ class Router(Struct):
     ] = defaultdict(set)
 
     # TODO: mapping of ems dialog ids to msg flow history
-    # - use the new ._util.OrderDialogs?
     # msgflows: defaultdict[
     #     str,
     #     ChainMap[dict[str, dict]],
@@ -583,25 +407,118 @@ class Router(Struct):
             yield relay
             return
 
-        async with open_brokerd_dialog(
-            brokermod=brokermod,
-            portal=portal,
-            exec_mode=exec_mode,
-            fqme=fqme,
-            loglevel=loglevel,
+        def mk_paper_ep():
+            nonlocal brokermod, exec_mode
 
-        ) as (
-            brokerd_stream,
-            pp_msg_table,
-            accounts,
-        ):
-            # create a new relay and sync it's state according
-            # to brokerd-backend reported position msgs.
-            relay = TradesRelay(
-                brokerd_stream=brokerd_stream,
-                positions=pp_msg_table,
-                accounts=tuple(accounts),
+            # for logging purposes
+            brokermod = paper
+
+            # for paper mode we need to mock this trades response feed
+            # so we load bidir stream to a new sub-actor running
+            # a paper-simulator clearing engine.
+
+            # load the paper trading engine
+            exec_mode = 'paper'
+            log.info(f'{broker}: Entering `paper` trading mode')
+
+            # load the paper trading engine as a subactor of this emsd
+            # actor to simulate the real IPC load it'll have when also
+            # pulling data from feeds
+            return paper.open_paperboi(
+                fqme=fqme,
+                loglevel=loglevel,
             )
+
+        trades_endpoint = getattr(brokermod, 'trades_dialogue', None)
+        if (
+            trades_endpoint is not None
+            or exec_mode != 'paper'
+        ):
+            # open live brokerd trades endpoint
+            open_trades_endpoint = portal.open_context(
+                trades_endpoint,
+                loglevel=loglevel,
+            )
+
+        else:
+            exec_mode: str = 'paper'
+
+        @acm
+        async def maybe_open_paper_ep():
+            if exec_mode == 'paper':
+                async with mk_paper_ep() as msg:
+                    yield msg
+                    return
+
+            # open trades-dialog endpoint with backend broker
+            async with open_trades_endpoint as msg:
+                ctx, first = msg
+
+                # runtime indication that the backend can't support live
+                # order ctrl yet, so boot the paperboi B0
+                if first == 'paper':
+                    async with mk_paper_ep() as msg:
+                        yield msg
+                        return
+                else:
+                    # working live ep case B)
+                    yield msg
+                    return
+
+        positions: list[BrokerdPosition]
+        accounts: tuple[str]
+        async with (
+            maybe_open_paper_ep() as (
+                brokerd_ctx,
+                (positions, accounts),
+            ),
+            brokerd_ctx.open_stream() as brokerd_trades_stream,
+        ):
+            # XXX: really we only want one stream per `emsd`
+            # actor to relay global `brokerd` order events
+            # unless we're going to expect each backend to
+            # relay only orders affiliated with a particular
+            # ``trades_dialogue()`` session (seems annoying
+            # for implementers). So, here we cache the relay
+            # task and instead of running multiple tasks
+            # (which will result in multiples of the same
+            # msg being relayed for each EMS client) we just
+            # register each client stream to this single
+            # relay loop in the dialog table.
+
+            # begin processing order events from the target
+            # brokerd backend by receiving order submission
+            # response messages, normalizing them to EMS
+            # messages and relaying back to the piker order
+            # client set.
+
+            # locally cache and track positions per account with
+            # a nested table of msgs:
+            #  tuple(brokername, acctid) ->
+            #      (fqme: str ->
+            #           `BrokerdPosition`)
+            relay = TradesRelay(
+                brokerd_stream=brokerd_trades_stream,
+                positions={},
+                accounts=accounts,
+            )
+            for msg in positions:
+
+                msg = BrokerdPosition(**msg)
+                log.info(
+                    f'loading pp for {brokermod.__name__}:\n'
+                    f'{pformat(msg.to_dict())}',
+                )
+
+                # TODO: state any mismatch here?
+                account = msg.account
+                assert account in accounts
+
+                relay.positions.setdefault(
+                    (broker, account),
+                    {},
+                )[msg.symbol] = msg
+
             self.relays[broker] = relay
 
             # this context should block here indefinitely until
@@ -633,17 +550,12 @@ class Router(Struct):
         indefinitely.
 
         '''
-        from ..data.feed import maybe_open_feed
-
         async with (
             maybe_open_feed(
                 [fqme],
                 loglevel=loglevel,
             ) as feed,
         ):
-            # extract expanded fqme in case input was of a less
-            # qualified form, eg. xbteur.kraken -> xbteur.spot.kraken
-            fqme: str = list(feed.flumes.keys())[0]
             brokername, _, _, _ = unpack_fqme(fqme)
             brokermod = feed.mods[brokername]
             broker = brokermod.name
@@ -678,7 +590,7 @@ class Router(Struct):
 
                 client_ready = trio.Event()
                 task_status.started(
-                    (fqme, relay, feed, client_ready)
+                    (relay, feed, client_ready)
                 )
 
                 # sync to the client side by waiting for the stream
@@ -827,8 +739,8 @@ async def translate_and_relay_brokerd_events(
                 # keep pps per account up to date locally in ``emsd`` mem
                 # sym, broker = pos_msg.symbol, pos_msg.broker
 
-                # NOTE: translate to a FQME!
                 relay.positions.setdefault(
+                    # NOTE: translate to a FQSN!
                     (broker, pos_msg.account),
                     {}
                 )[pos_msg.symbol] = pos_msg
@@ -884,7 +796,7 @@ async def translate_and_relay_brokerd_events(
                         BrokerdCancel(
                             oid=oid,
                             reqid=reqid,
-                            time_ns=time_ns(),
+                            time_ns=time.time_ns(),
                             account=status_msg.req.account,
                         )
                     )
@@ -899,75 +811,38 @@ async def translate_and_relay_brokerd_events(
                 continue
 
             # BrokerdError
-            # TODO: figure out how this will interact with EMS clients
-            # for ex. on an error do we react with a dark orders
-            # management response, like cancelling all dark orders?
-            # This looks like a supervision policy for pending orders on
-            # some unexpected failure - something we need to think more
-            # about.  In most default situations, with composed orders
-            # (ex.  brackets), most brokers seem to use a oca policy.
             case {
                 'name': 'error',
                 'oid': oid,  # ems order-dialog id
                 'reqid': reqid,  # brokerd generated order-request id
             }:
-                if (
-                    not oid
-                    # try to lookup any order dialog by
-                    # brokerd-side id..
-                    and not (
-                        oid := book._ems2brokerd_ids.inverse.get(reqid)
-                    )
-                ):
-                    log.warning(
-                        f'Rxed unusable error-msg:\n'
-                        f'{brokerd_msg}'
-                    )
-                    continue
-
+                status_msg = book._active.get(oid)
                 msg = BrokerdError(**brokerd_msg)
+                log.error(fmsg)  # XXX make one when it's blank?
 
-                # NOTE: retreive the last client-side response
-                # OR create an error when we have no last msg /dialog
-                # on record
-                status_msg: Status
-                if not (status_msg := book._active.get(oid)):
-                    status_msg = Error(
-                        time_ns=time_ns(),
-                        oid=oid,
-                        reqid=reqid,
-                        brokerd_msg=msg,
-                    )
-                else:
-                    # only modify last status if we have an active
-                    # ongoing dialog..
+                # TODO: figure out how this will interact with EMS clients
+                # for ex. on an error do we react with a dark orders
+                # management response, like cancelling all dark orders?
+                # This looks like a supervision policy for pending orders on
+                # some unexpected failure - something we need to think more
+                # about.  In most default situations, with composed orders
+                # (ex.  brackets), most brokers seem to use a oca policy.
+
+                # only relay to client side if we have an active
+                # ongoing dialog
+                if status_msg:
                     status_msg.resp = 'error'
                     status_msg.brokerd_msg = msg
+                    book._active[oid] = status_msg
 
-                book._active[oid] = status_msg
-
-                log.error(
-                    'Translating brokerd error to status:\n'
-                    f'{fmsg}'
-                    f'{status_msg.to_dict()}'
-                )
-                if req := status_msg.req:
-                    fqme: str = req.symbol
-                else:
-                    bdmsg: Struct = status_msg.brokerd_msg
-                    fqme: str = (
-                        bdmsg.symbol  # might be None
-                        or
-                        bdmsg.broker_details['flow']
-                        # NOTE: what happens in empty case in the
-                        # broadcast below? it's a problem?
-                        .get('symbol', '')
+                    await router.client_broadcast(
+                        status_msg.req.symbol,
+                        status_msg,
                     )
 
-                await router.client_broadcast(
-                    fqme,
-                    status_msg,
-                )
+                else:
+                    log.error(f'Error for unknown order flow:\n{msg}')
+                    continue
 
             # BrokerdStatus
             case {
@@ -1108,7 +983,7 @@ async def translate_and_relay_brokerd_events(
                     status_msg.req = order
 
                     assert status_msg.src  # source tag?
-                    oid: str = str(status_msg.reqid)
+                    oid = str(status_msg.reqid)
 
                     # attempt to avoid collisions
                     status_msg.reqid = oid
@@ -1125,28 +1000,38 @@ async def translate_and_relay_brokerd_events(
                         status_msg,
                     )
 
+                # don't fall through
+                continue
+
+            # brokerd error
+            case {
+                'name': 'status',
+                'status': 'error',
+            }:
+                log.error(f'Broker error:\n{fmsg}')
+                # XXX: we presume the brokerd cancels its own order
+                continue
+
             # TOO FAST ``BrokerdStatus`` that arrives
             # before the ``BrokerdAck``.
-            # NOTE XXX: sometimes there is a race with the backend (like
-            # `ib` where the pending status will be relayed *before*
-            # the ack msg, in which case we just ignore the faster
-            # pending msg and wait for our expected ack to arrive
-            # later (i.e. the first block below should enter).
             case {
+                # XXX: sometimes there is a race with the backend (like
+                # `ib` where the pending stauts will be related before
+                # the ack, in which case we just ignore the faster
+                # pending msg and wait for our expected ack to arrive
+                # later (i.e. the first block below should enter).
                 'name': 'status',
                 'status': status,
                 'reqid': reqid,
             }:
-                msg = (
-                    f'Unhandled broker status for dialog {reqid}:\n'
-                    f'{pformat(brokerd_msg)}'
-                )
-                if (
-                    oid := book._ems2brokerd_ids.inverse.get(reqid)
-                ):
+                oid = book._ems2brokerd_ids.inverse.get(reqid)
+                msg = f'Unhandled broker status for dialog {reqid}:\n'
+                if oid:
+                    status_msg = book._active.get(oid)
+                    # status msg may not have been set yet or popped?
                     # NOTE: have seen a key error here on kraken
                     # clearable limits..
-                    if status_msg := book._active.get(oid):
+                    if status_msg:
                         msg += (
                             f'last status msg: {pformat(status_msg)}\n\n'
                             f'this msg:{fmsg}\n'
@@ -1242,7 +1127,7 @@ async def process_client_order_cmds(
                         BrokerdCancel(
                             oid=oid,
                             reqid=reqid,
-                            time_ns=time_ns(),
+                            time_ns=time.time_ns(),
                             account=order.account,
                         )
                     )
@@ -1256,15 +1141,14 @@ async def process_client_order_cmds(
                 and status.resp == 'dark_open'
             ):
                 # remove from dark book clearing
-                entry: tuple | None = dark_book.triggers[fqme].pop(oid, None)
+                entry = dark_book.triggers[fqme].pop(oid, None)
                 if entry:
                     (
                         pred,
                         tickfilter,
                         cmd,
                         percent_away,
-                        abs_diff_away,
-                        min_tick_digits,
+                        abs_diff_away
                     ) = entry
 
                     # tell client side that we've cancelled the
@@ -1317,7 +1201,7 @@ async def process_client_order_cmds(
 
                 msg = BrokerdOrder(
                     oid=oid,  # no ib support for oids...
-                    time_ns=time_ns(),
+                    time_ns=time.time_ns(),
 
                     # if this is None, creates a new order
                     # otherwise will modify any existing one
@@ -1335,7 +1219,7 @@ async def process_client_order_cmds(
                         oid=oid,
                         reqid=reqid,
                         resp='pending',
-                        time_ns=time_ns(),
+                        time_ns=time.time_ns(),
                         brokerd_msg=msg,
                         req=req,
                     )
@@ -1399,36 +1283,33 @@ async def process_client_order_cmds(
                 # TODO: make this configurable from our top level
                 # config, prolly in a .clearing` section?
                 spread_slap: float = 5
-                min_tick = Decimal(flume.mkt.price_tick)
-                min_tick_digits: int = dec_digits(min_tick)
-
-                tickfilter: tuple[str, ...]
-                percent_away: float
+                min_tick = float(flume.mkt.size_tick)
+                min_tick_digits = float_digits(min_tick)
 
                 if action == 'buy':
                     tickfilter = ('ask', 'last', 'trade')
-                    percent_away: float = 0.005
+                    percent_away = 0.005
 
                     # TODO: we probably need to scale this based
                     # on some near term historical spread
                     # measure?
-                    abs_diff_away = float(round(
+                    abs_diff_away = round(
                         spread_slap * min_tick,
                         ndigits=min_tick_digits,
-                    ))
+                    )
 
                 elif action == 'sell':
                     tickfilter = ('bid', 'last', 'trade')
-                    percent_away: float = -0.005
-                    abs_diff_away: float = float(round(
+                    percent_away = -0.005
+                    abs_diff_away = round(
                         -spread_slap * min_tick,
                         ndigits=min_tick_digits,
-                    ))
+                    )
 
                 else:  # alert
                     tickfilter = ('trade', 'utrade', 'last')
-                    percent_away: float = 0
-                    abs_diff_away: float = 0
+                    percent_away = 0
+                    abs_diff_away = 0
 
                 # submit execution/order to EMS scan loop
                 # NOTE: this may result in an override of an existing
@@ -1440,8 +1321,7 @@ async def process_client_order_cmds(
                     tickfilter,
                     req,
                     percent_away,
-                    abs_diff_away,
-                    min_tick_digits,
+                    abs_diff_away
                 )
                 resp = 'dark_open'
 
@@ -1452,7 +1332,7 @@ async def process_client_order_cmds(
                 status = Status(
                     resp=resp,
                     oid=oid,
-                    time_ns=time_ns(),
+                    time_ns=time.time_ns(),
                     req=req,
                     src='dark',
                 )
@@ -1498,13 +1378,13 @@ async def maybe_open_trade_relays(
         loglevel: str = 'info',
     ):
 
-        fqme, relay, feed, client_ready = await _router.nursery.start(
+        relay, feed, client_ready = await _router.nursery.start(
             _router.open_trade_relays,
             fqme,
             exec_mode,
             loglevel,
         )
-        yield fqme, relay, feed, client_ready
+        yield relay, feed, client_ready
 
     async with tractor.trionics.maybe_open_context(
         acm_func=cached_mngr,
@@ -1517,13 +1397,13 @@ async def maybe_open_trade_relays(
         key=cache_on_fqme_unless_paper,
     ) as (
         cache_hit,
-        (fqme, relay, feed, client_ready)
+        (relay, feed, client_ready)
     ):
         if cache_hit:
             log.info(f'Reusing existing trades relay for {fqme}:\n'
                      f'{relay}\n')
 
-        yield fqme, relay, feed, client_ready
+        yield relay, feed, client_ready
 
 
 @tractor.context
@@ -1557,34 +1437,30 @@ async def _emsd_main(
     received in a stream from that client actor and then responses are
     streamed back up to the original calling task in the same client.
 
-    The primary ``emsd`` task tree is:
+    The primary ``emsd`` task trees are:
 
     - ``_setup_persistent_emsd()``:
-      is the ``emsd`` actor's primary *service-fixture* task which
-      is opened by the `pikerd` service manager and sets up
-      a process-global (actor-local) ``Router`` instance and opens
-      a service nursery which lives until the backend broker is
-      shutdown or the ems is terminated; all tasks are
-      *dynamically* started (and persisted) within this service
-      nursery when the below endpoint context is opened:
-        |
-        - ``_emsd_main()``:
-          attaches a real-time quote feed and trades dialogue with
-          a `brokerd` actor which connects to the backend broker's
-          trading api for every connecting client.
-           |
-            - ``clear_dark_triggers()``:
-              run (dark order) conditions on inputs and trigger brokerd
-              "live" order submissions.
-           |
-            - ``process_client_order_cmds()``:
-              accepts order cmds from requesting clients, registers
-              dark orders and alerts with above (dark) clearing loop.
-           |
-            - (maybe) ``translate_and_relay_brokerd_events()``:
-              accept normalized trades responses from brokerd, process and
-              relay to ems client(s); this is a effectively a "trade event
-              reponse" proxy-broker.
+      is the ``emsd`` actor's primary root task which sets up an
+      actor-global ``Router`` instance and starts a relay loop task
+      which lives until the backend broker is shutdown or the ems is
+      terminated.
+       |
+        - (maybe) ``translate_and_relay_brokerd_events()``:
+          accept normalized trades responses from brokerd, process and
+          relay to ems client(s); this is a effectively a "trade event
+          reponse" proxy-broker.
+
+    - ``_emsd_main()``:
+      attaches a brokerd real-time quote feed and trades dialogue with
+      brokderd trading api for every connecting client.
+       |
+        - ``clear_dark_triggers()``:
+          run (dark order) conditions on inputs and trigger brokerd "live"
+          order submissions.
+       |
+        - ``process_client_order_cmds()``:
+          accepts order cmds from requesting clients, registers dark orders and
+          alerts with clearing loop.
 
     '''
     global _router
@@ -1592,9 +1468,9 @@ async def _emsd_main(
 
     broker, _, _, _ = unpack_fqme(fqme)
 
-    # TODO: would be nice if in tractor we can require either a ctx
-    # arg, or a named arg with ctx in it and a type annotation of
-    # `tractor.Context` instead of strictly requiring a ctx arg.
+    # TODO: would be nice if in tractor we can require either a ctx arg,
+    # or a named arg with ctx in it and a type annotation of
+    # tractor.Context instead of strictly requiring a ctx arg.
     ems_ctx = ctx
 
     # spawn one task per broker feed
@@ -1610,7 +1486,7 @@ async def _emsd_main(
         fqme,
         exec_mode,
         loglevel,
-    ) as (fqme, relay, feed, client_ready):
+    ) as (relay, feed, client_ready):
 
         brokerd_stream = relay.brokerd_stream
         dark_book = _router.get_dark_book(broker)

piker/clearing/_messages.py

@@ -18,14 +18,40 @@
 Clearing sub-system message and protocols.
 
 """
-from __future__ import annotations
+# from collections import (
+#     ChainMap,
+#     deque,
+# )
 from typing import (
+    Optional,
     Literal,
 )
 
 from msgspec import field
 
-from piker.types import Struct
+from ..data.types import Struct
+
+
+# TODO: a composite for tracking msg flow on 2-legged
+# dialogs.
+# class Dialog(ChainMap):
+#     '''
+#     Msg collection abstraction to easily track the state changes of
+#     a msg flow in one high level, query-able and immutable construct.
+
+#     The main use case is to query data from a (long-running)
+#     msg-transaction-sequence
+
+
+#     '''
+#     def update(
+#         self,
+#         msg,
+#     ) -> None:
+#         self.maps.insert(0, msg.to_dict())
+
+#     def flatten(self) -> dict:
+#         return dict(self)
 
 
 # TODO: ``msgspec`` stuff worth paying attention to:
@@ -114,7 +140,7 @@ class Status(Struct):
 
     # this maps normally to the ``BrokerdOrder.reqid`` below, an id
     # normally allocated internally by the backend broker routing system
-    reqid: int | str | None = None
+    reqid: Optional[int | str] = None
 
     # the (last) source order/request msg if provided
     # (eg. the Order/Cancel which causes this msg) and
@@ -127,7 +153,7 @@ class Status(Struct):
     # event that wasn't originated by piker's emsd (eg. some external
     # trading system which does it's own order control but that you
     # might want to "track" using piker UIs/systems).
-    src: str | None = None
+    src: Optional[str] = None
 
     # set when a cancel request msg was set for this order flow dialog
     # but the brokerd dialog isn't yet in a cancelled state.
@@ -138,18 +164,6 @@ class Status(Struct):
     brokerd_msg: dict = {}
 
 
-class Error(Status):
-    resp: str = 'error'
-
-    # TODO: allow re-wrapping from existing (last) status?
-    @classmethod
-    def from_status(
-        cls,
-        msg: Status,
-    ) -> Error:
-        ...
-
-
 # ---------------
 # emsd -> brokerd
 # ---------------
@@ -167,7 +181,7 @@ class BrokerdCancel(Struct):
     # for setting a unique order id then this value will be relayed back
     # on the emsd order request stream as the ``BrokerdOrderAck.reqid``
     # field
-    reqid: int | str | None = None
+    reqid: Optional[int | str] = None
     action: str = 'cancel'
 
 
@@ -191,7 +205,7 @@ class BrokerdOrder(Struct):
     # for setting a unique order id then this value will be relayed back
     # on the emsd order request stream as the ``BrokerdOrderAck.reqid``
     # field
-    reqid: int | str | None = None
+    reqid: Optional[int | str] = None
 
 
 # ---------------
@@ -213,27 +227,24 @@ class BrokerdOrderAck(Struct):
 
     # emsd id originally sent in matching request msg
     oid: str
-    # TODO: do we need this?
     account: str = ''
     name: str = 'ack'
 
 
 class BrokerdStatus(Struct):
 
-    time_ns: int
     reqid: int | str
+    time_ns: int
     status: Literal[
         'open',
         'canceled',
+        'fill',
         'pending',
-        # 'error',  # NOTE: use `BrokerdError`
-        'closed',
+        'error',
     ]
-    name: str = 'status'
 
-    oid: str = ''
-    # TODO: do we need this?
-    account: str | None = None,
+    account: str
+    name: str = 'status'
     filled: float = 0.0
     reason: str = ''
     remaining: float = 0.0
@@ -248,24 +259,24 @@ class BrokerdStatus(Struct):
 
 class BrokerdFill(Struct):
     '''
-    A single message indicating a "fill-details" event from the
-    broker if avaiable.
+    A single message indicating a "fill-details" event from the broker
+    if avaiable.
 
     '''
     # brokerd timestamp required for order mode arrow placement on x-axis
     # TODO: maybe int if we force ns?
     # we need to normalize this somehow since backends will use their
     # own format and likely across many disparate epoch clocks...
-    time_ns: int
     broker_time: float
     reqid: int | str
+    time_ns: int
 
     # order exeuction related
     size: float
     price: float
 
     name: str = 'fill'
-    action: str | None = None
+    action: Optional[str] = None
     broker_details: dict = {}  # meta-data (eg. commisions etc.)
 
 
@@ -276,22 +287,18 @@ class BrokerdError(Struct):
     This is still a TODO thing since we're not sure how to employ it yet.
 
     '''
+    oid: str
+    symbol: str
     reason: str
 
-    # TODO: drop this right?
-    symbol: str | None = None
-
-    oid: str | None = None
     # if no brokerd order request was actually submitted (eg. we errored
     # at the ``pikerd`` layer) then there will be ``reqid`` allocated.
-    reqid: str | None = None
+    reqid: Optional[int | str] = None
 
     name: str = 'error'
     broker_details: dict = {}
 
 
-# TODO: yeah, so we REALLY need to completely deprecate
-# this and use the `.accounting.Position` msg-type instead..
 class BrokerdPosition(Struct):
     '''
     Position update event from brokerd.

piker/clearing/_paper_engine.py

@@ -26,12 +26,10 @@ from contextlib import asynccontextmanager as acm
 from datetime import datetime
 from operator import itemgetter
 import itertools
-from pprint import pformat
 import time
 from typing import (
     Callable,
 )
-from types import ModuleType
 import uuid
 
 from bidict import bidict
@@ -39,29 +37,24 @@ import pendulum
 import trio
 import tractor
 
-from piker.brokers import get_brokermod
-from piker.service import find_service
-from piker.accounting import (
-    Account,
+from ..brokers import get_brokermod
+from .. import data
+from ..data.types import Struct
+from ..accounting._mktinfo import (
     MktPair,
+)
+from ..accounting import (
     Position,
+    PpTable,
     Transaction,
     TransactionLedger,
-    open_account,
     open_trade_ledger,
-    unpack_fqme,
+    open_pps,
 )
-from piker.data import (
-    Feed,
-    SymbologyCache,
-    iterticks,
-    open_feed,
-    open_symcache,
-)
-from piker.types import Struct
+from ..data._normalize import iterticks
+from ..accounting import unpack_fqme
 from ._util import (
     log,  # sub-sys logger
-    get_console_log,
 )
 from ._messages import (
     BrokerdCancel,
@@ -83,10 +76,11 @@ class PaperBoi(Struct):
 
     '''
     broker: str
+
     ems_trades_stream: tractor.MsgStream
-    acnt: Account
+
+    ppt: PpTable
     ledger: TransactionLedger
-    fees: Callable
 
     # map of paper "live" orders which be used
     # to simulate fills based on paper engine settings
@@ -130,7 +124,7 @@ class PaperBoi(Struct):
         # for dark orders since we want the dark_executed
         # to trigger first thus creating a lookup entry
         # in the broker trades event processing loop
-        await trio.sleep(0.01)
+        await trio.sleep(0.05)
 
         if (
             action == 'sell'
@@ -197,7 +191,7 @@ class PaperBoi(Struct):
             self._sells[symbol].pop(oid, None)
 
         # TODO: net latency model
-        await trio.sleep(0.01)
+        await trio.sleep(0.05)
 
         msg = BrokerdStatus(
             status='canceled',
@@ -230,7 +224,7 @@ class PaperBoi(Struct):
 
         '''
         # TODO: net latency model
-        await trio.sleep(0.01)
+        await trio.sleep(0.05)
         fill_time_ns = time.time_ns()
         fill_time_s = time.time()
 
@@ -268,42 +262,29 @@ class PaperBoi(Struct):
         # we don't actually have any unique backend symbol ourselves
         # other then this thing, our fqme address.
         bs_mktid: str = fqme
-        if fees := self.fees:
-            cost: float = fees(price, size)
-        else:
-            cost: float = 0
-
         t = Transaction(
             fqme=fqme,
+            sym=self._mkts[fqme],
             tid=oid,
             size=size,
             price=price,
-            cost=cost,
+            cost=0,  # TODO: cost model
             dt=pendulum.from_timestamp(fill_time_s),
             bs_mktid=bs_mktid,
         )
 
         # update in-mem ledger and pos table
         self.ledger.update_from_t(t)
-        self.acnt.update_from_ledger(
-            {oid: t},
-            symcache=self.ledger._symcache,
-
-            # XXX when a backend has no symcache support yet we can
-            # simply pass in the gmi() retreived table created
-            # during init :o
-            _mktmap_table=self._mkts,
-        )
+        self.ppt.update_from_trans({oid: t})
 
         # transmit pp msg to ems
-        pp: Position = self.acnt.pps[bs_mktid]
-
+        pp = self.ppt.pps[bs_mktid]
         pp_msg = BrokerdPosition(
             broker=self.broker,
             account='paper',
             symbol=fqme,
 
-            size=pp.cumsize,
+            size=pp.size,
             avg_price=pp.ppu,
 
             # TODO: we need to look up the asset currency from
@@ -314,7 +295,7 @@ class PaperBoi(Struct):
         # write all updates to filesys immediately
         # (adds latency but that works for simulation anyway)
         self.ledger.write_config()
-        self.acnt.write_config()
+        self.ppt.write_config()
 
         await self.ems_trades_stream.send(pp_msg)
 
@@ -343,7 +324,6 @@ async def simulate_fills(
     # this stream may eventually contain multiple symbols
     async for quotes in quote_stream:
         for sym, quote in quotes.items():
-            # print(sym)
             for tick in iterticks(
                 quote,
                 # dark order price filter(s)
@@ -547,7 +527,7 @@ _sells: defaultdict[
 
 
 @tractor.context
-async def open_trade_dialog(
+async def trades_dialogue(
 
     ctx: tractor.Context,
     broker: str,
@@ -556,188 +536,141 @@ async def open_trade_dialog(
 
 ) -> None:
 
-    # enable piker.clearing console log for *this* subactor
-    get_console_log(loglevel)
+    tractor.log.get_console_log(loglevel)
 
-    symcache: SymbologyCache
-    async with open_symcache(get_brokermod(broker)) as symcache:
+    ppt: PpTable
+    ledger: TransactionLedger
+    with (
+        open_pps(
+            broker,
+            'paper',
+            write_on_exit=True,
+        ) as ppt,
 
-        acnt: Account
-        ledger: TransactionLedger
-        with (
+        open_trade_ledger(
+            broker,
+            'paper',
+        ) as ledger
+    ):
+        # NOTE: retreive market(pair) info from the backend broker
+        # since ledger entries (in their backend native format) often
+        # don't contain necessary market info per trade record entry..
+        # - if no fqme was passed in, we presume we're running in
+        #   "ledger-sync-only mode" and thus we load mkt info for
+        #   each symbol found in the ledger to a ppt table manually.
 
-            # TODO: probably do the symcache and ledger loading
-            # implicitly behind this? Deliver an account, and ledger
-            # pair or make the ledger an attr of the account?
-            open_account(
-                broker,
-                'paper',
-                write_on_exit=True,
-            ) as acnt,
+        # TODO: how to process ledger info from backends?
+        # - should we be rolling our own actor-cached version of these
+        # client API refs or using portal IPC to send requests to the
+        # existing brokerd daemon?
+        # - alternatively we can possibly expect and use
+        # a `.broker.norm_trade_records()` ep?
+        brokermod = get_brokermod(broker)
+        gmi = getattr(brokermod, 'get_mkt_info', None)
 
-            open_trade_ledger(
-                broker,
-                'paper',
-                symcache=symcache,
-            ) as ledger
-        ):
-            # NOTE: WE MUST retreive market(pair) info from each
-            # backend broker since ledger entries (in their
-            # provider-native format) often don't contain necessary
-            # market info per trade record entry..
-            # FURTHER, if no fqme was passed in, we presume we're
-            # running in "ledger-sync-only mode" and thus we load
-            # mkt info for each symbol found in the ledger to
-            # an acnt table manually.
+        # update all transactions with mkt info before
+        # loading any pps
+        mkt_by_fqme: dict[str, MktPair] = {}
+        if fqme:
+            bs_fqme, _, broker = fqme.rpartition('.')
+            mkt, _ = await brokermod.get_mkt_info(bs_fqme)
+            mkt_by_fqme[fqme] = mkt
 
-            # TODO: how to process ledger info from backends?
-            # - should we be rolling our own actor-cached version of these
-            #   client API refs or using portal IPC to send requests to the
-            #   existing brokerd daemon?
-            # - alternatively we can possibly expect and use
-            #   a `.broker.ledger.norm_trade()` ep?
-            brokermod: ModuleType = get_brokermod(broker)
-            gmi: Callable = getattr(brokermod, 'get_mkt_info', None)
+        # for each sym in the ledger load it's `MktPair` info
+        for tid, txdict in ledger.data.items():
+            l_fqme: str = txdict.get('fqme') or txdict['fqsn']
 
-            # update all transactions with mkt info before
-            # loading any pps
-            mkt_by_fqme: dict[str, MktPair] = {}
             if (
-                fqme
-                and fqme not in symcache.mktmaps
+                gmi
+                and l_fqme not in mkt_by_fqme
             ):
-                log.warning(
-                    f'Symcache for {broker} has no `{fqme}` entry?\n'
-                    'Manually requesting mkt map data via `.get_mkt_info()`..'
+                mkt, pair = await brokermod.get_mkt_info(
+                    l_fqme.rstrip(f'.{broker}'),
                 )
+                mkt_by_fqme[l_fqme] = mkt
 
-                bs_fqme, _, broker = fqme.rpartition('.')
-                mkt, pair = await gmi(bs_fqme)
-                mkt_by_fqme[mkt.fqme] = mkt
+            # if an ``fqme: str`` input was provided we only
+            # need a ``MktPair`` for that one market, since we're
+            # running in real simulated-clearing mode, not just ledger
+            # syncing.
+            if (
+                fqme is not None
+                and fqme in mkt_by_fqme
+            ):
+                break
 
-            # for each sym in the ledger load its `MktPair` info
-            for tid, txdict in ledger.data.items():
-                l_fqme: str = txdict.get('fqme') or txdict['fqsn']
+        # update pos table from ledger history and provide a ``MktPair``
+        # lookup for internal position accounting calcs.
+        ppt.update_from_trans(ledger.to_trans(mkt_by_fqme=mkt_by_fqme))
 
-                if (
-                    gmi
-                    and l_fqme not in symcache.mktmaps
-                    and l_fqme not in mkt_by_fqme
-                ):
-                    log.warning(
-                        f'Symcache for {broker} has no `{l_fqme}` entry?\n'
-                        'Manually requesting mkt map data via `.get_mkt_info()`..'
-                    )
-                    mkt, pair = await gmi(
-                        l_fqme.rstrip(f'.{broker}'),
-                    )
-                    mkt_by_fqme[l_fqme] = mkt
-
-                # if an ``fqme: str`` input was provided we only
-                # need a ``MktPair`` for that one market, since we're
-                # running in real simulated-clearing mode, not just ledger
-                # syncing.
-                if (
-                    fqme is not None
-                    and fqme in mkt_by_fqme
-                ):
-                    break
-
-            # update pos table from ledger history and provide a ``MktPair``
-            # lookup for internal position accounting calcs.
-            acnt.update_from_ledger(
-                ledger,
-
-                # NOTE: if the symcache fails on fqme lookup
-                # (either sycache not yet supported or not filled
-                # in) use manually constructed table from calling
-                # the `.get_mkt_info()` provider EP above.
-                _mktmap_table=mkt_by_fqme,
-            )
-
-            pp_msgs: list[BrokerdPosition] = []
-            pos: Position
-            token: str  # f'{symbol}.{self.broker}'
-            for token, pos in acnt.pps.items():
-
-                pp_msgs.append(BrokerdPosition(
-                    broker=broker,
-                    account='paper',
-                    symbol=pos.mkt.fqme,
-                    size=pos.cumsize,
-                    avg_price=pos.ppu,
-                ))
-
-            await ctx.started((
-                pp_msgs,
-                ['paper'],
+        pp_msgs: list[BrokerdPosition] = []
+        pos: Position
+        token: str  # f'{symbol}.{self.broker}'
+        for token, pos in ppt.pps.items():
+            pp_msgs.append(BrokerdPosition(
+                broker=broker,
+                account='paper',
+                symbol=pos.mkt.fqme,
+                size=pos.size,
+                avg_price=pos.ppu,
             ))
 
-            # write new positions state in case ledger was
-            # newer then that tracked in pps.toml
-            acnt.write_config()
+        await ctx.started((
+            pp_msgs,
+            ['paper'],
+        ))
 
-            # exit early since no fqme was passed,
-            # normally this case is just to load
-            # positions "offline".
-            if fqme is None:
-                log.warning(
-                    'Paper engine only running in position delivery mode!\n'
-                    'NO SIMULATED CLEARING LOOP IS ACTIVE!'
-                )
-                await trio.sleep_forever()
-                return
+        # write new positions state in case ledger was
+        # newer then that tracked in pps.toml
+        ppt.write_config()
+
+        # exit early since no fqme was passed,
+        # normally this case is just to load
+        # positions "offline".
+        if fqme is None:
+            log.warning(
+                'Paper engine only running in position delivery mode!\n'
+                'NO SIMULATED CLEARING LOOP IS ACTIVE!'
+            )
+            await trio.sleep_forever()
+            return
+
+        async with (
+            data.open_feed(
+                [fqme],
+                loglevel=loglevel,
+            ) as feed,
+        ):
+            # sanity check all the mkt infos
+            for fqme, flume in feed.flumes.items():
+                assert mkt_by_fqme[fqme] == flume.mkt
 
-            feed: Feed
             async with (
-                open_feed(
-                    [fqme],
-                    loglevel=loglevel,
-                ) as feed,
+                ctx.open_stream() as ems_stream,
+                trio.open_nursery() as n,
             ):
-                # sanity check all the mkt infos
-                for fqme, flume in feed.flumes.items():
-                    mkt: MktPair = symcache.mktmaps.get(fqme) or mkt_by_fqme[fqme]
-                    if mkt != flume.mkt:
-                        diff: tuple = mkt - flume.mkt
-                        log.warning(
-                            'MktPair sig mismatch?\n'
-                            f'{pformat(diff)}'
-                        )
+                client = PaperBoi(
+                    broker=broker,
+                    ems_trades_stream=ems_stream,
+                    ppt=ppt,
+                    ledger=ledger,
+
+                    _buys=_buys,
+                    _sells=_sells,
+                    _reqids=_reqids,
+
+                    _mkts=mkt_by_fqme,
 
-                get_cost: Callable = getattr(
-                    brokermod,
-                    'get_cost',
-                    None,
                 )
 
-                async with (
-                    ctx.open_stream() as ems_stream,
-                    trio.open_nursery() as n,
-                ):
-                    client = PaperBoi(
-                        broker=broker,
-                        ems_trades_stream=ems_stream,
-                        acnt=acnt,
-                        ledger=ledger,
-                        fees=get_cost,
+                n.start_soon(
+                    handle_order_requests,
+                    client,
+                    ems_stream,
+                )
 
-                        _buys=_buys,
-                        _sells=_sells,
-                        _reqids=_reqids,
-
-                        _mkts=mkt_by_fqme,
-
-                    )
-
-                    n.start_soon(
-                        handle_order_requests,
-                        client,
-                        ems_stream,
-                    )
-
-                    # paper engine simulator clearing task
-                    await simulate_fills(feed.streams[broker], client)
+                # paper engine simulator clearing task
+                await simulate_fills(feed.streams[broker], client)
 
 
 @acm
@@ -761,22 +694,22 @@ async def open_paperboi(
     service_name = f'paperboi.{broker}'
 
     async with (
-        find_service(service_name) as portal,
-        tractor.open_nursery() as an,
+        tractor.find_actor(service_name) as portal,
+        tractor.open_nursery() as tn,
     ):
         # NOTE: only spawn if no paperboi already is up since we likely
         # don't need more then one actor for simulated order clearing
         # per broker-backend.
         if portal is None:
             log.info('Starting new paper-engine actor')
-            portal = await an.start_actor(
+            portal = await tn.start_actor(
                 service_name,
                 enable_modules=[__name__]
             )
             we_spawned = True
 
         async with portal.open_context(
-            open_trade_dialog,
+            trades_dialogue,
             broker=broker,
             fqme=fqme,
             loglevel=loglevel,
@@ -784,59 +717,7 @@ async def open_paperboi(
         ) as (ctx, first):
             yield ctx, first
 
-            # ALWAYS tear down connection AND any newly spawned
-            # paperboi actor on exit!
+            # tear down connection and any spawned actor on exit
             await ctx.cancel()
-
             if we_spawned:
                 await portal.cancel_actor()
-
-
-def norm_trade(
-    tid: str,
-    txdict: dict,
-    pairs: dict[str, Struct],
-    symcache: SymbologyCache | None = None,
-
-    brokermod: ModuleType | None = None,
-
-) -> Transaction:
-    from pendulum import (
-        DateTime,
-        parse,
-    )
-
-    # special field handling for datetimes
-    # to ensure pendulum is used!
-    dt: DateTime = parse(txdict['dt'])
-    expiry: str | None = txdict.get('expiry')
-    fqme: str = txdict.get('fqme') or txdict.pop('fqsn')
-
-    price: float = txdict['price']
-    size: float =  txdict['size']
-    cost: float = txdict.get('cost', 0)
-    if (
-        brokermod
-        and (get_cost := getattr(
-            brokermod,
-            'get_cost',
-            False,
-        ))
-    ):
-        cost = get_cost(
-            price,
-            size,
-            is_taker=True,
-        )
-
-    return Transaction(
-        fqme=fqme,
-        tid=txdict['tid'],
-        dt=dt,
-        price=price,
-        size=size,
-        cost=cost,
-        bs_mktid=txdict['bs_mktid'],
-        expiry=parse(expiry) if expiry else None,
-        etype='clear',
-    )

piker/clearing/_util.py

@@ -17,15 +17,12 @@
 Sub-sys module commons.
 
 """
-from collections import ChainMap
 from functools import partial
-from typing import Any
 
 from ..log import (
     get_logger,
     get_console_log,
 )
-from piker.types import Struct
 subsys: str = 'piker.clearing'
 
 log = get_logger(subsys)
@@ -34,60 +31,3 @@ get_console_log = partial(
     get_console_log,
     name=subsys,
 )
-
-
-class OrderDialogs(Struct):
-    '''
-    Order control dialog (and thus transaction) tracking via
-    message recording.
-
-    Allows easily recording messages associated with a given set of
-    order control transactions and looking up the latest field
-    state using the entire (reverse chronological) msg flow.
-
-    '''
-    _flows: dict[str, ChainMap] = {}
-
-    def add_msg(
-        self,
-        oid: str,
-        msg: dict,
-    ) -> None:
-
-        # NOTE: manually enter a new map on the first msg add to
-        # avoid creating one with an empty dict first entry in
-        # `ChainMap.maps` which is the default if none passed at
-        # init.
-        cm: ChainMap = self._flows.get(oid)
-        if cm:
-            cm.maps.insert(0, msg)
-        else:
-            cm = ChainMap(msg)
-            self._flows[oid] = cm
-
-    # TODO: wrap all this in the `collections.abc.Mapping` interface?
-    def get(
-        self,
-        oid: str,
-
-    ) -> ChainMap[str, Any]:
-        '''
-        Return the dialog `ChainMap` for provided id.
-
-        '''
-        return self._flows.get(oid, None)
-
-    def pop(
-        self,
-        oid: str,
-
-    ) -> ChainMap[str, Any]:
-        '''
-        Pop and thus remove the `ChainMap` containing the msg flow
-        for the given order id.
-
-        '''
-        if (flow := self._flows.pop(oid, None)) is None:
-            log.warning(f'No flow found for oid: {oid}')
-
-        return flow

piker/cli/__init__.py

@@ -1,33 +1,29 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present Tyler Goodlet
-# (in stewardship for pikers, everywhere.)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
 
-# This program is free software: you can redistribute it and/or
-# modify it under the terms of the GNU Affero General Public
-# License as published by the Free Software Foundation, either
-# version 3 of the License, or (at your option) any later version.
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
 
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Affero General Public License for more details.
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
 
-# You should have received a copy of the GNU Affero General Public
-# License along with this program.  If not, see
-# <https://www.gnu.org/licenses/>.
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
 CLI commons.
 
 '''
 import os
-# from contextlib import AsyncExitStack
-from types import ModuleType
+from contextlib import AsyncExitStack
 
 import click
 import trio
 import tractor
-from tractor._multiaddr import parse_maddr
 
 from ..log import (
     get_console_log,
@@ -42,178 +38,93 @@ from ..service import (
 from .. import config
 
 
-log = get_logger('piker.cli')
-
-
-def load_trans_eps(
-    network: dict | None = None,
-    maddrs: list[tuple] | None = None,
-
-) -> dict[str, dict[str, dict]]:
-
-    # transport-oriented endpoint multi-addresses
-    eps: dict[
-        str,  # service name, eg. `pikerd`, `emsd`..
-
-        # libp2p style multi-addresses parsed into prot layers
-        list[dict[str, str | int]]
-    ] = {}
-
-    if (
-        network
-        and not maddrs
-    ):
-        # load network section and (attempt to) connect all endpoints
-        # which are reachable B)
-        for key, maddrs in network.items():
-            match key:
-
-                # TODO: resolve table across multiple discov
-                # prots Bo
-                case 'resolv':
-                    pass
-
-                case 'pikerd':
-                    dname: str = key
-                    for maddr in maddrs:
-                        layers: dict = parse_maddr(maddr)
-                        eps.setdefault(
-                            dname,
-                            [],
-                        ).append(layers)
-
-    elif maddrs:
-        # presume user is manually specifying the root actor ep.
-        eps['pikerd'] = [parse_maddr(maddr)]
-
-    return eps
+log = get_logger('cli')
 
 
 @click.command()
+@click.option('--loglevel', '-l', default='warning', help='Logging level')
+@click.option('--tl', is_flag=True, help='Enable tractor logging')
+@click.option('--pdb', is_flag=True, help='Enable tractor debug mode')
+@click.option('--host', '-h', default=None, help='Host addr to bind')
+@click.option('--port', '-p', default=None, help='Port number to bind')
 @click.option(
-    '--loglevel',
-    '-l',
-    default='warning',
-    help='Logging level',
-)
-@click.option(
-    '--tl',
+    '--tsdb',
     is_flag=True,
-    help='Enable tractor-runtime logs',
+    help='Enable local ``marketstore`` instance'
 )
 @click.option(
-    '--pdb',
+    '--es',
     is_flag=True,
-    help='Enable tractor debug mode',
+    help='Enable local ``elasticsearch`` instance'
 )
-@click.option(
-    '--maddr',
-    '-m',
-    default=None,
-    help='Multiaddrs to bind or contact',
-)
-# @click.option(
-#     '--tsdb',
-#     is_flag=True,
-#     help='Enable local ``marketstore`` instance'
-# )
-# @click.option(
-#     '--es',
-#     is_flag=True,
-#     help='Enable local ``elasticsearch`` instance'
-# )
 def pikerd(
-    maddr: list[str] | None,
     loglevel: str,
+    host: str,
+    port: int,
     tl: bool,
     pdb: bool,
-    # tsdb: bool,
-    # es: bool,
+    tsdb: bool,
+    es: bool,
 ):
     '''
     Spawn the piker broker-daemon.
 
     '''
-    from tractor.devx import maybe_open_crash_handler
-    with maybe_open_crash_handler(pdb=pdb):
-        log = get_console_log(loglevel, name='cli')
+    from .. import service
 
-        if pdb:
-            log.warning((
-                "\n"
-                "!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
-                "When a `piker` daemon crashes it will block the "
-                "task-thread until resumed from console!\n"
-                "\n"
-            ))
+    log = get_console_log(loglevel)
 
-        # service-actor registry endpoint socket-address set
-        regaddrs: list[tuple[str, int]] = []
+    if pdb:
+        log.warning((
+            "\n"
+            "!!! You have enabled daemon DEBUG mode !!!\n"
+            "If a daemon crashes it will likely block"
+            " the service until resumed from console!\n"
+            "\n"
+        ))
 
-        conf, _ = config.load(
-            conf_name='conf',
+    reg_addr: None | tuple[str, int] = None
+    if host or port:
+        reg_addr = (
+            host or _default_registry_host,
+            int(port) or _default_registry_port,
         )
-        network: dict = conf.get('network')
-        if (
-            network is None
-            and not maddr
+
+    async def main():
+        service_mngr: service.Services
+
+        async with (
+            service.open_pikerd(
+                loglevel=loglevel,
+                debug_mode=pdb,
+                registry_addr=reg_addr,
+
+            ) as service_mngr,  # normally delivers a ``Services`` handle
+            trio.open_nursery() as n,
+
+            AsyncExitStack() as stack,
         ):
-            regaddrs = [(
-                _default_registry_host,
-                _default_registry_port,
-            )]
+            if tsdb:
+                dname, conf = await stack.enter_async_context(
+                    service.marketstore.start_ahab_daemon(
+                        service_mngr,
+                        loglevel=loglevel,
+                    )
+                )
+                log.info(f'TSDB `{dname}` up with conf:\n{conf}')
 
-        else:
-            eps: dict = load_trans_eps(
-                network,
-                maddr,
-            )
-            for layers in eps['pikerd']:
-                regaddrs.append((
-                    layers['ipv4']['addr'],
-                    layers['tcp']['port'],
-                ))
+            if es:
+                dname, conf = await stack.enter_async_context(
+                    service.elastic.start_ahab_daemon(
+                        service_mngr,
+                        loglevel=loglevel,
+                    )
+                )
+                log.info(f'DB `{dname}` up with conf:\n{conf}')
 
-        from .. import service
+            await trio.sleep_forever()
 
-        async def main():
-            service_mngr: service.Services
-
-            async with (
-                service.open_pikerd(
-                    registry_addrs=regaddrs,
-                    loglevel=loglevel,
-                    debug_mode=pdb,
-
-                ) as service_mngr,  # normally delivers a ``Services`` handle
-
-                # AsyncExitStack() as stack,
-            ):
-                # TODO: spawn all other sub-actor daemons according to
-                # multiaddress endpoint spec defined by user config
-                assert service_mngr
-
-                # if tsdb:
-                #     dname, conf = await stack.enter_async_context(
-                #         service.marketstore.start_ahab_daemon(
-                #             service_mngr,
-                #             loglevel=loglevel,
-                #         )
-                #     )
-                #     log.info(f'TSDB `{dname}` up with conf:\n{conf}')
-
-                # if es:
-                #     dname, conf = await stack.enter_async_context(
-                #         service.elastic.start_ahab_daemon(
-                #             service_mngr,
-                #             loglevel=loglevel,
-                #         )
-                #     )
-                #     log.info(f'DB `{dname}` up with conf:\n{conf}')
-
-                await trio.sleep_forever()
-
-        trio.run(main)
+    trio.run(main)
 
 
 @click.group(context_settings=config._context_defaults)
@@ -226,24 +137,8 @@ def pikerd(
 @click.option('--loglevel', '-l', default='warning', help='Logging level')
 @click.option('--tl', is_flag=True, help='Enable tractor logging')
 @click.option('--configdir', '-c', help='Configuration directory')
-@click.option(
-    '--pdb',
-    is_flag=True,
-    help='Enable runtime debug mode ',
-)
-@click.option(
-    '--maddr',
-    '-m',
-    default=None,
-    multiple=True,
-    help='Multiaddr to bind',
-)
-@click.option(
-    '--regaddr',
-    '-r',
-    default=None,
-    help='Registrar addr to contact',
-)
+@click.option('--host', '-h', default=None, help='Host addr to bind')
+@click.option('--port', '-p', default=None, help='Port number to bind')
 @click.pass_context
 def cli(
     ctx: click.Context,
@@ -251,19 +146,14 @@ def cli(
     loglevel: str,
     tl: bool,
     configdir: str,
-    pdb: bool,
-
-    # TODO: make these list[str] with multiple -m maddr0 -m maddr1
-    maddr: list[str],
-    regaddr: str,
+    host: str,
+    port: int,
 
 ) -> None:
     if configdir is not None:
         assert os.path.isdir(configdir), f"`{configdir}` is not a valid path"
         config._override_config_dir(configdir)
 
-    # TODO: for typer see
-    # https://typer.tiangolo.com/tutorial/commands/context/
     ctx.ensure_object(dict)
 
     if not brokers:
@@ -271,25 +161,15 @@ def cli(
         from piker.brokers import __brokers__
         brokers = __brokers__
 
-    brokermods: dict[str, ModuleType] = {
-        broker: get_brokermod(broker) for broker in brokers
-    }
+    brokermods = [get_brokermod(broker) for broker in brokers]
     assert brokermods
 
-    # TODO: load endpoints from `conf::[network].pikerd`
-    # - pikerd vs. regd, separate registry daemon?
-    # - expose datad vs. brokerd?
-    # - bind emsd with certain perms on public iface?
-    regaddrs: list[tuple[str, int]] = regaddr or [(
-        _default_registry_host,
-        _default_registry_port,
-    )]
-
-    # TODO: factor [network] section parsing out from pikerd
-    # above and call it here as well.
-    # if maddr:
-    #     for addr in maddr:
-    #         layers: dict = parse_maddr(addr)
+    reg_addr: None | tuple[str, int] = None
+    if host or port:
+        reg_addr = (
+            host or _default_registry_host,
+            int(port) or _default_registry_port,
+        )
 
     ctx.obj.update({
         'brokers': brokers,
@@ -299,12 +179,7 @@ def cli(
         'log': get_console_log(loglevel),
         'confdir': config._config_dir,
         'wl_path': config._watchlists_data_path,
-        'registry_addrs': regaddrs,
-        'pdb': pdb,  # debug mode flag
-
-        # TODO: endpoint parsing, pinging and binding
-        # on no existing server.
-        # 'maddrs': maddr,
+        'registry_addr': reg_addr,
     })
 
     # allow enabling same loglevel in ``tractor`` machinery
@@ -351,15 +226,11 @@ def services(config, tl, ports):
 
 
 def _load_clis() -> None:
-    # from ..service import elastic  # noqa
+    from ..data import cli  # noqa
     from ..brokers import cli  # noqa
     from ..ui import cli  # noqa
     from ..watchlists import cli  # noqa
 
-    # typer implemented
-    from ..storage import cli  # noqa
-    from ..accounting import cli  # noqa
-
 
 # load downstream cli modules
 _load_clis()

piker/config.py

@@ -22,6 +22,7 @@ import platform
 import sys
 import os
 import shutil
+import time
 from typing import (
     Callable,
     MutableMapping,
@@ -104,15 +105,14 @@ def get_app_dir(
     # `tractor`) with the testing dir and check for it whenever we
     # detect `pytest` is being used (which it isn't under normal
     # operation).
-    # if "pytest" in sys.modules:
-    #     import tractor
-    #     actor = tractor.current_actor(err_on_no_runtime=False)
-    #     if actor:  # runtime is up
-    #         rvs = tractor._state._runtime_vars
-    #         import pdbp; pdbp.set_trace()
-    #         testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
-    #         assert testdirpath.exists(), 'piker test harness might be borked!?'
-    #         app_name = str(testdirpath)
+    if "pytest" in sys.modules:
+        import tractor
+        actor = tractor.current_actor(err_on_no_runtime=False)
+        if actor:  # runtime is up
+            rvs = tractor._state._runtime_vars
+            testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
+            assert testdirpath.exists(), 'piker test harness might be borked!?'
+            app_name = str(testdirpath)
 
     if platform.system() == 'Windows':
         key = "APPDATA" if roaming else "LOCALAPPDATA"
@@ -135,19 +135,14 @@ def get_app_dir(
 
 _click_config_dir: Path = Path(get_app_dir('piker'))
 _config_dir: Path = _click_config_dir
+_parent_user: str = os.environ.get('SUDO_USER')
 
-# NOTE: when using `sudo` we attempt to determine the non-root user
-# and still use their normal config dir.
-if (
-    (_parent_user := os.environ.get('SUDO_USER'))
-    and
-    _parent_user != 'root'
-):
+if _parent_user:
     non_root_user_dir = Path(
         os.path.expanduser(f'~{_parent_user}')
     )
     root: str = 'root'
-    _ccds: str = str(_click_config_dir)  # click config dir as string
+    _ccds: str = str(_click_config_dir)  # click config dir string
     i_tail: int = int(_ccds.rfind(root) + len(root))
     _config_dir = (
         non_root_user_dir
@@ -178,14 +173,6 @@ _context_defaults = dict(
 )
 
 
-class ConfigurationError(Exception):
-    'Misconfigured settings, likely in a TOML file.'
-
-
-class NoSignature(ConfigurationError):
-    'No credentials setup for broker backend!'
-
-
 def _override_config_dir(
     path: str
 ) -> None:
@@ -200,15 +187,6 @@ def _conf_fn_w_ext(
     return f'{name}.toml'
 
 
-def get_conf_dir() -> Path:
-    '''
-    Return the user configuration directory ``Path``
-    on the local filesystem.
-
-    '''
-    return _config_dir
-
-
 def get_conf_path(
     conf_name: str = 'brokers',
 
@@ -252,8 +230,7 @@ def repodir() -> Path:
 
 
 def load(
-    # NOTE: always appended with .toml suffix
-    conf_name: str = 'conf',
+    conf_name: str = 'brokers',  # appended with .toml suffix
     path: Path | None = None,
 
     decode: Callable[
@@ -316,6 +293,92 @@ def load(
     return config, path
 
 
+def load_account(
+    brokername: str,
+    acctid: str,
+
+) -> tuple[dict, Path]:
+    '''
+    Load a accounting (with positions) file from
+    $PIKER_CONFIG_DIR/accounting/account.<brokername>.<acctid>.toml.
+
+    Where normally $PIKER_CONFIG_DIR = ~/.config/piker/
+    and we implicitly create a accounting subdir which should
+    normally be linked to a git repo managed by the user B)
+
+    '''
+    legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
+    fn: str = f'account.{brokername}.{acctid}.toml'
+
+    dirpath: Path = _config_dir / 'accounting'
+    if not dirpath.is_dir():
+        dirpath.mkdir()
+
+    config, path = load(
+        path=dirpath / fn,
+        decode=tomlkit.parse,
+        touch_if_dne=True,
+    )
+
+    if not config:
+        legacypath = dirpath / legacy_fn
+        log.warning(
+            f'Your account file is using the legacy `pps.` prefix..\n'
+            f'Rewriting contents to new name -> {path}\n'
+            'Please delete the old file!\n'
+            f'|-> {legacypath}\n'
+        )
+        if legacypath.is_file():
+            legacy_config, _ = load(
+                path=legacypath,
+
+                # TODO: move to tomlkit:
+                # - needs to be fixed to support bidict?
+                #   https://github.com/sdispater/tomlkit/issues/289
+                # - we need to use or fork's fix to do multiline array
+                #   indenting.
+                decode=tomlkit.parse,
+            )
+            config.update(legacy_config)
+
+            # XXX: override the presumably previously non-existant
+            # file with legacy's contents.
+            write(
+                config,
+                path=path,
+                fail_empty=False,
+            )
+
+    return config, path
+
+
+def load_ledger(
+    brokername: str,
+    acctid: str,
+
+) -> tuple[dict, Path]:
+
+    ldir: Path = _config_dir / 'accounting' / 'ledgers'
+    if not ldir.is_dir():
+        ldir.mkdir()
+
+    fname = f'trades_{brokername}_{acctid}.toml'
+    fpath: Path = ldir / fname
+
+    if not fpath.is_file():
+        log.info(
+            f'Creating new local trades ledger: {fpath}'
+        )
+        fpath.touch()
+
+    with fpath.open(mode='rb') as cf:
+        start = time.time()
+        ledger_dict = tomllib.load(cf)
+        log.debug(f'Ledger load took {time.time() - start}s')
+
+    return ledger_dict, fpath
+
+
 def write(
     config: dict,  # toml config as dict
 
@@ -364,9 +427,7 @@ def load_accounts(
 
 ) -> bidict[str, str | None]:
 
-    conf, path = load(
-        conf_name='brokers',
-    )
+    conf, path = load(touch_if_dne=True)
     accounts = bidict()
     for provider_name, section in conf.items():
         accounts_section = section.get('accounts')

piker/data/__init__.py

@@ -22,7 +22,13 @@ and storing data from your brokers as well as
 sharing live streams over a network.
 
 """
-from .ticktools import iterticks
+import tractor
+import trio
+
+from ._util import (
+    get_console_log,
+)
+from ._normalize import iterticks
 from ._sharedmem import (
     maybe_open_shm_array,
     attach_shm_array,
@@ -30,42 +36,17 @@ from ._sharedmem import (
     get_shm_token,
     ShmArray,
 )
-from ._source import (
-    def_iohlcv_fields,
-    def_ohlcv_fields,
-)
 from .feed import (
-    Feed,
     open_feed,
 )
-from .flows import Flume
-from ._symcache import (
-    SymbologyCache,
-    open_symcache,
-    get_symcache,
-    match_from_pairs,
-)
-from ._sampling import open_sample_stream
-from ..types import Struct
 
 
-__all__: list[str] = [
-    'Flume',
-    'Feed',
+__all__ = [
     'open_feed',
     'ShmArray',
     'iterticks',
     'maybe_open_shm_array',
-    'match_from_pairs',
     'attach_shm_array',
     'open_shm_array',
     'get_shm_token',
-    'def_iohlcv_fields',
-    'def_ohlcv_fields',
-    'open_symcache',
-    'open_sample_stream',
-    'get_symcache',
-    'Struct',
-    'SymbologyCache',
-    'types',
 ]

piker/data/_formatters.py

@@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -13,10 +13,10 @@
 
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 Pre-(path)-graphics formatted x/y nd/1d rendering subsystem.
 
-'''
+"""
 from __future__ import annotations
 from typing import (
     Optional,
@@ -39,12 +39,7 @@ if TYPE_CHECKING:
     from ._dataviz import (
         Viz,
     )
-    from piker.toolz import Profiler
-
-# default gap between bars: "bar gap multiplier"
-# - 0.5 is no overlap between OC arms,
-# - 1.0 is full overlap on each neighbor sample
-BGM: float = 0.16
+    from .._profile import Profiler
 
 
 class IncrementalFormatter(msgspec.Struct):
@@ -227,7 +222,6 @@ class IncrementalFormatter(msgspec.Struct):
         profiler: Profiler,
 
         slice_to_inview: bool = True,
-        force_full_realloc: bool = False,
 
     ) -> tuple[
         np.ndarray,
@@ -254,10 +248,7 @@ class IncrementalFormatter(msgspec.Struct):
 
         # we first need to allocate xy data arrays
         # from the source data.
-        if (
-            self.y_nd is None
-            or force_full_realloc
-        ):
+        if self.y_nd is None:
             self.xy_nd_start = shm._first.value
             self.xy_nd_stop = shm._last.value
             self.x_nd, self.y_nd = self.allocate_xy_nd(
@@ -518,7 +509,6 @@ class IncrementalFormatter(msgspec.Struct):
 
 
 class OHLCBarsFmtr(IncrementalFormatter):
-
     x_offset: np.ndarray = np.array([
         -0.5,
         0,
@@ -610,9 +600,8 @@ class OHLCBarsFmtr(IncrementalFormatter):
         vr: tuple[int, int],
 
         start: int = 0,  # XXX: do we need this?
-
         # 0.5 is no overlap between arms, 1.0 is full overlap
-        gap: float = BGM,
+        w: float = 0.16,
 
     ) -> tuple[
         np.ndarray,
@@ -629,7 +618,7 @@ class OHLCBarsFmtr(IncrementalFormatter):
             array[:-1],
             start,
             bar_w=self.index_step_size,
-            bar_gap=gap * self.index_step_size,
+            bar_gap=w * self.index_step_size,
 
             # XXX: don't ask, due to a ``numba`` bug..
             use_time_index=(self.index_field == 'time'),

piker/data/_normalize.py

@@ -0,0 +1,82 @@
+# piker: trading gear for hackers
+# Copyright (C) Tyler Goodlet (in stewardship for piker0)
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Stream format enforcement.
+
+'''
+from itertools import chain
+from typing import AsyncIterator
+
+
+def iterticks(
+    quote: dict,
+    types: tuple[str] = (
+        'trade',
+        'dark_trade',
+    ),
+    deduplicate_darks: bool = False,
+
+) -> AsyncIterator:
+    '''
+    Iterate through ticks delivered per quote cycle.
+
+    '''
+    if deduplicate_darks:
+        assert 'dark_trade' in types
+
+    # print(f"{quote}\n\n")
+    ticks = quote.get('ticks', ())
+    trades = {}
+    darks = {}
+
+    if ticks:
+
+        # do a first pass and attempt to remove duplicate dark
+        # trades with the same tick signature.
+        if deduplicate_darks:
+            for tick in ticks:
+                ttype = tick.get('type')
+
+                time = tick.get('time', None)
+                if time:
+                    sig = (
+                        time,
+                        tick['price'],
+                        tick.get('size')
+                    )
+
+                    if ttype == 'dark_trade':
+                        darks[sig] = tick
+
+                    elif ttype == 'trade':
+                        trades[sig] = tick
+
+            # filter duplicates
+            for sig, tick in trades.items():
+                tick = darks.pop(sig, None)
+                if tick:
+                    ticks.remove(tick)
+                    # print(f'DUPLICATE {tick}')
+
+            # re-insert ticks
+            ticks.extend(list(chain(trades.values(), darks.values())))
+
+        for tick in ticks:
+            # print(f"{quote['symbol']}: {tick}")
+            ttype = tick.get('type')
+            if ttype in types:
+                yield tick

piker/data/_pathops.py

@@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -17,6 +17,11 @@
 Super fast ``QPainterPath`` generation related operator routines.
 
 """
+from math import (
+    ceil,
+    floor,
+)
+
 import numpy as np
 from numpy.lib import recfunctions as rfn
 from numba import (
@@ -30,6 +35,11 @@ from numba import (
 # TODO: for ``numba`` typing..
 # from ._source import numba_ohlc_dtype
 from ._m4 import ds_m4
+from .._profile import (
+    Profiler,
+    pg_profile_enabled,
+    ms_slower_then,
+)
 
 
 def xy_downsample(
@@ -125,7 +135,7 @@ def path_arrays_from_ohlc(
     half_w: float = bar_w/2
 
     # TODO: report bug for assert @
-    # ../piker/env/lib/python3.8/site-packages/numba/core/typing/builtins.py:991
+    # /home/goodboy/repos/piker/env/lib/python3.8/site-packages/numba/core/typing/builtins.py:991
     for i, q in enumerate(data[start:], start):
 
         open = q['open']
@@ -227,20 +237,20 @@ def trace_hl(
 
     for i in range(hl.size):
         row = hl[i]
-        lo, hi = row['low'], row['high']
+        l, h = row['low'], row['high']
 
-        up_diff = hi - last_l
-        down_diff = last_h - lo
+        up_diff = h - last_l
+        down_diff = last_h - l
 
         if up_diff > down_diff:
-            out[2*i + 1] = hi
+            out[2*i + 1] = h
             out[2*i] = last_l
         else:
-            out[2*i + 1] = lo
+            out[2*i + 1] = l
             out[2*i] = last_h
 
-        last_l = lo
-        last_h = hi
+        last_l = l
+        last_h = h
 
         x[2*i] = int(i) - margin
         x[2*i + 1] = int(i) + margin
@@ -279,3 +289,158 @@ def ohlc_flatten(
             num=len(flat),
         )
     return x, flat
+
+
+def slice_from_time(
+    arr: np.ndarray,
+    start_t: float,
+    stop_t: float,
+    step: float,  # sampler period step-diff
+
+) -> slice:
+    '''
+    Calculate array indices mapped from a time range and return them in
+    a slice.
+
+    Given an input array with an epoch `'time'` series entry, calculate
+    the indices which span the time range and return in a slice. Presume
+    each `'time'` step increment is uniform and when the time stamp
+    series contains gaps (the uniform presumption is untrue) use
+    ``np.searchsorted()`` binary search to look up the appropriate
+    index.
+
+    '''
+    profiler = Profiler(
+        msg='slice_from_time()',
+        disabled=not pg_profile_enabled(),
+        ms_threshold=ms_slower_then,
+    )
+
+    times = arr['time']
+    t_first = floor(times[0])
+    t_last = ceil(times[-1])
+
+    # the greatest index we can return which slices to the
+    # end of the input array.
+    read_i_max = arr.shape[0]
+
+    # compute (presumed) uniform-time-step index offsets
+    i_start_t = floor(start_t)
+    read_i_start = floor(((i_start_t - t_first) // step)) - 1
+
+    i_stop_t = ceil(stop_t)
+
+    # XXX: edge case -> always set stop index to last in array whenever
+    # the input stop time is detected to be greater then the equiv time
+    # stamp at that last entry.
+    if i_stop_t >= t_last:
+        read_i_stop = read_i_max
+    else:
+        read_i_stop = ceil((i_stop_t - t_first) // step) + 1
+
+    # always clip outputs to array support
+    # for read start:
+    # - never allow a start < the 0 index
+    # - never allow an end index > the read array len
+    read_i_start = min(
+        max(0, read_i_start),
+        read_i_max - 1,
+    )
+    read_i_stop = max(
+        0,
+        min(read_i_stop, read_i_max),
+    )
+
+    # check for larger-then-latest calculated index for given start
+    # time, in which case we do a binary search for the correct index.
+    # NOTE: this is usually the result of a time series with time gaps
+    # where it is expected that each index step maps to a uniform step
+    # in the time stamp series.
+    t_iv_start = times[read_i_start]
+    if (
+        t_iv_start > i_start_t
+    ):
+        # do a binary search for the best index mapping to ``start_t``
+        # given we measured an overshoot using the uniform-time-step
+        # calculation from above.
+
+        # TODO: once we start caching these per source-array,
+        # we can just overwrite ``read_i_start`` directly.
+        new_read_i_start = np.searchsorted(
+            times,
+            i_start_t,
+            side='left',
+        )
+
+        # TODO: minimize binary search work as much as possible:
+        # - cache these remap values which compensate for gaps in the
+        #   uniform time step basis where we calc a later start
+        #   index for the given input ``start_t``.
+        # - can we shorten the input search sequence by heuristic?
+        #   up_to_arith_start = index[:read_i_start]
+
+        if (
+            new_read_i_start <= read_i_start
+        ):
+            # t_diff = t_iv_start - start_t
+            # print(
+            #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
+            #     f'start_t:{start_t} -> 0index start_t:{t_iv_start}\n'
+            #     f'diff: {t_diff}\n'
+            #     f'REMAPPED START i: {read_i_start} -> {new_read_i_start}\n'
+            # )
+            read_i_start = new_read_i_start
+
+    t_iv_stop = times[read_i_stop - 1]
+    if (
+        t_iv_stop > i_stop_t
+    ):
+        # t_diff = stop_t - t_iv_stop
+        # print(
+        #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
+        #     f'calced iv stop:{t_iv_stop} -> stop_t:{stop_t}\n'
+        #     f'diff: {t_diff}\n'
+        #     # f'SHOULD REMAP STOP: {read_i_start} -> {new_read_i_start}\n'
+        # )
+        new_read_i_stop = np.searchsorted(
+            times[read_i_start:],
+            # times,
+            i_stop_t,
+            side='right',
+        )
+
+        if (
+            new_read_i_stop <= read_i_stop
+        ):
+            read_i_stop = read_i_start + new_read_i_stop + 1
+
+    # sanity checks for range size
+    # samples = (i_stop_t - i_start_t) // step
+    # index_diff = read_i_stop - read_i_start + 1
+    # if index_diff > (samples + 3):
+    #     breakpoint()
+
+    # read-relative indexes: gives a slice where `shm.array[read_slc]`
+    # will be the data spanning the input time range `start_t` ->
+    # `stop_t`
+    read_slc = slice(
+        int(read_i_start),
+        int(read_i_stop),
+    )
+
+    profiler(
+        'slicing complete'
+        # f'{start_t} -> {abs_slc.start} | {read_slc.start}\n'
+        # f'{stop_t} -> {abs_slc.stop} | {read_slc.stop}\n'
+    )
+
+    # NOTE: if caller needs absolute buffer indices they can
+    # slice the buffer abs index like so:
+    # index = arr['index']
+    # abs_indx = index[read_slc]
+    # abs_slc = slice(
+    #     int(abs_indx[0]),
+    #     int(abs_indx[-1]),
+    # )
+
+    return read_slc

piker/data/_sampling.py

@@ -27,27 +27,17 @@ from collections import (
 from contextlib import asynccontextmanager as acm
 import time
 from typing import (
-    Any,
     AsyncIterator,
     TYPE_CHECKING,
 )
 
 import tractor
-from tractor import (
-    Context,
-    MsgStream,
-    Channel,
-)
 from tractor.trionics import (
     maybe_open_nursery,
 )
 import trio
 from trio_typing import TaskStatus
 
-from .ticktools import (
-    frame_ticks,
-    _tick_groups,
-)
 from ._util import (
     log,
     get_console_log,
@@ -58,10 +48,7 @@ if TYPE_CHECKING:
     from ._sharedmem import (
         ShmArray,
     )
-    from .feed import (
-        _FeedsBus,
-        Sub,
-    )
+    from .feed import _FeedsBus
 
 
 # highest frequency sample step is 1 second by default, though in
@@ -102,7 +89,7 @@ class Sampler:
         float,
         list[
             float,
-            set[MsgStream]
+            set[tractor.MsgStream]
         ],
     ] = defaultdict(
         lambda: [
@@ -243,7 +230,6 @@ class Sampler:
         self,
         period_s: float,
         time_stamp: float | None = None,
-        info: dict | None = None,
 
     ) -> None:
         '''
@@ -266,20 +252,16 @@ class Sampler:
             f'broadcasting {period_s} -> {last_ts}\n'
             # f'consumers: {subs}'
         )
-        borked: set[MsgStream] = set()
-        sent: set[MsgStream] = set()
+        borked: set[tractor.MsgStream] = set()
+        sent: set[tractor.MsgStream] = set()
         while True:
             try:
                 for stream in (subs - sent):
                     try:
-                        msg = {
+                        await stream.send({
                             'index': time_stamp or last_ts,
                             'period': period_s,
-                        }
-                        if info:
-                            msg.update(info)
-
-                        await stream.send(msg)
+                        })
                         sent.add(stream)
 
                     except (
@@ -305,24 +287,14 @@ class Sampler:
                 )
 
     @classmethod
-    async def broadcast_all(
-        self,
-        info: dict | None = None,
-    ) -> None:
-
-        # NOTE: take a copy of subs since removals can happen
-        # during the broadcast checkpoint which can cause
-        # a `RuntimeError` on interation of the underlying `dict`.
-        for period_s in list(self.subscribers):
-            await self.broadcast(
-                period_s,
-                info=info,
-            )
+    async def broadcast_all(self) -> None:
+        for period_s in self.subscribers:
+            await self.broadcast(period_s)
 
 
 @tractor.context
 async def register_with_sampler(
-    ctx: Context,
+    ctx: tractor.Context,
     period_s: float,
     shms_by_period: dict[float, dict] | None = None,
 
@@ -387,21 +359,14 @@ async def register_with_sampler(
 
                         # except broadcast requests from the subscriber
                         async for msg in stream:
-                            if 'broadcast_all' in msg:
-                                await Sampler.broadcast_all(
-                                    info=msg['broadcast_all'],
-                                )
+                            if msg == 'broadcast_all':
+                                await Sampler.broadcast_all()
                 finally:
                     if (
                         sub_for_broadcasts
                         and subs
                     ):
-                        try:
-                            subs.remove(stream)
-                        except KeyError:
-                            log.warning(
-                                f'{stream._ctx.chan.uid} sub already removed!?'
-                            )
+                        subs.remove(stream)
             else:
                 # if no shms are passed in we just wait until cancelled
                 # by caller.
@@ -498,8 +463,6 @@ async def open_sample_stream(
     cache_key: str | None = None,
     allow_new_sampler: bool = True,
 
-    ensure_is_active: bool = False,
-
 ) -> AsyncIterator[dict[str, float]]:
     '''
     Subscribe to OHLC sampling "step" events: when the time aggregation
@@ -542,20 +505,11 @@ async def open_sample_stream(
             },
         ) as (ctx, first)
     ):
-        if ensure_is_active:
-            assert len(first) > 1
-
         async with (
-            ctx.open_stream(
-                allow_overruns=True,
-            ) as istream,
+            ctx.open_stream() as istream,
 
-            # TODO: we DO need this task-bcasting so that
-            # for eg. the history chart update loop eventually
-            # receceives all backfilling event msgs such that
-            # the underlying graphics format arrays are
-            # re-allocated until all history is loaded!
-            istream.subscribe() as istream,
+            # TODO: we don't need this task-bcasting right?
+            # istream.subscribe() as istream,
         ):
             yield istream
 
@@ -600,9 +554,9 @@ async def sample_and_broadcast(
             # TODO: we should probably not write every single
             # value to an OHLC sample stream XD
             # for a tick stream sure.. but this is excessive..
-            ticks: list[dict] = quote['ticks']
+            ticks = quote['ticks']
             for tick in ticks:
-                ticktype: str = tick['type']
+                ticktype = tick['type']
 
                 # write trade events to shm last OHLC sample
                 if ticktype in ('trade', 'utrade'):
@@ -612,14 +566,13 @@ async def sample_and_broadcast(
                     # more compact inline-way to do this assignment
                     # to both buffers?
                     for shm in [rt_shm, hist_shm]:
-
                         # update last entry
                         # benchmarked in the 4-5 us range
                         o, high, low, v = shm.array[-1][
                             ['open', 'high', 'low', 'volume']
                         ]
 
-                        new_v: float = tick.get('size', 0)
+                        new_v = tick.get('size', 0)
 
                         if v == 0 and new_v:
                             # no trades for this bar yet so the open
@@ -638,14 +591,14 @@ async def sample_and_broadcast(
                             'high',
                             'low',
                             'close',
-                            # 'bar_wap',  # can be optionally provided
+                            'bar_wap',  # can be optionally provided
                             'volume',
                         ]][-1] = (
                             o,
                             max(high, last),
                             min(low, last),
                             last,
-                            # quote.get('bar_wap', 0),
+                            quote.get('bar_wap', 0),
                             volume,
                         )
 
@@ -657,49 +610,48 @@ async def sample_and_broadcast(
             # eventually block this producer end of the feed and
             # thus other consumers still attached.
             sub_key: str = broker_symbol.lower()
-            subs: set[Sub] = bus.get_subs(sub_key)
+            subs: list[
+                tuple[
+                    tractor.MsgStream | trio.MemorySendChannel,
+                    float | None,  # tick throttle in Hz
+                ]
+            ] = bus.get_subs(sub_key)
 
             # NOTE: by default the broker backend doesn't append
             # it's own "name" into the fqme schema (but maybe it
             # should?) so we have to manually generate the correct
             # key here.
-            fqme: str = f'{broker_symbol}.{brokername}'
+            fqme = f'{broker_symbol}.{brokername}'
             lags: int = 0
 
-            # XXX TODO XXX: speed up this loop in an AOT compiled
-            # lang (like rust or nim or zig)!
-            # AND/OR instead of doing a fan out to TCP sockets
-            # here, we add a shm-style tick queue which readers can
-            # pull from instead of placing the burden of broadcast
-            # on solely on this `brokerd` actor. see issues:
+            # TODO: speed up this loop in an AOT compiled lang (like
+            # rust or nim or zig) and/or instead of doing a fan out to
+            # TCP sockets here, we add a shm-style tick queue which
+            # readers can pull from instead of placing the burden of
+            # broadcast on solely on this `brokerd` actor. see issues:
             # - https://github.com/pikers/piker/issues/98
             # - https://github.com/pikers/piker/issues/107
 
-            # for (stream, tick_throttle) in subs.copy():
-            for sub in subs.copy():
-                ipc: MsgStream = sub.ipc
-                throttle: float = sub.throttle_rate
+            for (stream, tick_throttle) in subs.copy():
                 try:
                     with trio.move_on_after(0.2) as cs:
-                        if throttle:
-                            send_chan: trio.abc.SendChannel = sub.send_chan
-
+                        if tick_throttle:
                             # this is a send mem chan that likely
                             # pushes to the ``uniform_rate_send()`` below.
                             try:
-                                send_chan.send_nowait(
+                                stream.send_nowait(
                                     (fqme, quote)
                                 )
                             except trio.WouldBlock:
                                 overruns[sub_key] += 1
-                                ctx: Context = ipc._ctx
-                                chan: Channel = ctx.chan
+                                ctx = stream._ctx
+                                chan = ctx.chan
 
                                 log.warning(
                                     f'Feed OVERRUN {sub_key}'
                                     '@{bus.brokername} -> \n'
                                     f'feed @ {chan.uid}\n'
-                                    f'throttle = {throttle} Hz'
+                                    f'throttle = {tick_throttle} Hz'
                                 )
 
                                 if overruns[sub_key] > 6:
@@ -716,33 +668,33 @@ async def sample_and_broadcast(
                                             f'{sub_key}:'
                                             f'{ctx.cid}@{chan.uid}'
                                         )
-                                        await ipc.aclose()
+                                        await stream.aclose()
                                         raise trio.BrokenResourceError
                         else:
-                            await ipc.send(
+                            await stream.send(
                                 {fqme: quote}
                             )
 
                     if cs.cancelled_caught:
                         lags += 1
                         if lags > 10:
-                            await tractor.pause()
+                            await tractor.breakpoint()
 
                 except (
                     trio.BrokenResourceError,
                     trio.ClosedResourceError,
                     trio.EndOfChannel,
                 ):
-                    ctx: Context = ipc._ctx
-                    chan: Channel = ctx.chan
+                    ctx = stream._ctx
+                    chan = ctx.chan
                     if ctx:
                         log.warning(
                             'Dropped `brokerd`-quotes-feed connection:\n'
                             f'{broker_symbol}:'
                             f'{ctx.cid}@{chan.uid}'
                         )
-                    if sub.throttle_rate:
-                        assert ipc._closed
+                    if tick_throttle:
+                        assert stream._closed
 
                     # XXX: do we need to deregister here
                     # if it's done in the fee bus code?
@@ -751,15 +703,69 @@ async def sample_and_broadcast(
                     # since there seems to be some kinda race..
                     bus.remove_subs(
                         sub_key,
-                        {sub},
+                        {(stream, tick_throttle)},
                     )
 
 
+# a working tick-type-classes template
+_tick_groups = {
+    'clears': {'trade', 'dark_trade', 'last'},
+    'bids': {'bid', 'bsize'},
+    'asks': {'ask', 'asize'},
+}
+
+
+def frame_ticks(
+    first_quote: dict,
+    last_quote: dict,
+    ticks_by_type: dict,
+) -> None:
+    # append quotes since last iteration into the last quote's
+    # tick array/buffer.
+    ticks = last_quote.get('ticks')
+
+    # TODO: once we decide to get fancy really we should
+    # have a shared mem tick buffer that is just
+    # continually filled and the UI just ready from it
+    # at it's display rate.
+    if ticks:
+        # TODO: do we need this any more or can we just
+        # expect the receiver to unwind the below
+        # `ticks_by_type: dict`?
+        # => undwinding would potentially require a
+        # `dict[str, set | list]` instead with an
+        # included `'types' field which is an (ordered)
+        # set of tick type fields in the order which
+        # types arrived?
+        first_quote['ticks'].extend(ticks)
+
+        # XXX: build a tick-by-type table of lists
+        # of tick messages. This allows for less
+        # iteration on the receiver side by allowing for
+        # a single "latest tick event" look up by
+        # indexing the last entry in each sub-list.
+        # tbt = {
+        #     'types': ['bid', 'asize', 'last', .. '<type_n>'],
+
+        #     'bid': [tick0, tick1, tick2, .., tickn],
+        #     'asize': [tick0, tick1, tick2, .., tickn],
+        #     'last': [tick0, tick1, tick2, .., tickn],
+        #     ...
+        #     '<type_n>': [tick0, tick1, tick2, .., tickn],
+        # }
+
+        # append in reverse FIFO order for in-order iteration on
+        # receiver side.
+        for tick in ticks:
+            ttype = tick['type']
+            ticks_by_type[ttype].append(tick)
+
+
 async def uniform_rate_send(
 
     rate: float,
     quote_stream: trio.abc.ReceiveChannel,
-    stream: MsgStream,
+    stream: tractor.MsgStream,
 
     task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
 
@@ -788,10 +794,10 @@ async def uniform_rate_send(
     diff = 0
 
     task_status.started()
-    ticks_by_type: dict[
+    ticks_by_type: defaultdict[
         str,
-        list[dict[str, Any]],
-    ] = {}
+        list[dict],
+    ] = defaultdict(list)
 
     clear_types = _tick_groups['clears']
 
@@ -819,9 +825,9 @@ async def uniform_rate_send(
                     # expired we aren't supposed to send yet so append
                     # to the tick frame.
                     frame_ticks(
+                        first_quote,
                         last_quote,
-                        ticks_in_order=first_quote['ticks'],
-                        ticks_by_type=ticks_by_type,
+                        ticks_by_type,
                     )
 
                     # send cycle isn't due yet so continue waiting
@@ -841,8 +847,8 @@ async def uniform_rate_send(
 
                 frame_ticks(
                     first_quote,
-                    ticks_in_order=first_quote['ticks'],
-                    ticks_by_type=ticks_by_type,
+                    first_quote,
+                    ticks_by_type,
                 )
 
             # we have a quote already so send it now.
@@ -858,9 +864,9 @@ async def uniform_rate_send(
                     break
 
                 frame_ticks(
+                    first_quote,
                     last_quote,
-                    ticks_in_order=first_quote['ticks'],
-                    ticks_by_type=ticks_by_type,
+                    ticks_by_type,
                 )
 
         # measured_rate = 1 / (time.time() - last_send)

piker/data/_sharedmem.py

@@ -33,8 +33,17 @@ from numpy.lib import recfunctions as rfn
 import tractor
 
 from ._util import log
-from ._source import def_iohlcv_fields
-from piker.types import Struct
+from ._source import base_iohlc_dtype
+from .types import Struct
+
+
+# how  much is probably dependent on lifestyle
+_secs_in_day = int(60 * 60 * 24)
+# we try for a buncha times, but only on a run-every-other-day kinda week.
+_days_worth = 16
+_default_size = _days_worth * _secs_in_day
+# where to start the new data append index
+_rt_buffer_start = int((_days_worth - 1) * _secs_in_day)
 
 
 def cuckoff_mantracker():
@@ -61,6 +70,7 @@ def cuckoff_mantracker():
     mantracker._resource_tracker = ManTracker()
     mantracker.register = mantracker._resource_tracker.register
     mantracker.ensure_running = mantracker._resource_tracker.ensure_running
+    # ensure_running = mantracker._resource_tracker.ensure_running
     mantracker.unregister = mantracker._resource_tracker.unregister
     mantracker.getfd = mantracker._resource_tracker.getfd
 
@@ -158,7 +168,7 @@ def _make_token(
     to access a shared array.
 
     '''
-    dtype = def_iohlcv_fields if dtype is None else dtype
+    dtype = base_iohlc_dtype if dtype is None else dtype
     return _Token(
         shm_name=key,
         shm_first_index_name=key + "_first",
@@ -248,6 +258,7 @@ class ShmArray:
         # to load an empty array..
         if len(a) == 0 and self._post_init:
             raise RuntimeError('Empty array race condition hit!?')
+            # breakpoint()
 
         return a
 
@@ -312,7 +323,7 @@ class ShmArray:
         field_map: Optional[dict[str, str]] = None,
         prepend: bool = False,
         update_first: bool = True,
-        start: int | None = None,
+        start: Optional[int] = None,
 
     ) -> int:
         '''
@@ -354,11 +365,7 @@ class ShmArray:
             # tries to access ``.array`` (which due to the index
             # overlap will be empty). Pretty sure we've fixed it now
             # but leaving this here as a reminder.
-            if (
-                prepend
-                and update_first
-                and length
-            ):
+            if prepend and update_first and length:
                 assert index < self._first.value
 
             if (
@@ -432,10 +439,10 @@ class ShmArray:
 
 
 def open_shm_array(
-    size: int,
-    key: str | None = None,
-    dtype: np.dtype | None = None,
-    append_start_index: int | None = None,
+
+    key: Optional[str] = None,
+    size: int = _default_size,  # see above
+    dtype: Optional[np.dtype] = None,
     readonly: bool = False,
 
 ) -> ShmArray:
@@ -500,13 +507,10 @@ def open_shm_array(
     # ``ShmArray._start.value: int = 0`` and the yet-to-be written
     # real-time section will start at ``ShmArray.index: int``.
 
-    # this sets the index to nearly 2/3rds into the the length of
-    # the buffer leaving at least a "days worth of second samples"
-    # for the real-time section.
-    if append_start_index is None:
-        append_start_index = round(size * 0.616)
-
-    last.value = first.value = append_start_index
+    # this sets the index to 3/4 of the length of the buffer
+    # leaving a "days worth of second samples" for the real-time
+    # section.
+    last.value = first.value = _rt_buffer_start
 
     shmarr = ShmArray(
         array,
@@ -520,6 +524,7 @@ def open_shm_array(
 
     # "unlink" created shm on process teardown by
     # pushing teardown calls onto actor context stack
+
     stack = tractor.current_actor().lifetime_stack
     stack.callback(shmarr.close)
     stack.callback(shmarr.destroy)
@@ -614,10 +619,7 @@ def attach_shm_array(
 
 def maybe_open_shm_array(
     key: str,
-    size: int,
-    dtype: np.dtype | None = None,
-    append_start_index: int | None = None,
-    readonly: bool = False,
+    dtype: Optional[np.dtype] = None,
     **kwargs,
 
 ) -> tuple[ShmArray, bool]:
@@ -638,16 +640,11 @@ def maybe_open_shm_array(
     use ``attach_shm_array``.
 
     '''
+    size = kwargs.pop('size', _default_size)
     try:
         # see if we already know this key
         token = _known_tokens[key]
-        return (
-            attach_shm_array(
-                token=token,
-                readonly=readonly,
-            ),
-            False,
-        )
+        return attach_shm_array(token=token, **kwargs), False
     except KeyError:
         log.debug(f"Could not find {key} in shms cache")
         if dtype:
@@ -666,16 +663,8 @@ def maybe_open_shm_array(
         # Attempt to open a block and expect
         # to fail if a block has been allocated
         # on the OS by someone else.
-        return (
-            open_shm_array(
-                key=key,
-                size=size,
-                dtype=dtype,
-                append_start_index=append_start_index,
-                readonly=readonly,
-            ),
-            True,
-        )
+        return open_shm_array(key=key, dtype=dtype, **kwargs), True
+
 
 def try_read(
     array: np.ndarray

piker/data/_source.py

@@ -23,42 +23,26 @@ from bidict import bidict
 import numpy as np
 
 
-def_iohlcv_fields: list[tuple[str, type]] = [
-
-    # YES WE KNOW, this isn't needed in polars but we use it for doing
-    # ring-buffer like pre/append ops our our `ShmArray` real-time
-    # numpy-array buffering system such that there is a master index
-    # that can be used for index-arithmetic when write data to the
-    # "middle" of the array. See the ``tractor.ipc.shm`` pkg for more
-    # details.
-    ('index', int),
-
-    # presume int for epoch stamps since it's most common
-    # and makes the most sense to avoid float rounding issues.
-    # TODO: if we want higher reso we should use the new
-    # ``time.time_ns()`` in python 3.10+
-    ('time', int),
+ohlc_fields = [
+    ('time', float),
     ('open', float),
     ('high', float),
     ('low', float),
     ('close', float),
     ('volume', float),
-
-    # TODO: can we elim this from default field set to save on mem?
-    # i think only kraken really uses this in terms of what we get from
-    # their ohlc history API?
-    # ('bar_wap', float),  # shouldn't be default right?
+    ('bar_wap', float),
 ]
 
-# remove index field
-def_ohlcv_fields: list[tuple[str, type]] = def_iohlcv_fields.copy()
-def_ohlcv_fields.pop(0)
-assert (len(def_iohlcv_fields) - len(def_ohlcv_fields)) == 1
+ohlc_with_index = ohlc_fields.copy()
+ohlc_with_index.insert(0, ('index', int))
+
+# our minimum structured array layout for ohlc data
+base_iohlc_dtype = np.dtype(ohlc_with_index)
+base_ohlc_dtype = np.dtype(ohlc_fields)
 
 # TODO: for now need to construct this manually for readonly arrays, see
 # https://github.com/numba/numba/issues/4511
 # from numba import from_dtype
-# base_ohlc_dtype = np.dtype(def_ohlc_fields)
 # numba_ohlc_dtype = from_dtype(base_ohlc_dtype)
 
 # map time frame "keys" to seconds values

piker/data/_symcache.py

@@ -1,510 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Mega-simple symbology cache via TOML files.
-
-Allow backend data providers and/or brokers to stash their
-symbology sets (aka the meta data we normalize into our
-`.accounting.MktPair` type) to the filesystem for faster lookup and
-offline usage.
-
-'''
-from __future__ import annotations
-from contextlib import (
-    asynccontextmanager as acm,
-)
-from pathlib import Path
-from pprint import pformat
-from typing import (
-    Any,
-    Sequence,
-    Hashable,
-    TYPE_CHECKING,
-)
-from types import ModuleType
-
-from rapidfuzz import process as fuzzy
-import tomli_w  # for fast symbol cache writing
-import tractor
-import trio
-try:
-    import tomllib
-except ModuleNotFoundError:
-    import tomli as tomllib
-from msgspec import field
-
-from piker.log import get_logger
-from piker import config
-from piker.types import Struct
-from piker.brokers import (
-    open_cached_client,
-    get_brokermod,
-)
-
-if TYPE_CHECKING:
-    from ..accounting import (
-        Asset,
-        MktPair,
-    )
-
-log = get_logger('data.cache')
-
-
-class SymbologyCache(Struct):
-    '''
-    Asset meta-data cache which holds lookup tables for 3 sets of
-    market-symbology related struct-types required by the
-    `.accounting` and `.data` subsystems.
-
-    '''
-    mod: ModuleType
-    fp: Path
-
-    # all asset-money-systems descriptions as minimally defined by
-    # in `.accounting.Asset`
-    assets: dict[str, Asset] = field(default_factory=dict)
-
-    # backend-system pairs loaded in provider (schema) specific
-    # structs.
-    pairs: dict[str, Struct] = field(default_factory=dict)
-    # serialized namespace path to the backend's pair-info-`Struct`
-    # defn B)
-    pair_ns_path: tractor.msg.NamespacePath | None = None
-
-    # TODO: piker-normalized `.accounting.MktPair` table?
-    # loaded from the `.pairs` and a normalizer
-    # provided by the backend pkg.
-    mktmaps: dict[str, MktPair] = field(default_factory=dict)
-
-    def write_config(self) -> None:
-
-        # put the backend's pair-struct type ref at the top
-        # of file if possible.
-        cachedict: dict[str, Any] = {
-            'pair_ns_path': str(self.pair_ns_path) or '',
-        }
-
-        # serialize all tables as dicts for TOML.
-        for key, table in {
-            'assets': self.assets,
-            'pairs': self.pairs,
-            'mktmaps': self.mktmaps,
-        }.items():
-            if not table:
-                log.warning(
-                    f'Asset cache table for `{key}` is empty?'
-                )
-                continue
-
-            dct = cachedict[key] = {}
-            for key, struct in table.items():
-                dct[key] = struct.to_dict(include_non_members=False)
-
-        try:
-            with self.fp.open(mode='wb') as fp:
-                tomli_w.dump(cachedict, fp)
-        except TypeError:
-            self.fp.unlink()
-            raise
-
-    async def load(self) -> None:
-        '''
-        Explicitly load the "symbology set" for this provider by using
-        2 required `Client` methods:
-
-          - `.get_assets()`: returning a table of `Asset`s
-          - `.get_mkt_pairs()`: returning a table of pair-`Struct`
-            types, custom defined by the particular backend.
-
-        AND, the required `.get_mkt_info()` module-level endpoint
-        which maps `fqme: str` -> `MktPair`s.
-
-        These tables are then used to fill out the `.assets`, `.pairs` and
-        `.mktmaps` tables on this cache instance, respectively.
-
-        '''
-        async with open_cached_client(self.mod.name) as client:
-
-            if get_assets := getattr(client, 'get_assets', None):
-                assets: dict[str, Asset] = await get_assets()
-                for bs_mktid, asset in assets.items():
-                    self.assets[bs_mktid] = asset
-            else:
-                log.warning(
-                    'No symbology cache `Asset` support for `{provider}`..\n'
-                    'Implement `Client.get_assets()`!'
-                )
-
-            if get_mkt_pairs := getattr(client, 'get_mkt_pairs', None):
-
-                pairs: dict[str, Struct] = await get_mkt_pairs()
-                for bs_fqme, pair in pairs.items():
-
-                    # NOTE: every backend defined pair should
-                    # declare it's ns path for roundtrip
-                    # serialization lookup.
-                    if not getattr(pair, 'ns_path', None):
-                        raise TypeError(
-                            f'Pair-struct for {self.mod.name} MUST define a '
-                            '`.ns_path: str`!\n'
-                            f'{pair}'
-                        )
-
-                    entry = await self.mod.get_mkt_info(pair.bs_fqme)
-                    if not entry:
-                        continue
-
-                    mkt: MktPair
-                    pair: Struct
-                    mkt, _pair = entry
-                    assert _pair is pair, (
-                        f'`{self.mod.name}` backend probably has a '
-                        'keying-symmetry problem between the pair-`Struct` '
-                        'returned from `Client.get_mkt_pairs()`and the '
-                        'module level endpoint: `.get_mkt_info()`\n\n'
-                        "Here's the struct diff:\n"
-                        f'{_pair - pair}'
-                    )
-                    # NOTE XXX: this means backends MUST implement
-                    # a `Struct.bs_mktid: str` field to provide
-                    # a native-keyed map to their own symbol
-                    # set(s).
-                    self.pairs[pair.bs_mktid] = pair
-
-                    # NOTE: `MktPair`s are keyed here using piker's
-                    # internal FQME schema so that search,
-                    # accounting and feed init can be accomplished
-                    # a sane, uniform, normalized basis.
-                    self.mktmaps[mkt.fqme] = mkt
-
-                self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
-                    pair,
-                )
-
-            else:
-                log.warning(
-                    'No symbology cache `Pair` support for `{provider}`..\n'
-                    'Implement `Client.get_mkt_pairs()`!'
-                )
-
-        return self
-
-    @classmethod
-    def from_dict(
-        cls: type,
-        data: dict,
-        **kwargs,
-    ) -> SymbologyCache:
-
-        # normal init inputs
-        cache = cls(**kwargs)
-
-        # XXX WARNING: this may break if backend namespacing
-        # changes (eg. `Pair` class def is moved to another
-        # module) in which case you can manually update the
-        # `pair_ns_path` in the symcache file and try again.
-        # TODO: probably a verbose error about this?
-        Pair: type = tractor.msg.NamespacePath(
-            str(data['pair_ns_path'])
-        ).load_ref()
-
-        pairtable = data.pop('pairs')
-        for key, pairtable in pairtable.items():
-
-            # allow each serialized pair-dict-table to declare its
-            # specific struct type's path in cases where a backend
-            # supports multiples (normally with different
-            # schemas..) and we are storing them in a flat `.pairs`
-            # table.
-            ThisPair = Pair
-            if this_pair_type := pairtable.get('ns_path'):
-                ThisPair: type = tractor.msg.NamespacePath(
-                    str(this_pair_type)
-                ).load_ref()
-
-            pair: Struct = ThisPair(**pairtable)
-            cache.pairs[key] = pair
-
-        from ..accounting import (
-            Asset,
-            MktPair,
-        )
-
-        # load `dict` -> `Asset`
-        assettable = data.pop('assets')
-        for name, asdict in assettable.items():
-            cache.assets[name] = Asset.from_msg(asdict)
-
-        # load `dict` -> `MktPair`
-        dne: list[str] = []
-        mkttable = data.pop('mktmaps')
-        for fqme, mktdict in mkttable.items():
-
-            mkt = MktPair.from_msg(mktdict)
-            assert mkt.fqme == fqme
-
-            # sanity check asset refs from those (presumably)
-            # loaded asset set above.
-            src: Asset = cache.assets[mkt.src.name]
-            assert src == mkt.src
-            dst: Asset
-            if not (dst := cache.assets.get(mkt.dst.name)):
-                dne.append(mkt.dst.name)
-                continue
-            else:
-                assert dst.name == mkt.dst.name
-
-            cache.mktmaps[fqme] = mkt
-
-        log.warning(
-            f'These `MktPair.dst: Asset`s DNE says `{cache.mod.name}`?\n'
-            f'{pformat(dne)}'
-        )
-        return cache
-
-    @staticmethod
-    async def from_scratch(
-        mod: ModuleType,
-        fp: Path,
-        **kwargs,
-
-    ) -> SymbologyCache:
-        '''
-        Generate (a) new symcache (contents) entirely from scratch
-        including all (TOML) serialized data and file.
-
-        '''
-        log.info(f'GENERATING symbology cache for `{mod.name}`')
-        cache = SymbologyCache(
-            mod=mod,
-            fp=fp,
-            **kwargs,
-        )
-        await cache.load()
-        cache.write_config()
-        return cache
-
-    def search(
-        self,
-        pattern: str,
-        table: str = 'mktmaps'
-
-    ) -> dict[str, Struct]:
-        '''
-        (Fuzzy) search this cache's `.mktmaps` table, which is
-        keyed by FQMEs, for `pattern: str` and return the best
-        matches in a `dict` including the `MktPair` values.
-
-        '''
-        matches = fuzzy.extract(
-            pattern,
-            getattr(self, table),
-            score_cutoff=50,
-        )
-
-        # repack in dict[fqme, MktPair] form
-        return {
-            item[0].fqme: item[0]
-            for item in matches
-        }
-
-
-# actor-process-local in-mem-cache of symcaches (by backend).
-_caches: dict[str, SymbologyCache] = {}
-
-
-def mk_cachefile(
-    provider: str,
-) -> Path:
-    cachedir: Path = config.get_conf_dir() / '_cache'
-    if not cachedir.is_dir():
-        log.info(f'Creating `nativedb` director: {cachedir}')
-        cachedir.mkdir()
-
-    cachefile: Path = cachedir / f'{str(provider)}.symcache.toml'
-    cachefile.touch()
-    return cachefile
-
-
-@acm
-async def open_symcache(
-    mod_or_name: ModuleType | str,
-
-    reload: bool = False,
-    only_from_memcache: bool = False,  # no API req
-    _no_symcache: bool = False,  # no backend support
-
-) -> SymbologyCache:
-
-    if isinstance(mod_or_name, str):
-        mod = get_brokermod(mod_or_name)
-    else:
-        mod: ModuleType = mod_or_name
-
-    provider: str = mod.name
-    cachefile: Path = mk_cachefile(provider)
-
-    # NOTE: certain backends might not support a symbology cache
-    # (easily) and thus we allow for an empty instance to be loaded
-    # and manually filled in at the whim of the caller presuming
-    # the backend pkg-module is annotated appropriately.
-    if (
-        getattr(mod, '_no_symcache', False)
-        or _no_symcache
-    ):
-        yield SymbologyCache(
-            mod=mod,
-            fp=cachefile,
-        )
-        # don't do nuttin
-        return
-
-    # actor-level cache-cache XD
-    global _caches
-    if not reload:
-        try:
-            yield _caches[provider]
-        except KeyError:
-            msg: str = (
-                f'No asset info cache exists yet for `{provider}`'
-            )
-            if only_from_memcache:
-                raise RuntimeError(msg)
-            else:
-                log.warning(msg)
-
-    # if no cache exists or an explicit reload is requested, load
-    # the provider API and call appropriate endpoints to populate
-    # the mkt and asset tables.
-    if (
-        reload
-        or not cachefile.is_file()
-    ):
-        cache = await SymbologyCache.from_scratch(
-            mod=mod,
-            fp=cachefile,
-        )
-
-    else:
-        log.info(
-            f'Loading EXISTING `{mod.name}` symbology cache:\n'
-            f'> {cachefile}'
-        )
-        import time
-        now = time.time()
-        with cachefile.open('rb') as existing_fp:
-            data: dict[str, dict] = tomllib.load(existing_fp)
-            log.runtime(f'SYMCACHE TOML LOAD TIME: {time.time() - now}')
-
-            # if there's an empty file for some reason we need
-            # to do a full reload as well!
-            if not data:
-                cache = await SymbologyCache.from_scratch(
-                    mod=mod,
-                    fp=cachefile,
-                )
-            else:
-                cache = SymbologyCache.from_dict(
-                    data,
-                    mod=mod,
-                    fp=cachefile,
-                )
-
-
-        # TODO: use a real profiling sys..
-        # https://github.com/pikers/piker/issues/337
-        log.info(f'SYMCACHE LOAD TIME: {time.time() - now}')
-
-    yield cache
-
-    # TODO: write only when changes detected? but that should
-    # never happen right except on reload?
-    # cache.write_config()
-
-
-def get_symcache(
-    provider: str,
-    force_reload: bool = False,
-
-) -> SymbologyCache:
-    '''
-    Get any available symbology/assets cache from sync code by
-    (maybe) manually running `trio` to do the work.
-
-    '''
-    # spawn tractor runtime and generate cache
-    # if not existing.
-    async def sched_gen_symcache():
-        async with (
-            # only for runtime's debug mode
-            tractor.open_nursery(debug_mode=True),
-
-            open_symcache(
-                get_brokermod(provider),
-                reload=force_reload,
-            ) as symcache,
-        ):
-            return symcache
-
-    try:
-        symcache: SymbologyCache = trio.run(sched_gen_symcache)
-        assert symcache
-    except BaseException:
-        import pdbp
-        pdbp.xpm()
-
-    return symcache
-
-
-def match_from_pairs(
-    pairs: dict[str, Struct],
-    query: str,
-    score_cutoff: int = 50,
-    **extract_kwargs,
-
-) -> dict[str, Struct]:
-    '''
-    Fuzzy search over a "pairs table" maintained by most backends
-    as part of their symbology-info caching internals.
-
-    Scan the native symbol key set and return best ranked
-    matches back in a new `dict`.
-
-    '''
-
-    # TODO: somehow cache this list (per call) like we were in
-    # `open_symbol_search()`?
-    keys: list[str] = list(pairs)
-    matches: list[tuple[
-        Sequence[Hashable],  # matching input key
-        Any,  # scores
-        Any,
-    ]] = fuzzy.extract(
-        # NOTE: most backends provide keys uppercased
-        query=query,
-        choices=keys,
-        score_cutoff=score_cutoff,
-        **extract_kwargs,
-    )
-
-    # pop and repack pairs in output dict
-    matched_pairs: dict[str, Struct] = {}
-    for item in matches:
-        pair_key: str = item[0]
-        matched_pairs[pair_key] = pairs[pair_key]
-
-    return matched_pairs

piker/data/_web_bs.py

@@ -50,8 +50,8 @@ from trio_websocket._impl import (
     ConnectionTimeout,
 )
 
-from piker.types import Struct
 from ._util import log
+from .types import Struct
 
 
 class NoBsWs:
@@ -234,13 +234,10 @@ async def _reconnect_forever(
             f'{url} trying (RE)CONNECT'
         )
 
-        ws: WebSocketConnection
-        try:
-            async with (
-                trio.open_nursery() as n,
-                open_websocket_url(url) as ws,
-            ):
-                cs = nobsws._cs = n.cancel_scope
+        async with trio.open_nursery() as n:
+            cs = nobsws._cs = n.cancel_scope
+            ws: WebSocketConnection
+            async with open_websocket_url(url) as ws:
                 nobsws._ws = ws
                 log.info(
                     f'{src_mod}\n'
@@ -272,11 +269,9 @@ async def _reconnect_forever(
                 # to let tasks run **inside** the ws open block above.
                 nobsws._connected.set()
                 await trio.sleep_forever()
-        except HandshakeError:
-            log.exception('Retrying connection')
-
-        # ws & nursery block ends
 
+            # ws open block end
+        # nursery block end
         nobsws._connected = trio.Event()
         if cs.cancelled_caught:
             log.cancel(
@@ -289,8 +284,7 @@ async def _reconnect_forever(
                 and not nobsws._connected.is_set()
             )
 
-        # -> from here, move to next reconnect attempt iteration
-        # in the while loop above Bp
+        # -> from here, move to next reconnect attempt
 
     else:
         log.exception(
@@ -359,8 +353,8 @@ async def open_autorecon_ws(
 
 
 '''
-JSONRPC response-request style machinery for transparent multiplexing
-of msgs over a `NoBsWs`.
+JSONRPC response-request style machinery for transparent multiplexing of msgs
+over a NoBsWs.
 
 '''
 
@@ -377,82 +371,43 @@ async def open_jsonrpc_session(
     url: str,
     start_id: int = 0,
     response_type: type = JSONRPCResult,
-    msg_recv_timeout: float = float('inf'),
-    # ^NOTE, since only `deribit` is using this jsonrpc stuff atm
-    # and options mkts are generally "slow moving"..
-    #
-    # FURTHER if we break the underlying ws connection then since we
-    # don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
-    # `_reconnect_forever()`, the jsonrpc "transport pipe" get's
-    # broken and never restored with wtv init sequence is required to
-    # re-establish a working req-resp session.
-
+    request_type: Optional[type] = None,
+    request_hook: Optional[Callable] = None,
+    error_hook: Optional[Callable] = None,
 ) -> Callable[[str, dict], dict]:
-    '''
-    Init a json-RPC-over-websocket connection to the provided `url`.
-
-    A `json_rpc: Callable[[str, dict], dict` is delivered to the
-    caller for sending requests and a bg-`trio.Task` handles
-    processing of response msgs including error reporting/raising in
-    the parent/caller task.
-
-    '''
-    # NOTE, store all request msgs so we can raise errors on the
-    # caller side!
-    req_msgs: dict[int, dict] = {}
 
     async with (
-        trio.open_nursery() as tn,
-        open_autorecon_ws(
-            url=url,
-            msg_recv_timeout=msg_recv_timeout,
-        ) as ws
+        trio.open_nursery() as n,
+        open_autorecon_ws(url) as ws
     ):
-        rpc_id: Iterable[int] = count(start_id)
+        rpc_id: Iterable = count(start_id)
         rpc_results: dict[int, dict] = {}
 
-        async def json_rpc(
-            method: str,
-            params: dict,
-        ) -> dict:
+        async def json_rpc(method: str, params: dict) -> dict:
             '''
             perform a json rpc call and wait for the result, raise exception in
             case of error field present on response
             '''
-            nonlocal req_msgs
-
-            req_id: int = next(rpc_id)
             msg = {
                 'jsonrpc': '2.0',
-                'id': req_id,
+                'id': next(rpc_id),
                 'method': method,
                 'params': params
             }
             _id = msg['id']
 
-            result = rpc_results[_id] = {
+            rpc_results[_id] = {
                 'result': None,
-                'error': None,
-                'event': trio.Event(),  # signal caller resp arrived
+                'event': trio.Event()
             }
-            req_msgs[_id] = msg
 
             await ws.send_msg(msg)
 
-            # wait for reponse before unblocking requester code
             await rpc_results[_id]['event'].wait()
 
-            if (maybe_result := result['result']):
-                ret = maybe_result
-                del rpc_results[_id]
+            ret = rpc_results[_id]['result']
 
-            else:
-                err = result['error']
-                raise Exception(
-                    f'JSONRPC request failed\n'
-                    f'req: {msg}\n'
-                    f'resp: {err}\n'
-                )
+            del rpc_results[_id]
 
             if ret.error is not None:
                 raise Exception(json.dumps(ret.error, indent=4))
@@ -467,7 +422,6 @@ async def open_jsonrpc_session(
             the server side.
 
             '''
-            nonlocal req_msgs
             async for msg in ws:
                 match msg:
                     case {
@@ -491,28 +445,19 @@ async def open_jsonrpc_session(
                         'params': _,
                     }:
                         log.debug(f'Recieved\n{msg}')
+                        if request_hook:
+                            await request_hook(request_type(**msg))
 
                     case {
                         'error': error
                     }:
-                        # retreive orig request msg, set error
-                        # response in original "result" msg,
-                        # THEN FINALLY set the event to signal caller
-                        # to raise the error in the parent task.
-                        req_id: int = error['id']
-                        req_msg: dict = req_msgs[req_id]
-                        result: dict = rpc_results[req_id]
-                        result['error'] = error
-                        result['event'].set()
-                        log.error(
-                            f'JSONRPC request failed\n'
-                            f'req: {req_msg}\n'
-                            f'resp: {error}\n'
-                        )
+                        log.warning(f'Recieved\n{error}')
+                        if error_hook:
+                            await error_hook(response_type(**msg))
 
                     case _:
                         log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
 
-        tn.start_soon(recv_task)
+        n.start_soon(recv_task)
         yield json_rpc
-        tn.cancel_scope.cancel()
+        n.cancel_scope.cancel()

piker/data/cli.py

@@ -0,0 +1,255 @@
+# piker: trading gear for hackers
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+"""
+marketstore cli.
+
+"""
+import trio
+import tractor
+import click
+
+# from ..service.marketstore import (
+#     # get_client,
+#     # stream_quotes,
+#     ingest_quote_stream,
+#     # _url,
+#     # _tick_tbk_ids,
+#     # mk_tbk,
+# )
+from ..cli import cli
+from .. import watchlists as wl
+from ._util import (
+    log,
+)
+
+
+@cli.command()
+@click.option(
+    '--url',
+    default='ws://localhost:5993/ws',
+    help='HTTP URL of marketstore instance'
+)
+@click.argument('names', nargs=-1)
+@click.pass_obj
+def ms_stream(
+    config: dict,
+    names: list[str],
+    url: str,
+) -> None:
+    '''
+    Connect to a marketstore time bucket stream for (a set of) symbols(s)
+    and print to console.
+
+    '''
+    async def main():
+        # async for quote in stream_quotes(symbols=names):
+        #    log.info(f"Received quote:\n{quote}")
+        ...
+
+    trio.run(main)
+
+
+# @cli.command()
+# @click.option(
+#     '--url',
+#     default=_url,
+#     help='HTTP URL of marketstore instance'
+# )
+# @click.argument('names', nargs=-1)
+# @click.pass_obj
+# def ms_destroy(config: dict, names: list[str], url: str) -> None:
+#     """Destroy symbol entries in the local marketstore instance.
+#     """
+#     async def main():
+#         nonlocal names
+#         async with get_client(url) as client:
+#
+#             if not names:
+#                 names = await client.list_symbols()
+#
+#             # default is to wipe db entirely.
+#             answer = input(
+#                 "This will entirely wipe you local marketstore db @ "
+#                 f"{url} of the following symbols:\n {pformat(names)}"
+#                 "\n\nDelete [N/y]?\n")
+#
+#             if answer == 'y':
+#                 for sym in names:
+#                     # tbk = _tick_tbk.format(sym)
+#                     tbk = tuple(sym, *_tick_tbk_ids)
+#                     print(f"Destroying {tbk}..")
+#                     await client.destroy(mk_tbk(tbk))
+#             else:
+#                 print("Nothing deleted.")
+#
+#     tractor.run(main)
+
+
+@cli.command()
+@click.option(
+    '--tsdb_host',
+    default='localhost'
+)
+@click.option(
+    '--tsdb_port',
+    default=5993
+)
+@click.argument('symbols', nargs=-1)
+@click.pass_obj
+def storesh(
+    config,
+    tl,
+    host,
+    port,
+    symbols: list[str],
+):
+    '''
+    Start an IPython shell ready to query the local marketstore db.
+
+    '''
+    from piker.data.marketstore import open_tsdb_client
+    from piker.service import open_piker_runtime
+
+    async def main():
+        nonlocal symbols
+
+        async with open_piker_runtime(
+            'storesh',
+            enable_modules=['piker.service._ahab'],
+        ):
+            symbol = symbols[0]
+
+            async with open_tsdb_client(symbol):
+                # TODO: ask if user wants to write history for detected
+                # available shm buffers?
+                from tractor.trionics import ipython_embed
+                await ipython_embed()
+
+    trio.run(main)
+
+
+@cli.command()
+@click.option(
+    '--host',
+    default='localhost'
+)
+@click.option(
+    '--port',
+    default=5993
+)
+@click.option(
+    '--delete',
+    '-d',
+    is_flag=True,
+    help='Delete history (1 Min) for symbol(s)',
+)
+@click.argument('symbols', nargs=-1)
+@click.pass_obj
+def storage(
+    config,
+    host,
+    port,
+    symbols: list[str],
+    delete: bool,
+
+):
+    '''
+    Start an IPython shell ready to query the local marketstore db.
+
+    '''
+    from piker.service.marketstore import open_tsdb_client
+    from piker.service import open_piker_runtime
+
+    async def main():
+        nonlocal symbols
+
+        async with open_piker_runtime(
+            'tsdb_storage',
+            enable_modules=['piker.service._ahab'],
+        ):
+            symbol = symbols[0]
+            async with open_tsdb_client(symbol) as storage:
+                if delete:
+                    for fqme in symbols:
+                        syms = await storage.client.list_symbols()
+
+                        resp60s = await storage.delete_ts(fqme, 60)
+
+                        msgish = resp60s.ListFields()[0][1]
+                        if 'error' in str(msgish):
+
+                            # TODO: MEGA LOL, apparently the symbols don't
+                            # flush out until you refresh something or other
+                            # (maybe the WALFILE)... #lelandorlulzone, classic
+                            # alpaca(Rtm) design here ..
+                            # well, if we ever can make this work we
+                            # probably want to dogsplain the real reason
+                            # for the delete errurz..llululu
+                            if fqme not in syms:
+                                log.error(f'Pair {fqme} dne in DB')
+
+                            log.error(f'Deletion error: {fqme}\n{msgish}')
+
+                        resp1s = await storage.delete_ts(fqme, 1)
+                        msgish = resp1s.ListFields()[0][1]
+                        if 'error' in str(msgish):
+                            log.error(f'Deletion error: {fqme}\n{msgish}')
+
+    trio.run(main)
+
+
+# @cli.command()
+# @click.option('--test-file', '-t', help='Test quote stream file')
+# @click.option('--tl', is_flag=True, help='Enable tractor logging')
+# @click.argument('name', nargs=1, required=True)
+# @click.pass_obj
+# def ingest(config, name, test_file, tl):
+#     '''
+#     Ingest real-time broker quotes and ticks to a marketstore instance.
+
+#     '''
+#     # global opts
+#     loglevel = config['loglevel']
+#     tractorloglevel = config['tractorloglevel']
+#     # log = config['log']
+
+#     watchlist_from_file = wl.ensure_watchlists(config['wl_path'])
+#     watchlists = wl.merge_watchlist(watchlist_from_file, wl._builtins)
+#     symbols = watchlists[name]
+
+#     grouped_syms = {}
+#     for sym in symbols:
+#         symbol, _, provider = sym.rpartition('.')
+#         if provider not in grouped_syms:
+#             grouped_syms[provider] = []
+
+#         grouped_syms[provider].append(symbol)
+
+#     async def entry_point():
+#         async with tractor.open_nursery() as n:
+#             for provider, symbols in grouped_syms.items():
+#                 await n.run_in_actor(
+#                     ingest_quote_stream,
+#                     name='ingest_marketstore',
+#                     symbols=symbols,
+#                     brokername=provider,
+#                     tries=1,
+#                     actorloglevel=loglevel,
+#                     loglevel=tractorloglevel
+#                 )
+
+#     tractor.run(entry_point)

piker/data/feed.py

@@ -28,7 +28,6 @@ module.
 from __future__ import annotations
 from collections import (
     defaultdict,
-    abc,
 )
 from contextlib import asynccontextmanager as acm
 from functools import partial
@@ -37,70 +36,51 @@ from types import ModuleType
 from typing import (
     Any,
     AsyncContextManager,
+    Optional,
     Awaitable,
     Sequence,
+    Union,
 )
 
 import trio
 from trio.abc import ReceiveChannel
 from trio_typing import TaskStatus
 import tractor
-from tractor import trionics
+from tractor.trionics import (
+    maybe_open_context,
+    gather_contexts,
+)
 
-from piker.accounting import (
-    MktPair,
-    unpack_fqme,
-)
-from piker.types import Struct
-from piker.brokers import get_brokermod
-from piker.service import (
-    maybe_spawn_brokerd,
-)
-from piker.calc import humanize
+from ..brokers import get_brokermod
+from ..calc import humanize
 from ._util import (
     log,
     get_console_log,
 )
+from ..service import (
+    maybe_spawn_brokerd,
+)
 from .flows import Flume
 from .validate import (
     FeedInit,
     validate_backend,
 )
-from ..tsp import (
+from .history import (
     manage_history,
 )
 from .ingest import get_ingestormod
+from .types import Struct
+from ..accounting import (
+    MktPair,
+    unpack_fqme,
+)
+from ..ui import _search
 from ._sampling import (
     sample_and_broadcast,
     uniform_rate_send,
 )
 
 
-class Sub(Struct, frozen=True):
-    '''
-    A live feed subscription entry.
-
-    Contains meta-data on the remote-actor type (in functionality
-    terms) as well as refs to IPC streams and sampler runtime
-    params.
-
-    '''
-    ipc: tractor.MsgStream
-    send_chan: trio.abc.SendChannel | None = None
-
-    # tick throttle rate in Hz; determines how live
-    # quotes/ticks should be downsampled before relay
-    # to the receiving remote consumer (process).
-    throttle_rate:  float | None  = None
-    _throttle_cs: trio.CancelScope | None = None
-
-    # TODO: actually stash comms info for the far end to allow
-    # `.tsp`, `.fsp` and `.data._sampling` sub-systems to re-render
-    # the data view as needed via msging with the `._remote_ctl`
-    # ipc ctx.
-    rc_ui: bool = False
-
-
 class _FeedsBus(Struct):
     '''
     Data feeds broadcaster and persistence management.
@@ -125,7 +105,13 @@ class _FeedsBus(Struct):
 
     _subscribers: defaultdict[
         str,
-        set[Sub]
+        set[
+            tuple[
+                tractor.MsgStream | trio.MemorySendChannel,
+                # tractor.Context,
+                float | None,  # tick throttle in Hz
+            ]
+        ]
     ] = defaultdict(set)
 
     async def start_task(
@@ -140,8 +126,6 @@ class _FeedsBus(Struct):
                 trio.CancelScope] = trio.TASK_STATUS_IGNORED,
         ) -> None:
             with trio.CancelScope() as cs:
-                # TODO: shouldn't this be a direct await to avoid
-                # cancellation contagion to the bus nursery!?!?!
                 await self.nursery.start(
                     target,
                     *args,
@@ -159,28 +143,33 @@ class _FeedsBus(Struct):
     def get_subs(
         self,
         key: str,
-
-    ) -> set[Sub]:
+    ) -> set[
+        tuple[
+            Union[tractor.MsgStream, trio.MemorySendChannel],
+            # tractor.Context,
+            float | None,  # tick throttle in Hz
+        ]
+    ]:
         '''
         Get the ``set`` of consumer subscription entries for the given key.
 
         '''
         return self._subscribers[key]
 
-    def subs_items(self) -> abc.ItemsView[str, set[Sub]]:
-        return self._subscribers.items()
-
     def add_subs(
         self,
         key: str,
-        subs: set[Sub],
-
-    ) -> set[Sub]:
+        subs: set[tuple[
+            tractor.MsgStream | trio.MemorySendChannel,
+            # tractor.Context,
+            float | None,  # tick throttle in Hz
+        ]],
+    ) -> set[tuple]:
         '''
         Add a ``set`` of consumer subscription entries for the given key.
 
         '''
-        _subs: set[Sub] = self._subscribers.setdefault(key, set())
+        _subs = self._subscribers[key]
         _subs.update(subs)
         return _subs
 
@@ -194,7 +183,7 @@ class _FeedsBus(Struct):
         Remove a ``set`` of consumer subscription entries for key.
 
         '''
-        _subs: set[tuple] = self.get_subs(key)
+        _subs = self.get_subs(key)
         _subs.difference_update(subs)
         return _subs
 
@@ -204,7 +193,7 @@ _bus: _FeedsBus = None
 
 def get_feed_bus(
     brokername: str,
-    nursery: trio.Nursery | None = None,
+    nursery: Optional[trio.Nursery] = None,
 
 ) -> _FeedsBus:
     '''
@@ -237,7 +226,6 @@ async def allocate_persistent_feed(
 
     loglevel: str,
     start_stream: bool = True,
-    init_timeout: float = 616,
 
     task_status: TaskStatus[FeedInit] = trio.TASK_STATUS_IGNORED,
 
@@ -279,23 +267,22 @@ async def allocate_persistent_feed(
     # TODO: probably make a struct msg type for this as well
     # since eventually we do want to have more efficient IPC..
     first_quote: dict[str, Any]
-    with trio.fail_after(init_timeout):
-        (
-            init_msgs,
-            first_quote,
-        ) = await bus.nursery.start(
-            partial(
-                mod.stream_quotes,
-                send_chan=send,
-                feed_is_live=feed_is_live,
+    (
+        init_msgs,
+        first_quote,
+    ) = await bus.nursery.start(
+        partial(
+            mod.stream_quotes,
+            send_chan=send,
+            feed_is_live=feed_is_live,
 
-                # NOTE / TODO: eventualy we may support providing more then
-                # one input here such that a datad daemon can multiplex
-                # multiple live feeds from one task, instead of getting
-                # a new request (and thus new task) for each subscription.
-                symbols=[symstr],
-            )
+            # NOTE / TODO: eventualy we may support providing more then
+            # one input here such that a datad daemon can multiplex
+            # multiple live feeds from one task, instead of getting
+            # a new request (and thus new task) for each subscription.
+            symbols=[symstr],
         )
+    )
 
     # TODO: this is indexed by symbol for now since we've planned (for
     # some time) to expect backends to handle single
@@ -345,6 +332,7 @@ async def allocate_persistent_feed(
     ) = await bus.nursery.start(
         manage_history,
         mod,
+        bus,
         mkt,
         some_data_ready,
         feed_is_live,
@@ -352,7 +340,7 @@ async def allocate_persistent_feed(
 
     # yield back control to starting nursery once we receive either
     # some history or a real-time quote.
-    log.info(f'loading OHLCV history: {fqme}')
+    log.info(f'waiting on history to load: {fqme}')
     await some_data_ready.wait()
 
     flume = Flume(
@@ -382,8 +370,7 @@ async def allocate_persistent_feed(
         mkt.bs_fqme: flume,
     })
 
-    # signal the ``open_feed_bus()`` caller task to continue since
-    # we now have (some) history pushed to the shm buffer.
+    # signal the ``open_feed_bus()`` caller task to continue
     task_status.started(init)
 
     if not start_stream:
@@ -421,13 +408,7 @@ async def allocate_persistent_feed(
         rt_shm.array['time'][1] = ts + 1
 
     elif hist_shm.array.size == 0:
-        for i in range(100):
-            await trio.sleep(0.1)
-            if hist_shm.array.size > 0:
-                break
-        else:
-            await tractor.pause()
-            raise RuntimeError(f'History (1m) Shm for {fqme} is empty!?')
+        raise RuntimeError(f'History (1m) Shm for {fqme} is empty!?')
 
     # wait the spawning parent task to register its subscriber
     # send-stream entry before we start the sample loop.
@@ -457,9 +438,8 @@ async def open_feed_bus(
     symbols: list[str],  # normally expected to the broker-specific fqme
 
     loglevel: str = 'error',
-    tick_throttle: float | None = None,
+    tick_throttle: Optional[float] = None,
     start_stream: bool = True,
-    allow_remote_ctl_ui: bool = False,
 
 ) -> dict[
     str,  # fqme
@@ -474,12 +454,8 @@ async def open_feed_bus(
     if loglevel is None:
         loglevel = tractor.current_actor().loglevel
 
-    # XXX: required to propagate ``tractor`` loglevel to piker
-    # logging
-    get_console_log(
-        loglevel
-        or tractor.current_actor().loglevel
-    )
+    # XXX: required to propagate ``tractor`` loglevel to piker logging
+    get_console_log(loglevel or tractor.current_actor().loglevel)
 
     # local state sanity checks
     # TODO: check for any stale shm entries for this symbol
@@ -489,7 +465,7 @@ async def open_feed_bus(
     assert 'brokerd' in servicename
     assert brokername in servicename
 
-    bus: _FeedsBus = get_feed_bus(brokername)
+    bus = get_feed_bus(brokername)
     sub_registered = trio.Event()
 
     flumes: dict[str, Flume] = {}
@@ -536,10 +512,10 @@ async def open_feed_bus(
         # pack for ``.started()`` sync msg
         flumes[fqme] = flume
 
-        # we use the broker-specific fqme (bs_fqme) for the sampler
-        # subscription since the backend isn't (yet) expected to
-        # append it's own name to the fqme, so we filter on keys
-        # which *do not* include that name (e.g .ib) .
+        # we use the broker-specific fqme (bs_fqme) for the
+        # sampler subscription since the backend isn't (yet) expected to
+        # append it's own name to the fqme, so we filter on keys which
+        # *do not* include that name (e.g .ib) .
         bus._subscribers.setdefault(bs_fqme, set())
 
     # sync feed subscribers with flume handles
@@ -578,60 +554,49 @@ async def open_feed_bus(
             # that the ``sample_and_broadcast()`` task (spawned inside
             # ``allocate_persistent_feed()``) will push real-time quote
             # (ticks) to this new consumer.
-            cs: trio.CancelScope | None = None
-            send: trio.MemorySendChannel | None = None
+
             if tick_throttle:
                 flume.throttle_rate = tick_throttle
 
-                # open a bg task which receives quotes over a mem
-                # chan and only pushes them to the target
-                # actor-consumer at a max ``tick_throttle``
-                # (instantaneous) rate.
+                # open a bg task which receives quotes over a mem chan
+                # and only pushes them to the target actor-consumer at
+                # a max ``tick_throttle`` instantaneous rate.
                 send, recv = trio.open_memory_channel(2**10)
 
-                # NOTE: the ``.send`` channel here is a swapped-in
-                # trio mem chan which gets `.send()`-ed by the normal
-                # sampler task but instead of being sent directly
-                # over the IPC msg stream it's the throttle task
-                # does the work of incrementally forwarding to the
-                # IPC stream at the throttle rate.
-                cs: trio.CancelScope = await bus.start_task(
+                cs = await bus.start_task(
                     uniform_rate_send,
                     tick_throttle,
                     recv,
                     stream,
                 )
+                # NOTE: so the ``send`` channel here is actually a swapped
+                # in trio mem chan which gets pushed by the normal sampler
+                # task but instead of being sent directly over the IPC msg
+                # stream it's the throttle task does the work of
+                # incrementally forwarding to the IPC stream at the throttle
+                # rate.
+                send._ctx = ctx  # mock internal ``tractor.MsgStream`` ref
+                sub = (send, tick_throttle)
 
-            sub = Sub(
-                ipc=stream,
-                send_chan=send,
-                throttle_rate=tick_throttle,
-                _throttle_cs=cs,
-                rc_ui=allow_remote_ctl_ui,
-            )
+            else:
+                sub = (stream, tick_throttle)
 
             # TODO: add an api for this on the bus?
             # maybe use the current task-id to key the sub list that's
             # added / removed? Or maybe we can add a general
             # pause-resume by sub-key api?
             bs_fqme = fqme.removesuffix(f'.{brokername}')
-            local_subs.setdefault(
-                bs_fqme,
-                set()
-            ).add(sub)
-            bus.add_subs(
-                bs_fqme,
-                {sub}
-            )
+            local_subs.setdefault(bs_fqme, set()).add(sub)
+            bus.add_subs(bs_fqme, {sub})
 
         # sync caller with all subs registered state
         sub_registered.set()
 
-        uid: tuple[str, str] = ctx.chan.uid
+        uid = ctx.chan.uid
         try:
-            # ctrl protocol for start/stop of live quote streams
-            # based on UI state (eg. don't need a stream when
-            # a symbol isn't being displayed).
+            # ctrl protocol for start/stop of quote streams based on UI
+            # state (eg. don't need a stream when a symbol isn't being
+            # displayed).
             async for msg in stream:
 
                 if msg == 'pause':
@@ -677,12 +642,9 @@ class Feed(Struct):
     '''
     mods: dict[str, ModuleType] = {}
     portals: dict[ModuleType, tractor.Portal] = {}
-    flumes: dict[
-        str,  # FQME
-        Flume,
-    ] = {}
+    flumes: dict[str, Flume] = {}
     streams: dict[
-        str,  # broker name
+        str,
         trio.abc.ReceiveChannel[dict[str, Any]],
     ] = {}
 
@@ -756,7 +718,7 @@ async def install_brokerd_search(
 
     async with portal.open_context(
         brokermod.open_symbol_search
-    ) as (ctx, _):
+    ) as (ctx, cache):
 
         # shield here since we expect the search rpc to be
         # cancellable by the user as they see fit.
@@ -769,7 +731,6 @@ async def install_brokerd_search(
                 except trio.EndOfChannel:
                     return {}
 
-            from piker.ui import _search
             async with _search.register_symbol_search(
 
                 provider_name=brokermod.name,
@@ -788,7 +749,7 @@ async def install_brokerd_search(
 async def maybe_open_feed(
 
     fqmes: list[str],
-    loglevel: str | None = None,
+    loglevel: Optional[str] = None,
 
     **kwargs,
 
@@ -804,7 +765,7 @@ async def maybe_open_feed(
     '''
     fqme = fqmes[0]
 
-    async with trionics.maybe_open_context(
+    async with maybe_open_context(
         acm_func=open_feed,
         kwargs={
             'fqmes': fqmes,
@@ -824,7 +785,7 @@ async def maybe_open_feed(
             # add a new broadcast subscription for the quote stream
             # if this feed is likely already in use
 
-            async with trionics.gather_contexts(
+            async with gather_contexts(
                 mngrs=[stream.subscribe() for stream in feed.streams.values()]
             ) as bstreams:
                 for bstream, flume in zip(bstreams, feed.flumes.values()):
@@ -848,8 +809,6 @@ async def open_feed(
     start_stream: bool = True,
     tick_throttle: float | None = None,  # Hz
 
-    allow_remote_ctl_ui: bool = False,
-
 ) -> Feed:
     '''
     Open a "data feed" which provides streamed real-time quotes.
@@ -886,7 +845,7 @@ async def open_feed(
         )
 
     portals: tuple[tractor.Portal]
-    async with trionics.gather_contexts(
+    async with gather_contexts(
         brokerd_ctxs,
     ) as portals:
 
@@ -932,19 +891,13 @@ async def open_feed(
                     # of these stream open sequences sequentially per
                     # backend? .. need some thot!
                     allow_overruns=True,
-
-                    # NOTE: UI actors (like charts) can allow
-                    # remote control of certain graphics rendering
-                    # capabilities via the
-                    # `.ui._remote_ctl.remote_annotate()` msg loop.
-                    allow_remote_ctl_ui=allow_remote_ctl_ui,
                 )
             )
 
         assert len(feed.mods) == len(feed.portals)
 
         async with (
-            trionics.gather_contexts(bus_ctxs) as ctxs,
+            gather_contexts(bus_ctxs) as ctxs,
         ):
             stream_ctxs: list[tractor.MsgStream] = []
             for (
@@ -954,7 +907,6 @@ async def open_feed(
 
                 for fqme, flume_msg in flumes_msg_dict.items():
                     flume = Flume.from_msg(flume_msg)
-
                     # assert flume.mkt.fqme == fqme
                     feed.flumes[fqme] = flume
 
@@ -986,7 +938,7 @@ async def open_feed(
             brokermod: ModuleType
             fqmes: list[str]
             async with (
-                trionics.gather_contexts(stream_ctxs) as streams,
+                gather_contexts(stream_ctxs) as streams,
             ):
                 for (
                     stream,
@@ -1002,12 +954,6 @@ async def open_feed(
                         if brokermod.name == flume.mkt.broker:
                             flume.stream = stream
 
-                assert (
-                    len(feed.mods)
-                    ==
-                    len(feed.portals)
-                    ==
-                    len(feed.streams)
-                )
+                assert len(feed.mods) == len(feed.portals) == len(feed.streams)
 
                 yield feed

piker/data/flows.py

@@ -30,27 +30,53 @@ import tractor
 import pendulum
 import numpy as np
 
-from piker.types import Struct
+from ..accounting import MktPair
+from ._util import log
+from .types import Struct
 from ._sharedmem import (
     attach_shm_array,
     ShmArray,
     _Token,
 )
+# from .._profile import (
+#     Profiler,
+#     pg_profile_enabled,
+# )
 
 if TYPE_CHECKING:
-    from ..accounting import MktPair
+    # from pyqtgraph import PlotItem
     from .feed import Feed
 
 
+# TODO: ideas for further abstractions as per
+# https://github.com/pikers/piker/issues/216 and
+# https://github.com/pikers/piker/issues/270:
+# - a ``Cascade`` would be the minimal "connection" of 2 ``Flumes``
+#   as per circuit parlance:
+#   https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
+#     - could cover the combination of our `FspAdmin` and the
+#       backend `.fsp._engine` related machinery to "connect" one flume
+#       to another?
+# - a (financial signal) ``Flow`` would be the a "collection" of such
+#    minmial cascades. Some engineering based jargon concepts:
+#     - https://en.wikipedia.org/wiki/Signal_chain
+#     - https://en.wikipedia.org/wiki/Daisy_chain_(electrical_engineering)
+#     - https://en.wikipedia.org/wiki/Audio_signal_flow
+#     - https://en.wikipedia.org/wiki/Digital_signal_processing#Implementation
+#     - https://en.wikipedia.org/wiki/Dataflow_programming
+#     - https://en.wikipedia.org/wiki/Signal_programming
+#     - https://en.wikipedia.org/wiki/Incremental_computing
+
+
 class Flume(Struct):
     '''
-    Composite reference type which points to all the addressing
-    handles and other meta-data necessary for the read, measure and
-    management of a set of real-time updated data flows.
+    Composite reference type which points to all the addressing handles
+    and other meta-data necessary for the read, measure and management
+    of a set of real-time updated data flows.
 
     Can be thought of as a "flow descriptor" or "flow frame" which
-    describes the high level properties of a set of data flows that
-    can be used seamlessly across process-memory boundaries.
+    describes the high level properties of a set of data flows that can
+    be used seamlessly across process-memory boundaries.
 
     Each instance's sub-components normally includes:
      - a msg oriented quote stream provided via an IPC transport
@@ -73,7 +99,6 @@ class Flume(Struct):
     # private shm refs loaded dynamically from tokens
     _hist_shm: ShmArray | None = None
     _rt_shm: ShmArray | None = None
-    _readonly: bool = True
 
     stream: tractor.MsgStream | None = None
     izero_hist: int = 0
@@ -90,7 +115,7 @@ class Flume(Struct):
         if self._rt_shm is None:
             self._rt_shm = attach_shm_array(
                 token=self._rt_shm_token,
-                readonly=self._readonly,
+                readonly=True,
             )
 
         return self._rt_shm
@@ -103,10 +128,12 @@ class Flume(Struct):
                 'No shm token has been set for the history buffer?'
             )
 
-        if self._hist_shm is None:
+        if (
+            self._hist_shm is None
+        ):
             self._hist_shm = attach_shm_array(
                 token=self._hist_shm_token,
-                readonly=self._readonly,
+                readonly=True,
             )
 
         return self._hist_shm
@@ -125,10 +152,10 @@ class Flume(Struct):
         period and ratio between them.
 
         '''
-        times: np.ndarray = self.hist_shm.array['time']
-        end: float | int = pendulum.from_timestamp(times[-1])
-        start: float | int = pendulum.from_timestamp(times[times != times[-1]][-1])
-        hist_step_size_s: float = (end - start).seconds
+        times = self.hist_shm.array['time']
+        end = pendulum.from_timestamp(times[-1])
+        start = pendulum.from_timestamp(times[times != times[-1]][-1])
+        hist_step_size_s = (end - start).seconds
 
         times = self.rt_shm.array['time']
         end = pendulum.from_timestamp(times[-1])
@@ -148,25 +175,17 @@ class Flume(Struct):
         msg = self.to_dict()
         msg['mkt'] = self.mkt.to_dict()
 
-        # NOTE: pop all un-msg-serializable fields:
-        # - `tractor.MsgStream`
-        # - `Feed`
-        # - `Shmarray`
-        # it's expected the `.from_msg()` on the other side
-        # will get instead some kind of msg-compat version
-        # that it can load.
+        # can't serialize the stream or feed objects, it's expected
+        # you'll have a ref to it since this msg should be rxed on
+        # a stream on whatever far end IPC..
         msg.pop('stream')
         msg.pop('feed')
-        msg.pop('_rt_shm')
-        msg.pop('_hist_shm')
-
         return msg
 
     @classmethod
     def from_msg(
         cls,
         msg: dict,
-        readonly: bool = True,
 
     ) -> dict:
         '''
@@ -175,13 +194,8 @@ class Flume(Struct):
 
         '''
         mkt_msg = msg.pop('mkt')
-        from ..accounting import MktPair  # cycle otherwise..
         mkt = MktPair.from_msg(mkt_msg)
-        msg |= {'_readonly': readonly}
-        return cls(
-            mkt=mkt,
-            **msg,
-        )
+        return cls(mkt=mkt, **msg)
 
     def get_index(
         self,
@@ -219,3 +233,5 @@ class Flume(Struct):
             np.all(np.isin(vlm, -1))
             or np.all(np.isnan(vlm))
         )
+
+

piker/data/history.py

@@ -0,0 +1,810 @@
+# piker: trading gear for hackers
+# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Historical data business logic for load, backfill and tsdb storage.
+
+'''
+from __future__ import annotations
+from collections import (
+    Counter,
+)
+from datetime import datetime
+from functools import partial
+import time
+from types import ModuleType
+from typing import (
+    Callable,
+    Optional,
+    TYPE_CHECKING,
+)
+
+import trio
+from trio_typing import TaskStatus
+import tractor
+import pendulum
+import numpy as np
+
+from .. import config
+from ..accounting import (
+    MktPair,
+)
+from ._util import (
+    log,
+)
+from ..service import (
+    check_for_service,
+)
+from ._sharedmem import (
+    maybe_open_shm_array,
+    ShmArray,
+    _secs_in_day,
+)
+from ._source import base_iohlc_dtype
+from ._sampling import (
+    open_sample_stream,
+)
+from ..brokers._util import (
+    DataUnavailable,
+)
+
+if TYPE_CHECKING:
+    from ..service.marketstore import Storage
+    from .feed import _FeedsBus
+
+
+def diff_history(
+    array: np.ndarray,
+    timeframe: int,
+    start_dt: datetime,
+    end_dt: datetime,
+    last_tsdb_dt: datetime | None = None
+
+) -> np.ndarray:
+
+    # no diffing with tsdb dt index possible..
+    if last_tsdb_dt is None:
+        return array
+
+    time = array['time']
+    return array[time > last_tsdb_dt.timestamp()]
+
+
+async def start_backfill(
+    mod: ModuleType,
+    mkt: MktPair,
+    shm: ShmArray,
+    timeframe: float,
+    sampler_stream: tractor.MsgStream,
+    feed_is_live: trio.Event,
+
+    last_tsdb_dt: Optional[datetime] = None,
+    storage: Optional[Storage] = None,
+    write_tsdb: bool = True,
+    tsdb_is_up: bool = False,
+
+    task_status: TaskStatus[tuple] = trio.TASK_STATUS_IGNORED,
+
+) -> int:
+
+    hist: Callable[
+        [int, datetime, datetime],
+        tuple[np.ndarray, str]
+    ]
+    config: dict[str, int]
+
+    async with mod.open_history_client(
+        mkt,
+    ) as (hist, config):
+        log.info(f'{mod} history client returned backfill config: {config}')
+
+        # get latest query's worth of history all the way
+        # back to what is recorded in the tsdb
+        array, start_dt, end_dt = await hist(
+            timeframe,
+            end_dt=None,
+        )
+        times = array['time']
+
+        # sample period step size in seconds
+        step_size_s = (
+            pendulum.from_timestamp(times[-1])
+            - pendulum.from_timestamp(times[-2])
+        ).seconds
+
+        # if the market is open (aka we have a live feed) but the
+        # history sample step index seems off we report the surrounding
+        # data and drop into a bp. this case shouldn't really ever
+        # happen if we're doing history retrieval correctly.
+        if (
+            step_size_s == 60
+            and feed_is_live.is_set()
+        ):
+            inow = round(time.time())
+            diff = inow - times[-1]
+            if abs(diff) > 60:
+                surr = array[-6:]
+                diff_in_mins = round(diff/60., ndigits=2)
+                log.warning(
+                    f'STEP ERROR `{mkt.fqme}` for period {step_size_s}s:\n'
+                    f'Off by `{diff}` seconds (or `{diff_in_mins}` mins)\n'
+                    'Surrounding 6 time stamps:\n'
+                    f'{list(surr["time"])}\n'
+                    'Here is surrounding 6 samples:\n'
+                    f'{surr}\nn'
+                )
+
+                # uncomment this for a hacker who wants to investigate
+                # this case manually..
+                # await tractor.breakpoint()
+
+        # frame's worth of sample-period-steps, in seconds
+        frame_size_s = len(array) * step_size_s
+
+        to_push = diff_history(
+            array,
+            timeframe,
+            start_dt,
+            end_dt,
+            last_tsdb_dt=last_tsdb_dt,
+        )
+
+        log.info(f'Pushing {to_push.size} to shm!')
+        shm.push(to_push, prepend=True)
+
+        # TODO: *** THIS IS A BUG ***
+        # we need to only broadcast to subscribers for this fqme..
+        # otherwise all fsps get reset on every chart..
+        await sampler_stream.send('broadcast_all')
+
+        # signal that backfilling to tsdb's end datum is complete
+        bf_done = trio.Event()
+
+        # let caller unblock and deliver latest history frame
+        task_status.started((
+            start_dt,
+            end_dt,
+            bf_done,
+        ))
+
+        # based on the sample step size, maybe load a certain amount history
+        if last_tsdb_dt is None:
+            if step_size_s not in (1, 60):
+                raise ValueError(
+                    '`piker` only needs to support 1m and 1s sampling '
+                    'but ur api is trying to deliver a longer '
+                    f'timeframe of {step_size_s} seconds..\n'
+                    'So yuh.. dun do dat brudder.'
+                )
+
+            # when no tsdb "last datum" is provided, we just load
+            # some near-term history.
+            periods = {
+                1: {'days': 1},
+                60: {'days': 14},
+            }
+
+            if tsdb_is_up:
+                # do a decently sized backfill and load it into storage.
+                periods = {
+                    1: {'days': 6},
+                    60: {'years': 6},
+                }
+
+            period_duration = periods[step_size_s]
+
+            # NOTE: manually set the "latest" datetime which we intend to
+            # backfill history "until" so as to adhere to the history
+            # settings above when the tsdb is detected as being empty.
+            last_tsdb_dt = start_dt.subtract(**period_duration)
+
+        # configure async query throttling
+        # rate = config.get('rate', 1)
+        # XXX: legacy from ``trimeter`` code but unsupported now.
+        # erlangs = config.get('erlangs', 1)
+
+        # avoid duplicate history frames with a set of datetime frame
+        # starts and associated counts of how many duplicates we see
+        # per time stamp.
+        starts: Counter[datetime] = Counter()
+
+        # inline sequential loop where we simply pass the
+        # last retrieved start dt to the next request as
+        # it's end dt.
+        while end_dt > last_tsdb_dt:
+            log.debug(
+                f'Requesting {step_size_s}s frame ending in {start_dt}'
+            )
+
+            try:
+                array, next_start_dt, end_dt = await hist(
+                    timeframe,
+                    end_dt=start_dt,
+                )
+
+            # broker says there never was or is no more history to pull
+            except DataUnavailable:
+                log.warning(
+                    f'NO-MORE-DATA: backend {mod.name} halted history!?'
+                )
+
+                # ugh, what's a better way?
+                # TODO: fwiw, we probably want a way to signal a throttle
+                # condition (eg. with ib) so that we can halt the
+                # request loop until the condition is resolved?
+                return
+
+            if (
+                next_start_dt in starts
+                and starts[next_start_dt] <= 6
+            ):
+                start_dt = min(starts)
+                log.warning(
+                    f"{mkt.fqme}: skipping duplicate frame @ {next_start_dt}"
+                )
+                starts[start_dt] += 1
+                continue
+
+            elif starts[next_start_dt] > 6:
+                log.warning(
+                    f'NO-MORE-DATA: backend {mod.name} before {next_start_dt}?'
+                )
+                return
+
+            # only update new start point if not-yet-seen
+            start_dt = next_start_dt
+            starts[start_dt] += 1
+
+            assert array['time'][0] == start_dt.timestamp()
+
+            diff = end_dt - start_dt
+            frame_time_diff_s = diff.seconds
+            expected_frame_size_s = frame_size_s + step_size_s
+
+            if frame_time_diff_s > expected_frame_size_s:
+
+                # XXX: query result includes a start point prior to our
+                # expected "frame size" and thus is likely some kind of
+                # history gap (eg. market closed period, outage, etc.)
+                # so just report it to console for now.
+                log.warning(
+                    f'History frame ending @ {end_dt} appears to have a gap:\n'
+                    f'{diff} ~= {frame_time_diff_s} seconds'
+                )
+
+            to_push = diff_history(
+                array,
+                timeframe,
+                start_dt,
+                end_dt,
+                last_tsdb_dt=last_tsdb_dt,
+            )
+            ln = len(to_push)
+            if ln:
+                log.info(f'{ln} bars for {start_dt} -> {end_dt}')
+
+            else:
+                log.warning(
+                    f'{ln} BARS TO PUSH after diff?!: {start_dt} -> {end_dt}'
+                )
+
+            # bail gracefully on shm allocation overrun/full condition
+            try:
+                shm.push(to_push, prepend=True)
+            except ValueError:
+                log.info(
+                    f'Shm buffer overrun on: {start_dt} -> {end_dt}?'
+                )
+                # can't push the entire frame? so
+                # push only the amount that can fit..
+                break
+
+            log.info(
+                f'Shm pushed {ln} frame:\n'
+                f'{start_dt} -> {end_dt}'
+            )
+
+            if (
+                storage is not None
+                and write_tsdb
+            ):
+                log.info(
+                    f'Writing {ln} frame to storage:\n'
+                    f'{start_dt} -> {end_dt}'
+                )
+
+                if mkt.dst.atype not in {'crypto', 'crypto_currency'}:
+                    # for now, our table key schema is not including
+                    # the dst[/src] source asset token.
+                    col_sym_key: str = mkt.get_fqme(
+                        delim_char='',
+                        without_src=True,
+                    )
+                else:
+                    col_sym_key: str = mkt.get_fqme(delim_char='')
+
+                await storage.write_ohlcv(
+                    col_sym_key,
+                    to_push,
+                    timeframe,
+                )
+
+        # TODO: can we only trigger this if the respective
+        # history in "in view"?!?
+
+        # XXX: extremely important, there can be no checkpoints
+        # in the block above to avoid entering new ``frames``
+        # values while we're pipelining the current ones to
+        # memory...
+        await sampler_stream.send('broadcast_all')
+
+        # short-circuit (for now)
+        bf_done.set()
+
+
+async def basic_backfill(
+    bus: _FeedsBus,
+    mod: ModuleType,
+    mkt: MktPair,
+    shms: dict[int, ShmArray],
+    sampler_stream: tractor.MsgStream,
+    feed_is_live: trio.Event,
+
+) -> None:
+
+    # do a legacy incremental backfill from the provider.
+    log.info('No TSDB (marketstored) found, doing basic backfill..')
+
+    # start history backfill task ``backfill_bars()`` is
+    # a required backend func this must block until shm is
+    # filled with first set of ohlc bars
+    for timeframe, shm in shms.items():
+        try:
+            await bus.nursery.start(
+                partial(
+                    start_backfill,
+                    mod,
+                    mkt,
+                    shm,
+                    timeframe,
+                    sampler_stream,
+                    feed_is_live,
+                )
+            )
+        except DataUnavailable:
+            # XXX: timeframe not supported for backend
+            continue
+
+
+async def tsdb_backfill(
+    mod: ModuleType,
+    marketstore: ModuleType,
+    bus: _FeedsBus,
+    storage: Storage,
+    mkt: MktPair,
+    shms: dict[int, ShmArray],
+    sampler_stream: tractor.MsgStream,
+    feed_is_live: trio.Event,
+
+    task_status: TaskStatus[
+        tuple[ShmArray, ShmArray]
+    ] = trio.TASK_STATUS_IGNORED,
+
+) -> None:
+
+    # TODO: this should be used verbatim for the pure
+    # shm backfiller approach below.
+    dts_per_tf: dict[int, datetime] = {}
+    fqme: str = mkt.fqme
+
+    # start history anal and load missing new data via backend.
+    for timeframe, shm in shms.items():
+        # loads a (large) frame of data from the tsdb depending
+        # on the db's query size limit.
+        tsdb_history, first_tsdb_dt, last_tsdb_dt = await storage.load(
+            fqme,
+            timeframe=timeframe,
+        )
+
+        try:
+            (
+                latest_start_dt,
+                latest_end_dt,
+                bf_done,
+            ) = await bus.nursery.start(
+                partial(
+                    start_backfill,
+                    mod,
+                    mkt,
+                    shm,
+                    timeframe,
+                    sampler_stream,
+                    feed_is_live,
+
+                    last_tsdb_dt=last_tsdb_dt,
+                    tsdb_is_up=True,
+                    storage=storage,
+                )
+            )
+        except DataUnavailable:
+            # XXX: timeframe not supported for backend
+            dts_per_tf[timeframe] = (
+                tsdb_history,
+                last_tsdb_dt,
+                None,
+                None,
+                None,
+            )
+            continue
+
+        # tsdb_history = series.get(timeframe)
+        dts_per_tf[timeframe] = (
+            tsdb_history,
+            last_tsdb_dt,
+            latest_start_dt,
+            latest_end_dt,
+            bf_done,
+        )
+
+        # if len(hist_shm.array) < 2:
+        # TODO: there's an edge case here to solve where if the last
+        # frame before market close (at least on ib) was pushed and
+        # there was only "1 new" row pushed from the first backfill
+        # query-iteration, then the sample step sizing calcs will
+        # break upstream from here since you can't diff on at least
+        # 2 steps... probably should also add logic to compute from
+        # the tsdb series and stash that somewhere as meta data on
+        # the shm buffer?.. no se.
+
+    # unblock the feed bus management task
+    # assert len(shms[1].array)
+    task_status.started()
+
+    async def back_load_from_tsdb(
+        timeframe: int,
+        shm: ShmArray,
+    ):
+        (
+            tsdb_history,
+            last_tsdb_dt,
+            latest_start_dt,
+            latest_end_dt,
+            bf_done,
+        ) = dts_per_tf[timeframe]
+
+        # sync to backend history task's query/load completion
+        if bf_done:
+            await bf_done.wait()
+
+        # TODO: eventually it'd be nice to not require a shm array/buffer
+        # to accomplish this.. maybe we can do some kind of tsdb direct to
+        # graphics format eventually in a child-actor?
+
+        # TODO: see if there's faster multi-field reads:
+        # https://numpy.org/doc/stable/user/basics.rec.html#accessing-multiple-fields
+        # re-index  with a `time` and index field
+        prepend_start = shm._first.value
+        array = shm.array
+        if len(array):
+            shm_last_dt = pendulum.from_timestamp(shm.array[0]['time'])
+        else:
+            shm_last_dt = None
+
+        if last_tsdb_dt:
+            assert shm_last_dt >= last_tsdb_dt
+
+        # do diff against start index of last frame of history and only
+        # fill in an amount of datums from tsdb allows for most recent
+        # to be loaded into mem *before* tsdb data.
+        if (
+            last_tsdb_dt
+            and latest_start_dt
+        ):
+            backfilled_size_s = (
+                latest_start_dt - last_tsdb_dt
+            ).seconds
+            # if the shm buffer len is not large enough to contain
+            # all missing data between the most recent backend-queried frame
+            # and the most recent dt-index in the db we warn that we only
+            # want to load a portion of the next tsdb query to fill that
+            # space.
+            log.info(
+                f'{backfilled_size_s} seconds worth of {timeframe}s loaded'
+            )
+
+        # Load TSDB history into shm buffer (for display) if there is
+        # remaining buffer space.
+
+        if (
+            len(tsdb_history)
+        ):
+            # load the first (smaller) bit of history originally loaded
+            # above from ``Storage.load()``.
+            to_push = tsdb_history[-prepend_start:]
+            shm.push(
+                to_push,
+
+                # insert the history pre a "days worth" of samples
+                # to leave some real-time buffer space at the end.
+                prepend=True,
+                # update_first=False,
+                # start=prepend_start,
+                field_map=marketstore.ohlc_key_map,
+            )
+
+            tsdb_last_frame_start = tsdb_history['Epoch'][0]
+
+            if timeframe == 1:
+                times = shm.array['time']
+                assert (times[1] - times[0]) == 1
+
+            # load as much from storage into shm possible (depends on
+            # user's shm size settings).
+            while shm._first.value > 0:
+
+                tsdb_history = await storage.read_ohlcv(
+                    fqme,
+                    timeframe=timeframe,
+                    end=tsdb_last_frame_start,
+                )
+
+                # empty query
+                if not len(tsdb_history):
+                    break
+
+                next_start = tsdb_history['Epoch'][0]
+                if next_start >= tsdb_last_frame_start:
+                    # no earlier data detected
+                    break
+                else:
+                    tsdb_last_frame_start = next_start
+
+                prepend_start = shm._first.value
+                to_push = tsdb_history[-prepend_start:]
+
+                # insert the history pre a "days worth" of samples
+                # to leave some real-time buffer space at the end.
+                shm.push(
+                    to_push,
+                    prepend=True,
+                    field_map=marketstore.ohlc_key_map,
+                )
+                log.info(f'Loaded {to_push.shape} datums from storage')
+
+                # manually trigger step update to update charts/fsps
+                # which need an incremental update.
+                # NOTE: the way this works is super duper
+                # un-intuitive right now:
+                # - the broadcaster fires a msg to the fsp subsystem.
+                # - fsp subsys then checks for a sample step diff and
+                #   possibly recomputes prepended history.
+                # - the fsp then sends back to the parent actor
+                #   (usually a chart showing graphics for said fsp)
+                #   which tells the chart to conduct a manual full
+                #   graphics loop cycle.
+                await sampler_stream.send('broadcast_all')
+
+                # TODO: write new data to tsdb to be ready to for next read.
+
+    # backload from db (concurrently per timeframe) once backfilling of
+    # recent dat a loaded from the backend provider (see
+    # ``bf_done.wait()`` call).
+    async with trio.open_nursery() as nurse:
+        for timeframe, shm in shms.items():
+            nurse.start_soon(
+                back_load_from_tsdb,
+                timeframe,
+                shm,
+            )
+
+
+async def manage_history(
+    mod: ModuleType,
+    bus: _FeedsBus,
+    mkt: MktPair,
+    some_data_ready: trio.Event,
+    feed_is_live: trio.Event,
+    timeframe: float = 60,  # in seconds
+
+    task_status: TaskStatus[
+        tuple[ShmArray, ShmArray]
+    ] = trio.TASK_STATUS_IGNORED,
+
+) -> None:
+    '''
+    Load and manage historical data including the loading of any
+    available series from `marketstore` as well as conducting real-time
+    update of both that existing db and the allocated shared memory
+    buffer.
+
+    '''
+
+    # TODO: is there a way to make each shm file key
+    # actor-tree-discovery-addr unique so we avoid collisions
+    # when doing tests which also allocate shms for certain instruments
+    # that may be in use on the system by some other running daemons?
+    # from tractor._state import _runtime_vars
+    # port = _runtime_vars['_root_mailbox'][1]
+
+    uid = tractor.current_actor().uid
+    name, uuid = uid
+    service = name.rstrip(f'.{mod.name}')
+
+    fqme: str = mkt.get_fqme(delim_char='')
+
+    # (maybe) allocate shm array for this broker/symbol which will
+    # be used for fast near-term history capture and processing.
+    hist_shm, opened = maybe_open_shm_array(
+        key=f'piker.{service}[{uuid[:16]}].{fqme}.hist',
+
+        # use any broker defined ohlc dtype:
+        dtype=getattr(mod, '_ohlc_dtype', base_iohlc_dtype),
+
+        # we expect the sub-actor to write
+        readonly=False,
+    )
+    hist_zero_index = hist_shm.index - 1
+
+    # TODO: history validation
+    if not opened:
+        raise RuntimeError(
+            "Persistent shm for sym was already open?!"
+        )
+
+    rt_shm, opened = maybe_open_shm_array(
+        key=f'piker.{service}[{uuid[:16]}].{fqme}.rt',
+
+        # use any broker defined ohlc dtype:
+        dtype=getattr(mod, '_ohlc_dtype', base_iohlc_dtype),
+
+        # we expect the sub-actor to write
+        readonly=False,
+        size=3*_secs_in_day,
+    )
+
+    # (for now) set the rt (hft) shm array with space to prepend
+    # only a few days worth of 1s history.
+    days = 2
+    start_index = days*_secs_in_day
+    rt_shm._first.value = start_index
+    rt_shm._last.value = start_index
+    rt_zero_index = rt_shm.index - 1
+
+    if not opened:
+        raise RuntimeError(
+            "Persistent shm for sym was already open?!"
+        )
+
+    # register 1s and 1m buffers with the global incrementer task
+    async with open_sample_stream(
+        period_s=1.,
+        shms_by_period={
+            1.: rt_shm.token,
+            60.: hist_shm.token,
+        },
+
+        # NOTE: we want to only open a stream for doing broadcasts on
+        # backfill operations, not receive the sample index-stream
+        # (since there's no code in this data feed layer that needs to
+        # consume it).
+        open_index_stream=True,
+        sub_for_broadcasts=False,
+
+    ) as sample_stream:
+
+        open_history_client = getattr(
+            mod,
+            'open_history_client',
+            None,
+        )
+        assert open_history_client
+
+        tsdb_is_up: bool = False
+        try_remote_tsdb: bool = False
+
+        conf, path = config.load('conf', touch_if_dne=True)
+        net = conf.get('network')
+        if net:
+            tsdbconf = net.get('tsdb')
+
+            # lookup backend tsdb module by name and load any user service
+            # settings for connecting to the tsdb service.
+            tsdb_backend: str = tsdbconf.pop('backend')
+            tsdb_host: str = tsdbconf['host']
+
+            # TODO: import and load storagemod by name
+            # mod = get_storagemod(tsdb_backend)
+            from ..service import marketstore
+            if tsdb_host == 'localhost':
+                log.info('Scanning for existing `{tsbd_backend}`')
+                tsdb_is_up: bool = await check_for_service(f'{tsdb_backend}d')
+
+            else:
+                try_remote_tsdb: bool = True
+
+        if (
+            tsdb_is_up
+            or try_remote_tsdb
+            and (
+                opened
+                and open_history_client
+            )
+        ):
+            log.info('Found existing `marketstored`')
+
+            async with (
+                marketstore.open_storage_client(
+                    **tsdbconf
+                ) as storage,
+            ):
+                # TODO: drop returning the output that we pass in?
+                await bus.nursery.start(
+                    tsdb_backfill,
+                    mod,
+                    marketstore,
+                    bus,
+                    storage,
+                    mkt,
+                    {
+                        1: rt_shm,
+                        60: hist_shm,
+                    },
+                    sample_stream,
+                    feed_is_live,
+                )
+
+                # yield back after client connect with filled shm
+                task_status.started((
+                    hist_zero_index,
+                    hist_shm,
+                    rt_zero_index,
+                    rt_shm,
+                ))
+
+                # indicate to caller that feed can be delivered to
+                # remote requesting client since we've loaded history
+                # data that can be used.
+                some_data_ready.set()
+
+                # history retreival loop depending on user interaction
+                # and thus a small RPC-prot for remotely controllinlg
+                # what data is loaded for viewing.
+                await trio.sleep_forever()
+
+        # load less history if no tsdb can be found
+        elif (
+            not tsdb_is_up
+            and opened
+        ):
+            await basic_backfill(
+                bus,
+                mod,
+                mkt,
+                {
+                    1: rt_shm,
+                    60: hist_shm,
+                },
+                sample_stream,
+                feed_is_live,
+            )
+            task_status.started((
+                hist_zero_index,
+                hist_shm,
+                rt_zero_index,
+                rt_shm,
+            ))
+            some_data_ready.set()
+            await trio.sleep_forever()

piker/data/ticktools.py

@@ -1,173 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Tick event stream processing, filter-by-types, format-normalization.
-
-'''
-from itertools import chain
-from typing import (
-    Any,
-    AsyncIterator,
-)
-
-# tick-type-classes template for all possible "lowest level" events
-# that can can be emitted by the "top of book" L1 queues and
-# price-matching (with eventual clearing) in a double auction
-# market (queuing) system.
-_tick_groups: dict[str, set[str]] = {
-    'clears': {'trade', 'dark_trade', 'last'},
-    'bids': {'bid', 'bsize'},
-    'asks': {'ask', 'asize'},
-}
-
-# XXX alo define the flattened set of all such "fundamental ticks"
-# so that it can be used as filter, eg. in the graphics display
-# loop to compute running windowed y-ranges B)
-_auction_ticks: set[str] = set.union(*_tick_groups.values())
-
-
-def frame_ticks(
-    quote: dict[str, Any],
-
-    ticks_by_type: dict | None = None,
-    ticks_in_order: list[dict[str, Any]] | None = None
-
-) -> dict[
-    str,
-    list[dict[str, Any]]
-]:
-    '''
-    XXX: build a tick-by-type table of lists
-    of tick messages. This allows for less
-    iteration on the receiver side by allowing for
-    a single "latest tick event" look up by
-    indexing the last entry in each sub-list.
-
-    tbt = {
-        'types': ['bid', 'asize', 'last', .. '<type_n>'],
-
-        'bid': [tick0, tick1, tick2, .., tickn],
-        'asize': [tick0, tick1, tick2, .., tickn],
-        'last': [tick0, tick1, tick2, .., tickn],
-        ...
-        '<type_n>': [tick0, tick1, tick2, .., tickn],
-    }
-
-    If `ticks_in_order` is provided, append any retrieved ticks
-    since last iteration into this array/buffer/list.
-
-    '''
-    # TODO: once we decide to get fancy really we should
-    # have a shared mem tick buffer that is just
-    # continually filled and the UI just ready from it
-    # at it's display rate.
-
-    tbt = ticks_by_type if ticks_by_type is not None else {}
-    if not (ticks := quote.get('ticks')):
-        return tbt
-
-    # append in reverse FIFO order for in-order iteration on
-    # receiver side.
-    tick: dict[str, Any]
-    for tick in ticks:
-        tbt.setdefault(
-            tick['type'],
-            [],
-        ).append(tick)
-
-    # TODO: do we need this any more or can we just
-    # expect the receiver to unwind the below
-    # `ticks_by_type: dict`?
-    # => undwinding would potentially require a
-    # `dict[str, set | list]` instead with an
-    # included `'types' field which is an (ordered)
-    # set of tick type fields in the order which
-    # types arrived?
-    if ticks_in_order:
-        ticks_in_order.extend(ticks)
-
-    return tbt
-
-
-def iterticks(
-    quote: dict,
-    types: tuple[str] = (
-        'trade',
-        'dark_trade',
-    ),
-    deduplicate_darks: bool = False,
-    reverse: bool = False,
-
-    # TODO: should we offer delegating to `frame_ticks()` above
-    # with this?
-    frame_by_type: bool = False,
-
-) -> AsyncIterator:
-    '''
-    Iterate through ticks delivered per quote cycle, filter and
-    yield any declared in `types`.
-
-    '''
-    if deduplicate_darks:
-        assert 'dark_trade' in types
-
-    # print(f"{quote}\n\n")
-    ticks = quote.get('ticks', ())
-    trades = {}
-    darks = {}
-
-    if ticks:
-
-        # do a first pass and attempt to remove duplicate dark
-        # trades with the same tick signature.
-        if deduplicate_darks:
-            for tick in ticks:
-                ttype = tick.get('type')
-
-                time = tick.get('time', None)
-                if time:
-                    sig = (
-                        time,
-                        tick['price'],
-                        tick.get('size')
-                    )
-
-                    if ttype == 'dark_trade':
-                        darks[sig] = tick
-
-                    elif ttype == 'trade':
-                        trades[sig] = tick
-
-            # filter duplicates
-            for sig, tick in trades.items():
-                tick = darks.pop(sig, None)
-                if tick:
-                    ticks.remove(tick)
-                    # print(f'DUPLICATE {tick}')
-
-            # re-insert ticks
-            ticks.extend(list(chain(trades.values(), darks.values())))
-
-        # most-recent-first
-        if reverse:
-            ticks = reversed(ticks)
-
-        for tick in ticks:
-            # print(f"{quote['symbol']}: {tick}")
-            ttype = tick.get('type')
-            if ttype in types:
-                yield tick

piker/data/types.py

@@ -0,0 +1,104 @@
+# piker: trading gear for hackers
+# Copyright (C) (in stewardship for pikers)
+#  - Tyler Goodlet
+#  - Guillermo Rodriguez
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Extensions to built-in or (heavily used but 3rd party) friend-lib
+types.
+
+'''
+from pprint import pformat
+
+from msgspec import (
+    msgpack,
+    Struct,
+    structs,
+)
+
+
+class Struct(
+    Struct,
+
+    # https://jcristharif.com/msgspec/structs.html#tagged-unions
+    # tag='pikerstruct',
+    # tag=True,
+):
+    '''
+    A "human friendlier" (aka repl buddy) struct subtype.
+
+    '''
+    def to_dict(self) -> dict:
+        '''
+        Like it sounds.. direct delegation to:
+        https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
+
+        TODO: probably just drop this method since it's now a built-int method?
+
+        '''
+        return structs.asdict(self)
+
+    def pformat(self) -> str:
+        return f'Struct({pformat(self.to_dict())})'
+
+    def copy(
+        self,
+        update: dict | None = None,
+
+    ) -> Struct:
+        '''
+        Validate-typecast all self defined fields, return a copy of
+        us with all such fields.
+
+        NOTE: This is kinda like the default behaviour in
+        `pydantic.BaseModel` except a copy of the object is
+        returned making it compat with `frozen=True`.
+
+        '''
+        if update:
+            for k, v in update.items():
+                setattr(self, k, v)
+
+        # NOTE: roundtrip serialize to validate
+        # - enode to msgpack binary format,
+        # - decode that back to a struct.
+        return msgpack.Decoder(type=type(self)).decode(
+            msgpack.Encoder().encode(self)
+        )
+
+    def typecast(
+        self,
+
+        # TODO: allow only casting a named subset?
+        # fields: set[str] | None = None,
+
+    ) -> None:
+        '''
+        Cast all fields using their declared type annotations
+        (kinda like what `pydantic` does by default).
+
+        NOTE: this of course won't work on frozen types, use
+        ``.copy()`` above in such cases.
+
+        '''
+        # https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
+        fi: structs.FieldInfo
+        for fi in structs.fields(self):
+            setattr(
+                self,
+                fi.name,
+                fi.type(getattr(self, fi.name)),
+            )

piker/data/validate.py

@@ -18,7 +18,6 @@ Data feed synchronization protocols, init msgs, and general
 data-provider-backend-agnostic schema definitions.
 
 '''
-from __future__ import annotations
 from decimal import Decimal
 from pprint import pformat
 from types import ModuleType
@@ -29,8 +28,8 @@ from typing import (
 
 from msgspec import field
 
-from piker.types import Struct
-from piker.accounting import (
+from .types import Struct
+from ..accounting import (
     Asset,
     MktPair,
 )
@@ -82,8 +81,8 @@ _eps: dict[str, list[str]] = {
     # live order control and trading
     'brokerd': [
         'trades_dialogue',
-        'open_trade_dialog',  # live order ctl
-        'norm_trade', # ledger normalizer for txns
+        # TODO: ledger normalizer helper?
+        # norm_trades(records: dict[str, Any]) -> TransactionLedger)
     ],
 }
 

piker/fsp/__init__.py

@@ -22,40 +22,17 @@ from typing import AsyncIterator
 
 import numpy as np
 
-from ._api import (
-    maybe_mk_fsp_shm,
-    Fsp,
-)
-from ._engine import (
-    cascade,
-    Cascade,
-)
-from ._volume import (
-    dolla_vlm,
-    flow_rates,
-    tina_vwap,
-)
+from ._engine import cascade
 
-__all__: list[str] = [
-    'cascade',
-    'Cascade',
-    'maybe_mk_fsp_shm',
-    'Fsp',
-    'dolla_vlm',
-    'flow_rates',
-    'tina_vwap',
-]
+__all__ = ['cascade']
 
 
 async def latency(
     source: 'TickStream[Dict[str, float]]',  # noqa
     ohlcv: np.ndarray
-
 ) -> AsyncIterator[np.ndarray]:
-    '''
-    Latency measurements, broker to piker.
-
-    '''
+    """Latency measurements, broker to piker.
+    """
     # TODO: do we want to offer yielding this async
     # before the rt data connection comes up?
 

piker/fsp/_api.py

@@ -177,7 +177,6 @@ def fsp(
 def maybe_mk_fsp_shm(
     sym: str,
     target: Fsp,
-    size: int,
     readonly: bool = True,
 
 ) -> (str, ShmArray, bool):
@@ -186,8 +185,7 @@ def maybe_mk_fsp_shm(
     exists, otherwise load the shm already existing for that token.
 
     '''
-    if not isinstance(sym, str):
-        raise ValueError('`sym: str` should be file-name-friendly')
+    assert isinstance(sym, str), '`sym` should be file-name-friendly `str`'
 
     # TODO: load output types from `Fsp`
     # - should `index` be a required internal field?
@@ -206,7 +204,7 @@ def maybe_mk_fsp_shm(
 
     shm, opened = maybe_open_shm_array(
         key,
-        size=size,
+        # TODO: create entry for each time frame
         dtype=fsp_dtype,
         readonly=True,
     )

piker/fsp/_engine.py

@@ -18,12 +18,13 @@
 core task logic for processing chains
 
 '''
-from __future__ import annotations
-from contextlib import asynccontextmanager as acm
+from dataclasses import dataclass
 from functools import partial
 from typing import (
     AsyncIterator,
     Callable,
+    Optional,
+    Union,
 )
 
 import numpy as np
@@ -32,9 +33,9 @@ from trio_typing import TaskStatus
 import tractor
 from tractor.msg import NamespacePath
 
-from piker.types import Struct
 from ..log import get_logger, get_console_log
 from .. import data
+from ..data import attach_shm_array
 from ..data.feed import (
     Flume,
     Feed,
@@ -50,11 +51,17 @@ from ._api import (
     _load_builtins,
     _Token,
 )
-from ..toolz import Profiler
+from .._profile import Profiler
 
 log = get_logger(__name__)
 
 
+@dataclass
+class TaskTracker:
+    complete: trio.Event
+    cs: trio.CancelScope
+
+
 async def filter_quotes_by_sym(
 
     sym: str,
@@ -75,168 +82,30 @@ async def filter_quotes_by_sym(
         if quote:
             yield quote
 
-# TODO: unifying the abstractions in this FSP subsys/layer:
-# -[ ] move the `.data.flows.Flume` type into this
-#   module/subsys/pkg?
-# -[ ] ideas for further abstractions as per
-#   - https://github.com/pikers/piker/issues/216,
-#   - https://github.com/pikers/piker/issues/270:
-#   - a (financial signal) ``Flow`` would be the a "collection" of such
-#     minmial cascades. Some engineering based jargon concepts:
-#     - https://en.wikipedia.org/wiki/Signal_chain
-#     - https://en.wikipedia.org/wiki/Daisy_chain_(electrical_engineering)
-#     - https://en.wikipedia.org/wiki/Audio_signal_flow
-#     - https://en.wikipedia.org/wiki/Digital_signal_processing#Implementation
-#     - https://en.wikipedia.org/wiki/Dataflow_programming
-#     - https://en.wikipedia.org/wiki/Signal_programming
-#     - https://en.wikipedia.org/wiki/Incremental_computing
-#     - https://en.wikipedia.org/wiki/Signal-flow_graph
-#     - https://en.wikipedia.org/wiki/Signal-flow_graph#Basic_components
 
-# -[ ] we probably want to eval THE BELOW design and unify with the
-#   proto `TaskManager` in the `tractor` dev branch as well as with
-#   our below idea for `Cascade`:
-#   - https://github.com/goodboy/tractor/pull/363
-class Cascade(Struct):
-    '''
-    As per sig-proc engineering parlance, this is a chaining of
-    `Flume`s, which are themselves collections of "Streams"
-    implemented currently via `ShmArray`s.
+async def fsp_compute(
 
-    A `Cascade` is be the minimal "connection" of 2 `Flumes`
-    as per circuit parlance:
-    https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
-
-    TODO:
-      -[ ] could cover the combination of our `FspAdmin` and the
-        backend `.fsp._engine` related machinery to "connect" one flume
-        to another?
-
-    '''
-    # TODO: make these `Flume`s
-    src: Flume
-    dst: Flume
-    tn: trio.Nursery
-    fsp: Fsp  # UI-side middleware ctl API
-
-    # filled during cascade/.bind_func() (fsp_compute) init phases
-    bind_func: Callable | None = None
-    complete: trio.Event | None = None
-    cs: trio.CancelScope | None = None
-    client_stream: tractor.MsgStream | None = None
-
-    async def resync(self) -> int:
-        # TODO: adopt an incremental update engine/approach
-        # where possible here eventually!
-        log.info(f're-syncing fsp {self.fsp.name} to source')
-        self.cs.cancel()
-        await self.complete.wait()
-        index: int = await self.tn.start(self.bind_func)
-
-        # always trigger UI refresh after history update,
-        # see ``piker.ui._fsp.FspAdmin.open_chain()`` and
-        # ``piker.ui._display.trigger_update()``.
-        dst_shm: ShmArray = self.dst.rt_shm
-        await self.client_stream.send({
-            'fsp_update': {
-                'key': dst_shm.token,
-                'first': dst_shm._first.value,
-                'last': dst_shm._last.value,
-            }
-        })
-        return index
-
-    def is_synced(self) -> tuple[bool, int, int]:
-        '''
-        Predicate to dertmine if a destination FSP
-        output array is aligned to its source array.
-
-        '''
-        src_shm: ShmArray = self.src.rt_shm
-        dst_shm: ShmArray = self.dst.rt_shm
-        step_diff = src_shm.index - dst_shm.index
-        len_diff = abs(len(src_shm.array) - len(dst_shm.array))
-        synced: bool = not (
-            # the source is likely backfilling and we must
-            # sync history calculations
-            len_diff > 2
-
-            # we aren't step synced to the source and may be
-            # leading/lagging by a step
-            or step_diff > 1
-            or step_diff < 0
-        )
-        if not synced:
-            fsp: Fsp = self.fsp
-            log.warning(
-                '***DESYNCED FSP***\n'
-                f'{fsp.ns_path}@{src_shm.token}\n'
-                f'step_diff: {step_diff}\n'
-                f'len_diff: {len_diff}\n'
-            )
-        return (
-            synced,
-            step_diff,
-            len_diff,
-        )
-
-    async def poll_and_sync_to_step(self) -> int:
-        synced, step_diff, _ = self.is_synced()
-        while not synced:
-            await self.resync()
-            synced, step_diff, _ = self.is_synced()
-
-        return step_diff
-
-    @acm
-    async def open_edge(
-        self,
-        bind_func: Callable,
-    ) -> int:
-        self.bind_func = bind_func
-        index = await self.tn.start(bind_func)
-        yield index
-        # TODO: what do we want on teardown/error?
-        # -[ ] dynamic reconnection after update?
-
-
-async def connect_streams(
-    casc: Cascade,
     mkt: MktPair,
+    flume: Flume,
     quote_stream: trio.abc.ReceiveChannel,
-    src: Flume,
-    dst: Flume,
 
-    edge_func: Callable,
+    src: ShmArray,
+    dst: ShmArray,
+
+    func: Callable,
 
     # attach_stream: bool = False,
     task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
 
 ) -> None:
-    '''
-    Stream and per-sample compute and write the cascade of
-    2 `Flumes`/streams given some operating `func`.
 
-    https://en.wikipedia.org/wiki/Signal-flow_graph#Basic_components
-
-    Not literally, but something like:
-
-        edge_func(Flume_in) -> Flume_out
-
-    '''
     profiler = Profiler(
         delayed=False,
         disabled=True
     )
 
-    # TODO: just pull it from src.mkt.fqme no?
-    # fqme: str = mkt.fqme
-    fqme: str = src.mkt.fqme
-
-    # TODO: dynamic introspection of what the underlying (vertex)
-    # function actually requires from input node (flumes) then
-    # deliver those inputs as part of a graph "compilation" step?
-    out_stream = edge_func(
+    fqme = mkt.fqme
+    out_stream = func(
 
         # TODO: do we even need this if we do the feed api right?
         # shouldn't a local stream do this before we get a handle
@@ -244,21 +113,20 @@ async def connect_streams(
         # async itertools style?
         filter_quotes_by_sym(fqme, quote_stream),
 
-        # XXX: currently the ``ohlcv`` arg, but we should allow
-        # (dynamic) requests for src flume (node) streams?
-        src.rt_shm,
+        # XXX: currently the ``ohlcv`` arg
+        flume.rt_shm,
     )
 
     # HISTORY COMPUTE PHASE
     # conduct a single iteration of fsp with historical bars input
     # and get historical output.
-    history_output: (
-        dict[str, np.ndarray]  # multi-output case
-        | np.ndarray,  # single output case
-    )
+    history_output: Union[
+        dict[str, np.ndarray],  # multi-output case
+        np.ndarray,  # single output case
+    ]
     history_output = await anext(out_stream)
 
-    func_name = edge_func.__name__
+    func_name = func.__name__
     profiler(f'{func_name} generated history')
 
     # build struct array with an 'index' field to push as history
@@ -266,12 +134,10 @@ async def connect_streams(
     # TODO: push using a[['f0', 'f1', .., 'fn']] = .. syntax no?
     # if the output array is multi-field then push
     # each respective field.
-    dst_shm: ShmArray = dst.rt_shm
-    fields = getattr(dst_shm.array.dtype, 'fields', None).copy()
+    fields = getattr(dst.array.dtype, 'fields', None).copy()
     fields.pop('index')
-    history_by_field: np.ndarray | None = None
-    src_shm: ShmArray = src.rt_shm
-    src_time = src_shm.array['time']
+    history_by_field: Optional[np.ndarray] = None
+    src_time = src.array['time']
 
     if (
         fields and
@@ -290,7 +156,7 @@ async def connect_streams(
                 if history_by_field is None:
 
                     if output is None:
-                        length = len(src_shm.array)
+                        length = len(src.array)
                     else:
                         length = len(output)
 
@@ -299,7 +165,7 @@ async def connect_streams(
                     # will be pushed to shm.
                     history_by_field = np.zeros(
                         length,
-                        dtype=dst_shm.array.dtype
+                        dtype=dst.array.dtype
                     )
 
                 if output is None:
@@ -316,13 +182,13 @@ async def connect_streams(
             )
         history_by_field = np.zeros(
             len(history_output),
-            dtype=dst_shm.array.dtype
+            dtype=dst.array.dtype
         )
         history_by_field[func_name] = history_output
 
     history_by_field['time'] = src_time[-len(history_by_field):]
 
-    history_output['time'] = src_shm.array['time']
+    history_output['time'] = src.array['time']
 
     # TODO: XXX:
     # THERE'S A BIG BUG HERE WITH THE `index` field since we're
@@ -335,11 +201,11 @@ async def connect_streams(
     #   is `index` aware such that historical data can be indexed
     #   relative to the true first datum? Not sure if this is sane
     #   for incremental compuations.
-    first = dst_shm._first.value = src_shm._first.value
+    first = dst._first.value = src._first.value
 
     # TODO: can we use this `start` flag instead of the manual
     # setting above?
-    index = dst_shm.push(
+    index = dst.push(
         history_by_field,
         start=first,
     )
@@ -350,9 +216,12 @@ async def connect_streams(
     # setup a respawn handle
     with trio.CancelScope() as cs:
 
-        casc.cs = cs
-        casc.complete = trio.Event()
-        task_status.started(index)
+        # TODO: might be better to just make a "restart" method where
+        # the target task is spawned implicitly and then the event is
+        # set via some higher level api? At that poing we might as well
+        # be writing a one-cancels-one nursery though right?
+        tracker = TaskTracker(trio.Event(), cs)
+        task_status.started((tracker, index))
 
         profiler(f'{func_name} yield last index')
 
@@ -366,12 +235,12 @@ async def connect_streams(
                 log.debug(f"{func_name}: {processed}")
                 key, output = processed
                 # dst.array[-1][key] = output
-                dst_shm.array[[key, 'time']][-1] = (
+                dst.array[[key, 'time']][-1] = (
                     output,
                     # TODO: what about pushing ``time.time_ns()``
                     # in which case we'll need to round at the graphics
                     # processing / sampling layer?
-                    src_shm.array[-1]['time']
+                    src.array[-1]['time']
                 )
 
                 # NOTE: for now we aren't streaming this to the consumer
@@ -383,7 +252,7 @@ async def connect_streams(
                 # N-consumers who subscribe for the real-time output,
                 # which we'll likely want to implement using local-mem
                 # chans for the fan out?
-                # index = src_shm.index
+                # index = src.index
                 # if attach_stream:
                 #     await client_stream.send(index)
 
@@ -393,7 +262,7 @@ async def connect_streams(
                 #     log.info(f'FSP quote too fast: {hz}')
                 # last = time.time()
         finally:
-            casc.complete.set()
+            tracker.complete.set()
 
 
 @tractor.context
@@ -404,15 +273,15 @@ async def cascade(
     # data feed key
     fqme: str,
 
-    # flume pair cascaded using an "edge function"
-    src_flume_addr: dict,
-    dst_flume_addr: dict,
+    src_shm_token: dict,
+    dst_shm_token: tuple[str, np.dtype],
+
     ns_path: NamespacePath,
 
     shm_registry: dict[str, _Token],
 
     zero_on_step: bool = False,
-    loglevel: str | None = None,
+    loglevel: Optional[str] = None,
 
 ) -> None:
     '''
@@ -428,14 +297,8 @@ async def cascade(
     if loglevel:
         get_console_log(loglevel)
 
-    src: Flume = Flume.from_msg(src_flume_addr)
-    dst: Flume = Flume.from_msg(
-        dst_flume_addr,
-        readonly=False,
-    )
-
-    # src: ShmArray = attach_shm_array(token=src_shm_token)
-    # dst: ShmArray = attach_shm_array(readonly=False, token=dst_shm_token)
+    src = attach_shm_array(token=src_shm_token)
+    dst = attach_shm_array(readonly=False, token=dst_shm_token)
 
     reg = _load_builtins()
     lines = '\n'.join([f'{key.rpartition(":")[2]} => {key}' for key in reg])
@@ -443,11 +306,11 @@ async def cascade(
         f'Registered FSP set:\n{lines}'
     )
 
-    # NOTE XXX: update actorlocal flows table which registers
-    # readonly "instances" of this fsp for symbol/source so that
-    # consumer fsps can look it up by source + fsp.
-    # TODO: ugh i hate this wind/unwind to list over the wire but
-    # not sure how else to do it.
+    # update actorlocal flows table which registers
+    # readonly "instances" of this fsp for symbol/source
+    # so that consumer fsps can look it up by source + fsp.
+    # TODO: ugh i hate this wind/unwind to list over the wire
+    # but not sure how else to do it.
     for (token, fsp_name, dst_token) in shm_registry:
         Fsp._flow_registry[(
             _Token.from_msg(token),
@@ -457,15 +320,12 @@ async def cascade(
     fsp: Fsp = reg.get(
         NamespacePath(ns_path)
     )
-    func: Callable = fsp.func
+    func = fsp.func
 
     if not func:
         # TODO: assume it's a func target path
         raise ValueError(f'Unknown fsp target: {ns_path}')
 
-    _fqme: str = src.mkt.fqme
-    assert _fqme == fqme
-
     # open a data feed stream with requested broker
     feed: Feed
     async with data.feed.maybe_open_feed(
@@ -479,142 +339,177 @@ async def cascade(
 
     ) as feed:
 
-        flume: Flume = feed.flumes[fqme]
-        # XXX: can't do this since flume.feed will be set XD
-        # assert flume == src
-        assert flume.mkt == src.mkt
-        mkt: MktPair = flume.mkt
-
-        # NOTE: FOR NOW, sanity checks around the feed as being
-        # always the src flume (until we get to fancier/lengthier
-        # chains/graphs.
-        assert src.rt_shm.token == flume.rt_shm.token
-
-        # XXX: won't work bc the _hist_shm_token value will be
-        # list[list] after IPC..
-        # assert flume.to_msg() == src_flume_addr
-
+        flume = feed.flumes[fqme]
+        mkt = flume.mkt
+        assert src.token == flume.rt_shm.token
         profiler(f'{func}: feed up')
 
-        func_name: str = func.__name__
+        func_name = func.__name__
         async with (
-            trio.open_nursery() as tn,
+            trio.open_nursery() as n,
         ):
-            # TODO: might be better to just make a "restart" method where
-            # the target task is spawned implicitly and then the event is
-            # set via some higher level api? At that poing we might as well
-            # be writing a one-cancels-one nursery though right?
-            casc = Cascade(
-                src,
-                dst,
-                tn,
-                fsp,
-            )
 
-            # TODO: this seems like it should be wrapped somewhere?
             fsp_target = partial(
-                connect_streams,
-                casc=casc,
+
+                fsp_compute,
                 mkt=mkt,
+                flume=flume,
                 quote_stream=flume.stream,
 
-                # flumes and shm passthrough
+                # shm
                 src=src,
                 dst=dst,
 
-                # chain function which takes src flume input(s)
-                # and renders dst flume output(s)
-                edge_func=func
+                # target
+                func=func
             )
-            async with casc.open_edge(
-                bind_func=fsp_target,
-            ) as index:
-                # casc.bind_func = fsp_target
-                # index = await tn.start(fsp_target)
-                dst_shm: ShmArray = dst.rt_shm
-                src_shm: ShmArray = src.rt_shm
 
-                if zero_on_step:
-                    last = dst.rt_shm.array[-1:]
-                    zeroed = np.zeros(last.shape, dtype=last.dtype)
+            tracker, index = await n.start(fsp_target)
 
-                profiler(f'{func_name}: fsp up')
+            if zero_on_step:
+                last = dst.array[-1:]
+                zeroed = np.zeros(last.shape, dtype=last.dtype)
 
-                # sync to client-side actor
-                await ctx.started(index)
+            profiler(f'{func_name}: fsp up')
 
-                # XXX:  rt stream with client which we MUST
-                # open here (and keep it open) in order to make
-                # incremental "updates" as history prepends take
-                # place.
-                async with ctx.open_stream() as client_stream:
-                    casc.client_stream: tractor.MsgStream = client_stream
+            # sync client
+            await ctx.started(index)
 
-                    s, step, ld = casc.is_synced()
+            # XXX:  rt stream with client which we MUST
+            # open here (and keep it open) in order to make
+            # incremental "updates" as history prepends take
+            # place.
+            async with ctx.open_stream() as client_stream:
 
-                    # detect sample period step for subscription to increment
-                    # signal
-                    times = src.rt_shm.array['time']
-                    if len(times) > 1:
-                        last_ts = times[-1]
-                        delay_s: float = float(last_ts - times[times != last_ts][-1])
-                    else:
-                        # our default "HFT" sample rate.
-                        delay_s: float = _default_delay_s
+                # TODO: these likely should all become
+                # methods of this ``TaskLifetime`` or wtv
+                # abstraction..
+                async def resync(
+                    tracker: TaskTracker,
 
-                    # sub and increment the underlying shared memory buffer
-                    # on every step msg received from the global `samplerd`
-                    # service.
-                    async with open_sample_stream(
-                        float(delay_s)
-                    ) as istream:
+                ) -> tuple[TaskTracker, int]:
+                    # TODO: adopt an incremental update engine/approach
+                    # where possible here eventually!
+                    log.info(f're-syncing fsp {func_name} to source')
+                    tracker.cs.cancel()
+                    await tracker.complete.wait()
+                    tracker, index = await n.start(fsp_target)
 
-                        profiler(f'{func_name}: sample stream up')
-                        profiler.finish()
+                    # always trigger UI refresh after history update,
+                    # see ``piker.ui._fsp.FspAdmin.open_chain()`` and
+                    # ``piker.ui._display.trigger_update()``.
+                    await client_stream.send({
+                        'fsp_update': {
+                            'key': dst_shm_token,
+                            'first': dst._first.value,
+                            'last': dst._last.value,
+                        }
+                    })
+                    return tracker, index
 
-                        async for i in istream:
-                            # print(f'FSP incrementing {i}')
+                def is_synced(
+                    src: ShmArray,
+                    dst: ShmArray
+                ) -> tuple[bool, int, int]:
+                    '''
+                    Predicate to dertmine if a destination FSP
+                    output array is aligned to its source array.
 
-                            # respawn the compute task if the source
-                            # array has been updated such that we compute
-                            # new history from the (prepended) source.
-                            synced, step_diff, _ = casc.is_synced()
-                            if not synced:
-                                step_diff: int = await casc.poll_and_sync_to_step()
+                    '''
+                    step_diff = src.index - dst.index
+                    len_diff = abs(len(src.array) - len(dst.array))
+                    return not (
+                        # the source is likely backfilling and we must
+                        # sync history calculations
+                        len_diff > 2
 
-                                # skip adding a last bar since we should already
-                                # be step alinged
-                                if step_diff == 0:
-                                    continue
+                        # we aren't step synced to the source and may be
+                        # leading/lagging by a step
+                        or step_diff > 1
+                        or step_diff < 0
+                    ), step_diff, len_diff
 
-                            # read out last shm row, copy and write new row
-                            array = dst_shm.array
+                async def poll_and_sync_to_step(
+                    tracker: TaskTracker,
+                    src: ShmArray,
+                    dst: ShmArray,
 
-                            # some metrics like vlm should be reset
-                            # to zero every step.
-                            if zero_on_step:
-                                last = zeroed
-                            else:
-                                last = array[-1:].copy()
+                ) -> tuple[TaskTracker, int]:
 
-                            dst.rt_shm.push(last)
+                    synced, step_diff, _ = is_synced(src, dst)
+                    while not synced:
+                        tracker, index = await resync(tracker)
+                        synced, step_diff, _ = is_synced(src, dst)
 
-                            # sync with source buffer's time step
-                            src_l2 = src_shm.array[-2:]
-                            src_li, src_lt = src_l2[-1][['index', 'time']]
-                            src_2li, src_2lt = src_l2[-2][['index', 'time']]
-                            dst_shm._array['time'][src_li] = src_lt
-                            dst_shm._array['time'][src_2li] = src_2lt
+                    return tracker, step_diff
 
-                            # last2 = dst.array[-2:]
-                            # if (
-                            #     last2[-1]['index'] != src_li
-                            #     or last2[-2]['index'] != src_2li
-                            # ):
-                            #     dstl2 = list(last2)
-                            #     srcl2 = list(src_l2)
-                            #     print(
-                            #         # f'{dst.token}\n'
-                            #         f'src: {srcl2}\n'
-                            #         f'dst: {dstl2}\n'
-                            #     )
+                s, step, ld = is_synced(src, dst)
+
+                # detect sample period step for subscription to increment
+                # signal
+                times = src.array['time']
+                if len(times) > 1:
+                    last_ts = times[-1]
+                    delay_s = float(last_ts - times[times != last_ts][-1])
+                else:
+                    # our default "HFT" sample rate.
+                    delay_s = _default_delay_s
+
+                # sub and increment the underlying shared memory buffer
+                # on every step msg received from the global `samplerd`
+                # service.
+                async with open_sample_stream(float(delay_s)) as istream:
+
+                    profiler(f'{func_name}: sample stream up')
+                    profiler.finish()
+
+                    async for i in istream:
+                        # print(f'FSP incrementing {i}')
+
+                        # respawn the compute task if the source
+                        # array has been updated such that we compute
+                        # new history from the (prepended) source.
+                        synced, step_diff, _ = is_synced(src, dst)
+                        if not synced:
+                            tracker, step_diff = await poll_and_sync_to_step(
+                                tracker,
+                                src,
+                                dst,
+                            )
+
+                            # skip adding a last bar since we should already
+                            # be step alinged
+                            if step_diff == 0:
+                                continue
+
+                        # read out last shm row, copy and write new row
+                        array = dst.array
+
+                        # some metrics like vlm should be reset
+                        # to zero every step.
+                        if zero_on_step:
+                            last = zeroed
+                        else:
+                            last = array[-1:].copy()
+
+                        dst.push(last)
+
+                        # sync with source buffer's time step
+                        src_l2 = src.array[-2:]
+                        src_li, src_lt = src_l2[-1][['index', 'time']]
+                        src_2li, src_2lt = src_l2[-2][['index', 'time']]
+                        dst._array['time'][src_li] = src_lt
+                        dst._array['time'][src_2li] = src_2lt
+
+                        # last2 = dst.array[-2:]
+                        # if (
+                        #     last2[-1]['index'] != src_li
+                        #     or last2[-2]['index'] != src_2li
+                        # ):
+                        #     dstl2 = list(last2)
+                        #     srcl2 = list(src_l2)
+                        #     print(
+                        #         # f'{dst.token}\n'
+                        #         f'src: {srcl2}\n'
+                        #         f'dst: {dstl2}\n'
+                        #     )

piker/fsp/_momo.py

@@ -24,7 +24,7 @@ import numpy as np
 from numba import jit, float64, optional, int64
 
 from ._api import fsp
-from ..data import iterticks
+from ..data._normalize import iterticks
 from ..data._sharedmem import ShmArray
 
 

piker/fsp/_volume.py

@@ -20,7 +20,7 @@ import numpy as np
 from tractor.trionics._broadcast import AsyncReceiver
 
 from ._api import fsp
-from ..data import iterticks
+from ..data._normalize import iterticks
 from ..data._sharedmem import ShmArray
 from ._momo import _wma
 from ..log import get_logger

piker/log.py

@@ -40,10 +40,7 @@ def get_logger(
     Return the package log or a sub-log for `name` if provided.
 
     '''
-    return tractor.log.get_logger(
-        name=name,
-        _root_name=_proj_name,
-    )
+    return tractor.log.get_logger(name=name, _root_name=_proj_name)
 
 
 def get_console_log(

piker/service/__init__.py

@@ -14,45 +14,49 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-'''
-Actor runtime primtives and (distributed) service APIs for,
+"""
+Actor-runtime service orchestration machinery.
 
-- daemon-service mgmt: `_daemon` (i.e. low-level spawn and supervise machinery
-  for sub-actors like `brokerd`, `emsd`, datad`, etc.)
+"""
+from __future__ import annotations
 
-- service-actor supervision (via `trio` tasks) API: `._mngr`
-
-- discovery interface (via light wrapping around `tractor`'s built-in
-  prot): `._registry`
-
-- `docker` cntr SC supervision for use with `trio`: `_ahab`
-  - wrappers for marketstore and elasticsearch dbs
-  => TODO: maybe to (re)move elsewhere?
-
-'''
-from ._mngr import Services as Services
-from ._registry import (
-    _tractor_kwargs as _tractor_kwargs,
-    _default_reg_addr as _default_reg_addr,
-    _default_registry_host as _default_registry_host,
-    _default_registry_port as _default_registry_port,
-
-    open_registry as open_registry,
-    find_service as find_service,
-    check_for_service as check_for_service,
+from ._mngr import Services
+from ._registry import (  # noqa
+    _tractor_kwargs,
+    _default_reg_addr,
+    _default_registry_host,
+    _default_registry_port,
+    open_registry,
+    find_service,
+    check_for_service,
 )
-from ._daemon import (
-    maybe_spawn_daemon as maybe_spawn_daemon,
-    spawn_emsd as spawn_emsd,
-    maybe_open_emsd as maybe_open_emsd,
+from ._daemon import (  # noqa
+    maybe_spawn_daemon,
+    spawn_emsd,
+    maybe_open_emsd,
 )
 from ._actor_runtime import (
-    open_piker_runtime as open_piker_runtime,
-    maybe_open_pikerd as maybe_open_pikerd,
-    open_pikerd as open_pikerd,
-    get_runtime_vars as get_runtime_vars,
+    open_piker_runtime,
+    maybe_open_pikerd,
+    open_pikerd,
+    get_tractor_runtime_kwargs,
 )
 from ..brokers._daemon import (
-    spawn_brokerd as spawn_brokerd,
-    maybe_spawn_brokerd as maybe_spawn_brokerd,
+    spawn_brokerd,
+    maybe_spawn_brokerd,
 )
+
+
+__all__ = [
+    'check_for_service',
+    'Services',
+    'maybe_spawn_daemon',
+    'spawn_brokerd',
+    'maybe_spawn_brokerd',
+    'spawn_emsd',
+    'maybe_open_emsd',
+    'open_piker_runtime',
+    'maybe_open_pikerd',
+    'open_pikerd',
+    'get_tractor_runtime_kwargs',
+]

piker/service/_actor_runtime.py

@@ -45,7 +45,7 @@ from ._registry import (  # noqa
 )
 
 
-def get_runtime_vars() -> dict[str, Any]:
+def get_tractor_runtime_kwargs() -> dict[str, Any]:
     '''
     Deliver ``tractor`` related runtime variables in a `dict`.
 
@@ -56,8 +56,6 @@ def get_runtime_vars() -> dict[str, Any]:
 @acm
 async def open_piker_runtime(
     name: str,
-    registry_addrs: list[tuple[str, int]] = [],
-
     enable_modules: list[str] = [],
     loglevel: Optional[str] = None,
 
@@ -65,6 +63,8 @@ async def open_piker_runtime(
     # for data daemons when running in production.
     debug_mode: bool = False,
 
+    registry_addr: None | tuple[str, int] = None,
+
     # TODO: once we have `rsyscall` support we will read a config
     # and spawn the service tree distributed per that.
     start_method: str = 'trio',
@@ -74,7 +74,7 @@ async def open_piker_runtime(
 
 ) -> tuple[
     tractor.Actor,
-    list[tuple[str, int]],
+    tuple[str, int],
 ]:
     '''
     Start a piker actor who's runtime will automatically sync with
@@ -84,31 +84,21 @@ async def open_piker_runtime(
     a root actor.
 
     '''
-    # check for existing runtime, boot it
-    # if not already running.
     try:
-        actor = tractor.current_actor()
+        # check for existing runtime
+        actor = tractor.current_actor().uid
+
     except tractor._exceptions.NoRuntime:
         tractor._state._runtime_vars[
-            'piker_vars'
-        ] = tractor_runtime_overrides
+            'piker_vars'] = tractor_runtime_overrides
 
-        # NOTE: if no registrar list passed used the default of just
-        # setting it as the root actor on localhost.
-        registry_addrs = (
-            registry_addrs
-            or [_default_reg_addr]
-        )
-
-        if ems := tractor_kwargs.pop('enable_modules', None):
-            # import pdbp; pdbp.set_trace()
-            enable_modules.extend(ems)
+        registry_addr = registry_addr or _default_reg_addr
 
         async with (
             tractor.open_root_actor(
 
                 # passed through to ``open_root_actor``
-                registry_addrs=registry_addrs,
+                arbiter_addr=registry_addr,
                 name=name,
                 loglevel=loglevel,
                 debug_mode=debug_mode,
@@ -120,30 +110,24 @@ async def open_piker_runtime(
                 enable_modules=enable_modules,
 
                 **tractor_kwargs,
-            ) as actor,
+            ) as _,
 
-            open_registry(
-                registry_addrs,
-                ensure_exists=False,
-            ) as addrs,
+            open_registry(registry_addr, ensure_exists=False) as addr,
         ):
-            assert actor is tractor.current_actor()
             yield (
-                actor,
-                addrs,
+                tractor.current_actor(),
+                addr,
             )
     else:
-        async with open_registry(
-            registry_addrs
-        ) as addrs:
+        async with open_registry(registry_addr) as addr:
             yield (
                 actor,
-                addrs,
+                addr,
             )
 
 
-_root_dname: str = 'pikerd'
-_root_modules: list[str] = [
+_root_dname = 'pikerd'
+_root_modules = [
     __name__,
     'piker.service._daemon',
     'piker.brokers._daemon',
@@ -157,13 +141,13 @@ _root_modules: list[str] = [
 
 @acm
 async def open_pikerd(
-    registry_addrs: list[tuple[str, int]],
 
     loglevel: str | None = None,
 
     # XXX: you should pretty much never want debug mode
     # for data daemons when running in production.
     debug_mode: bool = False,
+    registry_addr: None | tuple[str, int] = None,
 
     **kwargs,
 
@@ -175,39 +159,29 @@ async def open_pikerd(
     alive underling services (see below).
 
     '''
-    # NOTE: for the root daemon we always enable the root
-    # mod set and we `list.extend()` it into wtv the
-    # caller requested.
-    # TODO: make this mod set more strict?
-    # -[ ] eventually we should be able to avoid
-    #    having the root have more then permissions to spawn other
-    #    specialized daemons I think?
-    ems: list[str] = kwargs.setdefault('enable_modules', [])
-    ems.extend(_root_modules)
-
     async with (
         open_piker_runtime(
 
             name=_root_dname,
+            # TODO: eventually we should be able to avoid
+            # having the root have more then permissions to
+            # spawn other specialized daemons I think?
+            enable_modules=_root_modules,
             loglevel=loglevel,
             debug_mode=debug_mode,
-            registry_addrs=registry_addrs,
+            registry_addr=registry_addr,
 
             **kwargs,
 
-        ) as (
-            root_actor,
-            reg_addrs,
-        ),
+        ) as (root_actor, reg_addr),
         tractor.open_nursery() as actor_nursery,
         trio.open_nursery() as service_nursery,
     ):
-        for addr in reg_addrs:
-            if addr not in root_actor.accept_addrs:
-                raise RuntimeError(
-                    f'`pikerd` failed to bind on {addr}!\n'
-                    'Maybe you have another daemon already running?'
-                )
+        if root_actor.accept_addr != reg_addr:
+            raise RuntimeError(
+                f'`pikerd` failed to bind on {reg_addr}!\n'
+                'Maybe you have another daemon already running?'
+            )
 
         # assign globally for future daemon/task creation
         Services.actor_n = actor_nursery
@@ -251,9 +225,9 @@ async def open_pikerd(
 
 @acm
 async def maybe_open_pikerd(
-    registry_addrs: list[tuple[str, int]] | None = None,
+    loglevel: Optional[str] = None,
+    registry_addr: None | tuple = None,
 
-    loglevel: str | None = None,
     **kwargs,
 
 ) -> tractor._portal.Portal | ClassVar[Services]:
@@ -279,51 +253,32 @@ async def maybe_open_pikerd(
     #     async with open_portal(chan) as arb_portal:
     #         yield arb_portal
 
-    registry_addrs: list[tuple[str, int]] = (
-        registry_addrs
-        or [_default_reg_addr]
-    )
-
-    pikerd_portal: tractor.Portal | None
     async with (
         open_piker_runtime(
             name=query_name,
-            registry_addrs=registry_addrs,
+            registry_addr=registry_addr,
             loglevel=loglevel,
             **kwargs,
-        ) as (actor, addrs),
+        ) as _,
+
+        tractor.find_actor(
+            _root_dname,
+            arbiter_sockaddr=registry_addr,
+        ) as portal
     ):
-        if _root_dname in actor.uid:
-            yield None
+        # connect to any existing daemon presuming
+        # its registry socket was selected.
+        if (
+            portal is not None
+        ):
+            yield portal
             return
 
-        # NOTE: IFF running in disti mode, try to attach to any
-        # existing (host-local) `pikerd`.
-        else:
-            async with tractor.find_actor(
-                _root_dname,
-                registry_addrs=registry_addrs,
-                only_first=True,
-                # raise_on_none=True,
-            ) as pikerd_portal:
-
-                # connect to any existing remote daemon presuming its
-                # registry socket was selected.
-                if pikerd_portal is not None:
-
-                    # sanity check that we are actually connecting to
-                    # a remote process and not ourselves.
-                    assert actor.uid != pikerd_portal.channel.uid
-                    assert registry_addrs
-
-                    yield pikerd_portal
-                    return
-
     # presume pikerd role since no daemon could be found at
     # configured address
     async with open_pikerd(
         loglevel=loglevel,
-        registry_addrs=registry_addrs,
+        registry_addr=registry_addr,
 
         # passthrough to ``tractor`` init
         **kwargs,

piker/service/_ahab.py

@@ -15,8 +15,8 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-Supervisor for ``docker`` with included async and SC wrapping to
-ensure a cancellable container lifetime system.
+Supervisor for ``docker`` with included async and SC wrapping
+to ensure a cancellable container lifetime system.
 
 '''
 from __future__ import annotations

piker/service/_daemon.py

@@ -70,10 +70,7 @@ async def maybe_spawn_daemon(
     lock = Services.locks[service_name]
     await lock.acquire()
 
-    async with find_service(
-        service_name,
-        registry_addrs=[('127.0.0.1', 6116)],
-    ) as portal:
+    async with find_service(service_name) as portal:
         if portal is not None:
             lock.release()
             yield portal

piker/service/_mngr.py

@@ -27,12 +27,6 @@ from typing import (
 import trio
 from trio_typing import TaskStatus
 import tractor
-from tractor import (
-    current_actor,
-    ContextCancelled,
-    Context,
-    Portal,
-)
 
 from ._util import (
     log,  # sub-sys logger
@@ -44,8 +38,6 @@ from ._util import (
 #   library.
 # - wrap a "remote api" wherein you can get a method proxy
 #   to the pikerd actor for starting services remotely!
-# - prolly rename this to ActorServicesNursery since it spawns
-#   new actors and supervises them to completion?
 class Services:
 
     actor_n: tractor._supervise.ActorNursery
@@ -55,7 +47,7 @@ class Services:
         str,
         tuple[
             trio.CancelScope,
-            Portal,
+            tractor.Portal,
             trio.Event,
         ]
     ] = {}
@@ -65,12 +57,12 @@ class Services:
     async def start_service_task(
         self,
         name: str,
-        portal: Portal,
+        portal: tractor.Portal,
         target: Callable,
         allow_overruns: bool = False,
         **ctx_kwargs,
 
-    ) -> (trio.CancelScope, Context):
+    ) -> (trio.CancelScope, tractor.Context):
         '''
         Open a context in a service sub-actor, add to a stack
         that gets unwound at ``pikerd`` teardown.
@@ -109,30 +101,13 @@ class Services:
                         # wait on any context's return value
                         # and any final portal result from the
                         # sub-actor.
-                        ctx_res: Any = await ctx.result()
+                        ctx_res = await ctx.result()
 
                         # NOTE: blocks indefinitely until cancelled
                         # either by error from the target context
                         # function or by being cancelled here by the
                         # surrounding cancel scope.
                         return (await portal.result(), ctx_res)
-                    except ContextCancelled as ctxe:
-                        canceller: tuple[str, str] = ctxe.canceller
-                        our_uid: tuple[str, str] = current_actor().uid
-                        if (
-                            canceller != portal.channel.uid
-                            and
-                            canceller != our_uid
-                        ):
-                            log.cancel(
-                                f'Actor-service {name} was remotely cancelled?\n'
-                                f'remote canceller: {canceller}\n'
-                                f'Keeping {our_uid} alive, ignoring sub-actor cancel..\n'
-                            )
-                        else:
-                            raise
-
-
 
                     finally:
                         await portal.cancel_actor()

piker/service/_registry.py

@@ -27,7 +27,6 @@ from typing import (
 )
 
 import tractor
-from tractor import Portal
 
 from ._util import (
     log,  # sub-sys logger
@@ -47,9 +46,7 @@ _registry: Registry | None = None
 
 
 class Registry:
-    # TODO: should this be a set or should we complain
-    # on duplicates?
-    addrs: list[tuple[str, int]] = []
+    addr: None | tuple[str, int] = None
 
     # TODO: table of uids to sockaddrs
     peers: dict[
@@ -63,115 +60,69 @@ _tractor_kwargs: dict[str, Any] = {}
 
 @acm
 async def open_registry(
-    addrs: list[tuple[str, int]],
+    addr: None | tuple[str, int] = None,
     ensure_exists: bool = True,
 
-) -> list[tuple[str, int]]:
-    '''
-    Open the service-actor-discovery registry by returning a set of
-    tranport socket-addrs to registrar actors which may be
-    contacted and queried for similar addresses for other
-    non-registrar actors.
+) -> tuple[str, int]:
 
-    '''
     global _tractor_kwargs
     actor = tractor.current_actor()
     uid = actor.uid
-    preset_reg_addrs: list[tuple[str, int]] = Registry.addrs
     if (
-        preset_reg_addrs
-        and addrs
+        Registry.addr is not None
+        and addr
     ):
-        if preset_reg_addrs != addrs:
-            # if any(addr in preset_reg_addrs for addr in addrs):
-            diff: set[tuple[str, int]] = set(preset_reg_addrs) - set(addrs)
-            if diff:
-                log.warning(
-                    f'`{uid}` requested only subset of registrars: {addrs}\n'
-                    f'However there are more @{diff}'
-                )
-            else:
-                raise RuntimeError(
-                    f'`{uid}` has non-matching registrar addresses?\n'
-                    f'request: {addrs}\n'
-                    f'already set: {preset_reg_addrs}'
-                )
+        raise RuntimeError(
+            f'`{uid}` registry addr already bound @ {_registry.sockaddr}'
+        )
 
     was_set: bool = False
 
     if (
         not tractor.is_root_process()
-        and not Registry.addrs
+        and Registry.addr is None
     ):
-        Registry.addrs.extend(actor.reg_addrs)
+        Registry.addr = actor._arb_addr
 
     if (
         ensure_exists
-        and not Registry.addrs
+        and Registry.addr is None
     ):
         raise RuntimeError(
-            f"`{uid}` registry should already exist but doesn't?"
+            f"`{uid}` registry should already exist bug doesn't?"
         )
 
     if (
-        not Registry.addrs
+        Registry.addr is None
     ):
         was_set = True
-        Registry.addrs = addrs or [_default_reg_addr]
+        Registry.addr = addr or _default_reg_addr
 
-    # NOTE: only spot this seems currently used is inside
-    # `.ui._exec` which is the (eventual qtloops) bootstrapping
-    # with guest mode.
-    _tractor_kwargs['registry_addrs'] = Registry.addrs
+    _tractor_kwargs['arbiter_addr'] = Registry.addr
 
     try:
-        yield Registry.addrs
+        yield Registry.addr
     finally:
         # XXX: always clear the global addr if we set it so that the
         # next (set of) calls will apply whatever new one is passed
         # in.
         if was_set:
-            Registry.addrs = None
+            Registry.addr = None
 
 
 @acm
 async def find_service(
     service_name: str,
-    registry_addrs: list[tuple[str, int]] | None = None,
+) -> tractor.Portal | None:
 
-    first_only: bool = True,
-
-) -> (
-    Portal
-    | list[Portal]
-    | None
-):
-
-    reg_addrs: list[tuple[str, int]]
-    async with open_registry(
-        addrs=(
-            registry_addrs
-            # NOTE: if no addr set is passed assume the registry has
-            # already been opened and use the previously applied
-            # startup set.
-            or Registry.addrs
-        ),
-    ) as reg_addrs:
+    async with open_registry() as reg_addr:
         log.info(f'Scanning for service `{service_name}`')
-
-        maybe_portals: list[Portal] | Portal | None
-
         # attach to existing daemon by name if possible
         async with tractor.find_actor(
             service_name,
-            registry_addrs=reg_addrs,
-            only_first=first_only,  # if set only returns single ref
-        ) as maybe_portals:
-            if not maybe_portals:
-                yield None
-                return
-
-            yield maybe_portals
+            arbiter_sockaddr=reg_addr,
+        ) as maybe_portal:
+            yield maybe_portal
 
 
 async def check_for_service(
@@ -182,11 +133,9 @@ async def check_for_service(
     Service daemon "liveness" predicate.
 
     '''
-    async with (
-        open_registry(ensure_exists=False) as reg_addr,
-        tractor.query_actor(
+    async with open_registry(ensure_exists=False) as reg_addr:
+        async with tractor.query_actor(
             service_name,
             arbiter_sockaddr=reg_addr,
-        ) as sockaddr,
-    ):
-        return sockaddr
+        ) as sockaddr:
+            return sockaddr

piker/service/marketstore.py

@@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) Tyler Goodlet (in stewardship for piker0)
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -25,9 +25,11 @@
 '''
 from __future__ import annotations
 from contextlib import asynccontextmanager as acm
+from datetime import datetime
 from pprint import pformat
 from typing import (
     Any,
+    Union,
     TYPE_CHECKING,
 )
 import time
@@ -35,34 +37,31 @@ from math import isnan
 from pathlib import Path
 
 from bidict import bidict
-from msgspec.msgpack import (
-    encode,
-    decode,
-)
+from msgspec.msgpack import encode, decode
 # import pyqtgraph as pg
 import numpy as np
 import tractor
 from trio_websocket import open_websocket_url
-from anyio_marketstore import (  # noqa
+from anyio_marketstore import (
     open_marketstore_client,
     MarketstoreClient,
     Params,
 )
 import pendulum
-# TODO: import this for specific error set expected by mkts client
-# import purerpc
-
-from ..data.feed import maybe_open_feed
-from . import Services
-from ._util import (
-    log,  # sub-sys logger
-    get_console_log,
-)
+import purerpc
 
 if TYPE_CHECKING:
     import docker
     from ._ahab import DockerContainer
 
+from ._util import (
+    log,  # sub-sys logger
+    get_console_log,
+)
+from . import Services
+from ..data.feed import maybe_open_feed
+from .._profile import Profiler
+from .. import config
 
 
 # ahabd-supervisor and container level config
@@ -327,6 +326,16 @@ _ohlcv_dt = [
 ]
 
 
+ohlc_key_map = bidict({
+    'Epoch': 'time',
+    'Open': 'open',
+    'High': 'high',
+    'Low': 'low',
+    'Close': 'close',
+    'Volume': 'volume',
+})
+
+
 def mk_tbk(keys: tuple[str, str, str]) -> str:
     '''
     Generate a marketstore table key from a tuple.
@@ -378,6 +387,24 @@ def quote_to_marketstore_structarray(
     return np.array([tuple(array_input)], dtype=_quote_dt)
 
 
+@acm
+async def get_client(
+    host: str | None,
+    port: int | None,
+
+) -> MarketstoreClient:
+    '''
+    Load a ``anyio_marketstore`` grpc client connected
+    to an existing ``marketstore`` server.
+
+    '''
+    async with open_marketstore_client(
+        host or 'localhost',
+        port or _config['grpc_listen_port'],
+    ) as client:
+        yield client
+
+
 class MarketStoreError(Exception):
     "Generic marketstore client error"
 
@@ -405,6 +432,375 @@ tf_in_1s = bidict({
 })
 
 
+class Storage:
+    '''
+    High level storage api for both real-time and historical ingest.
+
+    '''
+    def __init__(
+        self,
+        client: MarketstoreClient,
+
+    ) -> None:
+        # TODO: eventually this should be an api/interface type that
+        # ensures we can support multiple tsdb backends.
+        self.client = client
+
+        # series' cache from tsdb reads
+        self._arrays: dict[str, np.ndarray] = {}
+
+    async def list_keys(self) -> list[str]:
+        return await self.client.list_symbols()
+
+    async def search_keys(self, pattern: str) -> list[str]:
+        '''
+        Search for time series key in the storage backend.
+
+        '''
+        ...
+
+    async def write_ticks(self, ticks: list) -> None:
+        ...
+
+    async def load(
+        self,
+        fqme: str,
+        timeframe: int,
+
+    ) -> tuple[
+        np.ndarray,  # timeframe sampled array-series
+        datetime | None,  # first dt
+        datetime | None,  # last dt
+    ]:
+
+        first_tsdb_dt, last_tsdb_dt = None, None
+        hist = await self.read_ohlcv(
+            fqme,
+            # on first load we don't need to pull the max
+            # history per request size worth.
+            limit=3000,
+            timeframe=timeframe,
+        )
+        log.info(f'Loaded tsdb history {hist}')
+
+        if len(hist):
+            times = hist['Epoch']
+            first, last = times[0], times[-1]
+            first_tsdb_dt, last_tsdb_dt = map(
+                pendulum.from_timestamp, [first, last]
+            )
+
+        return (
+            hist,  # array-data
+            first_tsdb_dt,  # start of query-frame
+            last_tsdb_dt,  # most recent
+        )
+
+    async def read_ohlcv(
+        self,
+        fqme: str,
+        timeframe: int | str,
+        end: int | None = None,
+        limit: int = int(800e3),
+
+    ) -> np.ndarray:
+
+        client = self.client
+        syms = await client.list_symbols()
+
+        if fqme not in syms:
+            return {}
+
+        # use the provided timeframe or 1s by default
+        tfstr = tf_in_1s.get(timeframe, tf_in_1s[1])
+
+        params = Params(
+            symbols=fqme,
+            timeframe=tfstr,
+            attrgroup='OHLCV',
+            end=end,
+            # limit_from_start=True,
+
+            # TODO: figure the max limit here given the
+            # ``purepc`` msg size limit of purerpc: 33554432
+            limit=limit,
+        )
+
+        for i in range(3):
+            try:
+                result = await client.query(params)
+                break
+            except purerpc.grpclib.exceptions.UnknownError as err:
+                if 'snappy' in err.args:
+                    await tractor.breakpoint()
+
+                # indicate there is no history for this timeframe
+                log.exception(
+                    f'Unknown mkts QUERY error: {params}\n'
+                    f'{err.args}'
+                )
+        else:
+            return {}
+
+        # TODO: it turns out column access on recarrays is actually slower:
+        # https://jakevdp.github.io/PythonDataScienceHandbook/02.09-structured-data-numpy.html#RecordArrays:-Structured-Arrays-with-a-Twist
+        # it might make sense to make these structured arrays?
+        data_set = result.by_symbols()[fqme]
+        array = data_set.array
+
+        # XXX: ensure sample rate is as expected
+        time = data_set.array['Epoch']
+        if len(time) > 1:
+            time_step = time[-1] - time[-2]
+            ts = tf_in_1s.inverse[data_set.timeframe]
+
+            if time_step != ts:
+                log.warning(
+                    f'MKTS BUG: wrong timeframe loaded: {time_step}'
+                    'YOUR DATABASE LIKELY CONTAINS BAD DATA FROM AN OLD BUG'
+                    f'WIPING HISTORY FOR {ts}s'
+                )
+                await self.delete_ts(fqme, timeframe)
+
+                # try reading again..
+                return await self.read_ohlcv(
+                    fqme,
+                    timeframe,
+                    end,
+                    limit,
+                )
+
+        return array
+
+    async def delete_ts(
+        self,
+        key: str,
+        timeframe: Union[int, str | None] = None,
+        fmt: str = 'OHLCV',
+
+    ) -> bool:
+
+        client = self.client
+        syms = await client.list_symbols()
+        if key not in syms:
+            await tractor.breakpoint()
+            raise KeyError(f'`{key}` table key not found in\n{syms}?')
+
+        tbk = mk_tbk((
+            key,
+            tf_in_1s.get(timeframe, tf_in_1s[60]),
+            fmt,
+        ))
+        return await client.destroy(tbk=tbk)
+
+    async def write_ohlcv(
+        self,
+        fqme: str,
+        ohlcv: np.ndarray,
+        timeframe: int,
+        append_and_duplicate: bool = True,
+        limit: int = int(800e3),
+
+    ) -> None:
+        # build mkts schema compat array for writing
+        mkts_dt = np.dtype(_ohlcv_dt)
+        mkts_array = np.zeros(
+            len(ohlcv),
+            dtype=mkts_dt,
+        )
+        # copy from shm array (yes it's this easy):
+        # https://numpy.org/doc/stable/user/basics.rec.html#assignment-from-other-structured-arrays
+        mkts_array[:] = ohlcv[[
+            'time',
+            'open',
+            'high',
+            'low',
+            'close',
+            'volume',
+        ]]
+
+        m, r = divmod(len(mkts_array), limit)
+
+        tfkey = tf_in_1s[timeframe]
+        for i in range(m, 1):
+            to_push = mkts_array[i-1:i*limit]
+
+            # write to db
+            resp = await self.client.write(
+                to_push,
+                tbk=f'{fqme}/{tfkey}/OHLCV',
+
+                # NOTE: will will append duplicates
+                # for the same timestamp-index.
+                # TODO: pre-deduplicate?
+                isvariablelength=append_and_duplicate,
+            )
+
+            log.info(
+                f'Wrote {mkts_array.size} datums to tsdb\n'
+            )
+
+            for resp in resp.responses:
+                err = resp.error
+                if err:
+                    raise MarketStoreError(err)
+
+        if r:
+            to_push = mkts_array[m*limit:]
+
+            # write to db
+            resp = await self.client.write(
+                to_push,
+                tbk=f'{fqme}/{tfkey}/OHLCV',
+
+                # NOTE: will will append duplicates
+                # for the same timestamp-index.
+                # TODO: pre deduplicate?
+                isvariablelength=append_and_duplicate,
+            )
+
+            log.info(
+                f'Wrote {mkts_array.size} datums to tsdb\n'
+            )
+
+            for resp in resp.responses:
+                err = resp.error
+                if err:
+                    raise MarketStoreError(err)
+
+    # XXX: currently the only way to do this is through the CLI:
+
+    # sudo ./marketstore connect --dir ~/.config/piker/data
+    # >> \show mnq.globex.20220617.ib/1Sec/OHLCV 2022-05-15
+    # and this seems to block and use up mem..
+    # >> \trim mnq.globex.20220617.ib/1Sec/OHLCV 2022-05-15
+
+    # relevant source code for this is here:
+    # https://github.com/alpacahq/marketstore/blob/master/cmd/connect/session/trim.go#L14
+    # def delete_range(self, start_dt, end_dt) -> None:
+    #     ...
+
+
+@acm
+async def open_storage_client(
+    host: str,
+    grpc_port: int,
+
+) -> tuple[Storage, dict[str, np.ndarray]]:
+    '''
+    Load a series by key and deliver in ``numpy`` struct array format.
+
+    '''
+    async with (
+        # eventually a storage backend endpoint
+        get_client(
+            host=host,
+            port=grpc_port,
+        ) as client,
+    ):
+        # slap on our wrapper api
+        yield Storage(client)
+
+
+@acm
+async def open_tsdb_client(
+    fqme: str,
+) -> Storage:
+
+    # TODO: real-time dedicated task for ensuring
+    # history consistency between the tsdb, shm and real-time feed..
+
+    # update sequence design notes:
+
+    # - load existing highest frequency data from mkts
+    #   * how do we want to offer this to the UI?
+    #    - lazy loading?
+    #    - try to load it all and expect graphics caching/diffing
+    #      to  hide extra bits that aren't in view?
+
+    # - compute the diff between latest data from broker and shm
+    #   * use sql api in mkts to determine where the backend should
+    #     start querying for data?
+    #   * append any diff with new shm length
+    #   * determine missing (gapped) history by scanning
+    #   * how far back do we look?
+
+    # - begin rt update ingest and aggregation
+    #   * could start by always writing ticks to mkts instead of
+    #     worrying about a shm queue for now.
+    #   * we have a short list of shm queues worth groking:
+    #     - https://github.com/pikers/piker/issues/107
+    #   * the original data feed arch blurb:
+    #     - https://github.com/pikers/piker/issues/98
+    #
+    profiler = Profiler(
+        disabled=True,  # not pg_profile_enabled(),
+        delayed=False,
+    )
+
+    # load any user service settings for connecting to
+    rootconf, path = config.load(
+        'conf',
+        touch_if_dne=True,
+    )
+    tsdbconf = rootconf['network'].get('tsdb')
+    # backend = tsdbconf.pop('backend')
+    async with (
+        open_storage_client(
+            **tsdbconf,
+        ) as storage,
+
+        maybe_open_feed(
+            [fqme],
+            start_stream=False,
+
+        ) as feed,
+    ):
+        profiler(f'opened feed for {fqme}')
+
+        # to_append = feed.hist_shm.array
+        # to_prepend = None
+
+        if fqme:
+            flume = feed.flumes[fqme]
+            symbol = flume.mkt
+            if symbol:
+                fqme = symbol.fqme
+
+            # diff db history with shm and only write the missing portions
+            # ohlcv = flume.hist_shm.array
+
+            # TODO: use pg profiler
+            # for secs in (1, 60):
+            #     tsdb_array = await storage.read_ohlcv(
+            #         fqme,
+            #         timeframe=timeframe,
+            #     )
+            #     # hist diffing:
+            #     # these aren't currently used but can be referenced from
+            #     # within the embedded ipython shell below.
+            #     to_append = ohlcv[ohlcv['time'] > ts['Epoch'][-1]]
+            #     to_prepend = ohlcv[ohlcv['time'] < ts['Epoch'][0]]
+
+            # profiler('Finished db arrays diffs')
+
+            _ = await storage.client.list_symbols()
+            # log.info(f'Existing tsdb symbol set:\n{pformat(syms)}')
+            # profiler(f'listed symbols {syms}')
+            yield storage
+
+        # for array in [to_append, to_prepend]:
+        #     if array is None:
+        #         continue
+
+        #     log.info(
+        #         f'Writing datums {array.size} -> to tsdb from shm\n'
+        #     )
+        #     await storage.write_ohlcv(fqme, array)
+
+        # profiler('Finished db writes')
+
+
 async def ingest_quote_stream(
     symbols: list[str],
     brokername: str,
@@ -416,7 +812,6 @@ async def ingest_quote_stream(
     Ingest a broker quote stream into a ``marketstore`` tsdb.
 
     '''
-    from piker.storage.marketstore import get_client
     async with (
         maybe_open_feed(brokername, symbols, loglevel=loglevel) as feed,
         get_client() as ms_client,
@@ -568,3 +963,5 @@ async def stream_quotes(
 
             if quotes:
                 yield quotes
+
+

piker/storage/__init__.py

@@ -1,320 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-(time-series) database middle ware layer.
-
-- APIs for read, write, delete, replicate over multiple
-  db systems.
-- backend agnostic tick msg ingest machinery.
-- broadcast systems for fan out of real-time ingested
-  data to live consumers.
-- test harness utilities for data-processing verification.
-
-'''
-from abc import abstractmethod
-from contextlib import asynccontextmanager as acm
-from functools import partial
-from importlib import import_module
-from datetime import datetime
-from types import ModuleType
-from typing import (
-    # Callable,
-    # Awaitable,
-    # Any,
-    # AsyncIterator,
-    Protocol,
-    # Generic,
-    # TypeVar,
-)
-
-import numpy as np
-
-
-from .. import config
-from ..service import (
-    check_for_service,
-)
-from ..log import (
-    get_logger,
-    get_console_log,
-)
-subsys: str = 'piker.storage'
-
-log = get_logger(subsys)
-get_console_log = partial(
-    get_console_log,
-    name=subsys,
-)
-
-
-__tsdbs__: list[str] = [
-    'nativedb',
-    # 'marketstore',
-]
-
-
-class StorageClient(
-    Protocol,
-):
-    '''
-    Api description that all storage backends must implement
-    in order to suffice the historical data mgmt layer.
-
-    '''
-    name: str
-
-    @abstractmethod
-    async def list_keys(self) -> list[str]:
-        ...
-
-    @abstractmethod
-    def search_keys(self) -> list[str]:
-        ...
-
-    # @abstractmethod
-    # async def write_ticks(
-    #     self,
-    #     ticks: list,
-    # ) -> ReceiveType:
-    #     ...
-
-    # ``trio.abc.AsyncResource`` methods
-    @abstractmethod
-    async def load(
-        self,
-        fqme: str,
-        timeframe: int,
-
-    ) -> tuple[
-        np.ndarray,  # timeframe sampled array-series
-        datetime | None,  # first dt
-        datetime | None,  # last dt
-    ]:
-        ...
-
-    @abstractmethod
-    async def delete_ts(
-        self,
-        key: str,
-        timeframe: int | str | None = None,
-        fmt: str = 'OHLCV',
-
-    ) -> bool:
-        ...
-
-    @abstractmethod
-    async def read_ohlcv(
-        self,
-        fqme: str,
-        timeframe: int | str,
-        end: int | None = None,
-        limit: int = int(800e3),
-
-    ) -> np.ndarray:
-        ...
-
-    async def write_ohlcv(
-        self,
-        fqme: str,
-        ohlcv: np.ndarray,
-        timeframe: int,
-        append_and_duplicate: bool = True,
-        limit: int = int(800e3),
-
-    ) -> None:
-        ...
-
-
-class TimeseriesNotFound(Exception):
-    '''
-    No timeseries entry can be found for this backend.
-
-    '''
-
-
-class StorageConnectionError(ConnectionError):
-    '''
-    Can't connect to the desired tsdb subsys/service.
-
-    '''
-
-def get_storagemod(name: str) -> ModuleType:
-    mod: ModuleType = import_module(
-        '.' + name,
-        'piker.storage',
-    )
-
-    # we only allow monkeying because it's for internal keying
-    mod.name = mod.__name__.split('.')[-1]
-    return mod
-
-
-@acm
-async def open_storage_client(
-    backend: str | None = None,
-
-) -> tuple[ModuleType, StorageClient]:
-    '''
-    Load the ``StorageClient`` for named backend.
-
-    '''
-    def_backend: str = 'nativedb'
-    tsdb_host: str = 'localhost'
-
-    # load root config and any tsdb user defined settings
-    conf, path = config.load(
-        conf_name='conf',
-        touch_if_dne=True,
-    )
-
-    # TODO: maybe not under a "network" section.. since
-    # no more chitty `marketstore`..
-    tsdbconf: dict = {}
-    service_section = conf.get('service')
-    if (
-        not backend
-        and service_section
-    ):
-        tsdbconf = service_section.get('tsdb')
-
-        # lookup backend tsdb module by name and load any user service
-        # settings for connecting to the tsdb service.
-        backend: str = tsdbconf.pop(
-            'name',
-            def_backend,
-        )
-        tsdb_host: str = tsdbconf.get('maddrs', [])
-
-    if backend is None:
-        backend: str = def_backend
-
-    # import and load storagemod by name
-    mod: ModuleType = get_storagemod(backend)
-    get_client = mod.get_client
-
-    log.info('Scanning for existing `{tsbd_backend}`')
-    if backend != def_backend:
-        tsdb_is_up: bool = await check_for_service(f'{backend}d')
-        if (
-            tsdb_host == 'localhost'
-            or tsdb_is_up
-        ):
-            log.info(f'Connecting to local: {backend}@{tsdbconf}')
-        else:
-            log.info(f'Attempting to connect to remote: {backend}@{tsdbconf}')
-    else:
-        log.info(f'Connecting to default storage: {backend}@{tsdbconf}')
-
-    async with (
-        get_client(**tsdbconf) as client,
-    ):
-        # slap on our wrapper api
-        yield mod, client
-
-
-# NOTE: pretty sure right now this is only being
-# called by a CLI entrypoint?
-@acm
-async def open_tsdb_client(
-    fqme: str,
-) -> StorageClient:
-
-    # TODO: real-time dedicated task for ensuring
-    # history consistency between the tsdb, shm and real-time feed..
-
-    # update sequence design notes:
-
-    # - load existing highest frequency data from mkts
-    #   * how do we want to offer this to the UI?
-    #    - lazy loading?
-    #    - try to load it all and expect graphics caching/diffing
-    #      to  hide extra bits that aren't in view?
-
-    # - compute the diff between latest data from broker and shm
-    #   * use sql api in mkts to determine where the backend should
-    #     start querying for data?
-    #   * append any diff with new shm length
-    #   * determine missing (gapped) history by scanning
-    #   * how far back do we look?
-
-    # - begin rt update ingest and aggregation
-    #   * could start by always writing ticks to mkts instead of
-    #     worrying about a shm queue for now.
-    #   * we have a short list of shm queues worth groking:
-    #     - https://github.com/pikers/piker/issues/107
-    #   * the original data feed arch blurb:
-    #     - https://github.com/pikers/piker/issues/98
-    #
-    from ..toolz import Profiler
-    profiler = Profiler(
-        disabled=True,  # not pg_profile_enabled(),
-        delayed=False,
-    )
-    from ..data.feed import maybe_open_feed
-
-    async with (
-        open_storage_client() as (_, storage),
-
-        maybe_open_feed(
-            [fqme],
-            start_stream=False,
-
-        ) as feed,
-    ):
-        profiler(f'opened feed for {fqme}')
-
-        # to_append = feed.hist_shm.array
-        # to_prepend = None
-
-        if fqme:
-            flume = feed.flumes[fqme]
-            symbol = flume.mkt
-            if symbol:
-                fqme = symbol.fqme
-
-            # diff db history with shm and only write the missing portions
-            # ohlcv = flume.hist_shm.array
-
-            # TODO: use pg profiler
-            # for secs in (1, 60):
-            #     tsdb_array = await storage.read_ohlcv(
-            #         fqme,
-            #         timeframe=timeframe,
-            #     )
-            #     # hist diffing:
-            #     # these aren't currently used but can be referenced from
-            #     # within the embedded ipython shell below.
-            #     to_append = ohlcv[ohlcv['time'] > ts['Epoch'][-1]]
-            #     to_prepend = ohlcv[ohlcv['time'] < ts['Epoch'][0]]
-
-            # profiler('Finished db arrays diffs')
-
-            _ = await storage.client.list_symbols()
-            # log.info(f'Existing tsdb symbol set:\n{pformat(syms)}')
-            # profiler(f'listed symbols {syms}')
-            yield storage
-
-        # for array in [to_append, to_prepend]:
-        #     if array is None:
-        #         continue
-
-        #     log.info(
-        #         f'Writing datums {array.size} -> to tsdb from shm\n'
-        #     )
-        #     await storage.write_ohlcv(fqme, array)
-
-        # profiler('Finished db writes')

piker/storage/cli.py

@@ -1,553 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-Storage middle-ware CLIs.
-
-"""
-from __future__ import annotations
-# from datetime import datetime
-# from contextlib import (
-#     AsyncExitStack,
-# )
-from pathlib import Path
-from math import copysign
-import time
-from types import ModuleType
-from typing import (
-    Any,
-    TYPE_CHECKING,
-)
-
-import polars as pl
-import numpy as np
-import tractor
-# import pendulum
-from rich.console import Console
-import trio
-# from rich.markdown import Markdown
-import typer
-
-from piker.service import open_piker_runtime
-from piker.cli import cli
-from piker.data import (
-    ShmArray,
-)
-from piker import tsp
-from piker.data._formatters import BGM
-from . import log
-from . import (
-    __tsdbs__,
-    open_storage_client,
-    StorageClient,
-)
-
-if TYPE_CHECKING:
-    from piker.ui._remote_ctl import AnnotCtl
-
-
-store = typer.Typer()
-
-
-@store.command()
-def ls(
-    backends: list[str] = typer.Argument(
-        default=None,
-        help='Storage backends to query, default is all.'
-    ),
-):
-    from rich.table import Table
-
-    if not backends:
-        backends: list[str] = __tsdbs__
-
-    console = Console()
-
-    async def query_all():
-        nonlocal backends
-
-        async with (
-            open_piker_runtime(
-                'tsdb_storage',
-            ),
-        ):
-            for i, backend in enumerate(backends):
-                table = Table()
-                try:
-                    async with open_storage_client(backend=backend) as (
-                        mod,
-                        client,
-                    ):
-                        table.add_column(f'{mod.name}@{client.address}')
-                        keys: list[str] = await client.list_keys()
-                        for key in keys:
-                            table.add_row(key)
-
-                    console.print(table)
-                except Exception:
-                    log.error(f'Unable to connect to storage engine: `{backend}`')
-
-    trio.run(query_all)
-
-
-# TODO: like ls but takes in a pattern and matches
-# @store.command()
-# def search(
-#     patt: str,
-#     backends: list[str] = typer.Argument(
-#         default=None,
-#         help='Storage backends to query, default is all.'
-#     ),
-# ):
-#     ...
-
-
-@store.command()
-def delete(
-    symbols: list[str],
-
-    backend: str = typer.Option(
-        default=None,
-        help='Storage backend to update'
-    ),
-    # TODO: expose this as flagged multi-option?
-    timeframes: list[int] = [1, 60],
-):
-    '''
-    Delete a storage backend's time series for (table) keys provided as
-    ``symbols``.
-
-    '''
-    from . import open_storage_client
-
-    async def main(symbols: list[str]):
-        async with (
-            open_piker_runtime(
-                'tsdb_storage',
-            ),
-            open_storage_client(backend) as (_, client),
-            trio.open_nursery() as n,
-        ):
-            # spawn queries as tasks for max conc!
-            for fqme in symbols:
-                for tf in timeframes:
-                    n.start_soon(
-                        client.delete_ts,
-                        fqme,
-                        tf,
-                    )
-
-    trio.run(main, symbols)
-
-
-@store.command()
-def anal(
-    fqme: str,
-    period: int = 60,
-    pdb: bool = False,
-
-) -> np.ndarray:
-    '''
-    Anal-ysis is when you take the data do stuff to it.
-
-    NOTE: This ONLY loads the offline timeseries data (by default
-    from a parquet file) NOT the in-shm version you might be seeing
-    in a chart.
-
-    '''
-    async def main():
-        async with (
-            open_piker_runtime(
-                # are you a bear or boi?
-                'tsdb_polars_anal',
-                debug_mode=pdb,
-            ),
-            open_storage_client() as (
-                mod,
-                client,
-            ),
-        ):
-            syms: list[str] = await client.list_keys()
-            log.info(f'{len(syms)} FOUND for {mod.name}')
-
-            history: ShmArray  # np buffer format
-            (
-                history,
-                first_dt,
-                last_dt,
-            ) = await client.load(
-                fqme,
-                period,
-            )
-            assert first_dt < last_dt
-
-            null_segs: tuple = tsp.get_null_segs(
-                frame=history,
-                period=period,
-            )
-            # TODO: do tsp queries to backcend to fill i missing
-            # history and then prolly write it to tsdb!
-
-            shm_df: pl.DataFrame = await client.as_df(
-                fqme,
-                period,
-            )
-
-            df: pl.DataFrame  # with dts
-            deduped: pl.DataFrame  # deduplicated dts
-            (
-                df,
-                deduped,
-                diff,
-            ) = tsp.dedupe(
-                shm_df,
-                period=period,
-            )
-
-            write_edits: bool = True
-            if (
-                write_edits
-                and (
-                    diff
-                    or null_segs
-                )
-            ):
-                await tractor.pause()
-                await client.write_ohlcv(
-                    fqme,
-                    ohlcv=deduped,
-                    timeframe=period,
-                )
-
-            else:
-                # TODO: something better with tab completion..
-                # is there something more minimal but nearly as
-                # functional as ipython?
-                await tractor.pause()
-                assert not null_segs
-
-    trio.run(main)
-
-
-async def markup_gaps(
-    fqme: str,
-    timeframe: float,
-    actl: AnnotCtl,
-    wdts: pl.DataFrame,
-    gaps: pl.DataFrame,
-
-) -> dict[int, dict]:
-    '''
-    Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
-    with rectangles.
-
-    '''
-    aids: dict[int] = {}
-    for i in range(gaps.height):
-
-        row: pl.DataFrame = gaps[i]
-
-        # the gap's RIGHT-most bar's OPEN value
-        # at that time (sample) step.
-        iend: int = row['index'][0]
-        # dt: datetime = row['dt'][0]
-        # dt_prev: datetime = row['dt_prev'][0]
-        # dt_end_t: float = dt.timestamp()
-
-
-        # TODO: can we eventually remove this
-        # once we figure out why the epoch cols
-        # don't match?
-        # TODO: FIX HOW/WHY these aren't matching
-        # and are instead off by 4hours (EST
-        # vs. UTC?!?!)
-        # end_t: float = row['time']
-        # assert (
-        #     dt.timestamp()
-        #     ==
-        #     end_t
-        # )
-
-        # the gap's LEFT-most bar's CLOSE value
-        # at that time (sample) step.
-        prev_r: pl.DataFrame = wdts.filter(
-            pl.col('index') == iend - 1
-        )
-        # XXX: probably a gap in the (newly sorted or de-duplicated)
-        # dt-df, so we might need to re-index first..
-        if prev_r.is_empty():
-            await tractor.pause()
-
-        istart: int = prev_r['index'][0]
-        # dt_start_t: float = dt_prev.timestamp()
-
-        # start_t: float = prev_r['time']
-        # assert (
-        #     dt_start_t
-        #     ==
-        #     start_t
-        # )
-
-        # TODO: implement px-col width measure
-        # and ensure at least as many px-cols
-        # shown per rect as configured by user.
-        # gap_w: float = abs((iend - istart))
-        # if gap_w < 6:
-        #     margin: float = 6
-        #     iend += margin
-        #     istart -= margin
-
-        rect_gap: float = BGM*3/8
-        opn: float = row['open'][0]
-        ro: tuple[float, float] = (
-            # dt_end_t,
-            iend + rect_gap + 1,
-            opn,
-        )
-        cls: float = prev_r['close'][0]
-        lc: tuple[float, float] = (
-            # dt_start_t,
-            istart - rect_gap, # + 1 ,
-            cls,
-        )
-
-        color: str = 'dad_blue'
-        diff: float = cls - opn
-        sgn: float = copysign(1, diff)
-        color: str = {
-            -1: 'buy_green',
-            1: 'sell_red',
-        }[sgn]
-
-        rect_kwargs: dict[str, Any] = dict(
-            fqme=fqme,
-            timeframe=timeframe,
-            start_pos=lc,
-            end_pos=ro,
-            color=color,
-        )
-
-        aid: int = await actl.add_rect(**rect_kwargs)
-        assert aid
-        aids[aid] = rect_kwargs
-
-    # tell chart to redraw all its
-    # graphics view layers Bo
-    await actl.redraw(
-        fqme=fqme,
-        timeframe=timeframe,
-    )
-    return aids
-
-
-@store.command()
-def ldshm(
-    fqme: str,
-    write_parquet: bool = True,
-    reload_parquet_to_shm: bool = True,
-
-) -> None:
-    '''
-    Linux ONLY: load any fqme file name matching shm buffer from
-    /dev/shm/ into an OHLCV numpy array and polars DataFrame,
-    optionally write to offline storage via `.parquet` file.
-
-    '''
-    async def main():
-        from piker.ui._remote_ctl import (
-            open_annot_ctl,
-        )
-        actl: AnnotCtl
-        mod: ModuleType
-        client: StorageClient
-        async with (
-            open_piker_runtime(
-                'polars_boi',
-                enable_modules=['piker.data._sharedmem'],
-                debug_mode=True,
-            ),
-            open_storage_client() as (
-                mod,
-                client,
-            ),
-            open_annot_ctl() as actl,
-        ):
-            shm_df: pl.DataFrame | None = None
-            tf2aids: dict[float, dict] = {}
-
-            for (
-                shmfile,
-                shm,
-                # parquet_path,
-                shm_df,
-            ) in tsp.iter_dfs_from_shms(fqme):
-
-                times: np.ndarray = shm.array['time']
-                d1: float = float(times[-1] - times[-2])
-                d2: float = float(times[-2] - times[-3])
-                med: float = np.median(np.diff(times))
-                if (
-                    d1 < 1.
-                    and d2 < 1.
-                    and med < 1.
-                ):
-                    raise ValueError(
-                        f'Something is wrong with time period for {shm}:\n{times}'
-                    )
-
-                period_s: float = float(max(d1, d2, med))
-
-                null_segs: tuple = tsp.get_null_segs(
-                    frame=shm.array,
-                    period=period_s,
-                )
-
-                # TODO: call null-seg fixer somehow?
-                if null_segs:
-                    await tractor.pause()
-                #     async with (
-                #         trio.open_nursery() as tn,
-                #         mod.open_history_client(
-                #             mkt,
-                #         ) as (get_hist, config),
-                #     ):
-                #         nulls_detected: trio.Event = await tn.start(partial(
-                #             tsp.maybe_fill_null_segments,
-
-                #             shm=shm,
-                #             timeframe=timeframe,
-                #             get_hist=get_hist,
-                #             sampler_stream=sampler_stream,
-                #             mkt=mkt,
-                #         ))
-
-                # over-write back to shm?
-                wdts: pl.DataFrame  # with dts
-                deduped: pl.DataFrame  # deduplicated dts
-                (
-                    wdts,
-                    deduped,
-                    diff,
-                ) = tsp.dedupe(
-                    shm_df,
-                    period=period_s,
-                )
-
-                # detect gaps from in expected (uniform OHLC) sample period
-                step_gaps: pl.DataFrame = tsp.detect_time_gaps(
-                    deduped,
-                    expect_period=period_s,
-                )
-
-                # TODO: by default we always want to mark these up
-                # with rects showing up/down gaps Bo
-                venue_gaps: pl.DataFrame = tsp.detect_time_gaps(
-                    deduped,
-                    expect_period=period_s,
-
-                    # TODO: actually pull the exact duration
-                    # expected for each venue operational period?
-                    gap_dt_unit='days',
-                    gap_thresh=1,
-                )
-
-                # TODO: find the disjoint set of step gaps from
-                # venue (closure) set!
-                # -[ ] do a set diff by checking for the unique
-                #    gap set only in the step_gaps?
-                if (
-                    not venue_gaps.is_empty()
-                    or (
-                        period_s < 60
-                        and not step_gaps.is_empty()
-                    )
-                ):
-                    # write repaired ts to parquet-file?
-                    if write_parquet:
-                        start: float = time.time()
-                        path: Path = await client.write_ohlcv(
-                            fqme,
-                            ohlcv=deduped,
-                            timeframe=period_s,
-                        )
-                        write_delay: float = round(
-                            time.time() - start,
-                            ndigits=6,
-                        )
-
-                        # read back from fs
-                        start: float = time.time()
-                        read_df: pl.DataFrame = pl.read_parquet(path)
-                        read_delay: float = round(
-                            time.time() - start,
-                            ndigits=6,
-                        )
-                        log.info(
-                            f'parquet write took {write_delay} secs\n'
-                            f'file path: {path}'
-                            f'parquet read took {read_delay} secs\n'
-                            f'polars df: {read_df}'
-                        )
-
-                        if reload_parquet_to_shm:
-                            new = tsp.pl2np(
-                                deduped,
-                                dtype=shm.array.dtype,
-                            )
-                            # since normally readonly
-                            shm._array.setflags(
-                                write=int(1),
-                            )
-                            shm.push(
-                                new,
-                                prepend=True,
-                                start=new['index'][-1],
-                                update_first=False,  # don't update ._first
-                            )
-
-                    do_markup_gaps: bool = True
-                    if do_markup_gaps:
-                        new_df: pl.DataFrame = tsp.np2pl(new)
-                        aids: dict = await markup_gaps(
-                            fqme,
-                            period_s,
-                            actl,
-                            new_df,
-                            step_gaps,
-                        )
-                        # last chance manual overwrites in REPL
-                        # await tractor.pause()
-                        assert aids
-                        tf2aids[period_s] = aids
-
-                else:
-                    # allow interaction even when no ts problems.
-                    assert not diff
-
-            await tractor.pause()
-            log.info('Exiting TSP shm anal-izer!')
-
-            if shm_df is None:
-                log.error(
-                    f'No matching shm buffers for {fqme} ?'
-
-                )
-
-    trio.run(main)
-
-
-typer_click_object = typer.main.get_command(store)
-cli.add_command(typer_click_object, 'store')

piker/storage/marketstore/__init__.py

@@ -1,384 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-marketstore tsdb backend:
-https://github.com/alpacahq/marketstore
-
-
-We wrote an async gGRPC client:
-https://github.com/pikers/anyio-marketstore
-
-which is normally preferred minus the discovered issues
-in https://github.com/pikers/piker/issues/443
-
-Which is the main reason for us moving away from this
-platform..
-
-'''
-from __future__ import annotations
-from contextlib import asynccontextmanager as acm
-from datetime import datetime
-# from pprint import pformat
-from typing import (
-    Union,
-)
-
-from bidict import bidict
-import tractor
-import numpy as np
-from anyio_marketstore import (
-    Params,
-)
-import pendulum
-# import purerpc
-
-from piker.service.marketstore import (
-    MarketstoreClient,
-    tf_in_1s,
-    mk_tbk,
-    _ohlcv_dt,
-    MarketStoreError,
-)
-from anyio_marketstore import (  # noqa
-    open_marketstore_client,
-    MarketstoreClient,
-    Params,
-)
-from piker.log import get_logger
-
-
-log = get_logger(__name__)
-
-
-class MktsStorageClient:
-    '''
-    High level storage api for both real-time and historical ingest.
-
-    '''
-    name: str = 'marketstore'
-
-    def __init__(
-        self,
-        client: MarketstoreClient,
-        config: dict,
-
-    ) -> None:
-        # TODO: eventually this should be an api/interface type that
-        # ensures we can support multiple tsdb backends.
-        self.client = client
-        self._config = config
-
-        # series' cache from tsdb reads
-        self._arrays: dict[str, np.ndarray] = {}
-
-    @property
-    def address(self) -> str:
-        conf = self._config
-        return f'grpc://{conf["host"]}:{conf["port"]}'
-
-    async def list_keys(self) -> list[str]:
-        return await self.client.list_symbols()
-
-    async def search_keys(self, pattern: str) -> list[str]:
-        '''
-        Search for time series key in the storage backend.
-
-        '''
-        ...
-
-    async def write_ticks(self, ticks: list) -> None:
-        ...
-
-    async def load(
-        self,
-        fqme: str,
-        timeframe: int,
-
-    ) -> tuple[
-        np.ndarray,  # timeframe sampled array-series
-        datetime | None,  # first dt
-        datetime | None,  # last dt
-    ]:
-        first_tsdb_dt, last_tsdb_dt = None, None
-        hist = await self.read_ohlcv(
-            fqme,
-            # on first load we don't need to pull the max
-            # history per request size worth.
-            limit=3000,
-            timeframe=timeframe,
-        )
-        log.info(f'Loaded tsdb history {hist}')
-
-        if len(hist):
-            # breakpoint()
-            times: np.ndarray = hist['Epoch']
-
-            first, last = times[0], times[-1]
-            first_tsdb_dt, last_tsdb_dt = map(
-                pendulum.from_timestamp,
-                [first, last]
-            )
-
-        return (
-            hist,  # array-data
-            first_tsdb_dt,  # start of query-frame
-            last_tsdb_dt,  # most recent
-        )
-
-    async def read_ohlcv(
-        self,
-        fqme: str,
-        timeframe: int | str,
-        end: float | None = None,  # epoch or none
-        limit: int = int(200e3),
-
-    ) -> np.ndarray:
-
-        client = self.client
-        syms = await client.list_symbols()
-        if fqme not in syms:
-            return {}
-
-        # ensure end time is in correct int format!
-        if (
-            end
-            and not isinstance(end, float)
-        ):
-            end = int(float(end))
-            # breakpoint()
-
-        # use the provided timeframe or 1s by default
-        tfstr = tf_in_1s.get(timeframe, tf_in_1s[1])
-
-        import pymarketstore as pymkts
-        sync_client = pymkts.Client()
-        param = pymkts.Params(
-            symbols=fqme,
-            timeframe=tfstr,
-            attrgroup='OHLCV',
-            end=end,
-
-            limit=limit,
-            # limit_from_start=True,
-        )
-        try:
-            reply = sync_client.query(param)
-        except Exception as err:
-            if 'no files returned from query parse: None' in err.args:
-                return []
-
-            raise
-
-        data_set: pymkts.results.DataSet = reply.first()
-        array: np.ndarray = data_set.array
-
-        # params = Params(
-        #     symbols=fqme,
-        #     timeframe=tfstr,
-        #     attrgroup='OHLCV',
-        #     end=end,
-        #     # limit_from_start=True,
-
-        #     # TODO: figure the max limit here given the
-        #     # ``purepc`` msg size limit of purerpc: 33554432
-        #     limit=limit,
-        # )
-
-        # for i in range(3):
-        #     try:
-        #         result = await client.query(params)
-        #         break
-        #     except purerpc.grpclib.exceptions.UnknownError as err:
-        #         if 'snappy' in err.args:
-        #             await tractor.pause()
-
-        #         # indicate there is no history for this timeframe
-        #         log.exception(
-        #             f'Unknown mkts QUERY error: {params}\n'
-        #             f'{err.args}'
-        #         )
-        # else:
-        #     return {}
-
-        # # TODO: it turns out column access on recarrays is actually slower:
-        # # https://jakevdp.github.io/PythonDataScienceHandbook/02.09-structured-data-numpy.html#RecordArrays:-Structured-Arrays-with-a-Twist
-        # # it might make sense to make these structured arrays?
-        # data_set = result.by_symbols()[fqme]
-        # array = data_set.array
-
-        # XXX: ensure sample rate is as expected
-        time = data_set.array['Epoch']
-        if len(time) > 1:
-            time_step = time[-1] - time[-2]
-            ts = tf_in_1s.inverse[data_set.timeframe]
-
-            if time_step != ts:
-                log.warning(
-                    f'MKTS BUG: wrong timeframe loaded: {time_step}\n'
-                    'YOUR DATABASE LIKELY CONTAINS BAD DATA FROM AN OLD BUG '
-                    f'WIPING HISTORY FOR {ts}s'
-                )
-                await tractor.pause()
-                # await self.delete_ts(fqme, timeframe)
-
-                # try reading again..
-                # return await self.read_ohlcv(
-                #     fqme,
-                #     timeframe,
-                #     end,
-                #     limit,
-                # )
-
-        return array
-
-    async def delete_ts(
-        self,
-        key: str,
-        timeframe: Union[int, str | None] = None,
-        fmt: str = 'OHLCV',
-
-    ) -> bool:
-
-        client = self.client
-        # syms = await client.list_symbols()
-        # if key not in syms:
-        #     raise KeyError(f'`{key}` table key not found in\n{syms}?')
-
-        tbk = mk_tbk((
-            key,
-            tf_in_1s.get(timeframe, tf_in_1s[60]),
-            fmt,
-        ))
-        return await client.destroy(tbk=tbk)
-
-    async def write_ohlcv(
-        self,
-        fqme: str,
-        ohlcv: np.ndarray,
-        timeframe: int,
-        append_and_duplicate: bool = True,
-        limit: int = int(800e3),
-
-    ) -> None:
-        # build mkts schema compat array for writing
-        mkts_dt = np.dtype(_ohlcv_dt)
-        mkts_array = np.zeros(
-            len(ohlcv),
-            dtype=mkts_dt,
-        )
-        # copy from shm array (yes it's this easy):
-        # https://numpy.org/doc/stable/user/basics.rec.html#assignment-from-other-structured-arrays
-        mkts_array[:] = ohlcv[[
-            'time',
-            'open',
-            'high',
-            'low',
-            'close',
-            'volume',
-        ]]
-
-        m, r = divmod(len(mkts_array), limit)
-
-        tfkey = tf_in_1s[timeframe]
-        for i in range(m, 1):
-            to_push = mkts_array[i-1:i*limit]
-
-            # write to db
-            resp = await self.client.write(
-                to_push,
-                tbk=f'{fqme}/{tfkey}/OHLCV',
-
-                # NOTE: will will append duplicates
-                # for the same timestamp-index.
-                # TODO: pre-deduplicate?
-                isvariablelength=append_and_duplicate,
-            )
-
-            log.info(
-                f'Wrote {mkts_array.size} datums to tsdb\n'
-            )
-
-            for resp in resp.responses:
-                err = resp.error
-                if err:
-                    raise MarketStoreError(err)
-
-        if r:
-            to_push = mkts_array[m*limit:]
-
-            # write to db
-            resp = await self.client.write(
-                to_push,
-                tbk=f'{fqme}/{tfkey}/OHLCV',
-
-                # NOTE: will will append duplicates
-                # for the same timestamp-index.
-                # TODO: pre deduplicate?
-                isvariablelength=append_and_duplicate,
-            )
-
-            log.info(
-                f'Wrote {mkts_array.size} datums to tsdb\n'
-            )
-
-            for resp in resp.responses:
-                err = resp.error
-                if err:
-                    raise MarketStoreError(err)
-
-    # XXX: currently the only way to do this is through the CLI:
-
-    # sudo ./marketstore connect --dir ~/.config/piker/data
-    # >> \show mnq.globex.20220617.ib/1Sec/OHLCV 2022-05-15
-    # and this seems to block and use up mem..
-    # >> \trim mnq.globex.20220617.ib/1Sec/OHLCV 2022-05-15
-
-    # relevant source code for this is here:
-    # https://github.com/alpacahq/marketstore/blob/master/cmd/connect/session/trim.go#L14
-    # def delete_range(self, start_dt, end_dt) -> None:
-    #     ...
-
-
-ohlc_key_map = bidict({
-    'Epoch': 'time',
-    'Open': 'open',
-    'High': 'high',
-    'Low': 'low',
-    'Close': 'close',
-    'Volume': 'volume',
-})
-
-
-@acm
-async def get_client(
-    grpc_port: int = 5995,  # required
-    host: str = 'localhost',
-
-) -> MarketstoreClient:
-    '''
-    Load a ``anyio_marketstore`` grpc client connected
-    to an existing ``marketstore`` server.
-
-    '''
-    async with open_marketstore_client(
-        host or 'localhost',
-        grpc_port,
-    ) as client:
-        yield MktsStorageClient(
-            client,
-            config={'host': host, 'port': grpc_port},
-        )

piker/storage/marketstore/_ingest.py

@@ -1,177 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Legacy marketstore ingest and streaming related clis.
-
-'''
-# from .. import watchlists as wl
-# from ..service.marketstore import (
-    # get_client,
-    # stream_quotes,
-    # ingest_quote_stream,
-    # _url,
-    # _tick_tbk_ids,
-    # mk_tbk,
-# )
-
-# @cli.command()
-# @click.option(
-#     '--url',
-#     default='ws://localhost:5993/ws',
-#     help='HTTP URL of marketstore instance'
-# )
-# @click.argument('names', nargs=-1)
-# @click.pass_obj
-# def ms_stream(
-#     config: dict,
-#     names: list[str],
-#     url: str,
-# ) -> None:
-#     '''
-#     Connect to a marketstore time bucket stream for (a set of) symbols(s)
-#     and print to console.
-
-#     '''
-#     async def main():
-#         # async for quote in stream_quotes(symbols=names):
-#         #    log.info(f"Received quote:\n{quote}")
-#         ...
-
-#     trio.run(main)
-
-
-# @cli.command()
-# @click.option(
-#     '--url',
-#     default=_url,
-#     help='HTTP URL of marketstore instance'
-# )
-# @click.argument('names', nargs=-1)
-# @click.pass_obj
-# def ms_destroy(config: dict, names: list[str], url: str) -> None:
-#     """Destroy symbol entries in the local marketstore instance.
-#     """
-#     async def main():
-#         nonlocal names
-#         async with get_client(url) as client:
-#
-#             if not names:
-#                 names = await client.list_symbols()
-#
-#             # default is to wipe db entirely.
-#             answer = input(
-#                 "This will entirely wipe you local marketstore db @ "
-#                 f"{url} of the following symbols:\n {pformat(names)}"
-#                 "\n\nDelete [N/y]?\n")
-#
-#             if answer == 'y':
-#                 for sym in names:
-#                     # tbk = _tick_tbk.format(sym)
-#                     tbk = tuple(sym, *_tick_tbk_ids)
-#                     print(f"Destroying {tbk}..")
-#                     await client.destroy(mk_tbk(tbk))
-#             else:
-#                 print("Nothing deleted.")
-#
-#     tractor.run(main)
-
-
-# @cli.command()
-# @click.option(
-#     '--tsdb_host',
-#     default='localhost'
-# )
-# @click.option(
-#     '--tsdb_port',
-#     default=5993
-# )
-# @click.argument('symbols', nargs=-1)
-# @click.pass_obj
-# def storesh(
-#     config,
-#     tl,
-#     host,
-#     port,
-#     symbols: list[str],
-# ):
-#     '''
-#     Start an IPython shell ready to query the local marketstore db.
-
-#     '''
-#     from piker.storage import open_tsdb_client
-#     from piker.service import open_piker_runtime
-
-#     async def main():
-#         nonlocal symbols
-
-#         async with open_piker_runtime(
-#             'storesh',
-#             enable_modules=['piker.service._ahab'],
-#         ):
-#             symbol = symbols[0]
-
-#             async with open_tsdb_client(symbol):
-#                 # TODO: ask if user wants to write history for detected
-#                 # available shm buffers?
-#                 from tractor.trionics import ipython_embed
-#                 await ipython_embed()
-
-#     trio.run(main)
-
-
-# @cli.command()
-# @click.option('--test-file', '-t', help='Test quote stream file')
-# @click.option('--tl', is_flag=True, help='Enable tractor logging')
-# @click.argument('name', nargs=1, required=True)
-# @click.pass_obj
-# def ingest(config, name, test_file, tl):
-#     '''
-#     Ingest real-time broker quotes and ticks to a marketstore instance.
-
-#     '''
-#     # global opts
-#     loglevel = config['loglevel']
-#     tractorloglevel = config['tractorloglevel']
-#     # log = config['log']
-
-#     watchlist_from_file = wl.ensure_watchlists(config['wl_path'])
-#     watchlists = wl.merge_watchlist(watchlist_from_file, wl._builtins)
-#     symbols = watchlists[name]
-
-#     grouped_syms = {}
-#     for sym in symbols:
-#         symbol, _, provider = sym.rpartition('.')
-#         if provider not in grouped_syms:
-#             grouped_syms[provider] = []
-
-#         grouped_syms[provider].append(symbol)
-
-#     async def entry_point():
-#         async with tractor.open_nursery() as n:
-#             for provider, symbols in grouped_syms.items():
-#                 await n.run_in_actor(
-#                     ingest_quote_stream,
-#                     name='ingest_marketstore',
-#                     symbols=symbols,
-#                     brokername=provider,
-#                     tries=1,
-#                     actorloglevel=loglevel,
-#                     loglevel=tractorloglevel
-#                 )
-
-#     tractor.run(entry_point)
-

piker/storage/nativedb.py

@@ -1,433 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-`nativedb`: a lulzy Apache-parquet file manager (that some might
-            call a poor man's tsdb).
-
-AKA a `piker`-native file-system native "time series database"
-without needing an extra process and no standard TSDB features,
-YET!
-
-'''
-# TODO: like there's soo much..
-# - better name like "parkdb" or "nativedb" (lel)? bundle this lib with
-#   others to make full system:
-#   - tractor for failover and reliablity?
-#   - borg for replication and sync?
-#
-# - use `fastparquet` for appends:
-#   https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
-#   (presuming it's actually faster then overwrites and
-#   makes sense in terms of impl?)
-#
-# - use `polars` support for lazy scanning, processing and schema
-#   validation?
-#   - https://pola-rs.github.io/polars-book/user-guide/io/parquet/#scan
-#   - https://pola-rs.github.io/polars-book/user-guide/concepts/lazy-vs-eager/
-# - consider delta writes for appends?
-#   - https://github.com/pola-rs/polars/blob/main/py-polars/polars/dataframe/frame.py#L3232
-# - consider multi-file appends with appropriate time-range naming?
-#   - https://pola-rs.github.io/polars-book/user-guide/io/multiple/
-#
-# - use `borg` for replication?
-#   - https://borgbackup.readthedocs.io/en/stable/quickstart.html#remote-repositories
-#   - https://github.com/borgbackup/borg
-#   - https://borgbackup.readthedocs.io/en/stable/faq.html#usage-limitations
-#   - https://github.com/borgbackup/community
-#   - https://github.com/spslater/borgapi
-#   - https://nixos.wiki/wiki/ZFS
-from __future__ import annotations
-from contextlib import asynccontextmanager as acm
-from datetime import datetime
-from pathlib import Path
-import time
-
-import numpy as np
-import polars as pl
-from pendulum import (
-    from_timestamp,
-)
-
-from piker import config
-from piker import tsp
-from piker.data import (
-    def_iohlcv_fields,
-    ShmArray,
-)
-from piker.log import get_logger
-from . import TimeseriesNotFound
-
-
-log = get_logger('storage.nativedb')
-
-
-def detect_period(shm: ShmArray) -> float:
-    '''
-    Attempt to detect the series time step sampling period
-    in seconds.
-
-    '''
-    # TODO: detect sample rate helper?
-    # calc ohlc sample period for naming
-    ohlcv: np.ndarray = shm.array
-    times: np.ndarray = ohlcv['time']
-    period: float = times[-1] - times[-2]
-    if period == 0:
-        # maybe just last sample is borked?
-        period: float = times[-2] - times[-3]
-
-    return period
-
-
-def mk_ohlcv_shm_keyed_filepath(
-    fqme: str,
-    period: float | int,  # ow known as the "timeframe"
-    datadir: Path,
-
-) -> Path:
-
-    if period < 1.:
-        raise ValueError('Sample period should be >= 1.!?')
-
-    path: Path = (
-        datadir
-        /
-        f'{fqme}.ohlcv{int(period)}s.parquet'
-    )
-    return path
-
-
-def unpack_fqme_from_parquet_filepath(path: Path) -> str:
-
-    filename: str = str(path.name)
-    fqme, fmt_descr, suffix = filename.split('.')
-    assert suffix == 'parquet'
-    return fqme
-
-
-ohlc_key_map = None
-
-
-class NativeStorageClient:
-    '''
-    High level storage api for OHLCV time series stored in
-    a (modern) filesystem as apache parquet files B)
-
-    Part of a grander scheme to use arrow and parquet as our main
-    lowlevel data framework: https://arrow.apache.org/faq/.
-
-    '''
-    name: str = 'nativedb'
-
-    def __init__(
-        self,
-        datadir: Path,
-
-    ) -> None:
-        self._datadir = datadir
-        self._index: dict[str, dict] = {}
-
-        # series' cache from tsdb reads
-        self._dfs: dict[str, dict[str, pl.DataFrame]] = {}
-
-    @property
-    def address(self) -> str:
-        return self._datadir.as_uri()
-
-    @property
-    def cardinality(self) -> int:
-        return len(self._index)
-
-    # @property
-    # def compression(self) -> str:
-    #     ...
-
-    async def list_keys(self) -> list[str]:
-        return list(self._index)
-
-    def index_files(self):
-        for path in self._datadir.iterdir():
-            if (
-                path.is_dir()
-                or
-                '.parquet' not in str(path)
-                # or
-                # path.name in {'borked', 'expired',}
-            ):
-                continue
-
-            key: str = path.name.rstrip('.parquet')
-            fqme, _, descr = key.rpartition('.')
-            prefix, _, suffix = descr.partition('ohlcv')
-            period: int = int(suffix.strip('s'))
-
-            # cache description data
-            self._index[fqme] = {
-                'path': path,
-                'period': period,
-            }
-
-        return self._index
-
-
-    # async def search_keys(self, pattern: str) -> list[str]:
-    #     '''
-    #     Search for time series key in the storage backend.
-
-    #     '''
-    #     ...
-
-    # async def write_ticks(self, ticks: list) -> None:
-    #     ...
-
-    async def load(
-        self,
-        fqme: str,
-        timeframe: int,
-
-    ) -> tuple[
-        np.ndarray,  # timeframe sampled array-series
-        datetime | None,  # first dt
-        datetime | None,  # last dt
-    ] | None:
-        try:
-            array: np.ndarray = await self.read_ohlcv(
-                fqme,
-                timeframe,
-            )
-        except FileNotFoundError as fnfe:
-
-            bs_fqme, _, *_ = fqme.rpartition('.')
-
-            possible_matches: list[str] = []
-            for tskey in self._index:
-                if bs_fqme in tskey:
-                    possible_matches.append(tskey)
-
-            match_str: str = '\n'.join(sorted(possible_matches))
-            raise TimeseriesNotFound(
-                f'No entry for `{fqme}`?\n'
-                f'Maybe you need a more specific fqme-key like:\n\n'
-                f'{match_str}'
-            ) from fnfe
-
-        times = array['time']
-        return (
-            array,
-            from_timestamp(times[0]),
-            from_timestamp(times[-1]),
-        )
-
-    def mk_path(
-        self,
-        fqme: str,
-        period: float,
-
-    ) -> Path:
-        return mk_ohlcv_shm_keyed_filepath(
-            fqme=fqme,
-            period=period,
-            datadir=self._datadir,
-        )
-
-    def _cache_df(
-        self,
-        fqme: str,
-        df: pl.DataFrame,
-        timeframe: float,
-
-    ) -> None:
-        # cache df for later usage since we (currently) need to
-        # convert to np.ndarrays to push to our `ShmArray` rt
-        # buffers subsys but later we may operate entirely on
-        # pyarrow arrays/buffers so keeping the dfs around for
-        # a variety of purposes is handy.
-        self._dfs.setdefault(
-            timeframe,
-            {},
-        )[fqme] = df
-
-    async def read_ohlcv(
-        self,
-        fqme: str,
-        timeframe: int | str,
-        end: float | None = None,  # epoch or none
-        # limit: int = int(200e3),
-
-    ) -> np.ndarray:
-        path: Path = self.mk_path(
-            fqme,
-            period=int(timeframe),
-        )
-        df: pl.DataFrame = pl.read_parquet(path)
-
-        self._cache_df(
-            fqme=fqme,
-            df=df,
-            timeframe=timeframe,
-        )
-        # TODO: filter by end and limit inputs
-        # times: pl.Series = df['time']
-        array: np.ndarray = tsp.pl2np(
-            df,
-            dtype=np.dtype(def_iohlcv_fields),
-        )
-        return array
-
-    async def as_df(
-        self,
-        fqme: str,
-        period: int = 60,
-        load_from_offline: bool = True,
-
-    ) -> pl.DataFrame:
-        try:
-            return self._dfs[period][fqme]
-        except KeyError:
-            if not load_from_offline:
-                raise
-
-            await self.read_ohlcv(fqme, period)
-            return self._dfs[period][fqme]
-
-    def _write_ohlcv(
-        self,
-        fqme: str,
-        ohlcv: np.ndarray | pl.DataFrame,
-        timeframe: int,
-
-    ) -> Path:
-        '''
-        Sync version of the public interface meth, since we don't
-        currently actually need or support an async impl.
-
-        '''
-        path: Path = mk_ohlcv_shm_keyed_filepath(
-            fqme=fqme,
-            period=timeframe,
-            datadir=self._datadir,
-        )
-        if isinstance(ohlcv, np.ndarray):
-            df: pl.DataFrame = tsp.np2pl(ohlcv)
-        else:
-            df = ohlcv
-
-        self._cache_df(
-            fqme=fqme,
-            df=df,
-            timeframe=timeframe,
-        )
-
-        # TODO: in terms of managing the ultra long term data
-        # -[ ] use a proper profiler to measure all this IO and
-        #   roundtripping!
-        # -[ ] implement parquet append!? see issue:
-        #   https://github.com/pikers/piker/issues/536
-        #   -[ ] try out ``fastparquet``'s append writing:
-        #     https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
-        start = time.time()
-        df.write_parquet(path)
-        delay: float = round(
-            time.time() - start,
-            ndigits=6,
-        )
-        log.info(
-            f'parquet write took {delay} secs\n'
-            f'file path: {path}'
-        )
-        return path
-
-    async def write_ohlcv(
-        self,
-        fqme: str,
-        ohlcv: np.ndarray | pl.DataFrame,
-        timeframe: int,
-
-    ) -> Path:
-        '''
-        Write input ohlcv time series for fqme and sampling period
-        to (local) disk.
-
-        '''
-        return self._write_ohlcv(
-            fqme,
-            ohlcv,
-            timeframe,
-        )
-
-    async def delete_ts(
-        self,
-        key: str,
-        timeframe: int | None = None,
-
-    ) -> bool:
-        path: Path = mk_ohlcv_shm_keyed_filepath(
-            fqme=key,
-            period=timeframe,
-            datadir=self._datadir,
-        )
-        if path.is_file():
-            path.unlink()
-            log.warning(f'Deleting parquet entry:\n{path}')
-        else:
-            log.error(f'No path exists:\n{path}')
-
-        return path
-
-    # TODO: allow wiping and refetching a segment of the OHLCV timeseries
-    # data.
-    # def clear_range(
-    #     self,
-    #     key: str,
-    #     start_dt: datetime,
-    #     end_dt: datetime,
-    #     timeframe: int | None = None,
-    # ) -> pl.DataFrame:
-    #     '''
-    #     Clear and re-fetch a range of datums for the OHLCV time series.
-
-    #     Useful for series editing from a chart B)
-
-    #     '''
-    #     ...
-
-
-# TODO: does this need to be async on average?
-# I guess for any IPC connected backend yes?
-@acm
-async def get_client(
-
-    # TODO: eventually support something something apache arrow
-    # transport over ssh something..?
-    # host: str | None = None,
-
-    **kwargs,
-
-) -> NativeStorageClient:
-    '''
-    Load a ``anyio_marketstore`` grpc client connected
-    to an existing ``marketstore`` server.
-
-    '''
-    datadir: Path = config.get_conf_dir() / 'nativedb'
-    if not datadir.is_dir():
-        log.info(f'Creating `nativedb` dir: {datadir}')
-        datadir.mkdir()
-
-    client = NativeStorageClient(datadir)
-    client.index_files()
-    yield client

piker/toolz/__init__.py

@@ -1,29 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Toolz for debug, profile and trace of the distributed runtime :surfer:
-
-'''
-from tractor.devx import (
-    open_crash_handler as open_crash_handler,
-)
-from .profile import (
-    Profiler as Profiler,
-    pg_profile_enabled as pg_profile_enabled,
-    ms_slower_then as ms_slower_then,
-    timeit as timeit,
-)

piker/trionics.py

@@ -0,0 +1,80 @@
+# piker: trading gear for hackers
+# Copyright (C) Tyler Goodlet (in stewardship of piker0)
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+sugarz for trio/tractor conc peeps.
+
+'''
+from typing import AsyncContextManager
+from typing import TypeVar
+from contextlib import asynccontextmanager as acm
+
+import trio
+
+
+# A regular invariant generic type
+T = TypeVar("T")
+
+
+async def _enter_and_sleep(
+
+    mngr: AsyncContextManager[T],
+    to_yield: dict[int, T],
+    all_entered: trio.Event,
+    # task_status: TaskStatus[T] = trio.TASK_STATUS_IGNORED,
+
+) -> T:
+    '''Open the async context manager deliver it's value
+    to this task's spawner and sleep until cancelled.
+
+    '''
+    async with mngr as value:
+        to_yield[id(mngr)] = value
+
+        if all(to_yield.values()):
+            all_entered.set()
+
+        # sleep until cancelled
+        await trio.sleep_forever()
+
+
+@acm
+async def async_enter_all(
+
+    *mngrs: list[AsyncContextManager[T]],
+
+) -> tuple[T]:
+
+    to_yield = {}.fromkeys(id(mngr) for mngr in mngrs)
+
+    all_entered = trio.Event()
+
+    async with trio.open_nursery() as n:
+        for mngr in mngrs:
+            n.start_soon(
+                _enter_and_sleep,
+                mngr,
+                to_yield,
+                all_entered,
+            )
+
+        # deliver control once all managers have started up
+        await all_entered.wait()
+        yield tuple(to_yield.values())
+
+        # tear down all sleeper tasks thus triggering individual
+        # mngr ``__aexit__()``s.
+        n.cancel_scope.cancel()

piker/tsp/_anal.py

@@ -1,746 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Financial time series processing utilities usually
-pertaining to OHLCV style sampled data.
-
-Routines are generally implemented in either ``numpy`` or
-``polars`` B)
-
-'''
-from __future__ import annotations
-from functools import partial
-from math import (
-    ceil,
-    floor,
-)
-import time
-from typing import (
-    Literal,
-    # AsyncGenerator,
-    Generator,
-)
-
-import numpy as np
-import polars as pl
-from pendulum import (
-    DateTime,
-    from_timestamp,
-)
-
-from ..toolz.profile import (
-    Profiler,
-    pg_profile_enabled,
-    ms_slower_then,
-)
-from ..log import (
-    get_logger,
-    get_console_log,
-)
-# for "time series processing"
-subsys: str = 'piker.tsp'
-
-log = get_logger(subsys)
-get_console_log = partial(
-    get_console_log,
-    name=subsys,
-)
-
-# NOTE: union type-defs to handle generic `numpy` and `polars` types
-# side-by-side Bo
-# |_ TODO: schema spec typing?
-#   -[ ] nptyping!
-#   -[ ] wtv we can with polars?
-Frame = pl.DataFrame | np.ndarray
-Seq = pl.Series | np.ndarray
-
-
-def slice_from_time(
-    arr: np.ndarray,
-    start_t: float,
-    stop_t: float,
-    step: float,  # sampler period step-diff
-
-) -> slice:
-    '''
-    Calculate array indices mapped from a time range and return them in
-    a slice.
-
-    Given an input array with an epoch `'time'` series entry, calculate
-    the indices which span the time range and return in a slice. Presume
-    each `'time'` step increment is uniform and when the time stamp
-    series contains gaps (the uniform presumption is untrue) use
-    ``np.searchsorted()`` binary search to look up the appropriate
-    index.
-
-    '''
-    profiler = Profiler(
-        msg='slice_from_time()',
-        disabled=not pg_profile_enabled(),
-        ms_threshold=ms_slower_then,
-    )
-
-    times = arr['time']
-    t_first = floor(times[0])
-    t_last = ceil(times[-1])
-
-    # the greatest index we can return which slices to the
-    # end of the input array.
-    read_i_max = arr.shape[0]
-
-    # compute (presumed) uniform-time-step index offsets
-    i_start_t = floor(start_t)
-    read_i_start = floor(((i_start_t - t_first) // step)) - 1
-
-    i_stop_t = ceil(stop_t)
-
-    # XXX: edge case -> always set stop index to last in array whenever
-    # the input stop time is detected to be greater then the equiv time
-    # stamp at that last entry.
-    if i_stop_t >= t_last:
-        read_i_stop = read_i_max
-    else:
-        read_i_stop = ceil((i_stop_t - t_first) // step) + 1
-
-    # always clip outputs to array support
-    # for read start:
-    # - never allow a start < the 0 index
-    # - never allow an end index > the read array len
-    read_i_start = min(
-        max(0, read_i_start),
-        read_i_max - 1,
-    )
-    read_i_stop = max(
-        0,
-        min(read_i_stop, read_i_max),
-    )
-
-    # check for larger-then-latest calculated index for given start
-    # time, in which case we do a binary search for the correct index.
-    # NOTE: this is usually the result of a time series with time gaps
-    # where it is expected that each index step maps to a uniform step
-    # in the time stamp series.
-    t_iv_start = times[read_i_start]
-    if (
-        t_iv_start > i_start_t
-    ):
-        # do a binary search for the best index mapping to ``start_t``
-        # given we measured an overshoot using the uniform-time-step
-        # calculation from above.
-
-        # TODO: once we start caching these per source-array,
-        # we can just overwrite ``read_i_start`` directly.
-        new_read_i_start = np.searchsorted(
-            times,
-            i_start_t,
-            side='left',
-        )
-
-        # TODO: minimize binary search work as much as possible:
-        # - cache these remap values which compensate for gaps in the
-        #   uniform time step basis where we calc a later start
-        #   index for the given input ``start_t``.
-        # - can we shorten the input search sequence by heuristic?
-        #   up_to_arith_start = index[:read_i_start]
-
-        if (
-            new_read_i_start <= read_i_start
-        ):
-            # t_diff = t_iv_start - start_t
-            # print(
-            #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
-            #     f'start_t:{start_t} -> 0index start_t:{t_iv_start}\n'
-            #     f'diff: {t_diff}\n'
-            #     f'REMAPPED START i: {read_i_start} -> {new_read_i_start}\n'
-            # )
-            read_i_start = new_read_i_start
-
-    t_iv_stop = times[read_i_stop - 1]
-    if (
-        t_iv_stop > i_stop_t
-    ):
-        # t_diff = stop_t - t_iv_stop
-        # print(
-        #     f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
-        #     f'calced iv stop:{t_iv_stop} -> stop_t:{stop_t}\n'
-        #     f'diff: {t_diff}\n'
-        #     # f'SHOULD REMAP STOP: {read_i_start} -> {new_read_i_start}\n'
-        # )
-        new_read_i_stop = np.searchsorted(
-            times[read_i_start:],
-            # times,
-            i_stop_t,
-            side='right',
-        )
-
-        if (
-            new_read_i_stop <= read_i_stop
-        ):
-            read_i_stop = read_i_start + new_read_i_stop + 1
-
-    # sanity checks for range size
-    # samples = (i_stop_t - i_start_t) // step
-    # index_diff = read_i_stop - read_i_start + 1
-    # if index_diff > (samples + 3):
-    #     breakpoint()
-
-    # read-relative indexes: gives a slice where `shm.array[read_slc]`
-    # will be the data spanning the input time range `start_t` ->
-    # `stop_t`
-    read_slc = slice(
-        int(read_i_start),
-        int(read_i_stop),
-    )
-
-    profiler(
-        'slicing complete'
-        # f'{start_t} -> {abs_slc.start} | {read_slc.start}\n'
-        # f'{stop_t} -> {abs_slc.stop} | {read_slc.stop}\n'
-    )
-
-    # NOTE: if caller needs absolute buffer indices they can
-    # slice the buffer abs index like so:
-    # index = arr['index']
-    # abs_indx = index[read_slc]
-    # abs_slc = slice(
-    #     int(abs_indx[0]),
-    #     int(abs_indx[-1]),
-    # )
-
-    return read_slc
-
-
-def get_null_segs(
-    frame: Frame,
-    period: float,  # sampling step in seconds
-    imargin: int = 1,
-    col: str = 'time',
-
-) -> tuple[
-    # Seq,  # TODO: can we make it an array-type instead?
-    list[
-        list[int, int],
-    ],
-    Seq,
-    Frame
-] | None:
-    '''
-    Detect if there are any zero(-epoch stamped) valued
-    rows in for the provided `col: str` column; by default
-    presume the 'time' field/column.
-
-    Filter to all such zero (time) segments and return
-    the corresponding frame zeroed segment's,
-
-      - gap absolute (in buffer terms) indices-endpoints as
-        `absi_zsegs`
-      - abs indices of all rows with zeroed `col` values as `absi_zeros`
-      - the corresponding frame's row-entries (view) which are
-        zeroed for the `col` as `zero_t`
-
-    '''
-    times: Seq = frame['time']
-    zero_pred: Seq = (times == 0)
-
-    if isinstance(frame, np.ndarray):
-        tis_zeros: int = zero_pred.any()
-    else:
-        tis_zeros: int = zero_pred.any()
-
-    if not tis_zeros:
-        return None
-
-    # TODO: use ndarray for this?!
-    absi_zsegs: list[list[int, int]] = []
-
-    if isinstance(frame, np.ndarray):
-        # view of ONLY the zero segments as one continuous chunk
-        zero_t: np.ndarray = frame[zero_pred]
-        # abs indices of said zeroed rows
-        absi_zeros = zero_t['index']
-        # diff of abs index steps between each zeroed row
-        absi_zdiff: np.ndarray = np.diff(absi_zeros)
-
-        # scan for all frame-indices where the
-        # zeroed-row-abs-index-step-diff is greater then the
-        # expected increment of 1.
-        # data  1st zero seg  data  zeros
-        # ----  ------------  ----  -----  ------  ----
-        # ||||..000000000000..||||..00000..||||||..0000
-        # ----  ------------  ----  -----  ------  ----
-        #       ^zero_t[0]                            ^zero_t[-1]
-        #                           ^fi_zgaps[0]   ^fi_zgaps[1]
-        #       ^absi_zsegs[0][0]   ^---^ => absi_zsegs[1]: tuple
-        #  absi_zsegs[0][1]^
-        #
-        # NOTE: the first entry in `fi_zgaps` is where
-        # the first (absolute) index step diff is > 1.
-        # and it is a frame-relative index into `zero_t`.
-        fi_zgaps = np.argwhere(
-            absi_zdiff > 1
-            # NOTE: +1 here is ensure we index to the "start" of each
-            # segment (if we didn't the below loop needs to be
-            # re-written to expect `fi_end_rows`!
-        ) + 1
-        # the rows from the contiguous zeroed segments which have
-        # abs-index steps >1 compared to the previous zero row
-        # (indicating an end of zeroed segment).
-        fi_zseg_start_rows = zero_t[fi_zgaps]
-
-    # TODO: equiv for pl.DataFrame case!
-    else:
-        izeros: pl.Series = zero_pred.arg_true()
-        zero_t: pl.DataFrame = frame[izeros]
-
-        absi_zeros = zero_t['index']
-        absi_zdiff: pl.Series = absi_zeros.diff()
-        fi_zgaps = (absi_zdiff > 1).arg_true()
-
-    # XXX: our goal (in this func) is to select out slice index
-    # pairs (zseg0_start, zseg_end) in abs index units for each
-    # null-segment portion detected throughout entire input frame.
-
-    # only up to one null-segment in entire frame?
-    num_gaps: int = fi_zgaps.size + 1
-    if num_gaps < 1:
-        if absi_zeros.size > 1:
-            absi_zsegs = [[
-                # TODO: maybe mk these max()/min() limits func
-                # consts instead of called more then once?
-                max(
-                    absi_zeros[0] - 1,
-                    0,
-                ),
-                # NOTE: need the + 1 to guarantee we index "up to"
-                # the next non-null row-datum.
-                min(
-                    absi_zeros[-1] + 1,
-                    frame['index'][-1],
-                ),
-            ]]
-        else:
-            # XXX EDGE CASE: only one null-datum found so
-            # mark the start abs index as None to trigger
-            # a full frame-len query to the respective backend?
-            absi_zsegs = [[
-                # see `get_hist()` in backend, should ALWAYS be
-                # able to handle a `start_dt=None`!
-                # None,
-                None,
-                absi_zeros[0] + 1,
-            ]]
-
-    # XXX NOTE XXX: if >= 2 zeroed segments are found, there should
-    # ALWAYS be more then one zero-segment-abs-index-step-diff row
-    # in `absi_zdiff`, so loop through all such
-    # abs-index-step-diffs >1 (i.e. the entries of `absi_zdiff`)
-    # and add them as the "end index" entries for each segment.
-    # Then, iif NOT iterating the first such segment end, look back
-    # for the prior segments zero-segment start indext by relative
-    # indexing the `zero_t` frame by -1 and grabbing the abs index
-    # of what should be the prior zero-segment abs start index.
-    else:
-        # NOTE: since `absi_zdiff` will never have a row
-        # corresponding to the first zero-segment's row, we add it
-        # manually here.
-        absi_zsegs.append([
-            max(
-                absi_zeros[0] - 1,
-                0,
-            ),
-            None,
-        ])
-
-        # TODO: can we do it with vec ops?
-        for i, (
-            fi,  # frame index of zero-seg start
-            zseg_start_row,  # full row for ^
-        ) in enumerate(zip(
-            fi_zgaps,
-            fi_zseg_start_rows,
-        )):
-            assert (zseg_start_row == zero_t[fi]).all()
-            iabs: int = zseg_start_row['index'][0]
-            absi_zsegs.append([
-                iabs - 1,
-                None,  # backfilled on next iter
-            ])
-
-            # final iter case, backfill FINAL end iabs!
-            if (i + 1) == fi_zgaps.size:
-                absi_zsegs[-1][1] = absi_zeros[-1] + 1
-
-            # NOTE: only after the first segment (due to `.diff()`
-            # usage above) can we do a lookback to the prior
-            # segment's end row and determine it's abs index to
-            # retroactively insert to the prior
-            # `absi_zsegs[i-1][1]` entry Bo
-            last_end: int = absi_zsegs[i][1]
-            if last_end is None:
-                prev_zseg_row = zero_t[fi - 1]
-                absi_post_zseg = prev_zseg_row['index'][0] + 1
-                # XXX: MUST BACKFILL previous end iabs!
-                absi_zsegs[i][1] = absi_post_zseg
-
-        else:
-            if 0 < num_gaps < 2:
-                absi_zsegs[-1][1] = min(
-                    absi_zeros[-1] + 1,
-                    frame['index'][-1],
-                )
-
-            iabs_first: int = frame['index'][0]
-            for start, end in absi_zsegs:
-
-                ts_start: float = times[start - iabs_first]
-                ts_end: float = times[end - iabs_first]
-                if (
-                    (ts_start == 0 and not start == 0)
-                    or
-                    ts_end == 0
-                ):
-                    import pdbp
-                    pdbp.set_trace()
-
-                assert end
-                assert start < end
-
-    log.warning(
-        f'Frame has {len(absi_zsegs)} NULL GAPS!?\n'
-        f'period: {period}\n'
-        f'total null samples: {len(zero_t)}\n'
-    )
-
-    return (
-        absi_zsegs,  # [start, end] abs slice indices of seg
-        absi_zeros,  # all abs indices within all null-segs
-        zero_t,  # sliced-view of all null-segment rows-datums
-    )
-
-
-def iter_null_segs(
-    timeframe: float,
-    frame: Frame | None = None,
-    null_segs: tuple | None = None,
-
-) -> Generator[
-    tuple[
-        int, int,
-        int, int,
-        float, float,
-        float, float,
-
-        # Seq,  # TODO: can we make it an array-type instead?
-        # list[
-        #     list[int, int],
-        # ],
-        # Seq,
-        # Frame
-    ],
-    None,
-]:
-    if not (
-        null_segs := get_null_segs(
-            frame,
-            period=timeframe,
-        )
-    ):
-        return
-
-    absi_pairs_zsegs: list[list[float, float]]
-    izeros: Seq
-    zero_t: Frame
-    (
-        absi_pairs_zsegs,
-        izeros,
-        zero_t,
-    ) = null_segs
-
-    absi_first: int = frame[0]['index']
-    for (
-        absi_start,
-        absi_end,
-    ) in absi_pairs_zsegs:
-
-        fi_end: int = absi_end - absi_first
-        end_row: Seq = frame[fi_end]
-        end_t: float = end_row['time']
-        end_dt: DateTime = from_timestamp(end_t)
-
-        fi_start = None
-        start_row = None
-        start_t = None
-        start_dt = None
-        if (
-            absi_start is not None
-            and start_t != 0
-        ):
-            fi_start: int = absi_start - absi_first
-            start_row: Seq = frame[fi_start]
-            start_t: float = start_row['time']
-            start_dt: DateTime = from_timestamp(start_t)
-
-        if absi_start < 0:
-            import pdbp
-            pdbp.set_trace()
-
-        yield (
-            absi_start, absi_end,  # abs indices
-            fi_start, fi_end,  # relative "frame" indices
-            start_t, end_t,
-            start_dt, end_dt,
-        )
-
-
-def with_dts(
-    df: pl.DataFrame,
-    time_col: str = 'time',
-
-) -> pl.DataFrame:
-    '''
-    Insert datetime (casted) columns to a (presumably) OHLC sampled
-    time series with an epoch-time column keyed by `time_col: str`.
-
-    '''
-    return df.with_columns([
-        pl.col(time_col).shift(1).suffix('_prev'),
-        pl.col(time_col).diff().alias('s_diff'),
-        pl.from_epoch(pl.col(time_col)).alias('dt'),
-    ]).with_columns([
-        pl.from_epoch(
-            column=pl.col(f'{time_col}_prev'),
-        ).alias('dt_prev'),
-        pl.col('dt').diff().alias('dt_diff'),
-    ])
-
-
-t_unit: Literal = Literal[
-    'days',
-    'hours',
-    'minutes',
-    'seconds',
-    'miliseconds',
-    'microseconds',
-    'nanoseconds',
-]
-
-
-def detect_time_gaps(
-    w_dts: pl.DataFrame,
-
-    time_col: str = 'time',
-    # epoch sampling step diff
-    expect_period: float = 60,
-
-    # NOTE: legacy stock mkts have venue operating hours
-    # and thus gaps normally no more then 1-2 days at
-    # a time.
-    gap_thresh: float = 1.,
-
-    # TODO: allow passing in a frame of operating hours?
-    # -[ ] durations/ranges for faster legit gap checks?
-    # XXX -> must be valid ``polars.Expr.dt.<name>``
-    # like 'days' which a sane default for venue closures
-    # though will detect weekend gaps which are normal :o
-    gap_dt_unit: t_unit | None = None,
-
-) -> pl.DataFrame:
-    '''
-    Filter to OHLC datums which contain sample step gaps.
-
-    For eg. legacy markets which have venue close gaps and/or
-    actual missing data segments.
-
-    '''
-    # first select by any sample-period (in seconds unit) step size
-    # greater then expected.
-    step_gaps: pl.DataFrame = w_dts.filter(
-        pl.col('s_diff').abs() > expect_period
-    )
-
-    if gap_dt_unit is None:
-        return step_gaps
-
-    # NOTE: this flag is to indicate that on this (sampling) time
-    # scale we expect to only be filtering against larger venue
-    # closures-scale time gaps.
-    return step_gaps.filter(
-        # Second by an arbitrary dt-unit step size
-        getattr(
-            pl.col('dt_diff').dt,
-            gap_dt_unit,
-        )().abs() > gap_thresh
-    )
-
-
-def detect_price_gaps(
-    df: pl.DataFrame,
-    gt_multiplier: float = 2.,
-    price_fields: list[str] = ['high', 'low'],
-
-) -> pl.DataFrame:
-    '''
-    Detect gaps in clearing price over an OHLC series.
-
-    2 types of gaps generally exist; up gaps and down gaps:
-
-    - UP gap: when any next sample's lo price is strictly greater
-      then the current sample's hi price.
-
-    - DOWN gap: when any next sample's hi price is strictly
-      less then the current samples lo price.
-
-    '''
-    # return df.filter(
-    #     pl.col('high') - ) > expect_period,
-    # ).select([
-    #     pl.dt.datetime(pl.col(time_col).shift(1)).suffix('_previous'),
-    #     pl.all(),
-    # ]).select([
-    #     pl.all(),
-    #     (pl.col(time_col) - pl.col(f'{time_col}_previous')).alias('diff'),
-    # ])
-    ...
-
-# TODO: probably just use the null_segs impl above?
-def detect_vlm_gaps(
-    df: pl.DataFrame,
-    col: str = 'volume',
-
-) -> pl.DataFrame:
-
-    vnull: pl.DataFrame = w_dts.filter(
-        pl.col(col) == 0
-    )
-    return vnull
-
-
-def dedupe(
-    src_df: pl.DataFrame,
-
-    time_gaps: pl.DataFrame | None = None,
-    sort: bool = True,
-    period: float = 60,
-
-) -> tuple[
-    pl.DataFrame,  # with dts
-    pl.DataFrame,  # with deduplicated dts (aka gap/repeat removal)
-    int,  # len diff between input and deduped
-]:
-    '''
-    Check for time series gaps and if found
-    de-duplicate any datetime entries, check for
-    a frame height diff and return the newly
-    dt-deduplicated frame.
-
-    '''
-    wdts: pl.DataFrame = with_dts(src_df)
-
-    deduped = wdts
-
-    # remove duplicated datetime samples/sections
-    deduped: pl.DataFrame = wdts.unique(
-        # subset=['dt'],
-        subset=['time'],
-        maintain_order=True,
-    )
-
-    # maybe sort on any time field
-    if sort:
-        deduped = deduped.sort(by='time')
-        # TODO: detect out-of-order segments which were corrected!
-        # -[ ] report in log msg
-        # -[ ] possibly return segment sections which were moved?
-
-    diff: int = (
-        wdts.height
-        -
-        deduped.height
-    )
-    return (
-        wdts,
-        deduped,
-        diff,
-    )
-
-
-def sort_diff(
-    src_df: pl.DataFrame,
-    col: str = 'time',
-
-) -> tuple[
-    pl.DataFrame,  # with dts
-    pl.DataFrame,  # sorted
-    list[int],  # indices of segments that are out-of-order
-]:
-    ser: pl.Series = src_df[col]
-    sortd: pl.DataFrame = ser.sort()
-    diff: pl.Series = ser.diff()
-
-    sortd_diff: pl.Series = sortd.diff()
-    i_step_diff = (diff != sortd_diff).arg_true()
-    frame_reorders: int = i_step_diff.len()
-    if frame_reorders:
-        log.warn(
-            f'Resorted frame on col: {col}\n'
-            f'{frame_reorders}'
-
-        )
-        # import pdbp; pdbp.set_trace()
-
-# NOTE: thanks to this SO answer for the below conversion routines
-# to go from numpy struct-arrays to polars dataframes and back:
-# https://stackoverflow.com/a/72054819
-def np2pl(array: np.ndarray) -> pl.DataFrame:
-    start: float = time.time()
-
-    # XXX: thanks to this SO answer for this conversion tip:
-    # https://stackoverflow.com/a/72054819
-    df = pl.DataFrame({
-        field_name: array[field_name]
-        for field_name in array.dtype.fields
-    })
-    delay: float = round(
-        time.time() - start,
-        ndigits=6,
-    )
-    log.info(
-        f'numpy -> polars conversion took {delay} secs\n'
-        f'polars df: {df}'
-    )
-    return df
-
-
-def pl2np(
-    df: pl.DataFrame,
-    dtype: np.dtype,
-
-) -> np.ndarray:
-
-    # Create numpy struct array of the correct size and dtype
-    # and loop through df columns to fill in array fields.
-    array = np.empty(
-        df.height,
-        dtype,
-    )
-    for field, col in zip(
-        dtype.fields,
-        df.columns,
-    ):
-        array[field] = df.get_column(col).to_numpy()
-
-    return array

piker/types.py

@@ -1,250 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) (in stewardship for pikers)
-#  - Tyler Goodlet
-#  - Guillermo Rodriguez
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-Extensions to built-in or (heavily used but 3rd party) friend-lib
-types.
-
-'''
-from __future__ import annotations
-from collections import UserList
-from pprint import (
-    saferepr,
-)
-from typing import Any
-
-from msgspec import (
-    msgpack,
-    Struct as _Struct,
-    structs,
-)
-
-
-class DiffDump(UserList):
-    '''
-    Very simple list delegator that repr() dumps (presumed) tuple
-    elements of the form `tuple[str, Any, Any]` in a nice
-    multi-line readable form for analyzing `Struct` diffs.
-
-    '''
-    def __repr__(self) -> str:
-        if not len(self):
-            return super().__repr__()
-
-        # format by displaying item pair's ``repr()`` on multiple,
-        # indented lines such that they are more easily visually
-        # comparable when printed to console when printed to
-        # console.
-        repstr: str = '[\n'
-        for k, left, right in self:
-            repstr += (
-                f'({k},\n'
-                f'\t{repr(left)},\n'
-                f'\t{repr(right)},\n'
-                ')\n'
-            )
-        repstr += ']\n'
-        return repstr
-
-
-class Struct(
-    _Struct,
-
-    # https://jcristharif.com/msgspec/structs.html#tagged-unions
-    # tag='pikerstruct',
-    # tag=True,
-):
-    '''
-    A "human friendlier" (aka repl buddy) struct subtype.
-
-    '''
-    def _sin_props(self) -> Iterator[
-        tuple[
-            structs.FieldIinfo,
-            str,
-            Any,
-        ]
-    ]:
-        '''
-        Iterate over all non-@property fields of this struct.
-
-        '''
-        fi: structs.FieldInfo
-        for fi in structs.fields(self):
-            key: str = fi.name
-            val: Any = getattr(self, key)
-            yield fi, key, val
-
-    def to_dict(
-        self,
-        include_non_members: bool = True,
-
-    ) -> dict:
-        '''
-        Like it sounds.. direct delegation to:
-        https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
-
-        BUT, by default we pop all non-member (aka not defined as
-        struct fields) fields by default.
-
-        '''
-        asdict: dict = structs.asdict(self)
-        if include_non_members:
-            return asdict
-
-        # only return a dict of the struct members
-        # which were provided as input, NOT anything
-        # added as type-defined `@property` methods!
-        sin_props: dict = {}
-        fi: structs.FieldInfo
-        for fi, k, v in self._sin_props():
-            sin_props[k] = asdict[k]
-
-        return sin_props
-
-    def pformat(
-        self,
-        field_indent: int = 2,
-        indent: int = 0,
-
-    ) -> str:
-        '''
-        Recursion-safe `pprint.pformat()` style formatting of
-        a `msgspec.Struct` for sane reading by a human using a REPL.
-
-        '''
-        # global whitespace indent
-        ws: str = ' '*indent
-
-        # field whitespace indent
-        field_ws: str = ' '*(field_indent + indent)
-
-        # qtn: str = ws + self.__class__.__qualname__
-        qtn: str = self.__class__.__qualname__
-
-        obj_str: str = ''  # accumulator
-        fi: structs.FieldInfo
-        k: str
-        v: Any
-        for fi, k, v in self._sin_props():
-
-            # TODO: how can we prefer `Literal['option1',  'option2,
-            # ..]` over .__name__ == `Literal` but still get only the
-            # latter for simple types like `str | int | None` etc..?
-            ft: type = fi.type
-            typ_name: str = getattr(ft, '__name__', str(ft))
-
-            # recurse to get sub-struct's `.pformat()` output Bo
-            if isinstance(v, Struct):
-                val_str: str =  v.pformat(
-                    indent=field_indent + indent,
-                    field_indent=indent + field_indent,
-                )
-
-            else:  # the `pprint` recursion-safe format:
-                # https://docs.python.org/3.11/library/pprint.html#pprint.saferepr
-                val_str: str = saferepr(v)
-
-            obj_str += (field_ws + f'{k}: {typ_name} = {val_str},\n')
-
-        return (
-            f'{qtn}(\n'
-            f'{obj_str}'
-            f'{ws})'
-        )
-
-    # TODO: use a pprint.PrettyPrinter instance around ONLY rendering
-    # inside a known tty?
-    # def __repr__(self) -> str:
-    #     ...
-
-    # __str__ = __repr__ = pformat
-    __repr__ = pformat
-
-    def copy(
-        self,
-        update: dict | None = None,
-
-    ) -> Struct:
-        '''
-        Validate-typecast all self defined fields, return a copy of
-        us with all such fields.
-
-        NOTE: This is kinda like the default behaviour in
-        `pydantic.BaseModel` except a copy of the object is
-        returned making it compat with `frozen=True`.
-
-        '''
-        if update:
-            for k, v in update.items():
-                setattr(self, k, v)
-
-        # NOTE: roundtrip serialize to validate
-        # - enode to msgpack binary format,
-        # - decode that back to a struct.
-        return msgpack.Decoder(type=type(self)).decode(
-            msgpack.Encoder().encode(self)
-        )
-
-    def typecast(
-        self,
-
-        # TODO: allow only casting a named subset?
-        # fields: set[str] | None = None,
-
-    ) -> None:
-        '''
-        Cast all fields using their declared type annotations
-        (kinda like what `pydantic` does by default).
-
-        NOTE: this of course won't work on frozen types, use
-        ``.copy()`` above in such cases.
-
-        '''
-        # https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
-        fi: structs.FieldInfo
-        for fi in structs.fields(self):
-            setattr(
-                self,
-                fi.name,
-                fi.type(getattr(self, fi.name)),
-            )
-
-    def __sub__(
-        self,
-        other: Struct,
-
-    ) -> DiffDump[tuple[str, Any, Any]]:
-        '''
-        Compare fields/items key-wise and return a ``DiffDump``
-        for easy visual REPL comparison B)
-
-        '''
-        diffs: DiffDump[tuple[str, Any, Any]] = DiffDump()
-        for fi in structs.fields(self):
-            attr_name: str = fi.name
-            ours: Any = getattr(self, attr_name)
-            theirs: Any = getattr(other, attr_name)
-            if ours != theirs:
-                diffs.append((
-                    attr_name,
-                    ours,
-                    theirs,
-                ))
-
-        return diffs

piker/ui/__init__.py

@@ -14,8 +14,9 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-'''
-UI components built using `Qt` with major versions swapped in via
-the import indirection in the `.qt` sub-mod.
+"""
+Stuff for your eyes, aka super hawt Qt UI components.
 
-'''
+Currently we only support PyQt5 due to this issue in Pyside2:
+https://bugreports.qt.io/projects/PYSIDE/issues/PYSIDE-1313
+"""

piker/ui/_anchors.py

@@ -21,10 +21,8 @@ Anchor funtions for UI placement of annotions.
 from __future__ import annotations
 from typing import Callable, TYPE_CHECKING
 
-from piker.ui.qt import (
-    QPointF,
-    QGraphicsPathItem,
-)
+from PyQt5.QtCore import QPointF
+from PyQt5.QtWidgets import QGraphicsPathItem
 
 if TYPE_CHECKING:
     from ._chart import ChartPlotWidget

piker/ui/_annotate.py

@@ -20,22 +20,12 @@ Annotations for ur faces.
 """
 from typing import Callable
 
-from pyqtgraph import (
-    Point,
-    functions as fn,
-    Color,
-)
+from PyQt5 import QtCore, QtGui, QtWidgets
+from PyQt5.QtCore import QPointF, QRectF
+from PyQt5.QtWidgets import QGraphicsPathItem
+from pyqtgraph import Point, functions as fn, Color
 import numpy as np
 
-from piker.ui.qt import (
-    QtCore,
-    QtGui,
-    QtWidgets,
-    QPointF,
-    QRectF,
-    QGraphicsPathItem,
-)
-
 
 def mk_marker_path(