Added note to exception when missing field in SpotPair class

README.rst

@@ -1,199 +1,162 @@
 piker
 -----
-trading gear for hackers
+trading gear for hackers.
 
 |gh_actions|
 
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
     :target: https://actions-badge.atrox.dev/piker/pikers/goto
 
-``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
-real-time computational trading targeted at `hardcore Linux users
-<comp_trader>`_ .
+``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
+computational trading targeted at `hardcore Linux users <comp_trader>`_ .
 
-we use much bleeding edge tech including (but not limited to):
+we use as much bleeding edge tech as possible including (but not limited to):
 
 - latest python for glue_
-- uv_ for packaging and distribution
-- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
-- Qt_ for pristine low latency UIs
-- pyqtgraph_ (which we've extended) for real-time charting and graphics
-- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
-- `apache arrow and parquet`_ for time-series storage
+- trio_ & tractor_ for our distributed, multi-core, real-time streaming
+  `structured concurrency`_ runtime B)
+- Qt_ for pristine high performance UIs
+- pyqtgraph_ for real-time charting
+- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
+- `apache arrow and parquet`_ for time series history management
+  persistence and sharing
+- (prototyped) techtonicdb_ for L2 book storage
 
-potential projects we might integrate with soon,
-
-- (already prototyped in ) techtonicdb_ for L2 book storage
-
-.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
-.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
-.. _uv: https://docs.astral.sh/uv/
+.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
+    :target: https://travis-ci.org/pikers/piker
 .. _trio: https://github.com/python-trio/trio
 .. _tractor: https://github.com/goodboy/tractor
 .. _structured concurrency: https://trio.discourse.group/
+.. _marketstore: https://github.com/alpacahq/marketstore
+.. _techtonicdb: https://github.com/0b01/tectonicdb
 .. _Qt: https://www.qt.io/
 .. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
+.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _apache arrow and parquet: https://arrow.apache.org/faq/
 .. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
-.. _techtonicdb: https://github.com/0b01/tectonicdb
+.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
 
 
-focus and feats:
-****************
-fitting with these tenets, we're always open to new
-framework/lib/service interop suggestions and ideas!
+focus and features:
+*******************
+- 100% federated: your code, your hardware, your data feeds, your broker fills.
+- zero web: low latency, native software that doesn't try to re-invent the OS
+- maximal **privacy**: prevent brokers and mms from knowing your
+  planz; smack their spreads with dark volume.
+- zero clutter: modal, context oriented UIs that echew minimalism, reduce
+  thought noise and encourage un-emotion.
+- first class parallelism: built from the ground up on next-gen structured concurrency
+  primitives.
+- traders first: broker/exchange/asset-class agnostic
+- systems grounded: real-time financial signal processing that will
+  make any queuing or DSP eng juice their shorts.
+- non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
+- data collaboration: every process and protocol is multi-host scalable.
+- fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
 
-- **100% federated**:
-  your code, your hardware, your data feeds, your broker fills.
+fitting with these tenets, we're always open to new framework suggestions and ideas.
 
-- **zero web**:
-  low latency as a prime objective, native UIs and modern IPC
-  protocols without trying to re-invent the "OS-as-an-app"..
-
-- **maximal privacy**:
-  prevent brokers and mms from knowing your planz; smack their
-  spreads with dark volume from a VPN tunnel.
-
-- **zero clutter**:
-  modal, context oriented UIs that echew minimalism, reduce thought
-  noise and encourage un-emotion.
-
-- **first class parallelism**:
-  built from the ground up on a next-gen structured concurrency
-  supervision sys.
-
-- **traders first**:
-  broker/exchange/venue/asset-class/money-sys agnostic
-
-- **systems grounded**:
-  real-time financial signal processing (fsp) that will make any
-  queuing or DSP eng juice their shorts.
-
-- **non-tina UX**:
-  sleek, powerful keyboard driven interaction with expected use in
-  tiling wms (or maybe even a DDE).
-
-- **data collab at scale**:
-  every actor-process and protocol is multi-host aware.
-
-- **fight club ready**:
-  zero interest in adoption by suits; no corporate friendly license,
-  ever.
-
-building the hottest looking, fastest, most reliable, keyboard
-friendly FOSS trading platform is the dream; join the cause.
+building the best looking, most reliable, keyboard friendly trading
+platform is the dream; join the cause.
 
 
-a sane install with `uv`
-************************
-bc why install with `python` when you can faster with `rust` ::
-
-    uv sync
-
-    # ^ astral's docs,
-    # https://docs.astral.sh/uv/concepts/projects/sync/
-
-include all GUIs (ex. for charting)::
-
-    uv sync --group uis
-
-AND with **all** our normal hacking tools::
-
-    uv sync --dev
-
-AND if you want to try WIP integrations::
-
-    uv sync --all-groups
-
-Ensure you can run the root-daemon::
-
-    uv run pikerd [-l info --pdb]
+sane install with `poetry`
+**************************
+TODO!
 
 
-install on nix(os)
-******************
-``NixOS`` is our core devs' distro of choice for which we offer
-a stringently defined development shell envoirment that can currently
-be applied in one of 2 ways::
-
-    # ONLY if running on X11
-    nix-shell default.nix
-
-Or if you prefer flakes style and a modern DE::
-
-    # ONLY if also running on Wayland
-    nix develop  # for default bash
-    nix develop -c uv run xonsh  # for @goodboy's preferred sh B)
+rigorous install on ``nixos`` using ``poetry2nix``
+**************************************************
+TODO!
 
 
-start a chart
-*************
-run a realtime OHLCV chart stand-alone::
+hacky install on nixos
+**********************
+`NixOS` is our core devs' distro of choice for which we offer
+a stringently defined development shell envoirment that can be loaded with::
 
-    [uv run] piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
+    nix-shell develop.nix
 
-    # ^^^ iff you haven't activated the py-env,
-    # - https://docs.astral.sh/uv/concepts/projects/run/
-    #
-    # in order to create an explicit virt-env see,
-    # - https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment
-    # - https://docs.astral.sh/uv/pip/environments/
-    #
-    # use $UV_PROJECT_ENVIRONMENT to select any non-`.venv/`
-    # as the venv sudir in the repo's root.
-    # - https://docs.astral.sh/uv/reference/environment/#uv_project_environment
+this will setup the required python environment to run piker, make sure to
+run::
 
-this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
-overlayed on the same graph. Use of `piker` without first starting
-a daemon (`pikerd` - see below) means there is an implicit spawning of the
-multi-actor-runtime (implemented as a `tractor` app).
+    pip install -r requirements.txt -e .
 
-For additional subsystem feats available through our chart UI see the
-various sub-readmes:
-
-- order control using a mouse-n-keyboard UX B)
-- cross venue market-pair (what most call "symbol") search, select, overlay Bo
-- financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
-- src-asset derivatives scan for anal, like the infamous "max pain" XO
+once after loading the shell
 
 
-spawn a daemon standalone
-*************************
-we call the root actor-process the ``pikerd``. it can be (and is
-recommended normally to be) started separately from the ``piker
-chart`` program::
+install wild-west style via `pip`
+*********************************
+``piker`` is currently under heavy pre-alpha development and as such
+should be cloned from this repo and hacked on directly.
+
+for a development install::
+
+    git clone git@github.com:pikers/piker.git
+    cd piker
+    virtualenv env
+    source ./env/bin/activate
+    pip install -r requirements.txt -e .
+
+
+check out our charts
+********************
+bet you weren't expecting this from the foss::
+
+    piker -l info -b kraken -b binance chart btcusdt.binance --pdb
+
+
+this runs the main chart (currently with 1m sampled OHLC) in in debug
+mode and you can practice paper trading using the following
+micro-manual:
+
+``order_mode`` (
+    edge triggered activation by any of the following keys,
+    ``mouse-click`` on y-level to submit at that price
+    ):
+
+    - ``f``/ ``ctl-f`` to stage buy
+    - ``d``/ ``ctl-d`` to stage sell
+    - ``a`` to stage alert
+
+
+``search_mode`` (
+    ``ctl-l`` or ``ctl-space`` to open,
+    ``ctl-c`` or ``ctl-space`` to close
+    ) :
+
+    - begin typing to have symbol search automatically lookup
+      symbols from all loaded backend (broker) providers
+    - arrow keys and mouse click to navigate selection
+    - vi-like ``ctl-[hjkl]`` for navigation
+
+
+you can also configure your position allocation limits from the
+sidepane.
+
+
+run in distributed mode
+***********************
+start the service manager and data feed daemon in the background and
+connect to it::
 
     pikerd -l info --pdb
 
-the daemon does nothing until a ``piker``-client (like ``piker
-chart``) connects and requests some particular sub-system. for
-a connecting chart ``pikerd`` will spawn and manage at least,
 
-- a data-feed daemon: ``datad`` which does all the work of comms with
-  the backend provider (in this case the ``binance`` cex).
-- a paper-trading engine instance, ``paperboi.binance``, (if no live
-  account has been configured) which allows for auto/manual order
-  control against the live quote stream.
+connect your chart::
 
-*using* an actor-service (aka micro-daemon) manager which dynamically
-supervises various sub-subsystems-as-services throughout the ``piker``
-runtime-stack.
+    piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
 
-now you can (implicitly) connect your chart::
 
-    piker chart btcusdt.spot.binance
-
-since ``pikerd`` was started separately you can now enjoy a persistent
-real-time data stream tied to the daemon-tree's lifetime. i.e. the next
-time you spawn a chart it will obviously not only load much faster
-(since the underlying ``datad.binance`` is left running with its
-in-memory IPC data structures) but also the data-feed and any order
-mgmt states should be persistent until you finally cancel ``pikerd``.
+enjoy persistent real-time data feeds tied to daemon lifetime. the next
+time you spawn a chart it will load much faster since the data feed has
+been cached and is now always running live in the background until you
+kill ``pikerd``.
 
 
 if anyone asks you what this project is about
 *********************************************
-you don't talk about it; just use it.
+you don't talk about it.
 
 
 how do i get involved?
@@ -203,15 +166,6 @@ enter the matrix.
 
 how come there ain't that many docs
 ***********************************
-i mean we want/need them but building the core right has been higher
-prio then marketting (and likely will stay that way Bp).
-
-soo, suck it up bc,
-
-- no one is trying to sell you on anything
-- learning the code base is prolly way more valuable
-- the UI/UXs are intended to be "intuitive" for any hacker..
-
-we obviously need tonz help so if you want to start somewhere and
-can't necessarily write "advanced" concurrent python/rust code, this
-helping document literally anything might be the place for you!
+suck it up, learn the code; no one is trying to sell you on anything.
+also, we need lotsa help so if you want to start somewhere and can't
+necessarily write serious code, this might be the place for you!

config/brokers.toml

@@ -1,5 +1,6 @@
+################
 # ---- CEXY ----
-
+################
 [binance]
 accounts.paper = 'paper'
 
@@ -12,41 +13,28 @@ accounts.spot = 'spot'
 spot.use_testnet = false
 spot.api_key = ''
 spot.api_secret = ''
-# ------ binance ------
 
 
 [deribit]
-# std assets
 key_id = ''
 key_secret = ''
-# options
-accounts.option = 'option'
-option.use_testnet = false
-option.key_id = ''
-option.key_secret = ''
-# aux logging from `cryptofeed`
-option.log.filename = 'cryptofeed.log'
-option.log.level = 'DEBUG'
-option.log.disabled = true
-# ------ deribit ------
 
 
 [kraken]
 key_descr = ''
 api_key = ''
 secret = ''
-# ------ kraken ------
 
 
 [kucoin]
 key_id = ''
 key_secret = ''
 key_passphrase = ''
-# ------ kucoin ------
 
 
+################
 # -- BROKERZ ---
-
+################
 [questrade]
 refresh_token = ''
 access_token = ''
@@ -54,55 +42,44 @@ api_server = 'https://api06.iq.questrade.com/'
 expires_in = 1800
 token_type = 'Bearer'
 expires_at = 1616095326.355846
-# ------ questrade ------
 
 
 [ib]
-# define the (set of) host-port socketaddrs that
-# brokerd.ib will scan to connect to an API endpoint
-# (ib-gw or ib-tws listening instances)
 hosts = [
     '127.0.0.1',
 ]
+# XXX: the order in which ports will be scanned
+# (by the `brokerd` daemon-actor)
+# is determined # by the line order here.
+# TODO: when we eventually spawn gateways in our
+# container, we can just dynamically allocate these
+# using IBC.
 ports = [
     4002,  # gw
     7497,  # tws
 ]
 
-# When API endpoints are being scanned durin startup, the order
-# of user-defined-account "names" (as defined below) here
-# determines which py-client connection is given priority to be
-# used for data-feed-requests by according to whichever client
-# connected to an API endpoing which reported the equivalent
-# account number for that name.
+# XXX: for a paper account the flex web query service
+# is not supported so you have to manually download
+# and XML report and put it in a location that can be
+# accessed by the ``brokerd.ib`` backend code for parsing.
+flex_token = ''
+flex_trades_query_id = ''  # live account
+
+# when clients are being scanned this determines
+# which clients are preferred to be used for data
+# feeds based on the order of account names, if
+# detected as active on an API client.
 prefer_data_account = [
     'paper',
     'margin',
     'ira',
 ]
 
-# For long-term trades txn (transaction) history
-# processing (i.e your txn ledger with IB) you can
-# (automatically for live accounts) query the FLEX
-# report system for past history.
-#
-# (For paper accounts the web query service
-# is not supported so you have to manually download
-# an XML report and put it in a location that can be
-# accessed by our `brokerd.ib` backend code for parsing).
-#
-flex_token = ''
-flex_trades_query_id = ''  # live account
-
-# define "aliases" (names) for each account number
-# such that the names can be reffed and logged throughout
-# `piker.accounting` subsys and more easily
-# referred to  by the user.
-#
-# These keys will be the set exposed through the order-mode
-# account-selection UI so that numbers are never shown.
 [ib.accounts]
-paper = 'DU0000000'  # <- literal account #
-margin = 'U0000000'
-ira = 'U0000000'
-# ------ ib ------
+# the order in which accounts will be selectable
+# in the order mode UI (if found via clients during
+# API-app scanning)when a new symbol is loaded.
+paper = 'XX0000000'
+margin = 'X0000000'
+ira = 'X0000000'

config/conf.toml

@@ -1,9 +1,7 @@
 [network]
-pikerd = [
-  '/ipv4/127.0.0.1/tcp/6116',  # std localhost daemon-actor tree
-  # '/uds/6116',  # TODO std uds socket file
-]
-
+tsdb.backend = 'marketstore'
+tsdb.host = 'localhost'
+tsdb.grpc_port = 5995
 
 [ui]
 # set custom font + size which will scale entire UI

default.nix

@@ -1,135 +1,74 @@
 with (import <nixpkgs> {});
+with python311Packages;
 let
-  glibStorePath = lib.getLib glib;
-  zlibStorePath = lib.getLib zlib;
-  zstdStorePath = lib.getLib zstd;
-  dbusStorePath = lib.getLib dbus;
-  libGLStorePath = lib.getLib libGL;
-  freetypeStorePath = lib.getLib freetype;
-  qt6baseStorePath = lib.getLib qt6.qtbase;
-  fontconfigStorePath = lib.getLib fontconfig;
-  libxkbcommonStorePath = lib.getLib libxkbcommon;
-  xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
-
-  pypkgs = python313Packages;
-  qtpyStorePath = lib.getLib pypkgs.qtpy;
-  pyqt6StorePath = lib.getLib pypkgs.pyqt6;
-  pyqt6SipStorePath = lib.getLib pypkgs.pyqt6-sip;
-  rapidfuzzStorePath = lib.getLib pypkgs.rapidfuzz;
-  qdarkstyleStorePath = lib.getLib pypkgs.qdarkstyle;
-
-  xorgLibX11StorePath = lib.getLib xorg.libX11;
-  xorgLibxcbStorePath = lib.getLib xorg.libxcb;
-  xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
-  xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
-  xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
-  xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
-  xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
+  rapidfuzzStorePath = lib.getLib rapidfuzz;
+  qdarkstyleStorePath = lib.getLib qdarkstyle;
+  qtpyStorePath = lib.getLib qtpy;
+  pyqt5StorePath = lib.getLib pyqt5;
+  pyqt5SipStorePath = lib.getLib pyqt5_sip;
 in
 stdenv.mkDerivation {
-  name = "piker-qt6-uv";
+  name = "piker-poetry-shell-with-qt-fix";
   buildInputs = [
     # System requirements.
-    glib
-    zlib
-    dbus
-    zstd
-    libGL
-    freetype
-    qt6.qtbase
-    libgcc.lib
-    fontconfig
-    libxkbcommon
-
-    # Xorg requirements
-    xcb-util-cursor
-    xorg.libxcb
-    xorg.libX11
-    xorg.xcbutilwm
-    xorg.xcbutilimage
-    xorg.xcbutilerrors
-    xorg.xcbutilkeysyms
-    xorg.xcbutilrenderutil
+    libsForQt5.qt5.qtbase
 
     # Python requirements.
-    python313
-    uv
-    pypkgs.qdarkstyle
-    pypkgs.rapidfuzz
-    pypkgs.pyqt6
-    pypkgs.qtpy
+    python311Full
+    poetry-core
+    rapidfuzz
+    qdarkstyle
+    qtpy
+    pyqt5
   ];
   src = null;
   shellHook = ''
     set -e
 
+    QTBASE_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.version}"
+
     # Set the Qt plugin path
     # export QT_DEBUG_PLUGINS=1
+    export QT_PLUGIN_PATH="$QTBASE_PATH/plugins"
+    export QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
 
-    QTBASE_PATH="${qt6baseStorePath}/lib"
-    QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
-    QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
+    # Maybe create venv & install deps
+    poetry install --with=nix-shell
 
-    LIB_GCC_PATH="${libgcc.lib}/lib"
-    GLIB_PATH="${glibStorePath}/lib"
-    ZSTD_PATH="${zstdStorePath}/lib"
-    ZLIB_PATH="${zlibStorePath}/lib"
-    DBUS_PATH="${dbusStorePath}/lib"
-    LIBGL_PATH="${libGLStorePath}/lib"
-    FREETYPE_PATH="${freetypeStorePath}/lib"
-    FONTCONFIG_PATH="${fontconfigStorePath}/lib"
-    LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
+    # Use pyqt5 from System, patch activate script
+    ACTIVATE_SCRIPT_PATH="$(poetry env info --path)/bin/activate"
+    export RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.11/site-packages"
+    export QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.11/site-packages"
+    export QTPY_PATH="${qtpyStorePath}/lib/python3.11/site-packages"
+    export PYQT5_PATH="${pyqt5StorePath}/lib/python3.11/site-packages"
+    export PYQT5_SIP_PATH="${pyqt5SipStorePath}/lib/python3.11/site-packages"
+    echo "rapidfuzz at:   $RPDFUZZ_PATH"
+    echo "qdarkstyle at:  $QDRKSTYLE_PATH"
+    echo "qtpy at:        $QTPY_PATH"
+    echo "pyqt5 at:       $PYQT5_PATH"
+    echo "pyqt5-sip at:   $PYQT5_SIP_PATH"
+    echo ""
 
-    XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
-    XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
-    XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
-    XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
-    XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
-    XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
-    XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
-    XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
+    PATCH="export PYTHONPATH=\""
 
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
+    PATCH="$PATCH\$RPDFUZZ_PATH"
+    PATCH="$PATCH:\$QDRKSTYLE_PATH"
+    PATCH="$PATCH:\$QTPY_PATH"
+    PATCH="$PATCH:\$PYQT5_PATH"
+    PATCH="$PATCH:\$PYQT5_SIP_PATH"
 
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
+    PATCH="$PATCH\""
 
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
+    if grep -q "$PATCH" "$ACTIVATE_SCRIPT_PATH"; then
+        echo "venv is already patched."
+    else
+        echo "patching $ACTIVATE_SCRIPT_PATH to use pyqt5 from nixos..."
+        sed -i "\$i$PATCH" $ACTIVATE_SCRIPT_PATH
+    fi
 
-    export LD_LIBRARY_PATH
-
-    RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.13/site-packages"
-    QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.13/site-packages"
-    QTPY_PATH="${qtpyStorePath}/lib/python3.13/site-packages"
-    PYQT6_PATH="${pyqt6StorePath}/lib/python3.13/site-packages"
-    PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.13/site-packages"
-
-    PATCH="$PATCH:$RPDFUZZ_PATH"
-    PATCH="$PATCH:$QDRKSTYLE_PATH"
-    PATCH="$PATCH:$QTPY_PATH"
-    PATCH="$PATCH:$PYQT6_PATH"
-    PATCH="$PATCH:$PYQT6_SIP_PATH"
-
-    export PATCH
-
-    # install all dev and extras
-    uv sync --dev --all-extras
+    echo "qt plguin path: $QT_PLUGIN_PATH"
+    echo ""
 
+    poetry shell
   '';
 }

develop.nix

@@ -1,47 +0,0 @@
-with (import <nixpkgs> {});
-
-stdenv.mkDerivation {
-  name = "poetry-env";
-  buildInputs = [
-    # System requirements.
-    readline
-
-    # TODO: hacky non-poetry install stuff we need to get rid of!!
-    poetry
-    # virtualenv
-    # setuptools
-    # pip
-
-    # Python requirements (enough to get a virtualenv going).
-    python311Full
-
-    # obviously, and see below for hacked linking
-    python311Packages.pyqt5
-    python311Packages.pyqt5_sip
-    # python311Packages.qtpy
-
-    # numerics deps
-    python311Packages.levenshtein
-    python311Packages.fastparquet
-    python311Packages.polars
-
-  ];
-  # environment.sessionVariables = {
-  #   LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
-  # };
-  src = null;
-  shellHook = ''
-    # Allow the use of wheels.
-    SOURCE_DATE_EPOCH=$(date +%s)
-
-    # Augment the dynamic linker path
-    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
-    export QT_QPA_PLATFORM_PLUGIN_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.version}/plugins";
-
-    if [ ! -d ".venv" ]; then
-        poetry install --with uis
-    fi
-
-    poetry shell
-  '';
-}

dockering/ib/README.rst

@@ -1,138 +1,30 @@
 running ``ib`` gateway in ``docker``
 ------------------------------------
-We have a config based on a well maintained community
-image from `@gnzsnz`:
+We have a config based on the (now defunct)
+image from "waytrade":
 
-https://github.com/gnzsnz/ib-gateway-docker
+https://github.com/waytrade/ib-gateway-docker
 
-
-To startup this image simply run the command::
+To startup this image with our custom settings
+simply run the command::
 
     docker compose up
 
-(For further usage^ see the official `docker-compose`_ docs)
+And you should have the following socket-available services:
 
+- ``x11vnc1@127.0.0.1:3003``
+- ``ib-gw@127.0.0.1:4002``
 
-And you should have the following socket-available services by
-default:
+You can attach to the container via a VNC client
+without password auth.
 
-- ``x11vnc1 @ 127.0.0.1:5900``
-- ``ib-gw @ 127.0.0.1:4002``
-
-You can now attach to the container via a VNC client with password-auth;
-here is an example using ``vncclient`` on ``linux``::
-
-    vncviewer localhost:5900
-
-now enter the pw (password) you set via an (see second code blob)
-`.env file`_ or pw-file according to the `credentials section`_.
-
-If you want to change away from their default config see the example
-`docker-compose.yml`-config issue and config-section of the readme,
-
-  - https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
-  - https://github.com/gnzsnz/ib-gateway-docker/discussions/103
-
-.. _.env file: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#how-to-use-it
-.. _docker-compose: https://docs.docker.com/compose/
-.. _credentials section: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#credentials
-
-
-Connecting to the API from `piker`
----------------------------------
-In order to expose the container's API endpoint to the
-`brokerd/datad/ib` actor, we need to add a section to the user's
-`brokers.toml` config (note the below is similar to the repo-shipped
-template file),
-
-.. code:: toml
-
-    [ib]
-    # define the (set of) host-port socketaddrs that
-    # brokerd.ib will scan to connect to an API endpoint
-    # (ib-gw or ib-tws listening instances)
-    hosts = [
-        '127.0.0.1',
-    ]
-    ports = [
-        4002,  # gw
-        7497,  # tws
-    ]
-
-    # When API endpoints are being scanned durin startup, the order
-    # of user-defined-account "names" (as defined below) here
-    # determines which py-client connection is given priority to be
-    # used for data-feed-requests by according to whichever client
-    # connected to an API endpoing which reported the equivalent
-    # account number for that name.
-    prefer_data_account = [
-        'paper',
-        'margin',
-        'ira',
-    ]
-
-    # define "aliases" (names) for each account number
-    # such that the names can be reffed and logged throughout
-    # `piker.accounting` subsys and more easily
-    # referred to  by the user.
-    #
-    # These keys will be the set exposed through the order-mode
-    # account-selection UI so that numbers are never shown.
-    [ib.accounts]
-    paper = 'XX0000000'
-    margin = 'X0000000'
-    ira = 'X0000000'
-
-
-the broker daemon can also connect to the container's VNC server for
-added functionalies including,
-
-- viewing the API endpoint program's GUI for manual interventions,
-- workarounds for historical data throttling using hotkey hacks,
-
-Add a further section to `brokers.toml` which maps each API-ep's
-port to a table of VNC server connection info like,
-
-.. code:: toml
-
-    [ib.vnc_addrs]
-    4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
-
-The `pw = 'doggy'` here ^ should the same value as the particular
-container instances `.env` file setting (when it was run),
-
-.. code:: ini
-
-    VNC_SERVER_PASSWORD='doggy'
-
-
-IF you also want to run ``TWS``
--------------------------------
-You can also run it containerized,
-
-https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#using-tws
-
-
-SECURITY stuff (advanced, only if you're paranoid)
---------------------------------------------------
-First and foremost if doing a "distributed" container setup where you
-run the ``ib-gw`` docker container and your connecting API client
-(likely ``ib_async`` from python) on **different hosts** be sure to
-read the `security considerations`_ section!
-
-And for a further (somewhat paranoid) perspective from
-a long-time-ago serious devops eng..
-
-Though "``ib``" claims they filter remote host connections outside
-``localhost`` (aka ``127.0.0.1`` on ipv4) it's prolly justified if
-you'd like to filter the socket at the *OS level* using a stateless
-firewall rule::
+SECURITY STUFF!?!?!
+-------------------
+Though "``ib``" claims they host filter connections outside
+localhost (aka ``127.0.0.1``) it's probably better if you filter
+the socket at the OS level using a stateless firewall rule::
 
     ip rule add not unicast iif lo to 0.0.0.0/0 dport 4002
 
-
-We will soon have this either baked into our own custom derivative
-image (or patched into the current upstream one after further testin)
-but for now you'll have to do it urself, diggity dawg.
-
-.. _security considerations: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#security-considerations
+We will soon have this baked into our own custom image but for
+now you'll have to do it urself dawgy.

dockering/ib/docker-compose.yml

@@ -1,15 +1,10 @@
-# a community maintained IB API container!
-#
-# https://github.com/gnzsnz/ib-gateway-docker
-#
-# For piker we (currently) include some minor deviations
-# for some config files in the `volumes` section.
-#
-# See full configuration settings @
-# - https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
-# - https://github.com/gnzsnz/ib-gateway-docker/discussions/103
+# rework from the original @
+# https://github.com/waytrade/ib-gateway-docker/blob/master/docker-compose.yml
+version: "3.5"
+
 
 services:
+
   ib_gw_paper:
 
     # apparently java is a mega cukc:
@@ -55,22 +50,16 @@ services:
         target: /root/scripts/run_x11_vnc.sh
         read_only: true
 
-    # NOTE: an alt method to fill these out is to
-    # define an `.env` file in the same dir as
-    # this compose file.
+    # NOTE:to fill these out, define an `.env` file in the same dir as
+    # this compose file which looks something like:
+    # TWS_USERID='myuser'
+    # TWS_PASSWORD='guest'
     environment:
       TWS_USERID: ${TWS_USERID}
-      # TWS_USERID: 'myuser'
       TWS_PASSWORD: ${TWS_PASSWORD}
-      # TWS_PASSWORD: 'guest'
-      TRADING_MODE: ${TRADING_MODE}
-      # TRADING_MODE: 'paper'
-      VNC_SERVER_PASSWORD: ${VNC_SERVER_PASSWORD}
-      # VNC_SERVER_PASSWORD: 'doggy'
-
-      # TODO, see if we can get this supported like it
-      # was on the old `waytrade` image?
-      # VNC_SERVER_PORT: '3003'
+      TRADING_MODE: 'paper'
+      VNC_SERVER_PASSWORD: 'doggy'
+      VNC_SERVER_PORT: '3003'
 
     # ports:
     #   - target: 4002
@@ -87,9 +76,6 @@ services:
       # - "127.0.0.1:4002:4002"
       # - "127.0.0.1:5900:5900"
 
-  # TODO, a masked but working example of dual paper + live
-  # ib-gw instances running in a single app run!
-  #
   # ib_gw_live:
   #   image: waytrade/ib-gateway:1012.2i
   #   restart: no

examples/basic_order_bot.py

@@ -121,7 +121,6 @@ async def bot_main():
             # tick_throttle=10,
         ) as feed,
 
-        tractor.trionics.collapse_eg(),
         trio.open_nursery() as tn,
     ):
         assert accounts

flake.lock

@@ -1,24 +1,135 @@
 {
   "nodes": {
-    "nixpkgs": {
+    "flake-utils": {
+      "inputs": {
+        "systems": "systems"
+      },
       "locked": {
-        "lastModified": 1765779637,
-        "narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
-        "owner": "nixos",
-        "repo": "nixpkgs",
-        "rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
+        "lastModified": 1689068808,
+        "narHash": "sha256-6ixXo3wt24N/melDWjq70UuHQLxGV8jZvooRanIHXw0=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "919d646de7be200f3bf08cb76ae1f09402b6f9b4",
         "type": "github"
       },
       "original": {
-        "owner": "nixos",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "type": "github"
+      }
+    },
+    "flake-utils_2": {
+      "inputs": {
+        "systems": "systems_2"
+      },
+      "locked": {
+        "lastModified": 1689068808,
+        "narHash": "sha256-6ixXo3wt24N/melDWjq70UuHQLxGV8jZvooRanIHXw0=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "919d646de7be200f3bf08cb76ae1f09402b6f9b4",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "type": "github"
+      }
+    },
+    "nix-github-actions": {
+      "inputs": {
+        "nixpkgs": [
+          "poetry2nix",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1688870561,
+        "narHash": "sha256-4UYkifnPEw1nAzqqPOTL2MvWtm3sNGw1UTYTalkTcGY=",
+        "owner": "nix-community",
+        "repo": "nix-github-actions",
+        "rev": "165b1650b753316aa7f1787f3005a8d2da0f5301",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nix-github-actions",
+        "type": "github"
+      }
+    },
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1692174805,
+        "narHash": "sha256-xmNPFDi/AUMIxwgOH/IVom55Dks34u1g7sFKKebxUm0=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "caac0eb6bdcad0b32cb2522e03e4002c8975c62e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
         "ref": "nixos-unstable",
         "repo": "nixpkgs",
         "type": "github"
       }
     },
+    "poetry2nix": {
+      "inputs": {
+        "flake-utils": "flake-utils_2",
+        "nix-github-actions": "nix-github-actions",
+        "nixpkgs": [
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1692048894,
+        "narHash": "sha256-cDw03rso2V4CDc3Mll0cHN+ztzysAvdI8pJ7ybbz714=",
+        "ref": "refs/heads/pyqt6",
+        "rev": "b059ad4c3051f45d6c912e17747aae37a9ec1544",
+        "revCount": 2276,
+        "type": "git",
+        "url": "file:///home/lord_fomo/repos/poetry2nix"
+      },
+      "original": {
+        "type": "git",
+        "url": "file:///home/lord_fomo/repos/poetry2nix"
+      }
+    },
     "root": {
       "inputs": {
-        "nixpkgs": "nixpkgs"
+        "flake-utils": "flake-utils",
+        "nixpkgs": "nixpkgs",
+        "poetry2nix": "poetry2nix"
+      }
+    },
+    "systems": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
+    "systems_2": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
       }
     }
   },

flake.nix

@@ -1,103 +1,180 @@
-# An "impure" template thx to `pyproject.nix`,
-# https://pyproject-nix.github.io/pyproject.nix/templates.html#impure
-# https://github.com/pyproject-nix/pyproject.nix/blob/master/templates/impure/flake.nix
-{
-  description = "An impure `piker` overlay using `uv` with Nix(OS)";
+# NOTE: to convert to a poetry2nix env like this here are the
+# steps:
+# - install poetry in your system nix config
+# - convert the repo to use poetry using `poetry init`:
+#   https://python-poetry.org/docs/basic-usage/#initialising-a-pre-existing-project
+# - then manually ensuring all deps are converted over:
+# - add this file to the repo and commit it
+# - 
 
-  inputs = {
-    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
+# GROKin tips:
+# - CLI eps are (ostensibly) added via an `entry_points.txt`:
+#   - https://packaging.python.org/en/latest/specifications/entry-points/#file-format
+#   - https://github.com/nix-community/poetry2nix/blob/master/editable.nix#L49
+{
+  description = "piker: trading gear for hackers (pkged with poetry2nix)";
+
+  inputs.flake-utils.url = "github:numtide/flake-utils";
+  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+
+  # see https://github.com/nix-community/poetry2nix/tree/master#api
+  inputs.poetry2nix = {
+    # url = "github:nix-community/poetry2nix";
+    # url = "github:K900/poetry2nix/qt5-explicit-deps";
+    url = "/home/lord_fomo/repos/poetry2nix";
+
+    inputs.nixpkgs.follows = "nixpkgs";
   };
 
-  outputs =
-    { nixpkgs, ... }:
-    let
-      inherit (nixpkgs) lib;
-      forAllSystems = lib.genAttrs lib.systems.flakeExposed;
-    in
-    {
-      devShells = forAllSystems (
-        system:
-        let
-          pkgs = nixpkgs.legacyPackages.${system};
+  outputs = {
+    self,
+    nixpkgs,
+    flake-utils,
+    poetry2nix,
+  }:
+    # TODO: build cross-OS and use the `${system}` var thingy..
+    flake-utils.lib.eachDefaultSystem (system:
+      let
+        # use PWD as sources
+        projectDir = ./.;
+        pyproject = ./pyproject.toml;
+        poetrylock = ./poetry.lock;
 
-          # do store-path extractions
-          qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
-          # ?TODO? can remove below since manual linking not needed?
-          # qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
+        # TODO: port to 3.11 and support both versions?
+        python = "python3.10";
 
-          # XXX NOTE XXX, for now we overlay specific pkgs via
-          # a major-version-pinned-`cpython`
-          cpython = "python313";
-          pypkgs = pkgs."${cpython}Packages";
-        in
-        {
-          default = pkgs.mkShell {
+        # for more functions and examples.
+        # inherit
+        # (poetry2nix.legacyPackages.${system})
+        # mkPoetryApplication;
+        # pkgs = nixpkgs.legacyPackages.${system};
 
-            packages = with pkgs; [
-              # XXX, ensure sh completions active!
-              bashInteractive
-              bash-completion
+        pkgs = nixpkgs.legacyPackages.x86_64-linux;
+        lib = pkgs.lib;
+        p2npkgs = poetry2nix.legacyPackages.x86_64-linux;
 
-              # dev utils
-              ruff
-              pypkgs.ruff
+        # define all pkg overrides per dep, see edgecases.md:
+        # https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md
+        # TODO: add these into the json file:
+        # https://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json
+        pypkgs-build-requirements = {
+          asyncvnc = [ "setuptools" ];
+          eventkit = [ "setuptools" ];
+          ib-insync = [ "setuptools" "flake8" ];
+          msgspec = [ "setuptools"];
+          pdbp = [ "setuptools" ];
+          pyqt6-sip = [ "setuptools" ];
+          tabcompleter = [ "setuptools" ];
+          tractor = [ "setuptools" ];
+          tricycle = [ "setuptools" ];
+          trio-typing = [ "setuptools" ];
+          trio-util = [ "setuptools" ];
+          xonsh = [ "setuptools" ];
+        };
 
-              qt6.qtwayland
-              qt6.qtbase
+        # auto-generate override entries
+        p2n-overrides = p2npkgs.defaultPoetryOverrides.extend (self: super:
+          builtins.mapAttrs (package: build-requirements:
+            (builtins.getAttr package super).overridePythonAttrs (old: {
+              buildInputs = (
+                old.buildInputs or [ ]
+              ) ++ (
+                builtins.map (
+                  pkg: if builtins.isString pkg then builtins.getAttr pkg super else pkg
+                  ) build-requirements
+              );
+            })
+          ) pypkgs-build-requirements
+        );
 
-              uv
-              python313  # ?TODO^ how to set from `cpython` above?
-              pypkgs.pyqt6
-              pypkgs.pyqt6-sip
-              pypkgs.qtpy
-              pypkgs.qdarkstyle
-              pypkgs.rapidfuzz
-            ];
+        # override some ahead-of-time compiled extensions
+        # to be built with their wheels.
+        ahot_overrides = p2n-overrides.extend(
+          final: prev: {
 
-            shellHook = ''
-              # unmask to debug **this** dev-shell-hook
-              # set -e
+            # llvmlite = prev.llvmlite.override {
+            #   preferWheel = false;
+            # };
 
-              # set qt-base/plugin path(s)
-              QTBASE_PATH="${qt6baseStorePath}/lib"
-              QT_PLUGIN_PATH="${qt6baseStorePath}/lib/qt-6/plugins"
-              QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
+            # TODO: get this workin with p2n and nixpkgs..
+            # pyqt6 = prev.pyqt6.override {
+            #   preferWheel = true;
+            # };
 
-              # link in Qt cc lib paths from <nixpkgs>
-              LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
-              LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
-              LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
+            # NOTE: this DOESN'T work atm but after a fix
+            # to poetry2nix, it will and actually this line
+            # won't be needed - thanks @k900:
+            # https://github.com/nix-community/poetry2nix/pull/1257
+            pyqt5 = prev.pyqt5.override {
+              # withWebkit = false;
+              preferWheel = true;
+            };
 
-              # link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
-              LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
+            # see PR from @k900:
+            # https://github.com/nix-community/poetry2nix/pull/1257
+            # pyqt5-qt5 = prev.pyqt5-qt5.override {
+            #   withWebkit = false;
+            #   preferWheel = true;
+            # };
 
-              export LD_LIBRARY_PATH
-
-              # RUNTIME-SETTINGS
-              #
-              # ------ Qt ------
-              # XXX, unmask to debug qt .so linking/loading deats
-              # export QT_DEBUG_PLUGINS=1
-              #
-              # ALSO, for *modern linux* DEs,
-              # - maybe set wayland-mode (TODO, parametrtize this!)
-              #  * a chosen wayland-mode shell-integration
-              export QT_QPA_PLATFORM="wayland"
-              export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
-
-              # ------ uv ------
-              # - always use the ./py313/ venv-subdir
-              export UV_PROJECT_ENVIRONMENT="py313"
-              # sync project-env with all extras
-              uv sync --dev --all-extras --no-group lint
-
-              # ------ TIPS ------
-              # NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
-              # run the `nix develop` cmd with,
-              # >> nix develop -c uv run xonsh
-            '';
-          };
-        }
+            # TODO: patch in an override for polars to build
+            # from src! See the details likely needed from
+            # the cryptography entry:
+            # https://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix#L426-L435
+            polars = prev.polars.override {
+              preferWheel = true;
+            };
+          }
       );
-    };
+
+      # WHY!? -> output-attrs that `nix develop` scans for:
+      # https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html#flake-output-attributes
+      in
+      rec {
+        packages = {
+          # piker = poetry2nix.legacyPackages.x86_64-linux.mkPoetryEditablePackage {
+          #   editablePackageSources = { piker = ./piker; };
+
+          piker = p2npkgs.mkPoetryApplication {
+            projectDir = projectDir;
+
+            # SEE ABOVE for auto-genned input set, override
+            # buncha deps with extras.. like `setuptools` mostly.
+            # TODO: maybe propose a patch to p2n to show that you
+            # can even do this in the edgecases docs?
+            overrides = ahot_overrides;
+
+            # XXX: won't work on llvmlite..
+            # preferWheels = true;
+          };
+        };
+
+        # devShells.default = pkgs.mkShell {
+        #   projectDir = projectDir;
+        #   python = "python3.10";
+        #   overrides = ahot_overrides;
+        #   inputsFrom = [ self.packages.x86_64-linux.piker ];
+        #   packages = packages;
+        #   # packages = [ poetry2nix.packages.${system}.poetry ];
+        # };
+
+        # TODO: grok the difference here..
+        # - avoid re-cloning git repos on every develop entry..
+        # - ideally allow hacking on the src code of some deps
+        #   (tractor, pyqtgraph, tomlkit, etc.) WITHOUT having to
+        #   re-install them every time a change is made.
+        # - boot a usable xonsh inside the poetry virtualenv when
+        #   defined via a custom entry point?
+        devShells.default = p2npkgs.mkPoetryEnv {
+        # env = p2npkgs.mkPoetryEnv {
+            projectDir = projectDir;
+            python = pkgs.python310;
+            overrides = ahot_overrides;
+            editablePackageSources = packages;
+              # piker = "./";
+              # tractor = "../tractor/";
+            # };  # wut?
+        };
+      }
+    );  # end of .outputs scope
 }

piker/accounting/__init__.py

@@ -19,10 +19,8 @@
 for tendiez.
 
 '''
-from piker.log import (
-    get_console_log,
-    get_logger,
-)
+from ..log import get_logger
+
 from .calc import (
     iter_by_dt,
 )
@@ -35,6 +33,7 @@ from ._pos import (
     Account,
     load_account,
     load_account_from_ledger,
+    open_pps,
     open_account,
     Position,
 )
@@ -43,6 +42,7 @@ from ._mktinfo import (
     dec_digits,
     digits_to_dec,
     MktPair,
+    Symbol,
     unpack_fqme,
     _derivs as DerivTypes,
 )
@@ -53,23 +53,14 @@ from ._allocate import (
 
 
 log = get_logger(__name__)
-# ?TODO, enable console on import
-# [ ] necessary? or `open_brokerd_dialog()` doing it is sufficient?
-#
-# bc might as well enable whenev imported by
-# other sub-sys code (namely `.clearing`).
-get_console_log(
-    level='warning',
-    name=__name__,
-)
 
-# TODO, the `as <samename>` style?
 __all__ = [
     'Account',
     'Allocator',
     'Asset',
     'MktPair',
     'Position',
+    'Symbol',
     'Transaction',
     'TransactionLedger',
     'dec_digits',
@@ -79,6 +70,7 @@ __all__ = [
     'load_account_from_ledger',
     'mk_allocator',
     'open_account',
+    'open_pps',
     'open_trade_ledger',
     'unpack_fqme',
     'DerivTypes',

piker/accounting/_ledger.py

@@ -40,7 +40,7 @@ import tomli_w  # for fast ledger writing
 
 from piker.types import Struct
 from piker import config
-from piker.log import get_logger
+from ..log import get_logger
 from .calc import (
     iter_by_dt,
 )
@@ -239,9 +239,7 @@ class TransactionLedger(UserDict):
 
         symcache: SymbologyCache = self._symcache
         towrite: dict[str, Any] = {}
-        for tid, txdict in self.tx_sort(
-            self.data.copy()
-        ):
+        for tid, txdict in self.tx_sort(self.data.copy()):
             # write blank-str expiry for non-expiring assets
             if (
                 'expiry' in txdict
@@ -379,7 +377,7 @@ def open_trade_ledger(
         account,
         dirpath=_fp,
     )
-    cpy: dict = ledger_dict.copy()
+    cpy = ledger_dict.copy()
 
     # XXX NOTE: if not provided presume we are being called from
     # sync code and need to maybe run `trio` to generate..
@@ -408,13 +406,7 @@ def open_trade_ledger(
         account=account,
         mod=mod,
         symcache=symcache,
-
-        # NOTE: allow backends to provide custom ledger sorting
-        tx_sort=getattr(
-            mod,
-            'tx_sort',
-            tx_sort,
-        ),
+        tx_sort=getattr(mod, 'tx_sort', tx_sort),
     )
     try:
         yield ledger

piker/accounting/_mktinfo.py

@@ -305,8 +305,8 @@ class MktPair(Struct, frozen=True):
     # config right?
     # src_type: AssetTypeName
 
-    # for derivs, info describing contract, egs. strike price, call
-    # or put, swap type, exercise model, etc.
+    # for derivs, info describing contract, egs.
+    # strike price, call or put, swap type, exercise model, etc.
     contract_info: list[str] | None = None
 
     # TODO: rename to sectype since all of these can
@@ -390,8 +390,8 @@ class MktPair(Struct, frozen=True):
         cls,
         fqme: str,
 
-        price_tick: float|str,
-        size_tick: float|str,
+        price_tick: float | str,
+        size_tick: float | str,
         bs_mktid: str,
 
         broker: str | None = None,
@@ -677,3 +677,90 @@ def unpack_fqme(
         # '.'.join([mkt_ep, venue]),
         suffix,
     )
+
+
+class Symbol(Struct):
+    '''
+    I guess this is some kinda container thing for dealing with
+    all the different meta-data formats from brokers?
+
+    '''
+    key: str
+
+    broker: str = ''
+    venue: str = ''
+
+    # precision descriptors for price and vlm
+    tick_size: Decimal = Decimal('0.01')
+    lot_tick_size: Decimal = Decimal('0.0')
+
+    suffix: str = ''
+    broker_info: dict[str, dict[str, Any]] = {}
+
+    @classmethod
+    def from_fqme(
+        cls,
+        fqsn: str,
+        info: dict[str, Any],
+
+    ) -> Symbol:
+        broker, mktep, venue, suffix = unpack_fqme(fqsn)
+        tick_size = info.get('price_tick_size', 0.01)
+        lot_size = info.get('lot_tick_size', 0.0)
+
+        return Symbol(
+            broker=broker,
+            key=mktep,
+            tick_size=tick_size,
+            lot_tick_size=lot_size,
+            venue=venue,
+            suffix=suffix,
+            broker_info={broker: info},
+        )
+
+    @property
+    def type_key(self) -> str:
+        return list(self.broker_info.values())[0]['asset_type']
+
+    @property
+    def tick_size_digits(self) -> int:
+        return float_digits(self.tick_size)
+
+    @property
+    def lot_size_digits(self) -> int:
+        return float_digits(self.lot_tick_size)
+
+    @property
+    def price_tick(self) -> Decimal:
+        return Decimal(str(self.tick_size))
+
+    @property
+    def size_tick(self) -> Decimal:
+        return Decimal(str(self.lot_tick_size))
+
+    @property
+    def broker(self) -> str:
+        return list(self.broker_info.keys())[0]
+
+    @property
+    def fqme(self) -> str:
+        return maybe_cons_tokens([
+            self.key,  # final "pair name" (eg. qqq[/usd], btcusdt)
+            self.venue,
+            self.suffix,  # includes expiry and other con info
+            self.broker,
+        ])
+
+    def quantize(
+        self,
+        size: float,
+    ) -> Decimal:
+        digits = float_digits(self.lot_tick_size)
+        return Decimal(size).quantize(
+            Decimal(f'1.{"0".ljust(digits, "0")}'),
+            rounding=ROUND_HALF_EVEN
+        )
+
+    # NOTE: when cast to `str` return fqme
+    def __str__(self) -> str:
+        return self.fqme

piker/accounting/_pos.py

@@ -30,8 +30,7 @@ from types import ModuleType
 from typing import (
     Any,
     Iterator,
-    Generator,
-    TYPE_CHECKING,
+    Generator
 )
 
 import pendulum
@@ -60,16 +59,10 @@ from ..clearing._messages import (
     BrokerdPosition,
 )
 from piker.types import Struct
-from piker.log import (
-    get_logger,
-)
+from piker.data._symcache import SymbologyCache
+from ..log import get_logger
 
-if TYPE_CHECKING:
-    from piker.data._symcache import SymbologyCache
-
-log = get_logger(
-    name=__name__,
-)
+log = get_logger(__name__)
 
 
 class Position(Struct):
@@ -360,20 +353,17 @@ class Position(Struct):
     ) -> bool:
         '''
         Update clearing table by calculating the rolling ppu and
-        (accumulative) size in both the clears entry and local attrs
-        state.
+        (accumulative) size in both the clears entry and local
+        attrs state.
 
         Inserts are always done in datetime sorted order.
 
         '''
+        # added: bool = False
         tid: str = t.tid
         if tid in self._events:
-            log.debug(
-                f'Txn is already added?\n'
-                f'\n'
-                f'{t}\n'
-            )
-            return False
+            log.warning(f'{t} is already added?!')
+            # return added
 
         # TODO: apparently this IS possible with a dict but not
         # common and probably not that beneficial unless we're also
@@ -454,12 +444,6 @@ class Position(Struct):
     # def suggest_split(self) -> float:
     #     ...
 
-    # ?TODO, for sending rendered state over the wire?
-    # def summary(self) -> PositionSummary:
-        # do minimal conversion to a subset of fields
-        # currently defined in `.clearing._messages.BrokerdPosition`
-
-
 
 class Account(Struct):
     '''
@@ -503,23 +487,12 @@ class Account(Struct):
 
     def update_from_ledger(
         self,
-        ledger: TransactionLedger|dict[str, Transaction],
+        ledger: TransactionLedger | dict[str, Transaction],
         cost_scalar: float = 2,
-        symcache: SymbologyCache|None = None,
+        symcache: SymbologyCache | None = None,
 
         _mktmap_table: dict[str, MktPair] | None = None,
 
-        only_require: list[str]|True = True,
-        # ^list of fqmes that are "required" to be processed from
-        # this ledger pass; we often don't care about others and
-        # definitely shouldn't always error in such cases.
-        # (eg. broker backend loaded that doesn't yet supsport the
-        # symcache but also, inside the paper engine we don't ad-hoc
-        # request `get_mkt_info()` for every symbol in the ledger,
-        # only the one for which we're simulating against).
-        # TODO, not sure if there's a better soln for this, ideally
-        # all backends get symcache support afap i guess..
-
     ) -> dict[str, Position]:
         '''
         Update the internal `.pps[str, Position]` table from input
@@ -562,32 +535,11 @@ class Account(Struct):
                 if _mktmap_table is None:
                     raise
 
-                required: bool = (
-                    only_require is True
-                    or (
-                        only_require is not True
-                        and
-                        fqme in only_require
-                    )
-                )
                 # XXX: caller is allowed to provide a fallback
                 # mktmap table for the case where a new position is
                 # being added and the preloaded symcache didn't
                 # have this entry prior (eg. with frickin IB..)
-                if (
-                    not (mkt := _mktmap_table.get(fqme))
-                    and
-                    required
-                ):
-                    raise
-
-                elif not required:
-                    continue
-
-                else:
-                    # should be an entry retreived somewhere
-                    assert mkt
-
+                mkt = _mktmap_table[fqme]
 
             if not (pos := pps.get(bs_mktid)):
 
@@ -704,7 +656,7 @@ class Account(Struct):
     def write_config(self) -> None:
         '''
         Write the current account state to the user's account TOML file, normally
-        something like `pps.toml`.
+        something like ``pps.toml``.
 
         '''
         # TODO: show diff output?
@@ -744,7 +696,7 @@ class Account(Struct):
                     else:
                         # TODO: we reallly need a diff set of
                         # loglevels/colors per subsys.
-                        log.debug(
+                        log.warning(
                             f'Recent position for {fqme} was closed!'
                         )
 
@@ -758,7 +710,7 @@ class Account(Struct):
         # XXX WTF: if we use a tomlkit.Integer here we get this
         # super weird --1 thing going on for cumsize!?1!
         # NOTE: the fix was to always float() the size value loaded
-        # in open_account() below!
+        # in open_pps() below!
         config.write(
             config=self.conf,
             path=self.conf_path,
@@ -942,6 +894,7 @@ def open_account(
             clears_table['dt'] = dt
             trans.append(Transaction(
                 fqme=bs_mktid,
+                # sym=mkt,
                 bs_mktid=bs_mktid,
                 tid=tid,
                 # XXX: not sure why sometimes these are loaded as
@@ -964,22 +917,11 @@ def open_account(
         ):
             expiry: pendulum.DateTime = pendulum.parse(expiry)
 
-        # !XXX, should never be duplicates over
-        # a backend-(broker)-system's unique market-IDs!
-        if pos := pp_objs.get(bs_mktid):
-            if mkt != pos.mkt:
-                log.warning(
-                    f'Duplicated position but diff `MktPair.fqme` ??\n'
-                    f'bs_mktid: {bs_mktid!r}\n'
-                    f'pos.mkt: {pos.mkt}\n'
-                    f'mkt: {mkt}\n'
-                )
-        else:
-            pos = pp_objs[bs_mktid] = Position(
-                mkt,
-                split_ratio=split_ratio,
-                bs_mktid=bs_mktid,
-            )
+        pp = pp_objs[bs_mktid] = Position(
+            mkt,
+            split_ratio=split_ratio,
+            bs_mktid=bs_mktid,
+        )
 
         # XXX: super critical, we need to be sure to include
         # all pps.toml clears to avoid reusing clears that were
@@ -987,13 +929,8 @@ def open_account(
         # state, since today's records may have already been
         # processed!
         for t in trans:
-            added: bool = pos.add_clear(t)
-            if not added:
-                log.warning(
-                    f'Txn already recorded in pp ??\n'
-                    f'\n'
-                    f'{t}\n'
-                )
+            pp.add_clear(t)
+
     try:
         yield acnt
     finally:
@@ -1001,6 +938,20 @@ def open_account(
             acnt.write_config()
 
 
+# TODO: drop the old name and THIS!
+@cm
+def open_pps(
+    *args,
+    **kwargs,
+) -> Generator[Account, None, None]:
+    log.warning(
+        '`open_pps()` is now deprecated!\n'
+        'Please use `with open_account() as cnt:`'
+    )
+    with open_account(*args, **kwargs) as acnt:
+        yield acnt
+
+
 def load_account_from_ledger(
 
     brokername: str,

piker/accounting/calc.py

@@ -22,9 +22,7 @@ you know when you're losing money (if possible) XD
 from __future__ import annotations
 from collections.abc import ValuesView
 from contextlib import contextmanager as cm
-from functools import partial
 from math import copysign
-from pprint import pformat
 from typing import (
     Any,
     Callable,
@@ -32,7 +30,6 @@ from typing import (
     TYPE_CHECKING,
 )
 
-from tractor.devx import maybe_open_crash_handler
 import polars as pl
 from pendulum import (
     DateTime,
@@ -40,16 +37,12 @@ from pendulum import (
     parse,
 )
 
-from ..log import get_logger
-
 if TYPE_CHECKING:
     from ._ledger import (
         Transaction,
         TransactionLedger,
     )
 
-log = get_logger(__name__)
-
 
 def ppu(
     clears: Iterator[Transaction],
@@ -245,9 +238,6 @@ def iter_by_dt(
 
     def dyn_parse_to_dt(
         tx: tuple[str, dict[str, Any]] | Transaction,
-
-        debug: bool = False,
-        _invalid: list|None = None,
     ) -> DateTime:
 
         # handle `.items()` inputs
@@ -260,90 +250,33 @@ def iter_by_dt(
         # get best parser for this record..
         for k in parsers:
             if (
-                (v := getattr(tx, k, None))
-                or
-                (
-                    isdict
-                    and
-                    (v := tx.get(k))
-                )
+                isdict and k in tx
+                 or getattr(tx, k, None)
             ):
+                v = tx[k] if isdict else tx.dt
+                assert v is not None, f'No valid value for `{k}`!?'
+
                 # only call parser on the value if not None from
                 # the `parsers` table above (when NOT using
                 # `.get()`), otherwise pass through the value and
                 # sort on it directly
                 if (
                     not isinstance(v, DateTime)
-                    and
-                    (parser := parsers.get(k))
+                    and (parser := parsers.get(k))
                 ):
-                    ret = parser(v)
+                    return parser(v)
                 else:
-                    ret = v
+                    return v
 
-                return ret
-
-            else:
-                log.debug(
-                    f'Parser-field not found in txn\n'
-                    f'\n'
-                    f'parser-field: {k!r}\n'
-                    f'txn: {tx!r}\n'
-                    f'\n'
-                    f'Trying next..\n'
-                )
-                continue
-
-        # XXX: we should never really get here bc it means some kinda
-        # bad txn-record (field) data..
-        #
-        # -> set the `debug_mode = True` if you want to trace such
-        # cases from REPL ;)
         else:
-            # XXX: we should really never get here..
-            # only if a ledger record has no expected sort(able)
-            # field will we likely hit this.. like with ze IB.
-            # if no sortable field just deliver epoch?
-            log.warning(
-                'No (time) sortable field for TXN:\n'
-                f'{tx!r}\n'
-            )
-            report: str = (
-                f'No supported time-field found in txn !?\n'
-                f'\n'
-                f'supported-time-fields: {parsers!r}\n'
-                f'\n'
-                f'txn: {tx!r}\n'
-            )
-            if debug:
-                with maybe_open_crash_handler(
-                    pdb=debug,
-                    raise_on_exit=False,
-                ):
-                    raise ValueError(report)
-            else:
-                log.error(report)
+            # XXX: should never get here..
+            breakpoint()
 
-            if _invalid is not None:
-                _invalid.append(tx)
-                return from_timestamp(0.)
-
-    entry: tuple[str, dict]|Transaction
-    invalid: list = []
+    entry: tuple[str, dict] | Transaction
     for entry in sorted(
         records,
-        key=key or partial(
-            dyn_parse_to_dt,
-            _invalid=invalid,
-        ),
+        key=key or dyn_parse_to_dt,
     ):
-        if entry in invalid:
-            log.warning(
-                f'Ignoring txn w invalid timestamp ??\n'
-                f'{pformat(entry)}\n'
-            )
-            continue
-
         # NOTE the type sig above; either pairs or txns B)
         yield entry
 
@@ -406,7 +339,6 @@ def open_ledger_dfs(
     acctname: str,
 
     ledger: TransactionLedger | None = None,
-    debug_mode: bool = False,
 
     **kwargs,
 
@@ -421,10 +353,8 @@ def open_ledger_dfs(
     can update the ledger on exit.
 
     '''
-    with maybe_open_crash_handler(
-        pdb=debug_mode,
-        # raise_on_exit=False,
-    ):
+    from piker.toolz import open_crash_handler
+    with open_crash_handler():
         if not ledger:
             import time
             from ._ledger import open_trade_ledger
@@ -516,7 +446,7 @@ def ledger_to_dfs(
 
         df = dfs[key] = ldf.with_columns([
 
-            pl.cum_sum('size').alias('cumsize'),
+            pl.cumsum('size').alias('cumsize'),
 
             # amount of source asset "sent" (via buy txns in
             # the market) to acquire the dst asset, PER txn.
@@ -531,7 +461,7 @@ def ledger_to_dfs(
         ]).with_columns([
 
             # rolling balance in src asset units
-            (pl.col('dst_bot').cum_sum() * -1).alias('src_balance'),
+            (pl.col('dst_bot').cumsum() * -1).alias('src_balance'),
 
             # "position operation type" in terms of increasing the
             # amount in the dst asset (entering) or decreasing the
@@ -673,7 +603,7 @@ def ledger_to_dfs(
                 # cost that was included in the least-recently
                 # entered txn that is still part of the current CSi
                 # set.
-                # => we look up the cost-per-unit cum_sum and apply
+                # => we look up the cost-per-unit cumsum and apply
                 # if over the current txn size (by multiplication)
                 # and then reverse that previusly applied cost on
                 # the txn_cost for this record.

piker/accounting/cli.py

@@ -21,6 +21,7 @@ CLI front end for trades ledger and position tracking management.
 from __future__ import annotations
 from pprint import pformat
 
+
 from rich.console import Console
 from rich.markdown import Markdown
 import polars as pl
@@ -28,10 +29,7 @@ import tractor
 import trio
 import typer
 
-from piker.log import (
-    get_console_log,
-    get_logger,
-)
+from ..log import get_logger
 from ..service import (
     open_piker_runtime,
 )
@@ -47,7 +45,6 @@ from .calc import (
     open_ledger_dfs,
 )
 
-log = get_logger(name=__name__)
 
 ledger = typer.Typer()
 
@@ -82,10 +79,7 @@ def sync(
         "-l",
     ),
 ):
-    log = get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
+    log = get_logger(loglevel)
     console = Console()
 
     pair: tuple[str, str]
@@ -306,8 +300,7 @@ def disect(
         assert not df.is_empty()
 
         # muck around in pdbp REPL
-        # tractor.devx.mk_pdb().set_trace()
-        # breakpoint()
+        breakpoint()
 
         # TODO: we REALLY need a better console REPL for this
         # kinda thing..

piker/brokers/__init__.py

@@ -25,16 +25,15 @@ from types import ModuleType
 
 from tractor.trionics import maybe_open_context
 
-from piker.log import (
-    get_logger,
-)
 from ._util import (
+    log,
     BrokerError,
     SymbolNotFound,
     NoData,
     DataUnavailable,
     DataThrottle,
     resproc,
+    get_logger,
 )
 
 __all__: list[str] = [
@@ -44,13 +43,14 @@ __all__: list[str] = [
     'DataUnavailable',
     'DataThrottle',
     'resproc',
+    'get_logger',
 ]
 
 __brokers__: list[str] = [
     'binance',
     'ib',
     'kraken',
-    'kucoin',
+    'kucoin'
 
     # broken but used to work
     # 'questrade',
@@ -65,17 +65,13 @@ __brokers__: list[str] = [
     # bitso
 ]
 
-log = get_logger(
-    name=__name__,
-)
-
 
 def get_brokermod(brokername: str) -> ModuleType:
     '''
     Return the imported broker module by name.
 
     '''
-    module: ModuleType = import_module('.' + brokername, 'piker.brokers')
+    module = import_module('.' + brokername, 'piker.brokers')
     # we only allow monkeying because it's for internal keying
     module.name = module.__name__.split('.')[-1]
     return module
@@ -102,14 +98,13 @@ async def open_cached_client(
     If one has not been setup do it and cache it.
 
     '''
-    brokermod: ModuleType = get_brokermod(brokername)
-
-    # TODO: make abstract or `typing.Protocol`
-    # client: Client
+    brokermod = get_brokermod(brokername)
     async with maybe_open_context(
         acm_func=brokermod.get_client,
         kwargs=kwargs,
+
     ) as (cache_hit, client):
+
         if cache_hit:
             log.runtime(f'Reusing existing {client}')
 

piker/brokers/_daemon.py

@@ -33,18 +33,12 @@ import exceptiongroup as eg
 import tractor
 import trio
 
-from piker.log import (
-    get_logger,
-    get_console_log,
-)
 from . import _util
 from . import get_brokermod
 
 if TYPE_CHECKING:
     from ..data import _FeedsBus
 
-log = get_logger(name=__name__)
-
 # `brokerd` enabled modules
 # TODO: move this def to the `.data` subpkg..
 # NOTE: keeping this list as small as possible is part of our caps-sec
@@ -65,7 +59,7 @@ _data_mods: str = [
 async def _setup_persistent_brokerd(
     ctx: tractor.Context,
     brokername: str,
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
 ) -> None:
     '''
@@ -78,14 +72,13 @@ async def _setup_persistent_brokerd(
     # since all hosted daemon tasks will reference this same
     # log instance's (actor local) state and thus don't require
     # any further (level) configuration on their own B)
-    actor: tractor.Actor = tractor.current_actor()
-    tll: str = actor.loglevel
-    log = get_console_log(
-        level=loglevel or tll,
+    log = _util.get_console_log(
+        loglevel or tractor.current_actor().loglevel,
         name=f'{_util.subsys}.{brokername}',
-        with_tractor_log=bool(tll),
     )
-    assert log.name == _util.subsys
+
+    # set global for this actor to this new process-wide instance B)
+    _util.log = log
 
     # further, set the log level on any broker broker specific
     # logger instance.
@@ -103,10 +96,7 @@ async def _setup_persistent_brokerd(
     # - `open_symbol_search()`
     # NOTE: see ep invocation details inside `.data.feed`.
     try:
-        async with (
-            # tractor.trionics.collapse_eg(),
-            trio.open_nursery() as service_nursery
-        ):
+        async with trio.open_nursery() as service_nursery:
             bus: _FeedsBus = feed.get_feed_bus(
                 brokername,
                 service_nursery,
@@ -200,6 +190,7 @@ def broker_init(
 
 
 async def spawn_brokerd(
+
     brokername: str,
     loglevel: str | None = None,
 
@@ -207,10 +198,8 @@ async def spawn_brokerd(
 
 ) -> bool:
 
-    log.info(
-        f'Spawning broker-daemon,\n'
-        f'backend: {brokername!r}'
-    )
+    from piker.service._util import log  # use service mngr log
+    log.info(f'Spawning {brokername} broker daemon')
 
     (
         brokermode,
@@ -257,7 +246,7 @@ async def spawn_brokerd(
 async def maybe_spawn_brokerd(
 
     brokername: str,
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
     **pikerd_kwargs,
 
@@ -273,7 +262,8 @@ async def maybe_spawn_brokerd(
     from piker.service import maybe_spawn_daemon
 
     async with maybe_spawn_daemon(
-        service_name=f'brokerd.{brokername}',
+
+        f'brokerd.{brokername}',
         service_task_target=spawn_brokerd,
         spawn_args={
             'brokername': brokername,

piker/brokers/_util.py

@@ -18,14 +18,15 @@
 Handy cross-broker utils.
 
 """
-from __future__ import annotations
-# from functools import partial
+from functools import partial
 
 import json
-import httpx
+import asks
 import logging
 
-from piker.log import (
+from ..log import (
+    get_logger,
+    get_console_log,
     colorize_json,
 )
 subsys: str = 'piker.brokers'
@@ -33,22 +34,12 @@ subsys: str = 'piker.brokers'
 # NOTE: level should be reset by any actor that is spawned
 # as well as given a (more) explicit name/key such
 # as `piker.brokers.binance` matching the subpkg.
-# log = get_logger(subsys)
+log = get_logger(subsys)
 
-# ?TODO?? we could use this approach, but we need to be able
-# to pass multiple `name=` values so for example we can include the
-# emissions in `.accounting._pos` and others!
-# [ ] maybe we could do the `log = get_logger()` above,
-# then cycle through the list of subsys mods we depend on
-# and then get all their loggers and pass them to
-# `get_console_log(logger=)`??
-# [ ] OR just write THIS `get_console_log()` as a hook which does
-# that based on who calls it?.. i dunno
-#
-# get_console_log = partial(
-#     get_console_log,
-#     name=subsys,
-# )
+get_console_log = partial(
+    get_console_log,
+    name=subsys,
+)
 
 
 class BrokerError(Exception):
@@ -69,11 +60,11 @@ class NoData(BrokerError):
     def __init__(
         self,
         *args,
-        info: dict|None = None,
+        info: dict,
 
     ) -> None:
         super().__init__(*args)
-        self.info: dict|None = info
+        self.info: dict = info
 
         # when raised, machinery can check if the backend
         # set a "frame size" for doing datetime calcs.
@@ -99,18 +90,16 @@ class DataThrottle(BrokerError):
 
 
 def resproc(
-    resp: httpx.Response,
+    resp: asks.response_objects.Response,
     log: logging.Logger,
     return_json: bool = True,
     log_resp: bool = False,
 
-) -> httpx.Response:
-    '''
-    Process response and return its json content.
+) -> asks.response_objects.Response:
+    """Process response and return its json content.
 
     Raise the appropriate error on non-200 OK responses.
-
-    '''
+    """
     if not resp.status_code == 200:
         raise BrokerError(resp.body)
     try:

piker/brokers/binance/api.py

@@ -1,8 +1,8 @@
 # piker: trading gear for hackers
 # Copyright (C)
-#  Guillermo Rodriguez (aka ze jefe)
-#  Tyler Goodlet
-#  (in stewardship for pikers)
+#   Guillermo Rodriguez (aka ze jefe)
+#   Tyler Goodlet
+#   (in stewardship for pikers)
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -25,13 +25,14 @@ from __future__ import annotations
 from collections import ChainMap
 from contextlib import (
     asynccontextmanager as acm,
-    AsyncExitStack,
 )
 from datetime import datetime
 from pprint import pformat
 from typing import (
     Any,
     Callable,
+    Hashable,
+    Sequence,
     Type,
 )
 import hmac
@@ -42,7 +43,8 @@ import trio
 from pendulum import (
     now,
 )
-import httpx
+import asks
+from rapidfuzz import process as fuzzy
 import numpy as np
 
 from piker import config
@@ -52,7 +54,6 @@ from piker.clearing._messages import (
 from piker.accounting import (
     Asset,
     digits_to_dec,
-    MktPair,
 )
 from piker.types import Struct
 from piker.data import (
@@ -68,6 +69,7 @@ from .venues import (
     PAIRTYPES,
     Pair,
     MarketType,
+
     _spot_url,
     _futes_url,
     _testnet_futes_url,
@@ -77,18 +79,19 @@ from .venues import (
 log = get_logger('piker.brokers.binance')
 
 
-def get_config() -> dict[str, Any]:
+def get_config() -> dict:
+
     conf: dict
     path: Path
     conf, path = config.load(
         conf_name='brokers',
         touch_if_dne=True,
     )
-    section: dict = conf.get('binance')
+
+    section = conf.get('binance')
+
     if not section:
-        log.warning(
-            f'No config section found for binance in {path}'
-        )
+        log.warning(f'No config section found for binance in {path}')
         return {}
 
     return section
@@ -144,7 +147,7 @@ def binance_timestamp(
 
 class Client:
     '''
-    Async ReST API client using `trio` + `httpx` B)
+    Async ReST API client using ``trio`` + ``asks`` B)
 
     Supports all of the spot, margin and futures endpoints depending
     on method.
@@ -153,17 +156,10 @@ class Client:
     def __init__(
         self,
 
-        venue_sessions: dict[
-            str,  # venue key
-            tuple[httpx.AsyncClient, str]  # session, eps path
-        ],
-        conf: dict[str, Any],
         # TODO: change this to `Client.[mkt_]venue: MarketType`?
         mkt_mode: MarketType = 'spot',
 
     ) -> None:
-        self.conf = conf
-
         # build out pair info tables for each market type
         # and wrap in a chain-map view for search / query.
         self._spot_pairs: dict[str, Pair] = {}  # spot info table
@@ -190,13 +186,44 @@ class Client:
         # market symbols for use by search. See `.exch_info()`.
         self._pairs: ChainMap[str, Pair] = ChainMap()
 
+        # spot EPs sesh
+        self._sesh = asks.Session(connections=4)
+        self._sesh.base_location: str = _spot_url
+        # spot testnet
+        self._test_sesh: asks.Session = asks.Session(connections=4)
+        self._test_sesh.base_location: str = _testnet_spot_url
+
+        # margin and extended spot endpoints session.
+        self._sapi_sesh = asks.Session(connections=4)
+        self._sapi_sesh.base_location: str = _spot_url
+
+        # futes EPs sesh
+        self._fapi_sesh = asks.Session(connections=4)
+        self._fapi_sesh.base_location: str = _futes_url
+        # futes testnet
+        self._test_fapi_sesh: asks.Session = asks.Session(connections=4)
+        self._test_fapi_sesh.base_location: str = _testnet_futes_url
+
         # global client "venue selection" mode.
         # set this when you want to switch venues and not have to
         # specify the venue for the next request.
         self.mkt_mode: MarketType = mkt_mode
 
-        # per-mkt-venue API client table
-        self.venue_sesh = venue_sessions
+        # per 8
+        self.venue_sesh: dict[
+            str,  # venue key
+            tuple[asks.Session, str]  # session, eps path
+        ] = {
+            'spot': (self._sesh, '/api/v3/'),
+            'spot_testnet': (self._test_sesh, '/fapi/v1/'),
+
+            'margin': (self._sapi_sesh, '/sapi/v1/'),
+
+            'usdtm_futes': (self._fapi_sesh, '/fapi/v1/'),
+            'usdtm_futes_testnet': (self._test_fapi_sesh, '/fapi/v1/'),
+
+            # 'futes_coin': self._dapi,  # TODO
+        }
 
         # lookup for going from `.mkt_mode: str` to the config
         # subsection `key: str`
@@ -211,6 +238,40 @@ class Client:
             'futes': ['usdtm_futes'],
         }
 
+        # for creating API keys see,
+        # https://www.binance.com/en/support/faq/how-to-create-api-keys-on-binance-360002502072
+        self.conf: dict = get_config()
+
+        for key, subconf in self.conf.items():
+            if api_key := subconf.get('api_key', ''):
+                venue_keys: list[str] = self.confkey2venuekeys[key]
+
+                venue_key: str
+                sesh: asks.Session
+                for venue_key in venue_keys:
+                    sesh, _ = self.venue_sesh[venue_key]
+
+                    api_key_header: dict = {
+                        # taken from official:
+                        # https://github.com/binance/binance-futures-connector-python/blob/main/binance/api.py#L47
+                        "Content-Type": "application/json;charset=utf-8",
+
+                        # TODO: prolly should just always query and copy
+                        # in the real latest ver?
+                        "User-Agent": "binance-connector/6.1.6smbz6",
+                        "X-MBX-APIKEY": api_key,
+                    }
+                    sesh.headers.update(api_key_header)
+
+                    # if `.use_tesnet = true` in the config then
+                    # also add headers for the testnet session which
+                    # will be used for all order control
+                    if subconf.get('use_testnet', False):
+                        testnet_sesh, _ = self.venue_sesh[
+                            venue_key + '_testnet'
+                        ]
+                        testnet_sesh.headers.update(api_key_header)
+
     def _mk_sig(
         self,
         data: dict,
@@ -229,6 +290,7 @@ class Client:
                 'to define the creds for auth-ed endpoints!?'
             )
 
+
         # XXX: Info on security and authentification
         # https://binance-docs.github.io/apidocs/#endpoint-security-type
         if not (api_secret := subconf.get('api_secret')):
@@ -257,7 +319,7 @@ class Client:
         params: dict,
 
         method: str = 'get',
-        venue: str|None = None,  # if None use `.mkt_mode` state
+        venue: str | None = None,  # if None use `.mkt_mode` state
         signed: bool = False,
         allow_testnet: bool = False,
 
@@ -268,9 +330,8 @@ class Client:
         - /fapi/v3/ USD-M FUTURES, or
         - /api/v3/ SPOT/MARGIN
 
-        account/market endpoint request depending on either passed in
-        `venue: str` or the current setting `.mkt_mode: str` setting,
-        default `'spot'`.
+        account/market endpoint request depending on either passed in `venue: str`
+        or the current setting `.mkt_mode: str` setting, default `'spot'`.
 
 
         Docs per venue API:
@@ -299,6 +360,9 @@ class Client:
                 venue=venue_key,
             )
 
+        sesh: asks.Session
+        path: str
+
         # Check if we're configured to route order requests to the
         # venue equivalent's testnet.
         use_testnet: bool = False
@@ -323,12 +387,11 @@ class Client:
             # ctl machinery B)
             venue_key += '_testnet'
 
-        client: httpx.AsyncClient
-        path: str
-        client, path = self.venue_sesh[venue_key]
-        meth: Callable = getattr(client, method)
+        sesh, path = self.venue_sesh[venue_key]
+
+        meth: Callable = getattr(sesh, method)
         resp = await meth(
-            url=path + endpoint,
+            path=path + endpoint,
             params=params,
             timeout=float('inf'),
         )
@@ -373,16 +436,7 @@ class Client:
             try:
                 pair: Pair = pair_type(**item)
             except Exception as e:
-                e.add_note(
-                    f'\n'
-                    f'New or removed field we need to codify!\n'
-                    f'pair-type: {pair_type!r}\n'
-                    f'\n'
-                    f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
-                    f'Check out their API docs here:\n'
-                    f'\n'
-                    f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
-                )
+                e.add_note(f'\nDon\'t panic, check out this https://binance-docs.github.io/apidocs/spot/en/#exchange-information')
                 raise
             pair_table[pair.symbol.upper()] = pair
 
@@ -478,9 +532,7 @@ class Client:
 
         '''
         pair_table: dict[str, Pair] = self._venue2pairs[
-            venue
-            or
-            self.mkt_mode
+            venue or self.mkt_mode
         ]
         if (
             expiry
@@ -499,9 +551,9 @@ class Client:
             venues: list[str] = [venue]
 
         # batch per-venue download of all exchange infos
-        async with trio.open_nursery() as tn:
+        async with trio.open_nursery() as rn:
             for ven in venues:
-                tn.start_soon(
+                rn.start_soon(
                     self._cache_pairs,
                     ven,
                 )
@@ -554,11 +606,11 @@ class Client:
 
     ) -> dict[str, Any]:
 
-        fq_pairs: dict[str, Pair] = await self.exch_info()
+        fq_pairs: dict = await self.exch_info()
 
         # TODO: cache this list like we were in
         # `open_symbol_search()`?
-        # keys: list[str] = list(fq_pairs)
+        keys: list[str] = list(fq_pairs)
 
         return match_from_pairs(
             pairs=fq_pairs,
@@ -566,20 +618,9 @@ class Client:
             score_cutoff=50,
         )
 
-    def pair2venuekey(
-        self,
-        pair: Pair,
-    ) -> str:
-        return {
-            'USDTM': 'usdtm_futes',
-            'SPOT': 'spot',
-            # 'COINM': 'coin_futes',
-            # ^-TODO-^ bc someone might want it..?
-        }[pair.venue]
-
     async def bars(
         self,
-        mkt: MktPair,
+        symbol: str,
 
         start_dt: datetime | None = None,
         end_dt: datetime | None = None,
@@ -609,20 +650,16 @@ class Client:
         start_time = binance_timestamp(start_dt)
         end_time = binance_timestamp(end_dt)
 
-        bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
-
         # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
         bars = await self._api(
             'klines',
             params={
-                # NOTE: always query using their native symbology!
-                'symbol': mkt.bs_mktid.upper(),
+                'symbol': symbol.upper(),
                 'interval': '1m',
                 'startTime': start_time,
                 'endTime': end_time,
                 'limit': limit
             },
-            venue=self.pair2venuekey(bs_pair),
             allow_testnet=False,
         )
         new_bars: list[tuple] = []
@@ -939,148 +976,17 @@ class Client:
         await self.close_listen_key(key)
 
 
-_venue_urls: dict[str, str] = {
-    'spot': (
-        _spot_url,
-        '/api/v3/',
-    ),
-    'spot_testnet': (
-        _testnet_spot_url,
-        '/fapi/v1/'
-    ),
-    # margin and extended spot endpoints session.
-    # TODO: did this ever get implemented fully?
-    # 'margin': (
-    #     _spot_url,
-    #     '/sapi/v1/'
-    # ),
-
-    'usdtm_futes': (
-        _futes_url,
-        '/fapi/v1/',
-    ),
-
-    'usdtm_futes_testnet': (
-        _testnet_futes_url,
-        '/fapi/v1/',
-    ),
-
-    # TODO: for anyone who actually needs it ;P
-    # 'coin_futes': ()
-}
-
-
-def init_api_keys(
-    client: Client,
-    conf: dict[str, Any],
-) -> None:
-    '''
-    Set up per-venue API keys each http client according to the user's
-    `brokers.conf`.
-
-    For ex, to use spot-testnet and live usdt futures APIs:
-
-    ```toml
-        [binance]
-        # spot test net
-        spot.use_testnet = true
-        spot.api_key = '<spot_api_key_from_binance_account>'
-        spot.api_secret = '<spot_api_key_password>'
-
-        # futes live
-        futes.use_testnet = false
-        accounts.usdtm = 'futes'
-        futes.api_key = '<futes_api_key_from_binance>'
-        futes.api_secret = '<futes_api_key_password>''
-
-        # if uncommented will use the built-in paper engine and not
-        # connect to `binance` API servers for order ctl.
-        # accounts.paper = 'paper'
-    ```
-
-    '''
-    for key, subconf in conf.items():
-        if api_key := subconf.get('api_key', ''):
-            venue_keys: list[str] = client.confkey2venuekeys[key]
-
-            venue_key: str
-            client: httpx.AsyncClient
-            for venue_key in venue_keys:
-                client, _ = client.venue_sesh[venue_key]
-
-                api_key_header: dict = {
-                    # taken from official:
-                    # https://github.com/binance/binance-futures-connector-python/blob/main/binance/api.py#L47
-                    "Content-Type": "application/json;charset=utf-8",
-
-                    # TODO: prolly should just always query and copy
-                    # in the real latest ver?
-                    "User-Agent": "binance-connector/6.1.6smbz6",
-                    "X-MBX-APIKEY": api_key,
-                }
-                client.headers.update(api_key_header)
-
-                # if `.use_tesnet = true` in the config then
-                # also add headers for the testnet session which
-                # will be used for all order control
-                if subconf.get('use_testnet', False):
-                    testnet_sesh, _ = client.venue_sesh[
-                        venue_key + '_testnet'
-                    ]
-                    testnet_sesh.headers.update(api_key_header)
-
-
 @acm
-async def get_client(
-    mkt_mode: MarketType = 'spot',
-) -> Client:
-    '''
-    Construct an single `piker` client which composes multiple underlying venue
-    specific API clients both for live and test networks.
+async def get_client() -> Client:
 
-    '''
-    venue_sessions: dict[
-        str,  # venue key
-        tuple[httpx.AsyncClient, str]  # session, eps path
-    ] = {}
-    async with AsyncExitStack() as client_stack:
-        for name, (base_url, path) in _venue_urls.items():
-            api: httpx.AsyncClient = await client_stack.enter_async_context(
-                httpx.AsyncClient(
-                    base_url=base_url,
-                    # headers={},
+    client = Client()
+    await client.exch_info()
+    log.info(
+        f'{client} in {client.mkt_mode} mode: caching exchange infos..\n'
+        'Cached multi-market pairs:\n'
+        f'spot: {len(client._spot_pairs)}\n'
+        f'usdtm_futes: {len(client._ufutes_pairs)}\n'
+        f'Total: {len(client._pairs)}\n'
+    )
 
-                    # TODO: is there a way to numerate this?
-                    # https://www.python-httpx.org/advanced/clients/#why-use-a-client
-                    # connections=4
-                )
-            )
-            venue_sessions[name] = (
-                api,
-                path,
-            )
-
-        conf: dict[str, Any] = get_config()
-        # for creating API keys see,
-        # https://www.binance.com/en/support/faq/how-to-create-api-keys-on-binance-360002502072
-        client = Client(
-            venue_sessions=venue_sessions,
-            conf=conf,
-            mkt_mode=mkt_mode,
-        )
-        init_api_keys(
-            client=client,
-            conf=conf,
-        )
-        fq_pairs: dict[str, Pair] = await client.exch_info()
-        assert fq_pairs
-        log.info(
-            f'Loaded multi-venue `Client` in mkt_mode={client.mkt_mode!r}\n\n'
-            f'Symbology Summary:\n'
-            f'------ - ------\n'
-            f'spot: {len(client._spot_pairs)}\n'
-            f'usdtm_futes: {len(client._ufutes_pairs)}\n'
-            '------ - ------\n'
-            f'total: {len(client._pairs)}\n'
-        )
-        yield client
+    yield client

piker/brokers/binance/broker.py

@@ -37,9 +37,8 @@ import trio
 from piker.accounting import (
     Asset,
 )
-from piker.log import (
+from piker.brokers._util import (
     get_logger,
-    get_console_log,
 )
 from piker.data._web_bs import (
     open_autorecon_ws,
@@ -70,9 +69,7 @@ from .venues import (
 )
 from .api import Client
 
-log = get_logger(
-    name=__name__,
-)
+log = get_logger('piker.brokers.binance')
 
 
 # Fee schedule template, mostly for paper engine fees modelling.
@@ -248,16 +245,9 @@ async def handle_order_requests(
 @tractor.context
 async def open_trade_dialog(
     ctx: tractor.Context,
-    loglevel: str = 'warning',
 
 ) -> AsyncIterator[dict[str, Any]]:
 
-    # enable piker.clearing console log for *this* `brokerd` subactor
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
-
     # TODO: how do we set this from the EMS such that
     # positions are loaded from the correct venue on the user
     # stream at startup? (that is in an attempt to support both
@@ -274,20 +264,15 @@ async def open_trade_dialog(
     # do a open_symcache() call.. though maybe we can hide
     # this in a new async version of open_account()?
     async with open_cached_client('binance') as client:
-        subconf: dict|None = client.conf.get(venue_name)
+        subconf: dict = client.conf[venue_name]
+        use_testnet = subconf.get('use_testnet', False)
 
         # XXX: if no futes.api_key or spot.api_key has been set we
         # always fall back to the paper engine!
-        if (
-            not subconf
-            or
-            not subconf.get('api_key')
-        ):
+        if not subconf.get('api_key'):
             await ctx.started('paper')
             return
 
-        use_testnet: bool = subconf.get('use_testnet', False)
-
     async with (
         open_cached_client('binance') as client,
     ):
@@ -450,7 +435,6 @@ async def open_trade_dialog(
             #   - ledger: TransactionLedger
 
             async with (
-                tractor.trionics.collapse_eg(),
                 trio.open_nursery() as tn,
                 ctx.open_stream() as ems_stream,
             ):

piker/brokers/binance/feed.py

@@ -42,12 +42,12 @@ from trio_typing import TaskStatus
 from pendulum import (
     from_timestamp,
 )
+from rapidfuzz import process as fuzzy
 import numpy as np
 import tractor
 
 from piker.brokers import (
     open_cached_client,
-    NoData,
 )
 from piker._cacheables import (
     async_lifo_cache,
@@ -64,9 +64,9 @@ from piker.data._web_bs import (
     open_autorecon_ws,
     NoBsWs,
 )
-from piker.log import get_logger
 from piker.brokers._util import (
     DataUnavailable,
+    get_logger,
 )
 
 from .api import (
@@ -78,7 +78,7 @@ from .venues import (
     get_api_eps,
 )
 
-log = get_logger(name=__name__)
+log = get_logger('piker.brokers.binance')
 
 
 class L1(Struct):
@@ -94,26 +94,22 @@ class L1(Struct):
 
 
 # validation type
-# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Aggregate-Trade-Streams#response-example
 class AggTrade(Struct, frozen=True):
     e: str  # Event type
     E: int  # Event time
     s: str  # Symbol
     a: int  # Aggregate trade ID
     p: float  # Price
-    q: float  # Quantity with all the market trades
+    q: float  # Quantity
     f: int  # First trade ID
     l: int  # noqa Last trade ID
     T: int  # Trade time
     m: bool  # Is the buyer the market maker?
-    M: bool|None = None  # Ignore
-    nq: float|None = None # Normal quantity without the trades involving RPI orders
-    # ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
+    M: bool | None = None  # Ignore
 
 
 async def stream_messages(
     ws: NoBsWs,
-
 ) -> AsyncGenerator[NoBsWs, dict]:
 
     # TODO: match syntax here!
@@ -224,8 +220,6 @@ def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
     }
 
 
-# TODO, why aren't frame resp `log.info()`s showing in upstream
-# code?!
 @acm
 async def open_history_client(
     mkt: MktPair,
@@ -237,8 +231,8 @@ async def open_history_client(
 
         async def get_ohlc(
             timeframe: float,
-            end_dt: datetime|None = None,
-            start_dt: datetime|None = None,
+            end_dt: datetime | None = None,
+            start_dt: datetime | None = None,
 
         ) -> tuple[
             np.ndarray,
@@ -258,36 +252,24 @@ async def open_history_client(
             else:
                 client.mkt_mode = 'spot'
 
-            array: np.ndarray = await client.bars(
-                mkt=mkt,
+            # NOTE: always query using their native symbology!
+            mktid: str = mkt.bs_mktid
+            array = await client.bars(
+                mktid,
                 start_dt=start_dt,
                 end_dt=end_dt,
             )
-            if array.size == 0:
-                raise NoData(
-                    f'No frame for {start_dt} -> {end_dt}\n'
-                )
-
             times = array['time']
-            if not times.any():
-                raise ValueError(
-                    'Bad frame with null-times?\n\n'
-                    f'{times}'
-                )
-
-            # XXX, debug any case where the latest 1m bar we get is
-            # already another "sample's-step-old"..
-            if end_dt is None:
-                inow: int = round(time.time())
-                if (
-                    _time_step := (inow - times[-1])
-                    >
-                    timeframe * 2
-                ):
+            if (
+                end_dt is None
+            ):
+                inow = round(time.time())
+                if (inow - times[-1]) > 60:
                     await tractor.pause()
 
             start_dt = from_timestamp(times[0])
             end_dt = from_timestamp(times[-1])
+
             return array, start_dt, end_dt
 
         yield get_ohlc, {'erlangs': 3, 'rate': 3}
@@ -297,7 +279,7 @@ async def open_history_client(
 async def get_mkt_info(
     fqme: str,
 
-) -> tuple[MktPair, Pair]|None:
+) -> tuple[MktPair, Pair] | None:
 
     # uppercase since kraken bs_mktid is always upper
     if 'binance' not in fqme.lower():
@@ -374,7 +356,7 @@ async def get_mkt_info(
         if 'futes' in mkt_mode:
             assert isinstance(pair, FutesPair)
 
-        dst: Asset|None = assets.get(pair.bs_dst_asset)
+        dst: Asset | None = assets.get(pair.bs_dst_asset)
         if (
             not dst
             # TODO: a known asset DNE list?
@@ -433,7 +415,7 @@ async def subscribe(
     # might get ack from ws server, or maybe some
     # other msg still in transit..
     res = await ws.recv_msg()
-    subid: str|None = res.get('id')
+    subid: str | None = res.get('id')
     if subid:
         assert res['id'] == subid
 
@@ -457,6 +439,7 @@ async def subscribe(
 
 
 async def stream_quotes(
+
     send_chan: trio.abc.SendChannel,
     symbols: list[str],
     feed_is_live: trio.Event,
@@ -468,14 +451,11 @@ async def stream_quotes(
 ) -> None:
 
     async with (
-        tractor.trionics.maybe_raise_from_masking_exc(),
         send_chan as send_chan,
         open_cached_client('binance') as client,
     ):
         init_msgs: list[FeedInit] = []
         for sym in symbols:
-            mkt: MktPair
-            pair: Pair
             mkt, pair = await get_mkt_info(sym)
 
             # build out init msgs according to latest spec
@@ -524,6 +504,7 @@ async def stream_quotes(
 
             # start streaming
             async for typ, quote in msg_gen:
+
                 # period = time.time() - last
                 # hz = 1/period if period else float('inf')
                 # if hz > 60:
@@ -559,7 +540,7 @@ async def open_symbol_search(
                 )
 
                 # repack in fqme-keyed table
-                byfqme: dict[str, Pair] = {}
+                byfqme: dict[start, Pair] = {}
                 for pair in pairs.values():
                     byfqme[pair.bs_fqme] = pair
 

piker/brokers/binance/venues.py

@@ -97,16 +97,6 @@ class Pair(Struct, frozen=True, kw_only=True):
     baseAsset: str
     baseAssetPrecision: int
 
-    permissionSets: list[list[str]]
-
-    # https://developers.binance.com/docs/binance-spot-api-docs#2025-08-26
-    # will become non-optional 2025-08-28?
-    # https://developers.binance.com/docs/binance-spot-api-docs#future-changes
-    pegInstructionsAllowed: bool = False
-
-    # https://developers.binance.com/docs/binance-spot-api-docs#2025-12-02
-    opoAllowed: bool = False
-
     filters: dict[
         str,
         str | int | float,
@@ -152,11 +142,7 @@ class SpotPair(Pair, frozen=True):
     defaultSelfTradePreventionMode: str
     allowedSelfTradePreventionModes: list[str]
     permissions: list[str]
-
-    # can the paint botz creat liq gaps even easier on this asset?
-    # Bp
-    # https://developers.binance.com/docs/binance-spot-api-docs/faqs/order_amend_keep_priority
-    amendAllowed: bool
+    permissionSets: list[list[str]]
 
     # NOTE: see `.data._symcache.SymbologyCache.load()` for why
     ns_path: str = 'piker.brokers.binance:SpotPair'
@@ -195,6 +181,7 @@ class FutesPair(Pair):
     quoteAsset: str  # 'USDT',
     quotePrecision: int  # 8,
     requiredMarginPercent: float  # '5.0000',
+    settlePlan: int  # 0,
     timeInForce: list[str]  # ['GTC', 'IOC', 'FOK', 'GTX'],
     triggerProtect: float  # '0.0500',
     underlyingSubType: list[str]  # ['PoW'],
@@ -223,10 +210,7 @@ class FutesPair(Pair):
                 assert pair == self.pair  # sanity
                 return f'{expiry}'
 
-            case (
-                'PERPETUAL'
-                | 'TRADIFI_PERPETUAL'
-            ):
+            case 'PERPETUAL':
                 return 'PERP'
 
             case '':
@@ -255,10 +239,7 @@ class FutesPair(Pair):
         margin: str = self.marginAsset
 
         match ctype:
-            case (
-                'PERPETUAL'
-                | 'TRADIFI_PERPETUAL'
-            ):
+            case 'PERPETUAL':
                 return f'{margin}M'
 
             case (

piker/brokers/cli.py

@@ -27,12 +27,14 @@ import click
 import trio
 import tractor
 
-from piker.cli import cli
-from piker import watchlists as wl
-from piker.log import (
+from ..cli import cli
+from .. import watchlists as wl
+from ..log import (
     colorize_json,
+)
+from ._util import (
+    log,
     get_console_log,
-    get_logger,
 )
 from ..service import (
     maybe_spawn_brokerd,
@@ -43,15 +45,12 @@ from ..brokers import (
     get_brokermod,
     data,
 )
-
-log = get_logger(
-    name=__name__,
-)
-
 DEFAULT_BROKER = 'binance'
+
 _config_dir = click.get_app_dir('piker')
 _watchlists_data_path = os.path.join(_config_dir, 'watchlists.json')
 
+
 OK = '\033[92m'
 WARNING = '\033[93m'
 FAIL = '\033[91m'
@@ -346,10 +345,7 @@ def contracts(ctx, loglevel, broker, symbol, ids):
 
     '''
     brokermod = get_brokermod(broker)
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
+    get_console_log(loglevel)
 
     contracts = trio.run(partial(core.contracts, brokermod, symbol))
     if not ids:
@@ -475,18 +471,13 @@ def search(
 
     '''
     # global opts
-    brokermods: list[ModuleType] = list(config['brokermods'].values())
-
-    # TODO: this is coming from the `search --pdb` NOT from
-    # the `piker --pdb` XD ..
-    # -[ ] pull from the parent click ctx's values..dumdum
-    # assert pdb
-    loglevel: str = config['loglevel']
+    brokermods = list(config['brokermods'].values())
 
     # define tractor entrypoint
     async def main(func):
+
         async with maybe_open_pikerd(
-            loglevel=loglevel,
+            loglevel=config['loglevel'],
             debug_mode=pdb,
         ):
             return await func()
@@ -499,7 +490,6 @@ def search(
                 core.symbol_search,
                 brokermods,
                 pattern,
-                loglevel=loglevel,
             ),
         )
 

piker/brokers/core.py

@@ -22,26 +22,20 @@ routines should be primitive data types where possible.
 """
 import inspect
 from types import ModuleType
-from typing import (
-    Any,
-)
+from typing import List, Dict, Any, Optional
 
 import trio
 
-from piker.log import get_logger
+from ._util import log
 from . import get_brokermod
 from ..service import maybe_spawn_brokerd
 from . import open_cached_client
 from ..accounting import MktPair
 
-log = get_logger(name=__name__)
-
 
 async def api(brokername: str, methname: str, **kwargs) -> dict:
-    '''
-    Make (proxy through) a broker API call by name and return its result.
-
-    '''
+    """Make (proxy through) a broker API call by name and return its result.
+    """
     brokermod = get_brokermod(brokername)
     async with brokermod.get_client() as client:
         meth = getattr(client, methname, None)
@@ -68,14 +62,10 @@ async def api(brokername: str, methname: str, **kwargs) -> dict:
 
 async def stocks_quote(
     brokermod: ModuleType,
-    tickers: list[str]
-
-) -> dict[str, dict[str, Any]]:
-    '''
-    Return a `dict` of snapshot quotes for the provided input
-    `tickers`: a `list` of fqmes.
-
-    '''
+    tickers: List[str]
+) -> Dict[str, Dict[str, Any]]:
+    """Return quotes dict for ``tickers``.
+    """
     async with brokermod.get_client() as client:
         return await client.quote(tickers)
 
@@ -84,15 +74,13 @@ async def stocks_quote(
 async def option_chain(
     brokermod: ModuleType,
     symbol: str,
-    date: str|None = None,
-) -> dict[str, dict[str, dict[str, Any]]]:
-    '''
-    Return option chain for ``symbol`` for ``date``.
+    date: Optional[str] = None,
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
+    """Return option chain for ``symbol`` for ``date``.
 
     By default all expiries are returned. If ``date`` is provided
     then contract quotes for that single expiry are returned.
-
-    '''
+    """
     async with brokermod.get_client() as client:
         if date:
             id = int((await client.tickers2ids([symbol]))[symbol])
@@ -110,7 +98,7 @@ async def option_chain(
 # async def contracts(
 #     brokermod: ModuleType,
 #     symbol: str,
-# ) -> dict[str, dict[str, dict[str, Any]]]:
+# ) -> Dict[str, Dict[str, Dict[str, Any]]]:
 #     """Return option contracts (all expiries) for ``symbol``.
 #     """
 #     async with brokermod.get_client() as client:
@@ -122,24 +110,15 @@ async def bars(
     brokermod: ModuleType,
     symbol: str,
     **kwargs,
-) -> dict[str, dict[str, dict[str, Any]]]:
-    '''
-    Return option contracts (all expiries) for ``symbol``.
-
-    '''
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
+    """Return option contracts (all expiries) for ``symbol``.
+    """
     async with brokermod.get_client() as client:
         return await client.bars(symbol, **kwargs)
 
 
-async def search_w_brokerd(
-    name: str,
-    pattern: str,
-) -> dict:
+async def search_w_brokerd(name: str, pattern: str) -> dict:
 
-    # TODO: WHY NOT WORK!?!
-    # when we `step` through the next block?
-    # import tractor
-    # await tractor.pause()
     async with open_cached_client(name) as client:
 
         # TODO: support multiple asset type concurrent searches.
@@ -149,15 +128,14 @@ async def search_w_brokerd(
 async def symbol_search(
     brokermods: list[ModuleType],
     pattern: str,
-    loglevel: str = 'warning',
     **kwargs,
 
-) -> dict[str, dict[str, dict[str, Any]]]:
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
     '''
     Return symbol info from broker.
 
     '''
-    results: list[str] = []
+    results = []
 
     async def search_backend(
         brokermod: ModuleType
@@ -165,13 +143,6 @@ async def symbol_search(
 
         brokername: str = mod.name
 
-        # TODO: figure this the FUCK OUT
-        # -> ok so obvi in the root actor any async task that's
-        # spawned outside the main tractor-root-actor task needs to
-        # call this..
-        # await tractor.devx._debug.maybe_init_greenback()
-        # tractor.pause_from_sync()
-
         async with maybe_spawn_brokerd(
             mod.name,
             infect_asyncio=getattr(
@@ -179,7 +150,6 @@ async def symbol_search(
                 '_infect_asyncio',
                 False,
             ),
-            loglevel=loglevel
         ) as portal:
 
             results.append((
@@ -192,6 +162,7 @@ async def symbol_search(
             ))
 
     async with trio.open_nursery() as n:
+
         for mod in brokermods:
             n.start_soon(search_backend, mod.name)
 
@@ -201,13 +172,11 @@ async def symbol_search(
 async def mkt_info(
     brokermod: ModuleType,
     fqme: str,
-
     **kwargs,
 
 ) -> MktPair:
     '''
-    Return the `piker.accounting.MktPair` info struct from a given
-    backend broker tradable src/dst asset pair.
+    Return MktPair info from broker including src and dst assets.
 
     '''
     async with open_cached_client(brokermod.name) as client:

piker/brokers/data.py

@@ -41,15 +41,12 @@ import tractor
 from tractor.experimental import msgpub
 from async_generator import asynccontextmanager
 
-from piker.log import(
-    get_logger,
+from ._util import (
+    log,
     get_console_log,
 )
 from . import get_brokermod
 
-log = get_logger(
-    name='piker.brokers.binance',
-)
 
 async def wait_for_network(
     net_func: Callable,
@@ -246,10 +243,7 @@ async def start_quote_stream(
 
     '''
     # XXX: why do we need this again?
-    get_console_log(
-        level=tractor.current_actor().loglevel,
-        name=__name__,
-    )
+    get_console_log(tractor.current_actor().loglevel)
 
     # pull global vars from local actor
     symbols = list(symbols)

piker/brokers/deribit/api.py

@@ -31,7 +31,7 @@ from typing import (
     Callable,
 )
 
-from pendulum import now
+import pendulum
 import trio
 from trio_typing import TaskStatus
 from rapidfuzz import process as fuzzy
@@ -39,7 +39,6 @@ import numpy as np
 from tractor.trionics import (
     broadcast_receiver,
     maybe_open_context
-    collapse_eg,
 )
 from tractor import to_asyncio
 # XXX WOOPS XD
@@ -433,7 +432,6 @@ async def get_client(
 ) -> Client:
 
     async with (
-        collapse_eg(),
         trio.open_nursery() as n,
         open_jsonrpc_session(
             _testnet_ws_url, dtype=JSONRPCResult) as json_rpc

piker/brokers/ib/_util.py

@@ -20,11 +20,6 @@ runnable script-programs.
 
 '''
 from __future__ import annotations
-from datetime import (  # noqa
-    datetime,
-    date,
-    tzinfo as TzInfo,
-)
 from functools import partial
 from typing import (
     Literal,
@@ -34,13 +29,13 @@ import subprocess
 
 import tractor
 
-from piker.log import get_logger
+from piker.brokers._util import get_logger
 
 if TYPE_CHECKING:
     from .api import Client
-    import i3ipc
+    from ib_insync import IB
 
-log = get_logger(name=__name__)
+log = get_logger('piker.brokers.ib')
 
 _reset_tech: Literal[
     'vnc',
@@ -53,39 +48,8 @@ _reset_tech: Literal[
 ] = 'vnc'
 
 
-no_setup_msg:str = (
-    'No data reset hack test setup for {vnc_sockaddr}!\n'
-    'See config setup tips @\n'
-    'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
-)
-
-
-def try_xdo_manual(
-    client: Client,
-):
-    '''
-    Do the "manual" `xdo`-based screen switch + click
-    combo since apparently the `asyncvnc` client ain't workin..
-
-    Note this is only meant as a backup method for Xorg users,
-    ideally you can use a real vnc client and the `vnc_click_hack()`
-    impl!
-
-    '''
-    global _reset_tech
-    try:
-        i3ipc_xdotool_manual_click_hack()
-        _reset_tech = 'i3ipc_xdotool'
-        return True
-    except OSError:
-        vnc_sockaddr: str = client.conf.vnc_addrs
-        log.exception(
-            no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
-        )
-        return False
-
-
 async def data_reset_hack(
+    # vnc_host: str,
     client: Client,
     reset_type: Literal['data', 'connection'],
 
@@ -117,60 +81,65 @@ async def data_reset_hack(
           that need to be wrangle.
 
     '''
+    ib_client: IB = client.ib
+
     # look up any user defined vnc socket address mapped from
     # a particular API socket port.
-    vnc_addrs: tuple[str]|None = client.conf.get('vnc_addrs')
-    if not vnc_addrs:
+    api_port: str = str(ib_client.client.port)
+    vnc_host: str
+    vnc_port: int
+    vnc_sockaddr: tuple[str] | None = client.conf.get('vnc_addrs')
+
+    no_setup_msg:str = (
+        f'No data reset hack test setup for {vnc_sockaddr}!\n'
+        'See config setup tips @\n'
+        'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
+    )
+
+    if not vnc_sockaddr:
         log.warning(
-            no_setup_msg.format(vnc_sockaddr=client.conf)
+            no_setup_msg
             +
-            'REQUIRES A `vnc_addrs: array` ENTRY'
+            f'REQUIRES A `vnc_addrs: array` ENTRY'
         )
 
+    vnc_host, vnc_port = vnc_sockaddr.get(
+        api_port,
+        ('localhost', 3003)
+    )
     global _reset_tech
+
     match _reset_tech:
         case 'vnc':
             try:
                 await tractor.to_asyncio.run_task(
                     partial(
                         vnc_click_hack,
-                        client=client,
+                        host=vnc_host,
+                        port=vnc_port,
                     )
                 )
-            except (
-                OSError,  # no VNC server avail..
-                PermissionError,  # asyncvnc pw fail..
-            ):
+            except OSError:
+                if vnc_host != 'localhost':
+                    log.warning(no_setup_msg)
+                    return False
+
                 try:
                     import i3ipc  # noqa  (since a deps dynamic check)
                 except ModuleNotFoundError:
-                    log.warning(
-                        no_setup_msg.format(vnc_sockaddr=client.conf)
-                    )
+                    log.warning(no_setup_msg)
                     return False
 
-                # XXX, Xorg only workaround..
-                # TODO? remove now that we have `pyvnc`?
-                # if vnc_host not in {
-                #     'localhost',
-                #     '127.0.0.1',
-                # }:
-                #     focussed, matches = i3ipc_fin_wins_titled()
-                #     if not matches:
-                #         log.warning(
-                #             no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
-                #         )
-                #         return False
-                #     else:
-                #         try_xdo_manual(vnc_sockaddr)
-
-                # localhost but no vnc-client or it borked..
-                else:
-                    try_xdo_manual(client)
+                try:
+                    i3ipc_xdotool_manual_click_hack()
+                    _reset_tech = 'i3ipc_xdotool'
+                    return True
+                except OSError:
+                    log.exception(no_setup_msg)
+                    return False
 
         case 'i3ipc_xdotool':
-            try_xdo_manual(client)
-            # i3ipc_xdotool_manual_click_hack()
+            i3ipc_xdotool_manual_click_hack()
 
         case _ as tech:
             raise RuntimeError(f'{tech} is not supported for reset tech!?')
@@ -180,66 +149,21 @@ async def data_reset_hack(
 
 
 async def vnc_click_hack(
-    client: Client,
-    reset_type: str = 'data',
-    pw: str|None = None,
-
+    host: str,
+    port: int,
+    reset_type: str = 'data'
 ) -> None:
     '''
     Reset the data or network connection for the VNC attached
-    ib-gateway using a (magic) keybinding combo.
-
-    A vnc-server password can be set either by an input `pw` param or
-    set in the client's config with the latter loaded from the user's
-    `brokers.toml` in a vnc-addrs-port-mapping section,
-
-    .. code:: toml
-
-        [ib.vnc_addrs]
-        4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
+    ib gateway using magic combos.
 
     '''
-    api_port: str = str(client.ib.client.port)
-    conf: dict = client.conf
-    vnc_addrs: dict[int, tuple] = conf.get('vnc_addrs')
-    if not vnc_addrs:
-        return None
-
-    addr_entry: dict|tuple = vnc_addrs.get(
-        api_port,
-        ('localhost', 5900)  # a typical default
-    )
-    if pw is None:
-        match addr_entry:
-            case (
-                host,
-                port,
-            ):
-                pass
-
-            case {
-                'host': host,
-                'port': port,
-                'pw': pw
-            }:
-                pass
-
-            case _:
-                raise ValueError(
-                    f'Invalid `ib.vnc_addrs` entry ?\n'
-                    f'{addr_entry!r}\n'
-                )
     try:
-        from pyvnc import (
-            AsyncVNCClient,
-            VNCConfig,
-            Point,
-            MOUSE_BUTTON_LEFT,
-        )
+        import asyncvnc
     except ModuleNotFoundError:
         log.warning(
             "In order to leverage `piker`'s built-in data reset hacks, install "
-            "the `pyvnc` project: https://github.com/regulad/pyvnc.git"
+            "the `asyncvnc` project: https://github.com/barneygale/asyncvnc"
         )
         return
 
@@ -250,80 +174,24 @@ async def vnc_click_hack(
         'connection': 'r'
     }[reset_type]
 
-    with tractor.devx.open_crash_handler(
-        ignore={TimeoutError,},
-    ):
-        client = await AsyncVNCClient.connect(
-            VNCConfig(
-                host=host,
-                port=port,
-                password=pw,
-            )
+    async with asyncvnc.connect(
+        host,
+        port=port,
+
+        # TODO: doesn't work see:
+        # https://github.com/barneygale/asyncvnc/issues/7
+        # password='ibcansmbz',
+
+    ) as client:
+
+        # move to middle of screen
+        # 640x1800
+        client.mouse.move(
+            x=500,
+            y=500,
         )
-        async with client:
-            # move to middle of screen
-            # 640x1800
-            await client.move(
-                Point(
-                    500,
-                    500,
-                )
-            )
-            # ensure the ib-gw window is active
-            await client.click(MOUSE_BUTTON_LEFT)
-            # send the hotkeys combo B)
-            await client.press('Ctrl', 'Alt', key)  # keys are stacked
-
-
-def i3ipc_fin_wins_titled(
-    titles: list[str] = [
-        'Interactive Brokers',  # tws running in i3
-        'IB Gateway',  # gw running in i3
-        # 'IB',  # gw running in i3 (newer version?)
-
-        # !TODO, remote vnc instance
-        # -[ ] something in title (or other Con-props) that indicates
-        #   this is explicitly for ibrk sw?
-        #  |_[ ]  !can use modden spawn eventually!
-        'TigerVNC',
-        # 'vncviewer',  # the terminal..
-    ],
-) -> tuple[
-    i3ipc.Con,  # orig focussed win
-    list[tuple[str, i3ipc.Con]],  # matching wins by title
-]:
-    '''
-    Attempt to find a local-DE window titled with an entry in
-    `titles`.
-
-    If found deliver the current focussed window and all matching
-    `i3ipc.Con`s in a list.
-
-    '''
-    import i3ipc
-    ipc = i3ipc.Connection()
-
-    # TODO: might be worth offering some kinda api for grabbing
-    # the window id from the pid?
-    # https://stackoverflow.com/a/2250879
-    tree = ipc.get_tree()
-    focussed: i3ipc.Con = tree.find_focused()
-
-    matches: list[i3ipc.Con] = []
-    for name in titles:
-        results = tree.find_titled(name)
-        print(f'results for {name}: {results}')
-        if results:
-            con = results[0]
-            matches.append((
-                name,
-                con,
-            ))
-
-    return (
-        focussed,
-        matches,
-    )
+        client.mouse.click()
+        client.keyboard.press('Ctrl', 'Alt', key)  # keys are stacked
 
 
 def i3ipc_xdotool_manual_click_hack() -> None:
@@ -331,55 +199,67 @@ def i3ipc_xdotool_manual_click_hack() -> None:
     Do the data reset hack but expecting a local X-window using `xdotool`.
 
     '''
-    focussed, matches = i3ipc_fin_wins_titled()
-    try:
-        orig_win_id = focussed.window
-    except AttributeError:
-        # XXX if .window cucks we prolly aren't intending to
-        # use this and/or just woke up from suspend..
-        log.exception('xdotool invalid usage ya ??\n')
-        return
+    import i3ipc
+    i3 = i3ipc.Connection()
+
+    # TODO: might be worth offering some kinda api for grabbing
+    # the window id from the pid?
+    # https://stackoverflow.com/a/2250879
+    t = i3.get_tree()
+
+    orig_win_id = t.find_focused().window
+
+    # for tws
+    win_names: list[str] = [
+        'Interactive Brokers',  # tws running in i3
+        'IB Gateway',  # gw running in i3
+        # 'IB',  # gw running in i3 (newer version?)
+    ]
 
     try:
-        for name, con in matches:
-            print(f'Resetting data feed for {name}')
-            win_id = str(con.window)
-            w, h = con.rect.width, con.rect.height
+        for name in win_names:
+            results = t.find_titled(name)
+            print(f'results for {name}: {results}')
+            if results:
+                con = results[0]
+                print(f'Resetting data feed for {name}')
+                win_id = str(con.window)
+                w, h = con.rect.width, con.rect.height
 
-            # TODO: seems to be a few libs for python but not sure
-            # if they support all the sub commands we need, order of
-            # most recent commit history:
-            # https://github.com/rr-/pyxdotool
-            # https://github.com/ShaneHutter/pyxdotool
-            # https://github.com/cphyc/pyxdotool
+                # TODO: seems to be a few libs for python but not sure
+                # if they support all the sub commands we need, order of
+                # most recent commit history:
+                # https://github.com/rr-/pyxdotool
+                # https://github.com/ShaneHutter/pyxdotool
+                # https://github.com/cphyc/pyxdotool
 
-            # TODO: only run the reconnect (2nd) kc on a detected
-            # disconnect?
-            for key_combo, timeout in [
-                # only required if we need a connection reset.
-                # ('ctrl+alt+r', 12),
-                # data feed reset.
-                ('ctrl+alt+f', 6)
-            ]:
-                subprocess.call([
-                    'xdotool',
-                    'windowactivate', '--sync', win_id,
+                # TODO: only run the reconnect (2nd) kc on a detected
+                # disconnect?
+                for key_combo, timeout in [
+                    # only required if we need a connection reset.
+                    # ('ctrl+alt+r', 12),
+                    # data feed reset.
+                    ('ctrl+alt+f', 6)
+                ]:
+                    subprocess.call([
+                        'xdotool',
+                        'windowactivate', '--sync', win_id,
 
-                    # move mouse to bottom left of window (where
-                    # there should be nothing to click).
-                    'mousemove_relative', '--sync', str(w-4), str(h-4),
+                        # move mouse to bottom left of window (where
+                        # there should be nothing to click).
+                        'mousemove_relative', '--sync', str(w-4), str(h-4),
 
-                    # NOTE: we may need to stick a `--retry 3` in here..
-                    'click', '--window', win_id,
-                    '--repeat', '3', '1',
+                        # NOTE: we may need to stick a `--retry 3` in here..
+                        'click', '--window', win_id,
+                        '--repeat', '3', '1',
 
-                    # hackzorzes
-                    'key', key_combo,
-                    ],
-                    timeout=timeout,
-                )
+                        # hackzorzes
+                        'key', key_combo,
+                        ],
+                        timeout=timeout,
+                    )
 
-        # re-activate and focus original window
+    # re-activate and focus original window
         subprocess.call([
             'xdotool',
             'windowactivate', '--sync', str(orig_win_id),

piker/brokers/ib/api.py

@@ -48,13 +48,11 @@ from bidict import bidict
 import trio
 import tractor
 from tractor import to_asyncio
-from tractor import trionics
 from pendulum import (
+    from_timestamp,
     DateTime,
     Duration,
     duration as mk_duration,
-    from_timestamp,
-    Interval,
 )
 from eventkit import Event
 from ib_insync import (
@@ -92,15 +90,10 @@ from .symbols import (
     _exch_skip_list,
     _futes_venues,
 )
-from ...log import get_logger
-from .venues import (
-    is_venue_open,
-    sesh_times,
-    is_venue_closure,
-)
-
-log = get_logger(
-    name=__name__,
+from ._util import (
+    log,
+    # only for the ib_sync internal logging
+    get_logger,
 )
 
 _bar_load_dtype: list[tuple[str, type]] = [
@@ -186,7 +179,7 @@ class NonShittyIB(IB):
         # override `ib_insync` internal loggers so we can see wtf
         # it's doing..
         self._logger = get_logger(
-            name=__name__,
+            'ib_insync.ib',
         )
         self._createEvents()
 
@@ -194,7 +187,7 @@ class NonShittyIB(IB):
         self.wrapper = NonShittyWrapper(self)
         self.client = ib_client.Client(self.wrapper)
         self.client._logger = get_logger(
-            name='ib_insync.client',
+            'ib_insync.client',
         )
 
         # self.errorEvent += self._onError
@@ -266,16 +259,6 @@ def remove_handler_on_err(
         event.disconnect(handler)
 
 
-# (originally?) i thot that,
-# > "EST in ISO 8601 format is required.."
-#
-# XXX, but see `ib_async`'s impl,
-# - `ib_async.ib.IB.reqHistoricalDataAsync()`
-# - `ib_async.util.formatIBDatetime()`
-# below is EPOCH.
-_iso8601_epoch_in_est: str = "1970-01-01T00:00:00.000000-05:00"
-
-
 class Client:
     '''
     IB wrapped for our broker backend API.
@@ -304,31 +287,9 @@ class Client:
         self.conf = config
 
         # NOTE: the ib.client here is "throttled" to 45 rps by default
-        self.ib: IB = ib
+        self.ib = ib
         self.ib.RaiseRequestErrors: bool = True
 
-        # self._acnt_names: set[str] = {}
-        self._acnt_names: list[str] = []
-
-    @property
-    def acnts(self) -> list[str]:
-        # return list(self._acnt_names)
-        return self._acnt_names
-
-    def __repr__(self) -> str:
-        return (
-            f'<{type(self).__name__}('
-            f'ib={self.ib} '
-            f'acnts={self.acnts}'
-
-            # TODO: we need to mask out acnt-#s and other private
-            # infos if we're going to console this!
-            # f' |_.conf:\n'
-            # f'    {pformat(self.conf)}\n'
-
-            ')>'
-        )
-
     async def get_fills(self) -> list[Fill]:
         '''
         Return list of rents `Fills` from trading session.
@@ -349,32 +310,22 @@ class Client:
         self,
         fqme: str,
 
-        # EST in ISO 8601 format is required..
-        # XXX, see `ib_async.ib.IB.reqHistoricalDataAsync()`
-        # below is EPOCH.
-        start_dt: datetime|None = None,  # _iso8601_epoch_in_est,
-        end_dt: datetime|None = None,
+        # EST in ISO 8601 format is required... below is EPOCH
+        start_dt: datetime | str = "1970-01-01T00:00:00.000000-05:00",
+        end_dt: datetime | str = "",
 
         # ohlc sample period in seconds
         sample_period_s: int = 1,
 
         # optional "duration of time" equal to the
         # length of the returned history frame.
-        duration: str|None = None,
+        duration: str | None = None,
 
         **kwargs,
 
-    ) -> tuple[
-        BarDataList,
-        np.ndarray,
-        Duration,
-    ]:
+    ) -> tuple[BarDataList, np.ndarray, Duration]:
         '''
-        Retreive the `fqme`'s OHLCV-bars for the time-range "until `end_dt`".
-
-        Notes:
-        - IB's api doesn't support a `start_dt` (which is why default
-          is null) so we only use it for bar-frame duration checking.
+        Retreive OHLCV bars for a fqme over a range to the present.
 
         '''
         # See API docs here:
@@ -389,19 +340,13 @@ class Client:
 
         dt_duration: Duration = (
             duration
-            or
-            default_dt_duration
+            or default_dt_duration
         )
 
         # TODO: maybe remove all this?
         global _enters
-        if end_dt is None:
-            end_dt: str = ''
-
-        else:
-            est_end_dt = end_dt.in_tz('EST')
-            if est_end_dt != end_dt:
-                breakpoint()
+        if not end_dt:
+            end_dt = ''
 
         _enters += 1
 
@@ -431,155 +376,89 @@ class Client:
             # whatToShow='MIDPOINT',
             # whatToShow='TRADES',
         )
+        log.info(
+            f'REQUESTING {ib_duration_str} worth {bar_size} BARS\n'
+            f'fqme: {fqme}\n'
+            f'global _enters: {_enters}\n'
+            f'kwargs: {pformat(kwargs)}\n'
+        )
+
         bars = await self.ib.reqHistoricalDataAsync(
             **kwargs,
         )
 
-        query_info: str = (
-            f'REQUESTING IB history BARS\n'
-            f'    ------ - ------\n'
-            f'dt_duration: {dt_duration}\n'
-            f'ib_duration_str: {ib_duration_str}\n'
-            f'bar_size: {bar_size}\n'
-            f'fqme: {fqme}\n'
-            f'actor-global _enters: {_enters}\n'
-            f'kwargs: {pformat(kwargs)}\n'
-        )
         # tail case if no history for range or none prior.
-        # NOTE: there's actually 3 cases here to handle (and
-        # this should be read alongside the implementation of
-        # `.reqHistoricalDataAsync()`):
-        # - a timeout occurred in which case insync internals return
-        #   an empty list thing with bars.clear()...
-        # - no data exists for the period likely due to
-        #   a weekend, holiday or other non-trading period prior to
-        #   ``end_dt`` which exceeds the ``duration``,
-        # - LITERALLY this is the start of the mkt's history!
         if not bars:
-            # TODO: figure out wut's going on here.
+            # NOTE: there's actually 3 cases here to handle (and
+            # this should be read alongside the implementation of
+            # `.reqHistoricalDataAsync()`):
+            # - a timeout occurred in which case insync internals return
+            #   an empty list thing with bars.clear()...
+            # - no data exists for the period likely due to
+            #   a weekend, holiday or other non-trading period prior to
+            #   ``end_dt`` which exceeds the ``duration``,
+            # - LITERALLY this is the start of the mkt's history!
 
-            # TODO: is this handy, a sync requester for tinkering
-            # with empty frame cases?
-            # def get_hist():
-            #     return self.ib.reqHistoricalData(**kwargs)
-            # import pdbp
-            # pdbp.set_trace()
 
-            log.critical(
-                'STUPID IB SAYS NO HISTORY\n\n'
-                + query_info
-            )
+            # sync requester for debugging empty frame cases
+            def get_hist():
+                return self.ib.reqHistoricalData(**kwargs)
+
+            assert get_hist
+            import pdbp
+            pdbp.set_trace()
 
-            # TODO: we could maybe raise `NoData` instead if we
-            # rewrite the method in the first case?
-            # right now there's no way to detect a timeout..
             return [], np.empty(0), dt_duration
+            # TODO: we could maybe raise ``NoData`` instead if we
+            # rewrite the method in the first case? right now there's no
+            # way to detect a timeout.
 
-        log.info(query_info)
-
-        # ------ GAP-DETECTION ------
-        # NOTE XXX: ensure minimum duration in bars?
-        # => recursively call this method until we get at least as
-        #   many bars such that they sum in aggregate to the the
-        #   desired total time (duration) at most.
-        #  - if you query over a gap and get no data
-        #    that may short circuit the history
-        if end_dt:
+        # NOTE XXX: ensure minimum duration in bars B)
+        # => we recursively call this method until we get at least
+        # as many bars such that they sum in aggregate to the the
+        # desired total time (duration) at most.
+        # XXX XXX XXX
+        # WHY DID WE EVEN NEED THIS ORIGINALLY!?
+        # XXX XXX XXX
+        # - if you query over a gap and get no data
+        #   that may short circuit the history 
+        if (
+            end_dt
+            and False
+        ):
             nparr: np.ndarray = bars_to_np(bars)
             times: np.ndarray = nparr['time']
             first: float = times[0]
-            last: float = times[-1]
-            # frame_dur: float = times[-1] - first
-
-            details: ContractDetails = (
-                await self.ib.reqContractDetailsAsync(contract)
-            )[0]
-            # convert to makt-native tz
-            tz: str = details.timeZoneId
-            end_dt = end_dt.in_tz(tz)
-            first_dt: DateTime = from_timestamp(first).in_tz(tz)
-            last_dt: DateTime = from_timestamp(last).in_tz(tz)
-            tdiff: int = (
-                last_dt
-                -
-                first_dt
-            ).in_seconds() + sample_period_s
-            _open_now: bool = is_venue_open(
-                con_deats=details,
-            )
-
-            # XXX, do gap detections.
-            has_closure_gap: bool = False
-            if (
-                last_dt.add(seconds=sample_period_s)
-                <
-                end_dt
-            ):
-                open_time, close_time = sesh_times(details)
-                # XXX, always calc gap in mkt-venue-local timezone
-                gap: Interval = end_dt - last_dt
-                if not (
-                    has_closure_gap := is_venue_closure(
-                    gap=gap,
-                    con_deats=details,
-                    time_step_s=sample_period_s,
-                )):
-                    log.warning(
-                        f'Invalid non-closure gap for {fqme!r} ?!?\n'
-                        f'is-open-now: {_open_now}\n'
-                        f'\n'
-                        f'{gap}\n'
-                    )
-                    log.warning(
-                        f'Detected NON venue-closure GAP ??\n'
-                        f'{gap}\n'
-                    )
-                    breakpoint()
-                else:
-                    assert has_closure_gap
-                    log.debug(
-                        f'Detected venue closure gap (weekend),\n'
-                        f'{gap}\n'
-                    )
+            tdiff: float = times[-1] - first
 
             if (
-                start_dt is None
-                and (
-                    tdiff
-                    <
-                    dt_duration.in_seconds()
-                )
-                and
-                not has_closure_gap
+                # len(bars) * sample_period_s) < dt_duration.in_seconds()
+                tdiff < dt_duration.in_seconds()
+                # and False
             ):
-                log.error(
+                end_dt: DateTime = from_timestamp(first)
+                log.warning(
                     f'Frame result was shorter then {dt_duration}!?\n'
+                    'Recursing for more bars:\n'
                     f'end_dt: {end_dt}\n'
                     f'dt_duration: {dt_duration}\n'
-                    # f'\n'
-                    # f'Recursing for more bars:\n'
                 )
-                # XXX, debug!
-                breakpoint()
-                # XXX ? TODO? recursively try to re-request?
-                # => i think *NO* right?
-                #
-                # (
-                #     r_bars,
-                #     r_arr,
-                #     r_duration,
-                # ) = await self.bars(
-                #     fqme,
-                #     start_dt=start_dt,
-                #     end_dt=end_dt,
-                #     sample_period_s=sample_period_s,
+                (
+                    r_bars,
+                    r_arr,
+                    r_duration,
+                ) = await self.bars(
+                    fqme,
+                    start_dt=start_dt,
+                    end_dt=end_dt,
+                    sample_period_s=sample_period_s,
 
-                #     # TODO: make a table for Duration to
-                #     # the ib str values in order to use this?
-                #     # duration=duration,
-                # )
-                # r_bars.extend(bars)
-                # bars = r_bars
+                    # TODO: make a table for Duration to
+                    # the ib str values in order to use this?
+                    # duration=duration,
+                )
+                r_bars.extend(bars)
+                bars = r_bars
 
         nparr: np.ndarray = bars_to_np(bars)
 
@@ -806,8 +685,8 @@ class Client:
 
     async def find_contracts(
         self,
-        pattern: str|None = None,
-        contract: Contract|None = None,
+        pattern: str | None = None,
+        contract: Contract | None = None,
         qualify: bool = True,
         err_on_qualify: bool = True,
 
@@ -874,16 +753,9 @@ class Client:
         # crypto$
         elif exch == 'PAXOS':  # btc.paxos
             con = Crypto(
-                symbol=symbol.upper(),
-                currency='USD',
-                exchange='PAXOS',
+                symbol=symbol,
+                currency=currency,
             )
-            # XXX, on `ib_insync` when first tried this,
-            # > Error 10299, reqId 141: Expected what to show is
-            # > AGGTRADES, please use that instead of TRADES.,
-            # > contract: Crypto(conId=479624278, symbol='BTC',
-            # > exchange='PAXOS', currency='USD',
-            # > localSymbol='BTC.USD', tradingClass='BTC')
 
         # stonks
         else:
@@ -959,7 +831,7 @@ class Client:
         self,
         fqme: str,
 
-    ) -> datetime|None:
+    ) -> datetime | None:
         '''
         Return the first datetime stamp for `fqme` or `None`
         on request failure.
@@ -1015,7 +887,7 @@ class Client:
         tries: int = 100,
         raise_on_timeout: bool = False,
 
-    ) -> Ticker|None:
+    ) -> Ticker | None:
         '''
         Return a single (snap) quote for symbol.
 
@@ -1027,7 +899,7 @@ class Client:
         ready: ticker.TickerUpdateEvent = ticker.updateEvent
 
         # ensure a last price gets filled in before we deliver quote
-        timeouterr: Exception|None = None
+        timeouterr: Exception | None = None
         warnset: bool = False
         for _ in range(tries):
 
@@ -1041,7 +913,6 @@ class Client:
                     )
                     if tkr:
                         break
-
                 except TimeoutError as err:
                     timeouterr = err
                     await asyncio.sleep(0.01)
@@ -1050,25 +921,18 @@ class Client:
                 else:
                     if not warnset:
                         log.warning(
-                            f'Quote req timed out..\n'
-                            f'Maybe the venue is closed?\n'
-                            f'\n'
+                            f'Quote req timed out..maybe venue is closed?\n'
                             f'{asdict(contract)}'
                         )
                         warnset = True
 
             else:
-                log.info(
-                    'Got first quote for contract\n'
-                    f'{contract}\n'
-                )
+                log.info(f'Got first quote for {contract}')
                 break
         else:
-            if (
-                timeouterr
-                and
-                raise_on_timeout
-            ):
+            if timeouterr and raise_on_timeout:
+                import pdbp
+                pdbp.set_trace()
                 raise timeouterr
 
             if not warnset:
@@ -1127,12 +991,8 @@ class Client:
                     outsideRth=True,
 
                     optOutSmartRouting=True,
-                    # TODO: need to understand this setting better as
-                    # it pertains to shit ass mms..
                     routeMarketableToBbo=True,
-
                     designatedLocation='SMART',
-
                     # TODO: make all orders GTC?
                     # https://interactivebrokers.github.io/tws-api/classIBApi_1_1Order.html#a95539081751afb9980f4c6bd1655a6ba
                     # goodTillDate=f"yyyyMMdd-HH:mm:ss",
@@ -1260,8 +1120,8 @@ def get_config() -> dict[str, Any]:
     names = list(accounts.keys())
     accts = section['accounts'] = bidict(accounts)
     log.info(
-        f'{path} defines {len(accts)} account aliases:\n'
-        f'{pformat(names)}\n'
+        f'brokers.toml defines {len(accts)} accounts: '
+        f'{pformat(names)}'
     )
 
     if section is None:
@@ -1284,7 +1144,7 @@ async def load_aio_clients(
     # the API TCP in `ib_insync` connection can be flaky af so instead
     # retry a few times to get the client going..
     connect_retries: int = 3,
-    connect_timeout: float = 30,  # in case a remote-host
+    connect_timeout: float = 10,
     disconnect_on_exit: bool = True,
 
 ) -> dict[str, Client]:
@@ -1328,7 +1188,7 @@ async def load_aio_clients(
         try_ports = list(try_ports.values())
 
     _err = None
-    accounts_def: dict[str, str] = config.load_accounts(['ib'])
+    accounts_def = config.load_accounts(['ib'])
     ports = try_ports if port is None else [port]
     combos = list(itertools.product(hosts, ports))
     accounts_found: dict[str, Client] = {}
@@ -1353,12 +1213,6 @@ async def load_aio_clients(
 
         for i in range(connect_retries):
             try:
-                log.info(
-                    'Trying `ib_async` connect\n'
-                    f'{host}: {port}\n'
-                    f'clientId: {client_id}\n'
-                    f'timeout: {connect_timeout}\n'
-                )
                 await ib.connectAsync(
                     host,
                     port,
@@ -1373,9 +1227,7 @@ async def load_aio_clients(
                 client = Client(ib=ib, config=conf)
 
                 # update all actor-global caches
-                log.runtime(
-                    f'Connected and caching `Client` @ {sockaddr!r}'
-                )
+                log.info(f"Caching client for {sockaddr}")
                 _client_cache[sockaddr] = client
                 break
 
@@ -1390,59 +1242,37 @@ async def load_aio_clients(
                 OSError,
             ) as ce:
                 _err = ce
-                message: str = (
-                    f'Failed to connect on {host}:{port} after {i} tries with\n'
-                    f'{ib.client.apiError.value()!r}\n\n'
-                    'Retrying with a new client id..\n'
-                )
-                log.runtime(message)
-        else:
-            # XXX report loudly if we never established after all
-            # re-tries
-            log.warning(message)
+                log.warning(
+                    f'Failed to connect on {host}:{port} for {i} time with,\n'
+                    f'{ib.client.apiError.value()}\n'
+                    'retrying with a new client id..')
 
         # Pre-collect all accounts available for this
         # connection and map account names to this client
         # instance.
         for value in ib.accountValues():
-            acct_number: str = value.account
+            acct_number = value.account
 
-            acnt_alias: str = accounts_def.inverse.get(acct_number)
-            if not acnt_alias:
-
-                # TODO: should we constuct the below reco-ex from
-                # the existing config content?
-                _, path = config.load(
-                    conf_name='brokers',
-                )
+            entry = accounts_def.inverse.get(acct_number)
+            if not entry:
                 raise ValueError(
-                    'No alias in account section for account!\n'
-                    f'Please add an acnt alias entry to your {path}\n'
-                    'For example,\n\n'
-
-                    '[ib.accounts]\n'
-                    'margin = {accnt_number!r}\n'
-                    '^^^^^^ <- you need this part!\n\n'
-
-                    'This ensures `piker` will not leak private acnt info '
-                    'to console output by default!\n'
+                    'No section in brokers.toml for account:'
+                    f' {acct_number}\n'
+                    f'Please add entry to continue using this API client'
                 )
 
             # surjection of account names to operating clients.
-            if acnt_alias not in accounts_found:
-                accounts_found[acnt_alias] = client
-                # client._acnt_names.add(acnt_alias)
-                client._acnt_names.append(acnt_alias)
+            if acct_number not in accounts_found:
+                accounts_found[entry] = client
 
-        if accounts_found:
-            log.info(
-                f'Loaded accounts for api client\n\n'
-                f'{pformat(accounts_found)}\n'
-            )
+        log.info(
+            f'Loaded accounts for client @ {host}:{port}\n'
+            f'{pformat(accounts_found)}'
+        )
 
-            # XXX: why aren't we just updating this directy above
-            # instead of using the intermediary `accounts_found`?
-            _accounts2clients.update(accounts_found)
+        # XXX: why aren't we just updating this directy above
+        # instead of using the intermediary `accounts_found`?
+        _accounts2clients.update(accounts_found)
 
     # if we have no clients after the scan loop then error out.
     if not _client_cache:
@@ -1465,20 +1295,21 @@ async def load_aio_clients(
 
 
 async def load_clients_for_trio(
-    chan: tractor.to_asyncio.LinkedTaskChannel,
+    from_trio: asyncio.Queue,
+    to_trio: trio.abc.SendChannel,
+
 ) -> None:
     '''
     Pure async mngr proxy to ``load_aio_clients()``.
 
-    This is a bootstrap entrypoint to call from
-    a `tractor.to_asyncio.open_channel_from()`.
+    This is a bootstrap entrypoing to call from
+    a ``tractor.to_asyncio.open_channel_from()``.
 
     '''
-    async with load_aio_clients(
-        disconnect_on_exit=False,
-    ) as accts2clients:
+    async with load_aio_clients() as accts2clients:
+
+        to_trio.send_nowait(accts2clients)
 
-        chan.started_nowait(accts2clients)
         # TODO: maybe a sync event to wait on instead?
         await asyncio.sleep(float('inf'))
 
@@ -1491,10 +1322,7 @@ async def open_client_proxies() -> tuple[
     async with (
         tractor.trionics.maybe_open_context(
             acm_func=tractor.to_asyncio.open_channel_from,
-            kwargs={
-                'target': load_clients_for_trio,
-                # ^XXX, kwarg to `open_channel_from()`
-            },
+            kwargs={'target': load_clients_for_trio},
 
             # lock around current actor task access
             # TODO: maybe this should be the default in tractor?
@@ -1604,7 +1432,7 @@ class MethodProxy:
         self,
         pattern: str,
 
-    ) -> dict[str, Any]|trio.Event:
+    ) -> dict[str, Any] | trio.Event:
 
         ev = self.event_table.get(pattern)
 
@@ -1625,25 +1453,26 @@ class MethodProxy:
 
 
 async def open_aio_client_method_relay(
-    chan: tractor.to_asyncio.LinkedTaskChannel,
+    from_trio: asyncio.Queue,
+    to_trio: trio.abc.SendChannel,
     client: Client,
     event_consumers: dict[str, trio.Event],
 
 ) -> None:
 
     # sync with `open_client_proxy()` caller
-    chan.started_nowait(client)
+    to_trio.send_nowait(client)
 
     # TODO: separate channel for error handling?
-    client.inline_errors(chan)
+    client.inline_errors(to_trio)
 
     # relay all method requests to ``asyncio``-side client and deliver
     # back results
-    while not chan._to_trio._closed:  # <- TODO, better check like `._web_bs`?
-        msg: tuple[str, dict]|dict|None = await chan.get()
+    while not to_trio._closed:
+        msg: tuple[str, dict] | dict | None = await from_trio.get()
         match msg:
             case None:  # termination sentinel
-                log.info('asyncio `Client` method-proxy SHUTDOWN!')
+                print('asyncio PROXY-RELAY SHUTDOWN')
                 break
 
             case (meth_name, kwargs):
@@ -1653,7 +1482,7 @@ async def open_aio_client_method_relay(
                 try:
                     resp = await meth(**kwargs)
                     # echo the msg back
-                    chan.send_nowait({'result': resp})
+                    to_trio.send_nowait({'result': resp})
 
                 except (
                     RequestError,
@@ -1661,10 +1490,10 @@ async def open_aio_client_method_relay(
                     # TODO: relay all errors to trio?
                     # BaseException,
                 ) as err:
-                    chan.send_nowait({'exception': err})
+                    to_trio.send_nowait({'exception': err})
 
             case {'error': content}:
-                chan.send_nowait({'exception': content})
+                to_trio.send_nowait({'exception': content})
 
             case _:
                 raise ValueError(f'Unhandled msg {msg}')
@@ -1686,8 +1515,7 @@ async def open_client_proxy(
             event_consumers=event_table,
         ) as (first, chan),
 
-        trionics.collapse_eg(),  # loose-ify
-        trio.open_nursery() as relay_tn,
+        trio.open_nursery() as relay_n,
     ):
 
         assert isinstance(first, Client)
@@ -1727,7 +1555,7 @@ async def open_client_proxy(
 
                     continue
 
-        relay_tn.start_soon(relay_events)
+        relay_n.start_soon(relay_events)
 
         yield proxy
 

piker/brokers/ib/broker.py

@@ -34,7 +34,6 @@ import trio
 from trio_typing import TaskStatus
 import tractor
 from tractor.to_asyncio import LinkedTaskChannel
-from tractor import trionics
 from ib_insync.contract import (
     Contract,
 )
@@ -50,10 +49,6 @@ from ib_insync.objects import (
 )
 
 from piker import config
-from piker.log import (
-    get_logger,
-    get_console_log,
-)
 from piker.types import Struct
 from piker.accounting import (
     Position,
@@ -81,6 +76,7 @@ from piker.clearing._messages import (
     BrokerdFill,
     BrokerdError,
 )
+from ._util import log
 from .api import (
     _accounts2clients,
     get_config,
@@ -98,10 +94,6 @@ from .ledger import (
     update_ledger_from_api_trades,
 )
 
-log = get_logger(
-    name=__name__,
-)
-
 
 def pack_position(
     pos: IbPosition,
@@ -124,11 +116,7 @@ def pack_position(
             symbol=fqme,
             currency=con.currency,
             size=float(pos.position),
-            avg_price=(
-                float(pos.avgCost)
-                /
-                float(con.multiplier or 1.0)
-            ),
+            avg_price=float(pos.avgCost) / float(con.multiplier or 1.0),
         ),
     )
 
@@ -369,10 +357,6 @@ async def update_and_audit_pos_msg(
         size=ibpos.position,
 
         avg_price=pikerpos.ppu,
-
-        # XXX ensures matching even if multiple venue-names
-        # in `.bs_fqme`, likely from txn records..
-        bs_mktid=mkt.bs_mktid,
     )
 
     ibfmtmsg: str = pformat(ibpos._asdict())
@@ -423,7 +407,7 @@ async def update_and_audit_pos_msg(
 
     # TODO: make this a "propaganda" log level?
     if ibpos.avgCost != msg.avg_price:
-        log.debug(
+        log.warning(
             f'IB "FIFO" avg price for {msg.symbol} is DIFF:\n'
             f'ib: {ibfmtmsg}\n'
             '---------------------------\n'
@@ -441,8 +425,7 @@ async def aggr_open_orders(
 
 ) -> None:
     '''
-    Collect all open orders from client and fill in `order_msgs:
-    list`.
+    Collect all open orders from client and fill in `order_msgs: list`.
 
     '''
     trades: list[Trade] = client.ib.openTrades()
@@ -543,15 +526,9 @@ class IbAcnt(Struct):
 @tractor.context
 async def open_trade_dialog(
     ctx: tractor.Context,
-    loglevel: str = 'warning',
 
 ) -> AsyncIterator[dict[str, Any]]:
 
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
-
     # task local msg dialog tracking
     flows = OrderDialogs()
     accounts_def = config.load_accounts(['ib'])
@@ -569,10 +546,7 @@ async def open_trade_dialog(
         ),
 
         # TODO: do this as part of `open_account()`!?
-        open_symcache(
-            'ib',
-            only_from_memcache=True,
-        ) as symcache,
+        open_symcache('ib', only_from_memcache=True) as symcache,
     ):
         # Open a trade ledgers stack for appending trade records over
         # multiple accounts.
@@ -580,10 +554,8 @@ async def open_trade_dialog(
         ledgers: dict[str, TransactionLedger] = {}
         tables: dict[str, Account] = {}
         order_msgs: list[Status] = []
-        conf: dict = get_config()
-        accounts_def_inv: bidict[str, str] = bidict(
-            conf['accounts']
-        ).inverse
+        conf = get_config()
+        accounts_def_inv: bidict[str, str] = bidict(conf['accounts']).inverse
 
         with (
             ExitStack() as lstack,
@@ -733,11 +705,7 @@ async def open_trade_dialog(
             # client-account and build out position msgs to deliver to
             # EMS.
             for acctid, acnt in tables.items():
-                active_pps: dict[str, Position]
-                (
-                    active_pps,
-                    closed_pps,
-                ) = acnt.dump_active()
+                active_pps, closed_pps = acnt.dump_active()
 
                 for pps in [active_pps, closed_pps]:
                     piker_pps: list[Position] = list(pps.values())
@@ -753,7 +721,6 @@ async def open_trade_dialog(
                         )
                         if ibpos:
                             bs_mktid: str = str(ibpos.contract.conId)
-
                             msg = await update_and_audit_pos_msg(
                                 acctid,
                                 pikerpos,
@@ -771,7 +738,7 @@ async def open_trade_dialog(
                                 f'UNEXPECTED POSITION says IB => {msg.symbol}\n'
                                 'Maybe they LIQUIDATED YOU or your ledger is wrong?\n'
                             )
-                            log.debug(logmsg)
+                            log.error(logmsg)
 
             await ctx.started((
                 all_positions,
@@ -780,22 +747,21 @@ async def open_trade_dialog(
 
             async with (
                 ctx.open_stream() as ems_stream,
-                trionics.collapse_eg(),
-                trio.open_nursery() as tn,
+                trio.open_nursery() as n,
             ):
                 # relay existing open orders to ems
                 for msg in order_msgs:
                     await ems_stream.send(msg)
 
                 for client in set(aioclients.values()):
-                    trade_event_stream: LinkedTaskChannel = await tn.start(
+                    trade_event_stream: LinkedTaskChannel = await n.start(
                         open_trade_event_stream,
                         client,
                     )
 
                     # start order request handler **before** local trades
                     # event loop
-                    tn.start_soon(
+                    n.start_soon(
                         handle_order_requests,
                         ems_stream,
                         accounts_def,
@@ -803,7 +769,7 @@ async def open_trade_dialog(
                     )
 
                     # allocate event relay tasks for each client connection
-                    tn.start_soon(
+                    n.start_soon(
                         deliver_trade_events,
 
                         trade_event_stream,
@@ -1217,14 +1183,7 @@ async def deliver_trade_events(
                         pos
                         and fill
                     ):
-                        now_cr: CommissionReport = fill.commissionReport
-                        if (now_cr != cr):
-                            log.warning(
-                                'UhhHh ib updated the commission report mid-fill..?\n'
-                                f'was: {pformat(cr)}\n'
-                                f'now: {pformat(now_cr)}\n'
-                            )
-
+                        assert fill.commissionReport == cr
                         await emit_pp_update(
                             ems_stream,
                             accounts_def,
@@ -1275,47 +1234,32 @@ async def deliver_trade_events(
                 # never relay errors for non-broker related issues
                 # https://interactivebrokers.github.io/tws-api/message_codes.html
                 code: int = err['error_code']
-                reason: str = err['reason']
-                reqid: str = str(err['reqid'])
-
-                # "Warning:" msg codes,
-                # https://interactivebrokers.github.io/tws-api/message_codes.html#warning_codes
-                # - 2109: 'Outside Regular Trading Hours'
-                if 'Warning:' in reason:
-                    log.warning(
-                        f'Order-API-warning: {code!r}\n'
-                        f'reqid: {reqid!r}\n'
-                        f'\n'
-                        f'{pformat(err)}\n'
-                        # ^TODO? should we just print the `reason`
-                        # not the full `err`-dict?
-                    )
-                    continue
-
-                # XXX known special (ignore) cases
-                elif code in {
-                    200,  # uhh.. ni idea
+                if code in {
+                    200,  # uhh
 
                     # hist pacing / connectivity
                     162,
                     165,
 
+                    # WARNING codes:
+                    # https://interactivebrokers.github.io/tws-api/message_codes.html#warning_codes
+                    # Attribute 'Outside Regular Trading Hours' is
+                    # " 'ignored based on the order type and
+                    # destination. PlaceOrder is now ' 'being
+                    # processed.',
+                    2109,
+
                     # XXX: lol this isn't even documented..
                     # 'No market data during competing live session'
                     1669,
                 }:
-                    log.error(
-                        f'Order-API-error which is non-cancel-causing ?!\n'
-                        f'\n'
-                        f'{pformat(err)}\n'
-                    )
                     continue
 
+                reqid: str = str(err['reqid'])
+                reason: str = err['reason']
+
                 if err['reqid'] == -1:
-                    log.error(
-                        f'TWS external order error ??\n'
-                        f'{pformat(err)}\n'
-                    )
+                    log.error(f'TWS external order error:\n{pformat(err)}')
 
                 flow: dict = dict(
                     flows.get(reqid)

piker/brokers/ib/feed.py

@@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-forever Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) Tyler Goodlet (in stewardship for pikers)
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -13,12 +13,10 @@
 
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
+"""
+Data feed endpoints pre-wrapped and ready for use with ``tractor``/``trio``.
 
-'''
-Data feed endpoints pre-wrapped and ready for use with `tractor`/`trio`
-via "infected-asyncio-mode".
-
-'''
+"""
 from __future__ import annotations
 import asyncio
 from contextlib import (
@@ -28,6 +26,7 @@ from dataclasses import asdict
 from datetime import datetime
 from functools import partial
 from pprint import pformat
+from math import isnan
 import time
 from typing import (
     Any,
@@ -41,6 +40,7 @@ import numpy as np
 from pendulum import (
     now,
     from_timestamp,
+    # DateTime,
     Duration,
     duration as mk_duration,
 )
@@ -56,11 +56,11 @@ from piker.brokers._util import (
     NoData,
     DataUnavailable,
 )
-from piker.log import get_logger
 from .api import (
     # _adhoc_futes_set,
     Client,
     con2fqme,
+    log,
     load_aio_clients,
     MethodProxy,
     open_client_proxies,
@@ -69,18 +69,12 @@ from .api import (
     Contract,
     RequestError,
 )
-from .venues import is_venue_open
-from ._util import (
-    data_reset_hack,
-)
+from ._util import data_reset_hack
 from .symbols import get_mkt_info
 
 if TYPE_CHECKING:
     from trio._core._run import Task
 
-log = get_logger(
-    name=__name__,
-)
 
 # XXX NOTE: See available types table docs:
 # https://interactivebrokers.github.io/tws-api/tick_types.html
@@ -181,8 +175,8 @@ async def open_history_client(
 
         async def get_hist(
             timeframe: float,
-            end_dt: datetime|None = None,
-            start_dt: datetime|None = None,
+            end_dt: datetime | None = None,
+            start_dt: datetime | None = None,
 
         ) -> tuple[np.ndarray, str]:
 
@@ -190,8 +184,7 @@ async def open_history_client(
 
             if (
                 start_dt
-                and
-                start_dt.timestamp() == 0
+                and start_dt.timestamp() == 0
             ):
                 await tractor.pause()
 
@@ -206,27 +199,22 @@ async def open_history_client(
             latency = time.time() - query_start
             if (
                 not timedout
-                # and
-                # latency <= max_timeout
+                # and latency <= max_timeout
             ):
                 count += 1
                 mean += latency / count
-                log.debug(
+                print(
                     f'HISTORY FRAME QUERY LATENCY: {latency}\n'
                     f'mean: {mean}'
                 )
 
             # could be trying to retreive bars over weekend
             if out is None:
-                log.error(
-                    f"No bars starting at {end_dt!r} !?!?"
-                )
+                log.error(f"Can't grab bars starting at {end_dt}!?!?")
                 if (
                     end_dt
-                    and
-                    head_dt
-                    and
-                    end_dt <= head_dt
+                    and head_dt
+                    and end_dt <= head_dt
                 ):
                     raise DataUnavailable(
                         f'First timestamp is {head_dt}\n'
@@ -268,43 +256,12 @@ async def open_history_client(
             vlm = bars_array['volume']
             vlm[vlm < 0] = 0
 
-            # XXX, if a start-limit was passed ensure we only
-            # return history that far back!
-            if (
-                start_dt
-                and
-                first_dt < start_dt
-            ):
-                trimmed_bars = bars_array[
-                    bars_array['time'] >= start_dt.timestamp()
-                ]
-                if (
-                    trimmed_first_dt := from_timestamp(trimmed_bars['time'][0])
-                    !=
-                    start_dt
-                ):
-                    # TODO! rm this once we're more confident it never hits!
-                    # breakpoint()
-                    raise RuntimeError(
-                        f'OHLC-bars array start is gt `start_dt` limit !!\n'
-                        f'start_dt: {start_dt}\n'
-                        f'first_dt: {first_dt}\n'
-                        f'trimmed_first_dt: {trimmed_first_dt}\n'
-                    )
-
-                # XXX, overwrite with start_dt-limited frame
-                bars_array = trimmed_bars
-
-            return (
-                bars_array,
-                first_dt,
-                last_dt,
-            )
+            return bars_array, first_dt, last_dt
 
         # TODO: it seems like we can do async queries for ohlc
         # but getting the order right still isn't working and I'm not
         # quite sure why.. needs some tinkering and probably
-        # a lookthrough of the `ib_insync` machinery, for eg. maybe
+        # a lookthrough of the ``ib_insync`` machinery, for eg. maybe
         # we have to do the batch queries on the `asyncio` side?
         yield (
             get_hist,
@@ -328,9 +285,8 @@ _pacing: str = (
 
 async def wait_on_data_reset(
     proxy: MethodProxy,
-
     reset_type: str = 'data',
-    timeout: float = 16,
+    timeout: float = 16, # float('inf'),
 
     task_status: TaskStatus[
         tuple[
@@ -339,47 +295,29 @@ async def wait_on_data_reset(
         ]
     ] = trio.TASK_STATUS_IGNORED,
 ) -> bool:
-    '''
-    Wait on a (global-ish) "data-farm" event to be emitted
-    by the IB api server.
 
-    Allows syncing to reconnect event-messages emitted on the API
-    console, such as:
-
-     - 'HMDS data farm connection is OK:ushmds'
-     - 'Market data farm is connecting:usfuture'
-     - 'Market data farm connection is OK:usfuture'
-
-     Deliver a `(cs, done: Event)` pair to the caller to support it
-     waiting or cancelling the associated "data-reset-request";
-     normally a manual data-reset-req is expected to be the cause and
-     thus trigger such events (such as our click-hack-magic from
-     `.ib._util`).
-
-    '''
-    # ?TODO, do we need a task-lock around this method?
-    #
-    # register for an API "status event" wrapped for `trio`-sync.
-    hist_ev: trio.Event = proxy.status_event(
+    # TODO: we might have to put a task lock around this
+    # method..
+    hist_ev = proxy.status_event(
         'HMDS data farm connection is OK:ushmds'
     )
-    #
-    # ^TODO: other event-messages we might want to support waiting-for
-    # but i wasn't able to get reliable..
-    #
+
+    # TODO: other event messages we might want to try and
+    # wait for but i wasn't able to get any of this
+    # reliable..
     # reconnect_start = proxy.status_event(
     #     'Market data farm is connecting:usfuture'
     # )
     # live_ev = proxy.status_event(
     #     'Market data farm connection is OK:usfuture'
     # )
-
     # try to wait on the reset event(s) to arrive, a timeout
     # will trigger a retry up to 6 times (for now).
     client: Client = proxy._aio_ns
 
     done = trio.Event()
     with trio.move_on_after(timeout) as cs:
+
         task_status.started((cs, done))
 
         log.warning(
@@ -427,13 +365,14 @@ _failed_resets: int = 0
 
 
 async def get_bars(
+
     proxy: MethodProxy,
     fqme: str,
     timeframe: int,
 
     # blank to start which tells ib to look up the latest datum
-    end_dt: datetime|None = None,
-    start_dt: datetime|None = None,
+    end_dt: str = '',
+    start_dt: str | None = '',
 
     # TODO: make this more dynamic based on measured frame rx latency?
     # how long before we trigger a feed reset (seconds)
@@ -457,9 +396,8 @@ async def get_bars(
     bool,  # timed out hint
 ]:
     '''
-    Request-n-retrieve historical data frames from a `trio.Task`
-    using a `MethoProxy` to query the `asyncio`-side's
-    `.ib.api.Client` methods.
+    Retrieve historical data from a ``trio``-side task using
+    a ``MethoProxy``.
 
     '''
     global _data_resetter_task, _failed_resets
@@ -487,9 +425,6 @@ async def get_bars(
                     dt_duration,
                 ) = await proxy.bars(
                     fqme=fqme,
-                    # XXX TODO! LOL we're not using this and IB dun
-                    # support it anyway..
-                    # start_dt=start_dt,
                     end_dt=end_dt,
                     sample_period_s=timeframe,
 
@@ -652,7 +587,7 @@ async def get_bars(
                         data_cs.cancel()
 
                     # spawn new data reset task
-                    data_cs, reset_done = await tn.start(
+                    data_cs, reset_done = await nurse.start(
                         partial(
                             wait_on_data_reset,
                             proxy,
@@ -672,14 +607,11 @@ async def get_bars(
     # such that simultaneous symbol queries don't try data resettingn
     # too fast..
     unset_resetter: bool = False
-    async with (
-        tractor.trionics.collapse_eg(),
-        trio.open_nursery() as tn
-    ):
+    async with trio.open_nursery() as nurse:
 
         # start history request that we allow
         # to run indefinitely until a result is acquired
-        tn.start_soon(query)
+        nurse.start_soon(query)
 
         # start history reset loop which waits up to the timeout
         # for a result before triggering a data feed reset.
@@ -699,7 +631,7 @@ async def get_bars(
                 unset_resetter: bool = True
 
             # spawn new data reset task
-            data_cs, reset_done = await tn.start(
+            data_cs, reset_done = await nurse.start(
                 partial(
                     wait_on_data_reset,
                     proxy,
@@ -721,12 +653,14 @@ async def get_bars(
     )
 
 
-# per-actor cache of inter-eventloop-chans
 _quote_streams: dict[str, trio.abc.ReceiveStream] = {}
 
 
 async def _setup_quote_stream(
-    chan: tractor.to_asyncio.LinkedTaskChannel,
+
+    from_trio: asyncio.Queue,
+    to_trio: trio.abc.SendChannel,
+
     symbol: str,
     opts: tuple[int] = (
         '375',  # RT trade volume (excludes utrades)
@@ -737,20 +671,17 @@ async def _setup_quote_stream(
         # making them mostly useless and explains why the scanner
         # is always slow XD
         # '293',  # Trade count for day
-        # '294',  # Trade rate / minute
-        # '295',  # Vlm rate / minute
+        '294',  # Trade rate / minute
+        '295',  # Vlm rate / minute
     ),
-    contract: Contract|None = None,
+    contract: Contract | None = None,
 
 ) -> trio.abc.ReceiveChannel:
     '''
-    Stream L1 quotes via the `Ticker.updateEvent.connect(push)`
-    callback API by registering a `push` callback which simply
-    `chan.send_nowait()`s quote msgs back to the calling
-    parent-`trio.Task`-side.
+    Stream a ticker using the std L1 api.
 
-    NOTE, that this task-fn is run on the `asyncio.Task`-side ONLY
-    and is thus run via `tractor.to_asyncio.open_channel_from()`.
+    This task is ``asyncio``-side and must be called from
+    ``tractor.to_asyncio.open_channel_from()``.
 
     '''
     global _quote_streams
@@ -758,84 +689,37 @@ async def _setup_quote_stream(
     async with load_aio_clients(
         disconnect_on_exit=False,
     ) as accts2clients:
+        caccount_name, client = get_preferred_data_client(accts2clients)
+        contract = contract or (await client.find_contract(symbol))
+        to_trio.send_nowait(contract)  # cuz why not
+        ticker: Ticker = client.ib.reqMktData(contract, ','.join(opts))
 
-        # XXX since this is an `asyncio.Task`, we must use
-        # tractor.pause_from_sync()
-
-        (
-            _account_name,
-            client,
-        ) = get_preferred_data_client(
-            accts2clients,
-        )
-        contract = (
-            contract
-            or
-            (await client.find_contract(symbol))
-        )
-        chan.started_nowait(contract)  # cuz why not
-        ticker: Ticker = client.ib.reqMktData(
-            contract,
-            ','.join(opts),
-        )
-        maybe_exc: BaseException|None = None
-        handler_tries: int = 0
-        aio_task: asyncio.Task = asyncio.current_task()
-
-        # ?TODO? this API is batch-wise and quite slow-af but,
-        # - seems to be 5s updates?
-        # - maybe we could use it for backchecking?
-        #
+        # NOTE: it's batch-wise and slow af but I guess could
+        # be good for backchecking? Seems to be every 5s maybe?
         # ticker: Ticker = client.ib.reqTickByTickData(
         #     contract, 'Last',
         # )
 
-        # define a very naive queue-pushing callback that relays
-        # quote-packets directly the calling (parent) `trio.Task`.
-        # Ensure on teardown we cancel the feed via their cancel API.
-        #
+        # # define a simple queue push routine that streams quote packets
+        # # to trio over the ``to_trio`` memory channel.
+        # to_trio, from_aio = trio.open_memory_channel(2**8)  # type: ignore
         def teardown():
-            '''
-            Disconnect our `push`-er callback and cancel the data-feed
-            for `contract`.
-
-            '''
-            nonlocal maybe_exc
             ticker.updateEvent.disconnect(push)
-            report: str = f'Disconnected mkt-data for {symbol!r} due to '
-            if maybe_exc is not None:
-                report += (
-                    'error,\n'
-                    f'{maybe_exc!r}\n'
-                )
-                log.error(report)
-            else:
-                report += (
-                    'cancellation.\n'
-                )
-                log.cancel(report)
-
+            log.error(f"Disconnected stream for `{symbol}`")
             client.ib.cancelMktData(contract)
 
             # decouple broadcast mem chan
             _quote_streams.pop(symbol, None)
 
-        def push(
-            t: Ticker,
-            tries_before_raise: int = 6,
-        ) -> None:
-            '''
-            Push quotes verbatim to parent-side `trio.Task`.
+        def push(t: Ticker) -> None:
+            """
+            Push quotes to trio task.
 
-            '''
-            nonlocal maybe_exc, handler_tries
-            # log.debug(f'new IB quote: {t}\n')
+            """
+            # log.debug(t)
             try:
-                chan.send_nowait(t)
+                to_trio.send_nowait(t)
 
-                # XXX TODO XXX replicate in `tractor` tests
-                # as per `CancelledError`-handler notes below!
-                # assert 0
             except (
                 trio.BrokenResourceError,
 
@@ -850,107 +734,35 @@ async def _setup_quote_stream(
                 # resulting in tracebacks spammed to console..
                 # Manually do the dereg ourselves.
                 teardown()
-
-            # for slow debugging purposes to avoid clobbering prompt
-            # with log msgs
             except trio.WouldBlock:
-                log.exception(
-                    f'Asyncio->Trio `chan.send_nowait()` blocked !?\n'
-                    f'\n'
-                    f'{chan._to_trio.statistics()}\n'
-                )
-
-            # ?TODO, handle re-connection attempts?
-            except BaseException as _berr:
-                berr = _berr
-                if handler_tries >= tries_before_raise:
-                    # breakpoint()
-                    maybe_exc = _berr
-                    # task.set_exception(berr)
-                    aio_task.cancel(msg=berr.args)
-                    raise berr
-                else:
-                    handler_tries += 1
-
-                log.exception(
-                    f'Failed to push ticker quote !?\n'
-                    f'handler_tries={handler_tries!r}\n'
-                    f'ticker: {t!r}\n'
-                    f'\n'
-                    f'{chan._to_trio.statistics()}\n'
-                    f'\n'
-                    f'CAUSE: {berr}\n'
-                )
+                # log.warning(
+                #     f'channel is blocking symbol feed for {symbol}?'
+                #     f'\n{to_trio.statistics}'
+                # )
+                pass
 
+            # except trio.WouldBlock:
+            #     # for slow debugging purposes to avoid clobbering prompt
+            #     # with log msgs
+            #     pass
 
         ticker.updateEvent.connect(push)
         try:
             await asyncio.sleep(float('inf'))
-
-            # XXX, for debug.. TODO? can we rm again?
-            #
-            # tractor.pause_from_sync()
-            # while True:
-            #     await asyncio.sleep(1.6)
-            #     if ticker.ticks:
-            #         log.debug(
-            #             f'ticker.ticks = \n'
-            #             f'{ticker.ticks}\n'
-            #         )
-            #     else:
-            #         log.warning(
-            #             'UHH no ticker.ticks ??'
-            #         )
-
-        # XXX TODO XXX !?!?
-        # apparently **without this handler** and the subsequent
-        # re-raising of `maybe_exc from _taskc` cancelling the
-        # `aio_task` from the `push()`-callback will cause a very
-        # strange chain of exc raising that breaks alll sorts of
-        # downstream callers, tasks and remote-actor tasks!?
-        #
-        # -[ ] we need some lowlevel reproducting tests to replicate
-        #      those worst-case scenarios in `tractor` core!!
-        # -[ ] likely we should factor-out the `tractor.to_asyncio`
-        #      attempts at workarounds in `.translate_aio_errors()`
-        #      for failed `asyncio.Task.set_exception()` to either
-        #      call `aio_task.cancel()` and/or
-        #      `aio_task._fut_waiter.set_exception()` to a re-useable
-        #      toolset in something like a `.to_asyncio._utils`??
-        #
-        except asyncio.CancelledError as _taskc:
-            if maybe_exc is not None:
-                raise maybe_exc from _taskc
-
-            raise _taskc
-
-        except BaseException as _berr:
-            # stash any crash cause for reporting in `teardown()`
-            maybe_exc = _berr
-            raise _berr
-
         finally:
-            # always disconnect our `push()` and cancel the
-            # ib-"mkt-data-feed".
             teardown()
 
+        # return from_aio
+
 
 @acm
 async def open_aio_quote_stream(
+
     symbol: str,
-    contract: Contract|None = None,
+    contract: Contract | None = None,
 
-) -> (
-    trio.abc.Channel|  # iface
-    tractor.to_asyncio.LinkedTaskChannel  # actually
-):
-    '''
-    Open a real-time `Ticker` quote stream from an `asyncio.Task`
-    spawned via `tractor.to_asyncio.open_channel_from()`, deliver the
-    inter-event-loop channel to the `trio.Task` caller and cache it
-    globally for re-use.
+) -> trio.abc.ReceiveStream:
 
-    '''
     from tractor.trionics import broadcast_receiver
     global _quote_streams
 
@@ -966,7 +778,6 @@ async def open_aio_quote_stream(
             yield from_aio
             return
 
-    from_aio: tractor.to_asyncio.LinkedTaskChannel
     async with tractor.to_asyncio.open_channel_from(
         _setup_quote_stream,
         symbol=symbol,
@@ -976,10 +787,6 @@ async def open_aio_quote_stream(
 
         assert contract
 
-        # TODO? de-reg on teardown of last consumer task?
-        # -> why aren't we using `.trionics.maybe_open_context()`
-        # here again?? (we are in `open_client_proxies()` tho?)
-        #
         # cache feed for later consumers
         _quote_streams[symbol] = from_aio
 
@@ -994,12 +801,7 @@ def normalize(
     calc_price: bool = False
 
 ) -> dict:
-    '''
-    Translate `ib_async`'s `Ticker.ticks` values to a `piker`
-    normalized `dict` form for transmit to downstream `.data` layer
-    consumers.
 
-    '''
     # check for special contract types
     con = ticker.contract
     fqme, calc_price = con2fqme(con)
@@ -1018,7 +820,7 @@ def normalize(
 
             tbt = ticker.tickByTicks
             if tbt:
-                log.info(f'tickbyticks:\n {ticker.tickByTicks}')
+                print(f'tickbyticks:\n {ticker.tickByTicks}')
 
     ticker.ticks = new_ticks
 
@@ -1054,39 +856,27 @@ def normalize(
     return data
 
 
-# ?TODO? feels like this task-fn could be factored to reduce some
-# indentation levels?
-# -[ ] the reconnect while loop on ib-gw "data farm connection.."s
-# -[ ] everything embedded under the `async with aclosing(stream):`
-#     as the "meat" of the quote delivery once the connection is
-#     stable.
-#
 async def stream_quotes(
+
     send_chan: trio.abc.SendChannel,
     symbols: list[str],
     feed_is_live: trio.Event,
-
-    # TODO? we need to hook into the `ib_async` logger like
-    # we can with i3ipc from modden!
-    # loglevel: str|None = None,
+    loglevel: str = None,
 
     # startup sync
     task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
 
 ) -> None:
     '''
-    Stream `symbols[0]` quotes back via `send_chan`.
+    Stream symbol quotes.
 
-    The `feed_is_live: Event` is set to signal the caller that it can
-    begin processing msgs from the mem-chan.
+    This is a ``trio`` callable routine meant to be invoked
+    once the brokerd is up.
 
     '''
     # TODO: support multiple subscriptions
-    sym: str = symbols[0]
-    log.info(
-        f'request for real-time quotes\n'
-        f'sym: {sym!r}\n'
-    )
+    sym = symbols[0]
+    log.info(f'request for real-time quotes: {sym}')
 
     init_msgs: list[FeedInit] = []
 
@@ -1095,95 +885,77 @@ async def stream_quotes(
     details: ibis.ContractDetails
     async with (
         open_data_client() as proxy,
+        # trio.open_nursery() as tn,
     ):
         mkt, details = await get_mkt_info(
             sym,
             proxy=proxy,  # passed to avoid implicit client load
         )
 
-        # is venue active rn?
-        venue_is_open: bool = is_venue_open(
-            con_deats=details,
-        )
         init_msg = FeedInit(mkt_info=mkt)
 
-        # NOTE, tell sampler (via config) to skip vlm summing for dst
-        # assets which provide no vlm data..
         if mkt.dst.atype in {
             'fiat',
             'index',
             'commodity',
         }:
+            # tell sampler config that it shouldn't do vlm summing.
             init_msg.shm_write_opts['sum_tick_vlm'] = False
             init_msg.shm_write_opts['has_vlm'] = False
 
         init_msgs.append(init_msg)
 
         con: Contract = details.contract
-        first_ticker: Ticker|None = None
-        first_quote: dict[str, Any] = {}
-
-        timeout: float = 1.6
-        with trio.move_on_after(timeout) as quote_cs:
+        first_ticker: Ticker | None = None
+        with trio.move_on_after(1):
             first_ticker: Ticker = await proxy.get_quote(
                 contract=con,
                 raise_on_timeout=False,
             )
 
-        # XXX should never happen with this ep right?
-        # but if so then, more then likely mkt is closed?
-        if quote_cs.cancelled_caught:
-            log.warning(
-                f'First quote req timed out after {timeout!r}s'
-            )
-
         if first_ticker:
             first_quote: dict = normalize(first_ticker)
-
-            # TODO: we need a stack-oriented log levels filters for
-            # this!
-            # log.info(message, filter={'stack': 'live_feed'}) ?
-            log.runtime(
-                'Rxed init quote:\n\n'
-                f'{pformat(first_quote)}\n'
+            log.info(
+                'Rxed init quote:\n'
+                f'{pformat(first_quote)}'
             )
 
-        # XXX NOTE: whenever we're "outside regular trading hours"
-        # (only relevant for assets coming from the "legacy markets"
-        # space) so we basically (from an API/runtime-operational
-        # perspective) "pretend the feed is live" even if it's
-        # actually closed.
-        #
-        # IOW, we signal to the effective caller (task) that the live
-        # feed is "already up" but really we're just indicating that
-        # the OHLCV history can start being loaded immediately by the
-        # `piker.data`/`.tsp` layers.
-        #
-        # XXX, deats: the "pretend we're live" is just done by
-        # a `feed_is_live.set()` even though nothing is actually live
-        # Bp
-        if not venue_is_open:
-            log.warning(
-                f'Venue is closed, unable to establish real-time feed.\n'
-                f'mkt: {mkt!r}\n'
-                f'\n'
-                f'first_ticker: {first_ticker}\n'
-            )
+        # NOTE: it might be outside regular trading hours for
+        # assets with "standard venue operating hours" so we
+        # only "pretend the feed is live" when the dst asset
+        # type is NOT within the NON-NORMAL-venue set: aka not
+        # commodities, forex or crypto currencies which CAN
+        # always return a NaN on a snap quote request during
+        # normal venue hours. In the case of a closed venue
+        # (equitiies, futes, bonds etc.) we at least try to
+        # grab the OHLC history.
+        if (
+            first_ticker
+            and
+            isnan(first_ticker.last)
+            # SO, if the last quote price value is NaN we ONLY
+            # "pretend to do" `feed_is_live.set()` if it's a known
+            # dst asset venue with a lot of closed operating hours.
+            and mkt.dst.atype not in {
+                'commodity',
+                'fiat',
+                'crypto',
+            }
+        ):
             task_status.started((
                 init_msgs,
                 first_quote,
             ))
 
-            # block and let data history backfill code run.
-            # XXX obvi given the venue is closed, we never expect feed
-            # to come up; a taskc should be the only way to
-            # terminate this task.
-            await trio.sleep_forever()
-            #
-            # ^^XXX^^TODO! INSTEAD impl a `trio.sleep()` for the
-            # duration until the venue opens!!
+            # it's not really live but this will unblock
+            # the brokerd feed task to tell the ui to update?
+            feed_is_live.set()
 
-        # ?TODO, we could instead spawn a task that waits on a feed
+            # block and let data history backfill code run.
+            await trio.sleep_forever()
+            return  # we never expect feed to come up?
+
+        # TODO: we should instead spawn a task that waits on a feed
         # to start and let it wait indefinitely..instead of this
         # hard coded stuff.
         # async def wait_for_first_quote():
@@ -1197,38 +969,27 @@ async def stream_quotes(
             raise_on_timeout=True,
         )
         first_quote: dict = normalize(first_ticker)
-
-        # TODO: we need a stack-oriented log levels filters for
-        # this!
-        # log.info(message, filter={'stack': 'live_feed'}) ?
-        log.runtime(
+        log.info(
             'Rxed init quote:\n'
             f'{pformat(first_quote)}'
         )
-        # signal `.data.feed` layer that mkt quotes are LIVE
-        feed_is_live.set()
-
-        cs: trio.CancelScope|None = None
+        cs: trio.CancelScope | None = None
         startup: bool = True
-        iter_quotes: trio.abc.Channel
         while (
             startup
-            or
-            cs.cancel_called
+            or cs.cancel_called
         ):
             with trio.CancelScope() as cs:
                 async with (
-                    tractor.trionics.collapse_eg(),
-                    trio.open_nursery() as tn,
+                    trio.open_nursery() as nurse,
                     open_aio_quote_stream(
                         symbol=sym,
                         contract=con,
-                    ) as iter_quotes,
+                    ) as stream,
                 ):
-                    # ?TODO? can we rm this - particularly for `ib_async`?
                     # ugh, clear ticks since we've consumed them
                     # (ahem, ib_insync is stateful trash)
-                    # first_ticker.ticks = []
+                    first_ticker.ticks = []
 
                     # only on first entry at feed boot up
                     if startup:
@@ -1242,8 +1003,8 @@ async def stream_quotes(
                     # data feed event.
                     async def reset_on_feed():
 
-                        # ??TODO? this seems to be surpressed from the
-                        # traceback in `tractor`?
+                        # TODO: this seems to be surpressed from the
+                        # traceback in ``tractor``?
                         # assert 0
 
                         rt_ev = proxy.status_event(
@@ -1252,28 +1013,60 @@ async def stream_quotes(
                         await rt_ev.wait()
                         cs.cancel()  # cancel called should now be set
 
-                    tn.start_soon(reset_on_feed)
+                    nurse.start_soon(reset_on_feed)
+
+                    async with aclosing(stream):
+                        # if syminfo.get('no_vlm', False):
+                        if not init_msg.shm_write_opts['has_vlm']:
+
+                            # generally speaking these feeds don't
+                            # include vlm data.
+                            atype: str = mkt.dst.atype
+                            log.info(
+                                f'No-vlm {mkt.fqme}@{atype}, skipping quote poll'
+                            )
+
+                        else:
+                            # wait for real volume on feed (trading might be
+                            # closed)
+                            while True:
+                                ticker = await stream.receive()
+
+                                # for a real volume contract we rait for
+                                # the first "real" trade to take place
+                                if (
+                                    # not calc_price
+                                    # and not ticker.rtTime
+                                    not ticker.rtTime
+                                ):
+                                    # spin consuming tickers until we
+                                    # get a real market datum
+                                    log.debug(f"New unsent ticker: {ticker}")
+                                    continue
+                                else:
+                                    log.debug("Received first volume tick")
+                                    # ugh, clear ticks since we've
+                                    # consumed them (ahem, ib_insync is
+                                    # truly stateful trash)
+                                    ticker.ticks = []
+
+                                    # XXX: this works because we don't use
+                                    # ``aclosing()`` above?
+                                    break
+
+                            quote = normalize(ticker)
+                            log.debug(f"First ticker received {quote}")
 
-                    async with aclosing(iter_quotes):
                         # tell data-layer spawner-caller that live
-                        # quotes are now active desptie not having
-                        # necessarily received a first vlm/clearing
-                        # tick.
-                        ticker = await iter_quotes.receive()
-                        quote = normalize(ticker)
-                        fqme: str = quote['fqme']
-                        await send_chan.send({fqme: quote})
+                        # quotes are now streaming.
+                        feed_is_live.set()
 
                         # last = time.time()
-                        async for ticker in iter_quotes:
+                        async for ticker in stream:
                             quote = normalize(ticker)
-                            fqme: str = quote['fqme']
-                            log.debug(
-                                f'Sending quote\n'
-                                f'{quote}'
-                            )
+                            fqme = quote['fqme']
                             await send_chan.send({fqme: quote})
 
                             # ugh, clear ticks since we've consumed them
-                            # ticker.ticks = []
+                            ticker.ticks = []
                             # last = time.time()

piker/brokers/ib/ledger.py

@@ -31,11 +31,7 @@ from typing import (
 )
 
 from bidict import bidict
-from pendulum import (
-    DateTime,
-    parse,
-    from_timestamp,
-)
+import pendulum
 from ib_insync import (
     Contract,
     Commodity,
@@ -44,7 +40,6 @@ from ib_insync import (
     CommissionReport,
 )
 
-from piker.log import get_logger
 from piker.types import Struct
 from piker.data import (
     SymbologyCache,
@@ -58,6 +53,7 @@ from piker.accounting import (
     iter_by_dt,
 )
 from ._flex_reports import parse_flex_dt
+from ._util import log
 
 if TYPE_CHECKING:
     from .api import (
@@ -65,19 +61,15 @@ if TYPE_CHECKING:
         MethodProxy,
     )
 
-log = get_logger(
-    name=__name__,
-)
 
 tx_sort: Callable = partial(
     iter_by_dt,
     parsers={
         'dateTime': parse_flex_dt,
-        'datetime': parse,
-
-        # XXX: for some some fucking 2022 and
-        # back options records.. f@#$ me..
-        'date': parse,
+        'datetime': pendulum.parse,
+        # for some some fucking 2022 and
+        # back options records...fuck me.
+        'date': pendulum.parse,
     }
 )
 
@@ -97,38 +89,15 @@ def norm_trade(
 
     conid: int = str(record.get('conId') or record['conid'])
     bs_mktid: str = str(conid)
+    comms = record.get('commission')
+    if comms is None:
+        comms = -1*record['ibCommission']
 
-    # NOTE: sometimes weird records (like BTTX?)
-    # have no field for this?
-    comms: float = -1 * (
-        record.get('commission')
-        or record.get('ibCommission')
-        or 0
-    )
-    if not comms:
-        log.warning(
-            'No commissions found for record?\n'
-            f'{pformat(record)}\n'
-        )
-
-    price: float = (
-        record.get('price')
-        or record.get('tradePrice')
-    )
-    if price is None:
-        log.warning(
-            'No `price` field found in record?\n'
-            'Skipping normalization..\n'
-            f'{pformat(record)}\n'
-        )
-        return None
+    price = record.get('price') or record['tradePrice']
 
     # the api doesn't do the -/+ on the quantity for you but flex
     # records do.. are you fucking serious ib...!?
-    size: float|int = (
-        record.get('quantity')
-        or record['shares']
-    ) * {
+    size = record.get('quantity') or record['shares'] * {
         'BOT': 1,
         'SLD': -1,
     }[record['side']]
@@ -159,31 +128,26 @@ def norm_trade(
         # otype = tail[6]
         # strike = tail[7:]
 
-        log.warning(
-            f'Skipping option contract -> NO SUPPORT YET!\n'
-            f'{symbol}\n'
-        )
+        print(f'skipping opts contract {symbol}')
         return None
 
     # timestamping is way different in API records
-    dtstr: str = record.get('datetime')
-    date: str = record.get('date')
-    flex_dtstr: str = record.get('dateTime')
+    dtstr = record.get('datetime')
+    date = record.get('date')
+    flex_dtstr = record.get('dateTime')
 
     if dtstr or date:
-        dt: DateTime = parse(dtstr or date)
+        dt = pendulum.parse(dtstr or date)
 
     elif flex_dtstr:
         # probably a flex record with a wonky non-std timestamp..
-        dt: DateTime = parse_flex_dt(record['dateTime'])
+        dt = parse_flex_dt(record['dateTime'])
 
     # special handling of symbol extraction from
     # flex records using some ad-hoc schema parsing.
-    asset_type: str = (
-        record.get('assetCategory')
-        or record.get('secType')
-        or 'STK'
-    )
+    asset_type: str = record.get(
+        'assetCategory'
+    ) or record.get('secType', 'STK')
 
     if (expiry := (
             record.get('lastTradeDateOrContractMonth')
@@ -393,7 +357,6 @@ def norm_trade_records(
         if txn is None:
             continue
 
-        # inject txns sorted by datetime
         insort(
             records,
             txn,
@@ -442,7 +405,7 @@ def api_trades_to_ledger_entries(
                     txn_dict[attr_name] = val
 
         tid = str(txn_dict['execId'])
-        dt = from_timestamp(txn_dict['time'])
+        dt = pendulum.from_timestamp(txn_dict['time'])
         txn_dict['datetime'] = str(dt)
         acctid = accounts[txn_dict['acctNumber']]
 

piker/brokers/ib/symbols.py

@@ -42,7 +42,10 @@ from piker.accounting import (
 from piker._cacheables import (
     async_lifo_cache,
 )
-from piker.log import get_logger
+
+from ._util import (
+    log,
+)
 
 if TYPE_CHECKING:
     from .api import (
@@ -50,10 +53,6 @@ if TYPE_CHECKING:
         Client,
     )
 
-log = get_logger(
-    name=__name__,
-)
-
 _futes_venues = (
     'GLOBEX',
     'NYMEX',
@@ -135,7 +134,7 @@ _adhoc_fiat_set = set((
 
 # manually discovered tick discrepancies,
 # onl god knows how or why they'd cuck these up..
-_adhoc_mkt_infos: dict[int|str, dict] = {
+_adhoc_mkt_infos: dict[int | str, dict] = {
     'vtgn.nasdaq': {'price_tick': Decimal('0.01')},
 }
 
@@ -210,10 +209,7 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
                 break
 
             ib_client = proxy._aio_ns.ib
-            log.info(
-                f'Using API client for symbol-search\n'
-                f'{ib_client}\n'
-            )
+            log.info(f'Using {ib_client} for symbol search')
 
             last = time.time()
             async for pattern in stream:
@@ -298,7 +294,7 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
                     elif stock_results:
                         break
                     # else:
-                    # await tractor.pause()
+                    await tractor.pause()
 
                     # # match against our ad-hoc set immediately
                     # adhoc_matches = fuzzy.extract(
@@ -489,7 +485,8 @@ def con2fqme(
 @async_lifo_cache()
 async def get_mkt_info(
     fqme: str,
-    proxy: MethodProxy|None = None,
+
+    proxy: MethodProxy | None = None,
 
 ) -> tuple[MktPair, ibis.ContractDetails]:
 
@@ -525,21 +522,7 @@ async def get_mkt_info(
         venue = con.primaryExchange or con.exchange
 
     price_tick: Decimal = Decimal(str(details.minTick))
-    ib_min_tick_gt_2: Decimal = Decimal('0.01')
-    if (
-        price_tick < ib_min_tick_gt_2
-    ):
-        # TODO: we need to add some kinda dynamic rounding sys
-        # to our MktPair i guess?
-        # not sure where the logic should sit, but likely inside
-        # the `.clearing._ems` i suppose...
-        log.warning(
-            'IB seems to disallow a min price tick < 0.01 '
-            'when the price is > 2.0..?\n'
-            f'Decreasing min tick precision for {fqme} to 0.01'
-        )
-        # price_tick = ib_min_tick
-        # await tractor.pause()
+    # price_tick: Decimal = Decimal('0.01')
 
     if atype == 'stock':
         # XXX: GRRRR they don't support fractional share sizes for
@@ -550,7 +533,7 @@ async def get_mkt_info(
         size_tick: Decimal = Decimal(
             str(details.minSize).rstrip('0')
         )
-        # ?TODO, there is also the Contract.sizeIncrement, bt wtf is it?
+        # |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
 
     # NOTE: this is duplicate from the .broker.norm_trade_records()
     # routine, we should factor all this parsing somewhere..

piker/brokers/ib/venues.py

@@ -1,312 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-'''
-(Multi-)venue mgmt helpers.
-
-IB generally supports all "legacy" trading venues, those mostly owned
-by ICE and friends.
-
-'''
-from __future__ import annotations
-from datetime import (  # noqa
-    datetime,
-    date,
-    tzinfo as TzInfo,
-)
-from typing import (
-    Iterator,
-    TYPE_CHECKING,
-)
-
-import exchange_calendars as xcals
-from pendulum import (
-    now,
-    Duration,
-    Interval,
-    Time,
-)
-
-if TYPE_CHECKING:
-    from ib_insync import (
-        TradingSession,
-        ContractDetails,
-    )
-    from exchange_calendars.exchange_calendars import (
-        ExchangeCalendar,
-    )
-    from pandas import (
-        # DatetimeIndex,
-        TimeDelta,
-        Timestamp,
-    )
-
-
-def has_weekend(
-    period: Interval,
-) -> bool:
-    '''
-    Predicate to for a period being within
-    days 6->0 (sat->sun).
-
-    '''
-    has_weekend: bool = False
-    for dt in period:
-        if dt.day_of_week in [0, 6]: # 0=Sunday, 6=Saturday
-            has_weekend = True
-            break
-
-    return has_weekend
-
-
-def has_holiday(
-    con_deats: ContractDetails,
-    period: Interval,
-) -> bool:
-    '''
-    Using the `exchange_calendars` lib detect if a time-gap `period`
-    is contained in a known "cash hours" closure.
-
-    '''
-    tz: str = con_deats.timeZoneId
-    exch: str = con_deats.contract.primaryExchange
-    cal: ExchangeCalendar = xcals.get_calendar(exch)
-    end: datetime = period.end
-    # _start: datetime = period.start
-    # ?TODO, can rm ya?
-    # => not that useful?
-    # dti: DatetimeIndex = cal.sessions_in_range(
-    #     _start.date(),
-    #     end.date(),
-    # )
-    prev_close: Timestamp = cal.previous_close(
-        end.date()
-    ).tz_convert(tz)
-    prev_open: Timestamp = cal.previous_open(
-        end.date()
-    ).tz_convert(tz)
-    # now do relative from prev_ values ^
-    # to get the next open which should match
-    # "contain" the end of the gap.
-    next_open: Timestamp = cal.next_open(
-        prev_open,
-    ).tz_convert(tz)
-    next_open: Timestamp = cal.next_open(
-        prev_open,
-    ).tz_convert(tz)
-    _next_close: Timestamp = cal.next_close(
-        prev_close
-    ).tz_convert(tz)
-    cash_gap: TimeDelta = next_open - prev_close
-    is_holiday_gap = (
-        cash_gap
-        > 
-        period
-    )
-    # XXX, debug
-    # breakpoint()
-    return is_holiday_gap
-
-
-def is_current_time_in_range(
-    sesh: Interval,
-    when: datetime|None = None,
-) -> bool:
-    '''
-    Check if current time is within the datetime range.
-
-    Use any/the-same timezone as provided by `start_dt.tzinfo` value
-    in the range.
-
-    '''
-    when: datetime = when or now()
-    return when in sesh
-
-
-def iter_sessions(
-    con_deats: ContractDetails,
-) -> Iterator[Interval]:
-    '''
-    Yield `pendulum.Interval`s for all
-    `ibas.ContractDetails.tradingSessions() -> TradingSession`s.
-
-    '''
-    sesh: TradingSession
-    for sesh in con_deats.tradingSessions():
-        yield Interval(*sesh)
-
-
-def sesh_times(
-    con_deats: ContractDetails,
-) -> tuple[Time, Time]:
-    '''
-    Based on the earliest trading session provided by the IB API,
-    get the (day-agnostic) times for the start/end.
-
-    '''
-    earliest_sesh: Interval = next(iter_sessions(con_deats))
-    return (
-        earliest_sesh.start.time(),
-        earliest_sesh.end.time(),
-    )
-    # ^?TODO, use `.diff()` to get point-in-time-agnostic period?
-    # https://pendulum.eustace.io/docs/#difference
-
-
-def is_venue_open(
-    con_deats: ContractDetails,
-    when: datetime|Duration|None = None,
-) -> bool:
-    '''
-    Check if market-venue is open during `when`, which defaults to
-    "now".
-
-    '''
-    sesh: Interval
-    for sesh in iter_sessions(con_deats):
-        if is_current_time_in_range(
-            sesh=sesh,
-            when=when,
-        ):
-            return True
-
-    return False
-
-
-def is_venue_closure(
-    gap: Interval,
-    con_deats: ContractDetails,
-    time_step_s: int,
-) -> bool:
-    '''
-    Check if a provided time-`gap` is just an (expected) trading
-    venue closure period.
-
-    '''
-    open: Time
-    close: Time
-    open, close = sesh_times(con_deats)
-
-    # ensure times are in mkt-native timezone
-    tz: str = con_deats.timeZoneId
-    start = gap.start.in_tz(tz)
-    start_t = start.time()
-    end = gap.end.in_tz(tz)
-    end_t = end.time()
-    if (
-        (
-            start_t in (
-                close,
-                close.subtract(seconds=time_step_s)
-            )
-            and 
-            end_t in (
-                open,
-                open.add(seconds=time_step_s),
-            )
-        )
-        or
-        has_weekend(gap)
-        or
-        has_holiday(
-            con_deats=con_deats,
-            period=gap,
-        )
-    ):
-        return True
-
-    # breakpoint()
-    return False
-
-
-# TODO, put this into `._util` and call it from here!
-#
-# NOTE, this was generated by @guille from a gpt5 prompt
-# and was originally thot to be needed before learning about
-# `ib_insync.contract.ContractDetails._parseSessions()` and
-# it's downstream meths..
-#
-# This is still likely useful to keep for now to parse the
-# `.tradingHours: str` value manually if we ever decide
-# to move off `ib_async` and implement our own `trio`/`anyio`
-# based version Bp
-#
-# >attempt to parse the retarted ib "time stampy thing" they
-# >do for "venue hours" with this.. written by
-# >gpt5-"thinking",
-#
-
-
-def parse_trading_hours(
-    spec: str,
-    tz: TzInfo|None = None
-) -> dict[
-    date,
-    tuple[datetime, datetime]
-]|None:
-    '''
-    Parse venue hours like:
-    'YYYYMMDD:HHMM-YYYYMMDD:HHMM;YYYYMMDD:CLOSED;...'
-
-    Returns `dict[date] = (open_dt, close_dt)` or `None` if
-    closed.
-
-    '''
-    if (
-        not isinstance(spec, str)
-        or
-        not spec
-    ):
-        raise ValueError('spec must be a non-empty string')
-
-    out: dict[
-        date,
-        tuple[datetime, datetime]
-    ]|None = {}
-
-    for part in (p.strip() for p in spec.split(';') if p.strip()):
-        if part.endswith(':CLOSED'):
-            day_s, _ = part.split(':', 1)
-            d = datetime.strptime(day_s, '%Y%m%d').date()
-            out[d] = None
-            continue
-
-        try:
-            start_s, end_s = part.split('-', 1)
-            start_dt = datetime.strptime(start_s, '%Y%m%d:%H%M')
-            end_dt = datetime.strptime(end_s, '%Y%m%d:%H%M')
-        except ValueError as exc:
-            raise ValueError(f'invalid segment: {part}') from exc
-
-        if tz is not None:
-            start_dt = start_dt.replace(tzinfo=tz)
-            end_dt = end_dt.replace(tzinfo=tz)
-
-        out[start_dt.date()] = (start_dt, end_dt)
-
-    return out
-
-
-# ORIG desired usage,
-#
-# TODO, for non-drunk tomorrow,
-# - call above fn and check that `output[today] is not None`
-# trading_hrs: dict = parse_trading_hours(
-#     details.tradingHours
-# )
-# liq_hrs: dict = parse_trading_hours(
-#     details.liquidHours
-    # )

piker/brokers/kraken/api.py

@@ -27,14 +27,13 @@ from typing import (
 )
 import time
 
-import httpx
 import pendulum
+import asks
 import numpy as np
 import urllib.parse
 import hashlib
 import hmac
 import base64
-import tractor
 import trio
 
 from piker import config
@@ -61,11 +60,6 @@ log = get_logger('piker.brokers.kraken')
 
 # <uri>/<version>/
 _url = 'https://api.kraken.com/0'
-
-_headers: dict[str, str] = {
-    'User-Agent': 'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
-}
-
 # TODO: this is the only backend providing this right?
 # in which case we should drop it from the defaults and
 # instead make a custom fields descr in this module!
@@ -141,15 +135,16 @@ class Client:
     def __init__(
         self,
         config: dict[str, str],
-        httpx_client: httpx.AsyncClient,
-
         name: str = '',
         api_key: str = '',
         secret: str = ''
     ) -> None:
-
-        self._sesh: httpx.AsyncClient = httpx_client
-
+        self._sesh = asks.Session(connections=4)
+        self._sesh.base_location = _url
+        self._sesh.headers.update({
+            'User-Agent':
+                'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
+        })
         self._name = name
         self._api_key = api_key
         self._secret = secret
@@ -171,9 +166,10 @@ class Client:
         method: str,
         data: dict,
     ) -> dict[str, Any]:
-        resp: httpx.Response = await self._sesh.post(
-            url=f'/public/{method}',
+        resp = await self._sesh.post(
+            path=f'/public/{method}',
             json=data,
+            timeout=float('inf')
         )
         return resproc(resp, log)
 
@@ -184,18 +180,18 @@ class Client:
         uri_path: str
     ) -> dict[str, Any]:
         headers = {
-            'Content-Type': 'application/x-www-form-urlencoded',
-            'API-Key': self._api_key,
-            'API-Sign': get_kraken_signature(
-                uri_path,
-                data,
-                self._secret,
-            ),
+            'Content-Type':
+                'application/x-www-form-urlencoded',
+            'API-Key':
+                self._api_key,
+            'API-Sign':
+                get_kraken_signature(uri_path, data, self._secret)
         }
-        resp: httpx.Response = await self._sesh.post(
-            url=f'/private/{method}',
+        resp = await self._sesh.post(
+            path=f'/private/{method}',
             data=data,
             headers=headers,
+            timeout=float('inf')
         )
         return resproc(resp, log)
 
@@ -373,7 +369,8 @@ class Client:
         #     1658347714, 'status': 'Success'}]}
 
         if xfers:
-            await tractor.pause()
+            import tractor
+            await tractor.pp()
 
         trans: dict[str, Transaction] = {}
         for entry in xfers:
@@ -501,8 +498,7 @@ class Client:
             for xkey, data in resp['result'].items():
 
                 # NOTE: always cache in pairs tables for faster lookup
-                with tractor.devx.maybe_open_crash_handler(): # as bxerr:
-                    pair = Pair(xname=xkey, **data)
+                pair = Pair(xname=xkey, **data)
 
                 # register the above `Pair` structs for all
                 # key-sets/monikers: a set of 4 (frickin) tables
@@ -669,36 +665,24 @@ class Client:
 @acm
 async def get_client() -> Client:
 
-    conf: dict[str, Any] = get_config()
-    async with httpx.AsyncClient(
-        base_url=_url,
-        headers=_headers,
+    conf = get_config()
+    if conf:
+        client = Client(
+            conf,
 
-        # TODO: is there a way to numerate this?
-        # https://www.python-httpx.org/advanced/clients/#why-use-a-client
-        # connections=4
-    ) as trio_client:
-        if conf:
-            client = Client(
-                conf,
-                httpx_client=trio_client,
+            # TODO: don't break these up and just do internal
+            # conf lookups instead..
+            name=conf['key_descr'],
+            api_key=conf['api_key'],
+            secret=conf['secret']
+        )
+    else:
+        client = Client({})
 
-                # TODO: don't break these up and just do internal
-                # conf lookups instead..
-                name=conf['key_descr'],
-                api_key=conf['api_key'],
-                secret=conf['secret']
-            )
-        else:
-            client = Client(
-                conf={},
-                httpx_client=trio_client,
-            )
+    # at startup, load all symbols, and asset info in
+    # batch requests.
+    async with trio.open_nursery() as nurse:
+        nurse.start_soon(client.get_assets)
+        await client.get_mkt_pairs()
 
-        # at startup, load all symbols, and asset info in
-        # batch requests.
-        async with trio.open_nursery() as nurse:
-            nurse.start_soon(client.get_assets)
-            await client.get_mkt_pairs()
-
-        yield client
+    yield client

piker/brokers/kraken/broker.py

@@ -62,12 +62,9 @@ from piker.clearing._messages import (
 from piker.brokers import (
     open_cached_client,
 )
-from piker.log import (
-    get_console_log,
-    get_logger,
-)
 from piker.data import open_symcache
 from .api import (
+    log,
     Client,
     BrokerError,
 )
@@ -81,8 +78,6 @@ from .ledger import (
     verify_balances,
 )
 
-log = get_logger(name=__name__)
-
 MsgUnion = Union[
     BrokerdCancel,
     BrokerdError,
@@ -180,8 +175,9 @@ async def handle_order_requests(
 
             case {
                 'account': 'kraken.spot' as account,
-                'action': 'buy'|'sell',
-            }:
+                'action': action,
+            } if action in {'buy', 'sell'}:
+
                 # validate
                 order = BrokerdOrder(**msg)
 
@@ -266,12 +262,6 @@ async def handle_order_requests(
                 } | extra
 
                 log.info(f'Submitting WS order request:\n{pformat(req)}')
-
-                # NOTE HOWTO, debug order requests
-                #
-                # if 'XRP' in pair:
-                #     await tractor.pause()
-
                 await ws.send_msg(req)
 
                 # placehold for sanity checking in relay loop
@@ -436,15 +426,9 @@ def trades2pps(
 @tractor.context
 async def open_trade_dialog(
     ctx: tractor.Context,
-    loglevel: str = 'warning',
 
 ) -> AsyncIterator[dict[str, Any]]:
 
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
-
     async with (
         # TODO: maybe bind these together and deliver
         # a tuple from `.open_cached_client()`?
@@ -560,7 +544,7 @@ async def open_trade_dialog(
             # to be reloaded.
             balances: dict[str, float] = await client.get_balances()
 
-            await verify_balances(
+            verify_balances(
                 acnt,
                 src_fiat,
                 balances,
@@ -628,18 +612,18 @@ async def open_trade_dialog(
 
                 # enter relay loop
                 await handle_order_updates(
-                    client=client,
-                    ws=ws,
-                    ws_stream=stream,
-                    ems_stream=ems_stream,
-                    apiflows=apiflows,
-                    ids=ids,
-                    reqids2txids=reqids2txids,
-                    acnt=acnt,
-                    ledger=ledger,
-                    acctid=acctid,
-                    acc_name=acc_name,
-                    token=token,
+                    client,
+                    ws,
+                    stream,
+                    ems_stream,
+                    apiflows,
+                    ids,
+                    reqids2txids,
+                    acnt,
+                    api_trans,
+                    acctid,
+                    acc_name,
+                    token,
                 )
 
 
@@ -655,8 +639,7 @@ async def handle_order_updates(
 
     # transaction records which will be updated
     # on new trade clearing events (aka order "fills")
-    ledger: TransactionLedger,
-    # ledger_trans: dict[str, Transaction],
+    ledger_trans: dict[str, Transaction],
     acctid: str,
     acc_name: str,
     token: str,
@@ -716,8 +699,7 @@ async def handle_order_updates(
                     # if tid not in ledger_trans
                 }
                 for tid, trade in trades.items():
-                    # assert tid not in ledger_trans
-                    assert tid not in ledger
+                    assert tid not in ledger_trans
                     txid = trade['ordertxid']
                     reqid = trade.get('userref')
 
@@ -765,17 +747,11 @@ async def handle_order_updates(
                     client,
                     api_name_set='wsname',
                 )
-                ppmsgs: list[BrokerdPosition] = trades2pps(
-                    acnt=acnt,
-                    ledger=ledger,
-                    acctid=acctid,
-                    new_trans=new_trans,
+                ppmsgs = trades2pps(
+                    acnt,
+                    acctid,
+                    new_trans,
                 )
-                # ppmsgs = trades2pps(
-                #     acnt,
-                #     acctid,
-                #     new_trans,
-                # )
                 for pp_msg in ppmsgs:
                     await ems_stream.send(pp_msg)
 
@@ -1101,8 +1077,6 @@ async def handle_order_updates(
                         f'Failed to {action} order {reqid}:\n'
                         f'{errmsg}'
                     )
-                    # if tractor._state.debug_mode():
-                    #     await tractor.pause()
 
                     symbol: str = 'N/A'
                     if chain := apiflows.get(reqid):

piker/brokers/kraken/symbols.py

@@ -21,6 +21,7 @@ Symbology defs and search.
 from decimal import Decimal
 
 import tractor
+from rapidfuzz import process as fuzzy
 
 from piker._cacheables import (
     async_lifo_cache,
@@ -40,13 +41,8 @@ from piker.accounting._mktinfo import (
 )
 
 
+# https://www.kraken.com/features/api#get-tradable-pairs
 class Pair(Struct):
-    '''
-    A tradable asset pair as schema-defined by,
-
-    https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs
-
-    '''
     xname: str  # idiotic bs_mktid equiv i guess?
     altname: str  # alternate pair name
     wsname: str  # WebSocket pair name (if available)
@@ -57,6 +53,7 @@ class Pair(Struct):
     lot: str  # volume lot size
 
     cost_decimals: int
+    costmin: float
     pair_decimals: int  # scaling decimal places for pair
     lot_decimals: int  # scaling decimal places for volume
 
@@ -82,7 +79,6 @@ class Pair(Struct):
     tick_size: float  # min price step size
     status: str
 
-    costmin: str|None = None  # XXX, only some mktpairs?
     short_position_limit: float = 0
     long_position_limit: float = float('inf')
 

piker/brokers/kucoin.py

@@ -16,9 +16,10 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-Kucoin cex API backend.
+Kucoin broker backend
 
 '''
+
 from contextlib import (
     asynccontextmanager as acm,
     aclosing,
@@ -41,7 +42,7 @@ import wsproto
 from uuid import uuid4
 
 from trio_typing import TaskStatus
-import httpx
+import asks
 from bidict import bidict
 import numpy as np
 import pendulum
@@ -62,7 +63,7 @@ from piker._cacheables import (
 )
 from piker.log import get_logger
 from piker.data.validate import FeedInit
-from piker.types import Struct  # NOTE, this is already a `tractor.msg.Struct`
+from piker.types import Struct
 from piker.data import (
     def_iohlcv_fields,
     match_from_pairs,
@@ -98,18 +99,9 @@ class KucoinMktPair(Struct, frozen=True):
     def size_tick(self) -> Decimal:
         return Decimal(str(self.quoteMinSize))
 
-    callauctionFirstStageStartTime: None|float
-    callauctionIsEnabled: bool
-    callauctionPriceCeiling: float|None
-    callauctionPriceFloor: float|None
-    callauctionSecondStageStartTime: float|None
-    callauctionThirdStageStartTime: float|None
-
     enableTrading: bool
-    feeCategory: int
     feeCurrency: str
     isMarginEnabled: bool
-    makerFeeCoefficient: float
     market: str
     minFunds: float
     name: str
@@ -119,10 +111,7 @@ class KucoinMktPair(Struct, frozen=True):
     quoteIncrement: float
     quoteMaxSize: float
     quoteMinSize: float
-    st: bool
     symbol: str  # our bs_mktid, kucoin's internal id
-    takerFeeCoefficient: float
-    tradingStartTime: float|None
 
 
 class AccountTrade(Struct, frozen=True):
@@ -223,12 +212,8 @@ def get_config() -> BrokerConfig | None:
 
 class Client:
 
-    def __init__(
-        self,
-        httpx_client: httpx.AsyncClient,
-    ) -> None:
-        self._http: httpx.AsyncClient = httpx_client
-        self._config: BrokerConfig|None = get_config()
+    def __init__(self) -> None:
+        self._config: BrokerConfig | None = get_config()
         self._pairs: dict[str, KucoinMktPair] = {}
         self._fqmes2mktids: bidict[str, str] = bidict()
         self._bars: list[list[float]] = []
@@ -242,24 +227,18 @@ class Client:
 
     ) -> dict[str, str | bytes]:
         '''
-        Generate authenticated request headers:
-
+        Generate authenticated request headers
         https://docs.kucoin.com/#authentication
-        https://www.kucoin.com/docs/basic-info/connection-method/authentication/creating-a-request
-        https://www.kucoin.com/docs/basic-info/connection-method/authentication/signing-a-message
 
         '''
+
         if not self._config:
             raise ValueError(
-                'No config found when trying to send authenticated request'
-            )
+                'No config found when trying to send authenticated request')
 
         str_to_sign = (
             str(int(time.time() * 1000))
-            +
-            action
-            +
-            f'/api/{api}/{endpoint.lstrip("/")}'
+            + action + f'/api/{api}/{endpoint.lstrip("/")}'
         )
 
         signature = base64.b64encode(
@@ -270,7 +249,6 @@ class Client:
             ).digest()
         )
 
-        # TODO: can we cache this between calls?
         passphrase = base64.b64encode(
             hmac.new(
                 self._config.key_secret.encode('utf-8'),
@@ -292,10 +270,8 @@ class Client:
         self,
         action: Literal['POST', 'GET'],
         endpoint: str,
-
         api: str = 'v2',
         headers: dict = {},
-
     ) -> Any:
         '''
         Generic request wrapper for Kucoin API
@@ -308,19 +284,14 @@ class Client:
                 api,
             )
 
-        req_meth: Callable = getattr(
-            self._http,
-            action.lower(),
-        )
-        res = await req_meth(
-            url=f'/{api}/{endpoint}',
-            headers=headers,
-        )
-        json: dict = res.json()
-        if (data := json.get('data')) is not None:
-            return data
+        api_url = f'https://api.kucoin.com/api/{api}/{endpoint}'
+
+        res = await asks.request(action, api_url, headers=headers)
+
+        json = res.json()
+        if 'data' in json:
+            return json['data']
         else:
-            api_url: str = self._http.base_url
             log.error(
                 f'Error making request to {api_url} ->\n'
                 f'{pformat(res)}'
@@ -340,7 +311,7 @@ class Client:
         '''
         token_type = 'private' if private else 'public'
         try:
-            data: dict[str, Any]|None = await self._request(
+            data: dict[str, Any] | None = await self._request(
                 'POST',
                 endpoint=f'bullet-{token_type}',
                 api='v1'
@@ -378,8 +349,8 @@ class Client:
             currencies: dict[str, Currency] = {}
             entries: list[dict] = await self._request(
                 'GET',
-                endpoint='currencies',
                 api='v1',
+                endpoint='currencies',
             )
             for entry in entries:
                 curr = Currency(**entry).copy()
@@ -395,22 +366,13 @@ class Client:
         dict[str, KucoinMktPair],
         bidict[str, KucoinMktPair],
     ]:
-        entries = await self._request(
-            'GET',
-            endpoint='symbols',
-        )
+        entries = await self._request('GET', 'symbols')
         log.info(f' {len(entries)} Kucoin market pairs fetched')
 
         pairs: dict[str, KucoinMktPair] = {}
         fqmes2mktids: bidict[str, str] = bidict()
         for item in entries:
-            try:
-                pair = pairs[item['name']] = KucoinMktPair(**item)
-            except TypeError as te:
-                raise TypeError(
-                    '`KucoinMktPair` and reponse fields do not match ??\n'
-                    f'{KucoinMktPair.fields_diff(item)}\n'
-                ) from te
+            pair = pairs[item['name']] = KucoinMktPair(**item)
             fqmes2mktids[
                 item['name'].lower().replace('-', '')
             ] = pair.name
@@ -605,21 +567,13 @@ def fqme_to_kucoin_sym(
 
 @acm
 async def get_client() -> AsyncGenerator[Client, None]:
-    '''
-    Load an API `Client` preconfigured from user settings
+    client = Client()
 
-    '''
-    async with (
-        httpx.AsyncClient(
-            base_url='https://api.kucoin.com/api',
-        ) as trio_client,
-    ):
-        client = Client(httpx_client=trio_client)
-        async with trio.open_nursery() as tn:
-            tn.start_soon(client.get_mkt_pairs)
-            await client.get_currencies()
+    async with trio.open_nursery() as n:
+        n.start_soon(client.get_mkt_pairs)
+        await client.get_currencies()
 
-        yield client
+    yield client
 
 
 @tractor.context
@@ -655,7 +609,7 @@ async def open_ping_task(
                 await trio.sleep((ping_interval - 1000) / 1000)
                 await ws.send_msg({'id': connect_id, 'type': 'ping'})
 
-        log.warning('Starting ping task for kucoin ws connection')
+        log.info('Starting ping task for kucoin ws connection')
         n.start_soon(ping_server)
 
         yield
@@ -667,14 +621,9 @@ async def open_ping_task(
 async def get_mkt_info(
     fqme: str,
 
-) -> tuple[
-    MktPair,
-    KucoinMktPair,
-]:
+) -> tuple[MktPair, KucoinMktPair]:
     '''
-    Query for and return both a `piker.accounting.MktPair` and
-    `KucoinMktPair` from provided `fqme: str`
-    (fully-qualified-market-endpoint).
+    Query for and return a `MktPair` and `KucoinMktPair`.
 
     '''
     async with open_cached_client('kucoin') as client:
@@ -749,8 +698,6 @@ async def stream_quotes(
 
         log.info(f'Starting up quote stream(s) for {symbols}')
         for sym_str in symbols:
-            mkt: MktPair
-            pair: KucoinMktPair
             mkt, pair = await get_mkt_info(sym_str)
             init_msgs.append(
                 FeedInit(mkt_info=mkt)
@@ -758,11 +705,7 @@ async def stream_quotes(
 
         ws: NoBsWs
         token, ping_interval = await client._get_ws_token()
-        log.info('API reported ping_interval: {ping_interval}\n')
-
-        connect_id: str = str(uuid4())
-        typ: str
-        quote: dict
+        connect_id = str(uuid4())
         async with (
             open_autorecon_ws(
                 (
@@ -776,37 +719,20 @@ async def stream_quotes(
                 ),
             ) as ws,
             open_ping_task(ws, ping_interval, connect_id),
-            aclosing(
-                iter_normed_quotes(
-                    ws, sym_str
-                )
-            ) as iter_quotes,
+            aclosing(stream_messages(ws, sym_str)) as msg_gen,
         ):
-            typ, quote = await anext(iter_quotes)
+            typ, quote = await anext(msg_gen)
 
-            # take care to not unblock here until we get a real
-            # trade quote?
-            # ^TODO, remove this right?
-            # -[ ] what often blocks chart boot/new-feed switching
-            #   since we'ere waiting for a live quote instead of just
-            #   loading history afap..
-            #  |_ XXX, not sure if we require a bit of rework to core
-            #    feed init logic or if backends justg gotta be
-            #    changed up.. feel like there was some causality
-            #    dilema prolly only seen with IB too..
-            # while typ != 'trade':
-            #     typ, quote = await anext(iter_quotes)
+            while typ != 'trade':
+                # take care to not unblock here until we get a real
+                # trade quote
+                typ, quote = await anext(msg_gen)
 
             task_status.started((init_msgs, quote))
             feed_is_live.set()
 
-            # XXX NOTE, DO NOT include the `.<backend>` suffix!
-            # OW the sampling loop will not broadcast correctly..
-            # since `bus._subscribers.setdefault(bs_fqme, set())`
-            # is used inside `.data.open_feed_bus()` !!!
-            topic: str = mkt.bs_fqme
-            async for typ, quote in iter_quotes:
-                await send_chan.send({topic: quote})
+            async for typ, msg in msg_gen:
+                await send_chan.send({sym_str: msg})
 
 
 @acm
@@ -861,7 +787,7 @@ async def subscribe(
             )
 
 
-async def iter_normed_quotes(
+async def stream_messages(
     ws: NoBsWs,
     sym: str,
 
@@ -892,9 +818,6 @@ async def iter_normed_quotes(
 
                 yield 'trade', {
                     'symbol': sym,
-                    # TODO, is 'last' even used elsewhere/a-good
-                    # semantic? can't we just read the ticks with our
-                    # .data.ticktools.frame_ticks()`/
                     'last': trade_data.price,
                     'brokerd_ts': last_trade_ts,
                     'ticks': [
@@ -987,7 +910,7 @@ async def open_history_client(
             if end_dt is None:
                 inow = round(time.time())
 
-                log.debug(
+                print(
                     f'difference in time between load and processing'
                     f'{inow - times[-1]}'
                 )

piker/brokers/questrade.py

@@ -37,12 +37,6 @@ import tractor
 from async_generator import asynccontextmanager
 import numpy as np
 import wrapt
-
-# TODO, port to `httpx`/`trio-websocket` whenver i get back to
-# writing a proper ws-api streamer for this backend (since the data
-# feeds are free now) as per GH feat-req:
-# https://github.com/pikers/piker/issues/509
-#
 import asks
 
 from ..calc import humanize, percent_change
@@ -50,19 +44,13 @@ from . import open_cached_client
 from piker._cacheables import async_lifo_cache
 from .. import config
 from ._util import resproc, BrokerError, SymbolNotFound
-from piker.log import (
+from ..log import (
     colorize_json,
+)
+from ._util import (
+    log,
     get_console_log,
 )
-from piker.log import (
-    get_logger,
-)
-
-
-log = get_logger(
-    name=__name__,
-)
-
 
 _use_practice_account = False
 _refresh_token_ep = 'https://{}login.questrade.com/oauth2/'
@@ -1211,10 +1199,7 @@ async def stream_quotes(
     # feed_type: str = 'stock',
 ) -> AsyncGenerator[str, Dict[str, Any]]:
     # XXX: required to propagate ``tractor`` loglevel to piker logging
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
+    get_console_log(loglevel)
 
     async with open_cached_client('questrade') as client:
         if feed_type == 'stock':

piker/brokers/robinhood.py

@@ -30,16 +30,9 @@ import asks
 from ._util import (
     resproc,
     BrokerError,
+    log,
 )
-from piker.calc import percent_change
-from piker.log import (
-    get_logger,
-)
-
-log = get_logger(
-    name=__name__,
-)
-
+from ..calc import percent_change
 
 _service_ep = 'https://api.robinhood.com'
 

piker/clearing/README.rst

@@ -1,49 +0,0 @@
-piker.clearing
-______________
-trade execution-n-control subsys for both live and paper trading as
-well as algo-trading manual override/interaction across any backend
-broker and data provider.
-
-avail UIs
-*********
-
-order ctl
----------
-the `piker.clearing` subsys is exposed mainly though
-the `piker chart` GUI as a "chart trader" style UX and
-is automatically enabled whenever a chart is opened.
-
-.. ^TODO, more prose here!
-
-the "manual" order control features are exposed via the
-`piker.ui.order_mode` API and can pretty much always be
-used (at least) in simulated-trading mode, aka "paper"-mode, and
-the micro-manual is as follows:
-
-``order_mode`` (
-    edge triggered activation by any of the following keys,
-    ``mouse-click`` on y-level to submit at that price
-    ):
-
-    - ``f``/ ``ctl-f`` to stage buy
-    - ``d``/ ``ctl-d`` to stage sell
-    - ``a`` to stage alert
-
-
-``search_mode`` (
-    ``ctl-l`` or ``ctl-space`` to open,
-    ``ctl-c`` or ``ctl-space`` to close
-    ) :
-
-    - begin typing to have symbol search automatically lookup
-      symbols from all loaded backend (broker) providers
-    - arrow keys and mouse click to navigate selection
-    - vi-like ``ctl-[hjkl]`` for navigation
-
-
-position (pp) mgmt
-------------------
-you can also configure your position allocation limits from the
-sidepane.
-
-.. ^TODO, explain and provide tut once more refined!

piker/clearing/_client.py

@@ -25,10 +25,7 @@ from typing import TYPE_CHECKING
 
 import trio
 import tractor
-from tractor.trionics import (
-    broadcast_receiver,
-    collapse_eg,
-)
+from tractor.trionics import broadcast_receiver
 
 from ._util import (
     log,  # sub-sys logger
@@ -171,6 +168,7 @@ class OrderClient(Struct):
 
 
 async def relay_orders_from_sync_code(
+
     client: OrderClient,
     symbol_key: str,
     to_ems_stream: tractor.MsgStream,
@@ -215,7 +213,7 @@ async def relay_orders_from_sync_code(
 async def open_ems(
     fqme: str,
     mode: str = 'live',
-    loglevel: str = 'warning',
+    loglevel: str = 'error',
 
 ) -> tuple[
     OrderClient,  # client
@@ -244,11 +242,6 @@ async def open_ems(
 
     async with maybe_open_emsd(
         broker,
-        # XXX NOTE, LOL so this determines the daemon `emsd` loglevel
-        # then FYI.. that's kinda wrong no?
-        # -[ ] shouldn't it be set by `pikerd -l` or no?
-        # -[ ] would make a lot more sense to have a subsys ctl for
-        #     levels.. like `-l emsd.info` or something?
         loglevel=loglevel,
     ) as portal:
 
@@ -288,11 +281,8 @@ async def open_ems(
             client._ems_stream = trades_stream
 
             # start sync code order msg delivery task
-            async with (
-                collapse_eg(),
-                trio.open_nursery() as tn,
-            ):
-                tn.start_soon(
+            async with trio.open_nursery() as n:
+                n.start_soon(
                     relay_orders_from_sync_code,
                     client,
                     fqme,
@@ -308,4 +298,4 @@ async def open_ems(
                 )
 
                 # stop the sync-msg-relay task on exit.
-                tn.cancel_scope.cancel()
+                n.cancel_scope.cancel()

piker/clearing/_ems.py

@@ -42,12 +42,10 @@ from bidict import bidict
 import trio
 from trio_typing import TaskStatus
 import tractor
-from tractor import trionics
 
 from ._util import (
     log,  # sub-sys logger
     get_console_log,
-    subsys,
 )
 from ..accounting._mktinfo import (
     unpack_fqme,
@@ -78,6 +76,7 @@ if TYPE_CHECKING:
 
 # TODO: numba all of this
 def mk_check(
+
     trigger_price: float,
     known_last: float,
     action: str,
@@ -137,7 +136,7 @@ class DarkBook(Struct):
             tuple[
                 Callable[[float], bool],  # predicate
                 tuple[str, ...],  # tickfilter
-                dict|Order,  # cmd / msg type
+                dict | Order,  # cmd / msg type
 
                 # live submission constraint parameters
                 float,  # percent_away max price diff
@@ -163,7 +162,7 @@ async def clear_dark_triggers(
 
     router: Router,
     brokerd_orders_stream: tractor.MsgStream,
-    quote_stream: tractor.MsgStream,
+    quote_stream: tractor.ReceiveMsgStream,  # noqa
     broker: str,
     fqme: str,
 
@@ -179,7 +178,6 @@ async def clear_dark_triggers(
     '''
     # XXX: optimize this for speed!
     # TODO:
-    # - port to the new ringbuf stuff in `tractor.ipc`!
     # - numba all this!
     # - this stream may eventually contain multiple symbols
     quote_stream._raise_on_lag = False
@@ -279,7 +277,7 @@ async def clear_dark_triggers(
 
                     # remove exec-condition from set
                     log.info(f'Removing trigger for {oid}')
-                    trigger: tuple|None = execs.pop(oid, None)
+                    trigger: tuple | None = execs.pop(oid, None)
                     if not trigger:
                         log.warning(
                             f'trigger for {oid} was already removed!?'
@@ -337,8 +335,8 @@ async def open_brokerd_dialog(
     brokermod: ModuleType,
     portal: tractor.Portal,
     exec_mode: str,
-    fqme: str|None = None,
-    loglevel: str|None = None,
+    fqme: str | None = None,
+    loglevel: str | None = None,
 
 ) -> tuple[
     tractor.MsgStream,
@@ -352,21 +350,9 @@ async def open_brokerd_dialog(
     broker backend, configuration, or client code usage.
 
     '''
-    get_console_log(
-        level=loglevel,
-        name='clearing',
-    )
-    # enable `.accounting` console since normally used by
-    # each `brokerd`.
-    get_console_log(
-        level=loglevel,
-        name='piker.accounting',
-    )
     broker: str = brokermod.name
 
-    def mk_paper_ep(
-        loglevel: str,
-    ):
+    def mk_paper_ep():
         from . import _paper_engine as paper_mod
 
         nonlocal brokermod, exec_mode
@@ -401,7 +387,6 @@ async def open_brokerd_dialog(
     for ep_name in [
         'open_trade_dialog',  # probably final name?
         'trades_dialogue',  # legacy
-        # ^!TODO, rm this since all backends ported no ?!?
     ]:
         trades_endpoint = getattr(
             brokermod,
@@ -418,21 +403,17 @@ async def open_brokerd_dialog(
 
     if (
         trades_endpoint is not None
-        or
-        exec_mode != 'paper'
+        or exec_mode != 'paper'
     ):
         # open live brokerd trades endpoint
         open_trades_endpoint = portal.open_context(
             trades_endpoint,
-            loglevel=loglevel,
         )
 
     @acm
     async def maybe_open_paper_ep():
         if exec_mode == 'paper':
-            async with mk_paper_ep(
-                loglevel=loglevel,
-            ) as msg:
+            async with mk_paper_ep() as msg:
                 yield msg
                 return
 
@@ -443,9 +424,7 @@ async def open_brokerd_dialog(
             # runtime indication that the backend can't support live
             # order ctrl yet, so boot the paperboi B0
             if first == 'paper':
-                async with mk_paper_ep(
-                    loglevel=loglevel,
-                ) as msg:
+                async with mk_paper_ep() as msg:
                     yield msg
                     return
             else:
@@ -521,7 +500,7 @@ class Router(Struct):
 
     '''
     # setup at actor spawn time
-    _tn: trio.Nursery
+    nursery: trio.Nursery
 
     # broker to book map
     books: dict[str, DarkBook] = {}
@@ -674,11 +653,7 @@ class Router(Struct):
             flume = feed.flumes[fqme]
             first_quote: dict = flume.first_quote
             book: DarkBook = self.get_dark_book(broker)
-
-            if not (last := first_quote.get('last')):
-                last: float = flume.rt_shm.array[-1]['close']
-
-            book.lasts[fqme]: float = float(last)
+            book.lasts[fqme]: float = float(first_quote['last'])
 
             async with self.maybe_open_brokerd_dialog(
                 brokermod=brokermod,
@@ -691,7 +666,7 @@ class Router(Struct):
                 # dark book clearing loop, also lives with parent
                 # daemon to allow dark order clearing while no
                 # client is connected.
-                self._tn.start_soon(
+                self.nursery.start_soon(
                     clear_dark_triggers,
                     self,
                     relay.brokerd_stream,
@@ -714,7 +689,7 @@ class Router(Struct):
 
                 # spawn a ``brokerd`` order control dialog stream
                 # that syncs lifetime with the parent `emsd` daemon.
-                self._tn.start_soon(
+                self.nursery.start_soon(
                     translate_and_relay_brokerd_events,
                     broker,
                     relay.brokerd_stream,
@@ -741,14 +716,13 @@ class Router(Struct):
             subs = self.subscribers[sub_key]
 
         sent_some: bool = False
-        for client_stream in subs.copy():
+        for client_stream in subs:
             try:
                 await client_stream.send(msg)
                 sent_some = True
             except (
                 trio.ClosedResourceError,
                 trio.BrokenResourceError,
-                tractor.TransportClosed,
             ):
                 to_remove.add(client_stream)
                 log.warning(
@@ -780,25 +754,19 @@ _router: Router = None
 @tractor.context
 async def _setup_persistent_emsd(
     ctx: tractor.Context,
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
 ) -> None:
 
     if loglevel:
-        _log = get_console_log(
-            level=loglevel,
-            name=subsys,
-        )
-        assert _log.name == 'piker.clearing'
+        get_console_log(loglevel)
 
     global _router
 
-    # open a root "service task-nursery" for the `emsd`-actor
-    async with (
-        trionics.collapse_eg(),
-        trio.open_nursery() as tn
-    ):
-        _router = Router(_tn=tn)
+    # open a root "service nursery" for the ``emsd`` actor
+    async with trio.open_nursery() as service_nursery:
+
+        _router = Router(nursery=service_nursery)
 
         # TODO: send back the full set of persistent
         # orders/execs?
@@ -845,7 +813,7 @@ async def translate_and_relay_brokerd_events(
             f'Rx brokerd trade msg:\n'
             f'{fmsg}'
         )
-        status_msg: Status|None = None
+        status_msg: Status | None = None
 
         match brokerd_msg:
             # BrokerdPosition
@@ -1042,28 +1010,14 @@ async def translate_and_relay_brokerd_events(
                 status_msg.brokerd_msg = msg
                 status_msg.src = msg.broker_details['name']
 
-                if not status_msg.req:
-                    # likely some order change state?
-                    await tractor.pause()
-                else:
-                    await router.client_broadcast(
-                        status_msg.req.symbol,
-                        status_msg,
-                    )
+                await router.client_broadcast(
+                    status_msg.req.symbol,
+                    status_msg,
+                )
 
                 if status == 'closed':
-                    log.info(
-                        f'Execution is complete!\n'
-                        f'oid: {oid!r}\n'
-                    )
-                    status_msg = book._active.pop(oid, None)
-                    if status_msg is None:
-                        log.warning(
-                            f'Order was already cleared from book ??\n'
-                            f'oid: {oid!r}\n'
-                            f'\n'
-                            f'Maybe the order cancelled before submitted ??\n'
-                        )
+                    log.info(f'Execution for {oid} is complete!')
+                    status_msg = book._active.pop(oid)
 
                 elif status == 'canceled':
                     log.cancel(f'Cancellation for {oid} is complete!')
@@ -1228,16 +1182,12 @@ async def process_client_order_cmds(
     submitting live orders immediately if requested by the client.
 
     '''
-    # TODO, only allow `msgspec.Struct` form!
-    cmd: dict
+    # cmd: dict
     async for cmd in client_order_stream:
-        log.info(
-            f'Received order cmd:\n'
-            f'{pformat(cmd)}\n'
-        )
+        log.info(f'Received order cmd:\n{pformat(cmd)}')
 
         # CAWT DAMN we need struct support!
-        oid: str = str(cmd['oid'])
+        oid = str(cmd['oid'])
 
         # register this stream as an active order dialog (msg flow) for
         # this order id such that translated message from the brokerd
@@ -1306,7 +1256,7 @@ async def process_client_order_cmds(
                 and status.resp == 'dark_open'
             ):
                 # remove from dark book clearing
-                entry: tuple|None = dark_book.triggers[fqme].pop(oid, None)
+                entry: tuple | None = dark_book.triggers[fqme].pop(oid, None)
                 if entry:
                     (
                         pred,
@@ -1343,7 +1293,7 @@ async def process_client_order_cmds(
             case {
                 'oid': oid,
                 'symbol': fqme,
-                'price': price,
+                'price': trigger_price,
                 'size': size,
                 'action': ('buy' | 'sell') as action,
                 'exec_mode': ('live' | 'paper'),
@@ -1375,7 +1325,7 @@ async def process_client_order_cmds(
 
                     symbol=sym,
                     action=action,
-                    price=price,
+                    price=trigger_price,
                     size=size,
                     account=req.account,
                 )
@@ -1397,11 +1347,7 @@ async def process_client_order_cmds(
                 # (``translate_and_relay_brokerd_events()`` above) will
                 # handle relaying the ems side responses back to
                 # the client/cmd sender from this request
-                log.info(
-                    f'Sending live order to {broker}:\n'
-                    f'{pformat(msg)}'
-                )
-
+                log.info(f'Sending live order to {broker}:\n{pformat(msg)}')
                 await brokerd_order_stream.send(msg)
 
                 # an immediate response should be ``BrokerdOrderAck``
@@ -1417,7 +1363,7 @@ async def process_client_order_cmds(
             case {
                 'oid': oid,
                 'symbol': fqme,
-                'price': price,
+                'price': trigger_price,
                 'size': size,
                 'exec_mode': exec_mode,
                 'action': action,
@@ -1445,12 +1391,7 @@ async def process_client_order_cmds(
                 if isnan(last):
                     last = flume.rt_shm.array[-1]['close']
 
-                trigger_price: float = float(price)
-                pred = mk_check(
-                    trigger_price,
-                    last,
-                    action,
-                )
+                pred = mk_check(trigger_price, last, action)
 
                 # NOTE: for dark orders currently we submit
                 # the triggered live order at a price 5 ticks
@@ -1557,7 +1498,7 @@ async def maybe_open_trade_relays(
         loglevel: str = 'info',
     ):
 
-        fqme, relay, feed, client_ready = await _router._tn.start(
+        fqme, relay, feed, client_ready = await _router.nursery.start(
             _router.open_trade_relays,
             fqme,
             exec_mode,
@@ -1587,18 +1528,19 @@ async def maybe_open_trade_relays(
 
 @tractor.context
 async def _emsd_main(
-    ctx: tractor.Context,   # becomes `ems_ctx` below
+    ctx: tractor.Context,
     fqme: str,
     exec_mode: str,  # ('paper', 'live')
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
-) -> tuple[  # `ctx.started()` value!
-    dict[  # positions
-        tuple[str, str],  # brokername, acctid
+) -> tuple[
+    dict[
+        # brokername, acctid
+        tuple[str, str],
         list[BrokerdPosition],
     ],
-    list[str],  # accounts
-    dict[str, Status],  # dialogs
+    list[str],
+    dict[str, Status],
 ]:
     '''
     EMS (sub)actor entrypoint providing the execution management
@@ -1723,5 +1665,5 @@ async def _emsd_main(
                     if not client_streams:
                         log.warning(
                             f'Order dialog is not being monitored:\n'
-                            f'{oid!r} <-> {client_stream.chan.aid.reprol()}\n'
+                            f'{oid} ->\n{client_stream._ctx.chan.uid}'
                         )

piker/clearing/_messages.py

@@ -19,7 +19,6 @@ Clearing sub-system message and protocols.
 
 """
 from __future__ import annotations
-from decimal import Decimal
 from typing import (
     Literal,
 )
@@ -72,15 +71,7 @@ class Order(Struct):
     symbol: str  # | MktPair
     account: str  # should we set a default as '' ?
 
-    # https://docs.python.org/3/library/decimal.html#decimal-objects
-    #
-    # ?TODO? decimal usage throughout?
-    # -[ ] possibly leverage the `Encoder(decimal_format='number')`
-    #  bit?
-    # |_https://jcristharif.com/msgspec/supported-types.html#decimal
-    # -[ ] should we also use it for .size?
-    #
-    price: Decimal
+    price: float
     size: float  # -ve is "sell", +ve is "buy"
 
     brokers: list[str] = []
@@ -187,7 +178,7 @@ class BrokerdOrder(Struct):
     time_ns: int
 
     symbol: str  # fqme
-    price: Decimal
+    price: float
     size: float
 
     # TODO: if we instead rely on a +ve/-ve size to determine
@@ -301,9 +292,6 @@ class BrokerdError(Struct):
 
 # TODO: yeah, so we REALLY need to completely deprecate
 # this and use the `.accounting.Position` msg-type instead..
-# -[ ] an alternative might be to add a `Position.summary() ->
-#   `PositionSummary`-msg that we generate since `Position` has a lot
-#   of fields by default we likely don't want to send over the wire?
 class BrokerdPosition(Struct):
     '''
     Position update event from brokerd.
@@ -316,4 +304,3 @@ class BrokerdPosition(Struct):
     avg_price: float
     currency: str = ''
     name: str = 'position'
-    bs_mktid: str|int|None = None

piker/clearing/_paper_engine.py

@@ -59,9 +59,9 @@ from piker.data import (
     open_symcache,
 )
 from piker.types import Struct
-from piker.log import (
+from ._util import (
+    log,  # sub-sys logger
     get_console_log,
-    get_logger,
 )
 from ._messages import (
     BrokerdCancel,
@@ -73,8 +73,6 @@ from ._messages import (
     BrokerdError,
 )
 
-log = get_logger(name=__name__)
-
 
 class PaperBoi(Struct):
     '''
@@ -299,8 +297,6 @@ class PaperBoi(Struct):
 
         # transmit pp msg to ems
         pp: Position = self.acnt.pps[bs_mktid]
-        # TODO, this will break if `require_only=True` was passed to
-        # `.update_from_ledger()`
 
         pp_msg = BrokerdPosition(
             broker=self.broker,
@@ -512,7 +508,7 @@ async def handle_order_requests(
                 reqid = await client.submit_limit(
                     oid=order.oid,
                     symbol=f'{order.symbol}.{client.broker}',
-                    price=float(order.price),
+                    price=order.price,
                     action=order.action,
                     size=order.size,
                     # XXX: by default 0 tells ``ib_insync`` methods that
@@ -552,18 +548,16 @@ _sells: defaultdict[
 
 @tractor.context
 async def open_trade_dialog(
+
     ctx: tractor.Context,
     broker: str,
-    fqme: str|None = None,  # if empty, we only boot broker mode
+    fqme: str | None = None,  # if empty, we only boot broker mode
     loglevel: str = 'warning',
 
 ) -> None:
 
-    # enable piker.clearing console log for *this* `brokerd` subactor
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
+    # enable piker.clearing console log for *this* subactor
+    get_console_log(loglevel)
 
     symcache: SymbologyCache
     async with open_symcache(get_brokermod(broker)) as symcache:
@@ -659,7 +653,6 @@ async def open_trade_dialog(
                 # in) use manually constructed table from calling
                 # the `.get_mkt_info()` provider EP above.
                 _mktmap_table=mkt_by_fqme,
-                only_require=list(mkt_by_fqme),
             )
 
             pp_msgs: list[BrokerdPosition] = []

piker/clearing/_util.py

@@ -28,14 +28,11 @@ from ..log import (
 from piker.types import Struct
 subsys: str = 'piker.clearing'
 
-log = get_logger(
-    name='piker.clearing',
-)
+log = get_logger(subsys)
 
-# TODO, oof doesn't this ignore the `loglevel` then???
 get_console_log = partial(
     get_console_log,
-    name='clearing',
+    name=subsys,
 )
 
 

piker/cli/__init__.py

@@ -61,8 +61,7 @@ def load_trans_eps(
 
     if (
         network
-        and
-        not maddrs
+        and not maddrs
     ):
         # load network section and (attempt to) connect all endpoints
         # which are reachable B)
@@ -113,83 +112,108 @@ def load_trans_eps(
     default=None,
     help='Multiaddrs to bind or contact',
 )
+# @click.option(
+#     '--tsdb',
+#     is_flag=True,
+#     help='Enable local ``marketstore`` instance'
+# )
+# @click.option(
+#     '--es',
+#     is_flag=True,
+#     help='Enable local ``elasticsearch`` instance'
+# )
 def pikerd(
     maddr: list[str] | None,
     loglevel: str,
     tl: bool,
     pdb: bool,
+    # tsdb: bool,
+    # es: bool,
 ):
     '''
-    Start the "root service actor", `pikerd`, run it until
-    cancellation.
-
-    This "root daemon" operates as the top most service-mngr and
-    subsys-as-subactor supervisor, think of it as the "init proc" of
-    any of any `piker` application or daemon-process tree.
+    Spawn the piker broker-daemon.
 
     '''
-    # from tractor.devx import maybe_open_crash_handler
-    # with maybe_open_crash_handler(pdb=False):
-    log = get_console_log(
-        level=loglevel,
-        with_tractor_log=tl,
-    )
+    from tractor.devx import maybe_open_crash_handler
+    with maybe_open_crash_handler(pdb=pdb):
+        log = get_console_log(loglevel, name='cli')
 
-    if pdb:
-        log.warning((
-            "\n"
-            "!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
-            "When a `piker` daemon crashes it will block the "
-            "task-thread until resumed from console!\n"
-            "\n"
-        ))
-
-    # service-actor registry endpoint socket-address set
-    regaddrs: list[tuple[str, int]] = []
-
-    conf, _ = config.load(
-        conf_name='conf',
-    )
-    network: dict = conf.get('network')
-    if (
-        network is None
-        and not maddr
-    ):
-        regaddrs = [(
-            _default_registry_host,
-            _default_registry_port,
-        )]
-
-    else:
-        eps: dict = load_trans_eps(
-            network,
-            maddr,
-        )
-        for layers in eps['pikerd']:
-            regaddrs.append((
-                layers['ipv4']['addr'],
-                layers['tcp']['port'],
+        if pdb:
+            log.warning((
+                "\n"
+                "!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
+                "When a `piker` daemon crashes it will block the "
+                "task-thread until resumed from console!\n"
+                "\n"
             ))
 
-    from .. import service
+        # service-actor registry endpoint socket-address set
+        regaddrs: list[tuple[str, int]] = []
 
-    async def main():
-        service_mngr: service.Services
-        async with (
-            service.open_pikerd(
-                registry_addrs=regaddrs,
-                loglevel=loglevel,
-                debug_mode=pdb,
-                # enable_transports=['uds'],
-                enable_transports=['tcp'],
-            ) as service_mngr,
+        conf, _ = config.load(
+            conf_name='conf',
+        )
+        network: dict = conf.get('network')
+        if (
+            network is None
+            and not maddr
         ):
-            assert service_mngr
-            # ?TODO? spawn all other sub-actor daemons according to
-            # multiaddress endpoint spec defined by user config
-            await trio.sleep_forever()
+            regaddrs = [(
+                _default_registry_host,
+                _default_registry_port,
+            )]
 
-    trio.run(main)
+        else:
+            eps: dict = load_trans_eps(
+                network,
+                maddr,
+            )
+            for layers in eps['pikerd']:
+                regaddrs.append((
+                    layers['ipv4']['addr'],
+                    layers['tcp']['port'],
+                ))
+
+        from .. import service
+
+        async def main():
+            service_mngr: service.Services
+
+            async with (
+                service.open_pikerd(
+                    registry_addrs=regaddrs,
+                    loglevel=loglevel,
+                    debug_mode=pdb,
+
+                ) as service_mngr,  # normally delivers a ``Services`` handle
+
+                # AsyncExitStack() as stack,
+            ):
+                # TODO: spawn all other sub-actor daemons according to
+                # multiaddress endpoint spec defined by user config
+                assert service_mngr
+
+                # if tsdb:
+                #     dname, conf = await stack.enter_async_context(
+                #         service.marketstore.start_ahab_daemon(
+                #             service_mngr,
+                #             loglevel=loglevel,
+                #         )
+                #     )
+                #     log.info(f'TSDB `{dname}` up with conf:\n{conf}')
+
+                # if es:
+                #     dname, conf = await stack.enter_async_context(
+                #         service.elastic.start_ahab_daemon(
+                #             service_mngr,
+                #             loglevel=loglevel,
+                #         )
+                #     )
+                #     log.info(f'DB `{dname}` up with conf:\n{conf}')
+
+                await trio.sleep_forever()
+
+        trio.run(main)
 
 
 @click.group(context_settings=config._context_defaults)
@@ -234,14 +258,6 @@ def cli(
     regaddr: str,
 
 ) -> None:
-    '''
-    The "root" `piker`-cmd CLI endpoint.
-
-    NOTE, this def generally relies on and requires a sub-cmd to be
-    provided by the user, OW only a `--help` msg (listing said
-    subcmds) will be dumped to console.
-
-    '''
     if configdir is not None:
         assert os.path.isdir(configdir), f"`{configdir}` is not a valid path"
         config._override_config_dir(configdir)
@@ -300,93 +316,42 @@ def cli(
 @click.option('--tl', is_flag=True, help='Enable tractor logging')
 @click.argument('ports', nargs=-1, required=False)
 @click.pass_obj
-def services(
-    config,
-    tl: bool,
-    ports: list[int],
-):
-    '''
-    List all `piker` "service deamons" to the console in
-    a `json`-table which maps each actor's UID in the form,
+def services(config, tl, ports):
 
-    `{service_name}.{subservice_name}.{UUID}`
-
-    to its (primary) IPC server address.
-
-    (^TODO, should be its multiaddr form once we support it)
-
-    Note that by convention actors which operate as "headless"
-    processes (those without GUIs/graphics, and which generally
-    parent some noteworthy subsystem) are normally suffixed by
-    a "d" such as,
-
-    - pikerd: the root runtime supervisor
-    - brokerd: a broker-backend order ctl daemon
-    - emsd: the internal dark-clearing and order routing daemon
-    - datad: a data-provider-backend data feed daemon
-    - samplerd: the real-time data sampling and clock-syncing daemon
-
-    "Headed units" are normally just given an obvious app-like name
-    with subactors indexed by `.` such as,
-    - chart: the primary modal charting iface, a Qt app
-    - chart.fsp_0: a financial-sig-proc cascade instance which
-                   delivers graphics to a parent `chart` app.
-    - polars_boi: some (presumably) `polars` using console app.
-
-    '''
-    from piker.service import (
+    from ..service import (
         open_piker_runtime,
         _default_registry_port,
         _default_registry_host,
     )
 
-    # !TODO, mk this to work with UDS!
-    host: str = _default_registry_host
+    host = _default_registry_host
     if not ports:
-        ports: list[int] = [_default_registry_port]
-
-    addr = tractor._addr.wrap_address(
-        addr=(host, ports[0])
-    )
+        ports = [_default_registry_port]
 
     async def list_services():
         nonlocal host
         async with (
             open_piker_runtime(
                 name='service_query',
-                loglevel=(
-                    config['loglevel']
-                    if tl
-                    else None
-                ),
+                loglevel=config['loglevel'] if tl else None,
             ),
-            tractor.get_registry(
-                addr=addr,
+            tractor.get_arbiter(
+                host=host,
+                port=ports[0]
             ) as portal
         ):
-            registry = await portal.run_from_ns(
-                'self',
-                'get_registry',
-            )
+            registry = await portal.run_from_ns('self', 'get_registry')
             json_d = {}
             for key, socket in registry.items():
-                json_d[key] = f'{socket}'
-
+                host, port = socket
+                json_d[key] = f'{host}:{port}'
             click.echo(f"{colorize_json(json_d)}")
 
     trio.run(list_services)
 
 
 def _load_clis() -> None:
-    '''
-    Dynamically load and register all subsys CLI endpoints (at call
-    time).
-
-    NOTE, obviously this is normally expected to be called at
-    `import` time and implicitly relies on our use of various
-    `click`/`typer` decorator APIs.
-
-    '''
+    # from ..service import elastic  # noqa
     from ..brokers import cli  # noqa
     from ..ui import cli  # noqa
     from ..watchlists import cli  # noqa
@@ -396,5 +361,5 @@ def _load_clis() -> None:
     from ..accounting import cli  # noqa
 
 
-# load all subsytem cli eps
+# load downstream cli modules
 _load_clis()

piker/config.py

@@ -41,13 +41,10 @@ from .log import get_logger
 log = get_logger('broker-config')
 
 
-# XXX NOTE: taken from `click`
-# |_https://github.com/pallets/click/blob/main/src/click/utils.py#L449
-#
-# (since apparently they have some super weirdness with SIGINT and
-# sudo.. no clue we're probably going to slowly just modify it to our
-# own version over time..)
-#
+# XXX NOTE: taken from ``click`` since apparently they have some
+# super weirdness with sigint and sudo..no clue
+# we're probably going to slowly just modify it to our own version over
+# time..
 def get_app_dir(
     app_name: str,
     roaming: bool = True,
@@ -107,15 +104,14 @@ def get_app_dir(
     # `tractor`) with the testing dir and check for it whenever we
     # detect `pytest` is being used (which it isn't under normal
     # operation).
-    # if "pytest" in sys.modules:
-    #     import tractor
-    #     actor = tractor.current_actor(err_on_no_runtime=False)
-    #     if actor:  # runtime is up
-    #         rvs = tractor._state._runtime_vars
-    #         import pdbp; pdbp.set_trace()
-    #         testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
-    #         assert testdirpath.exists(), 'piker test harness might be borked!?'
-    #         app_name = str(testdirpath)
+    if "pytest" in sys.modules:
+        import tractor
+        actor = tractor.current_actor(err_on_no_runtime=False)
+        if actor:  # runtime is up
+            rvs = tractor._state._runtime_vars
+            testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
+            assert testdirpath.exists(), 'piker test harness might be borked!?'
+            app_name = str(testdirpath)
 
     if platform.system() == 'Windows':
         key = "APPDATA" if roaming else "LOCALAPPDATA"
@@ -264,7 +260,7 @@ def load(
         MutableMapping,
     ] = tomllib.loads,
 
-    touch_if_dne: bool = True,
+    touch_if_dne: bool = False,
 
     **tomlkws,
 
@@ -273,7 +269,7 @@ def load(
     Load config file by name.
 
     If desired config is not in the top level piker-user config path then
-    pass the `path: Path` explicitly.
+    pass the ``path: Path`` explicitly.
 
     '''
     # create the $HOME/.config/piker dir if dne
@@ -288,8 +284,7 @@ def load(
 
     if (
         not path.is_file()
-        and
-        touch_if_dne
+        and touch_if_dne
     ):
         # only do a template if no path provided,
         # just touch an empty file with same name.

piker/data/_sampling.py

@@ -80,27 +80,20 @@ class Sampler:
     This non-instantiated type is meant to be a singleton within
     a `samplerd` actor-service spawned once by the user wishing to
     time-step-sample (real-time) quote feeds, see
-    `.service.maybe_open_samplerd()` and the below
-    `register_with_sampler()`.
+    ``.service.maybe_open_samplerd()`` and the below
+    ``register_with_sampler()``.
 
     '''
-    service_nursery: None|trio.Nursery = None
+    service_nursery: None | trio.Nursery = None
 
-    # TODO: we could stick these in a composed type to avoid angering
-    # the "i hate module scoped variables crowd" (yawn).
+    # TODO: we could stick these in a composed type to avoid
+    # angering the "i hate module scoped variables crowd" (yawn).
     ohlcv_shms: dict[float, list[ShmArray]] = {}
 
     # holds one-task-per-sample-period tasks which are spawned as-needed by
     # data feed requests with a given detected time step usually from
     # history loading.
-    incr_task_cs: trio.CancelScope|None = None
-
-    bcast_errors: tuple[Exception] = (
-        trio.BrokenResourceError,
-        trio.ClosedResourceError,
-        trio.EndOfChannel,
-        tractor.TransportClosed,
-    )
+    incr_task_cs: trio.CancelScope | None = None
 
     # holds all the ``tractor.Context`` remote subscriptions for
     # a particular sample period increment event: all subscribers are
@@ -249,8 +242,8 @@ class Sampler:
     async def broadcast(
         self,
         period_s: float,
-        time_stamp: float|None = None,
-        info: dict|None = None,
+        time_stamp: float | None = None,
+        info: dict | None = None,
 
     ) -> None:
         '''
@@ -265,15 +258,14 @@ class Sampler:
         subs: set
         last_ts, subs = pair
 
-        # NOTE, for debugging pub-sub issues
-        # task = trio.lowlevel.current_task()
-        # log.debug(
-        #     f'AlL-SUBS@{period_s!r}: {self.subscribers}\n'
-        #     f'PAIR: {pair}\n'
-        #     f'TASK: {task}: {id(task)}\n'
-        #     f'broadcasting {period_s} -> {last_ts}\n'
-        #     f'consumers: {subs}'
-        # )
+        task = trio.lowlevel.current_task()
+        log.debug(
+            f'SUBS {self.subscribers}\n'
+            f'PAIR {pair}\n'
+            f'TASK: {task}: {id(task)}\n'
+            f'broadcasting {period_s} -> {last_ts}\n'
+            # f'consumers: {subs}'
+        )
         borked: set[MsgStream] = set()
         sent: set[MsgStream] = set()
         while True:
@@ -290,12 +282,12 @@ class Sampler:
                         await stream.send(msg)
                         sent.add(stream)
 
-                    except self.bcast_errors as err:
+                    except (
+                        trio.BrokenResourceError,
+                        trio.ClosedResourceError
+                    ):
                         log.error(
-                            f'Connection dropped for IPC ctx due to,\n'
-                            f'{type(err)!r}\n'
-                            f'\n'
-                            f'{stream._ctx}'
+                            f'{stream._ctx.chan.uid} dropped connection'
                         )
                         borked.add(stream)
                 else:
@@ -315,7 +307,7 @@ class Sampler:
     @classmethod
     async def broadcast_all(
         self,
-        info: dict|None = None,
+        info: dict | None = None,
     ) -> None:
 
         # NOTE: take a copy of subs since removals can happen
@@ -332,22 +324,14 @@ class Sampler:
 async def register_with_sampler(
     ctx: Context,
     period_s: float,
-    shms_by_period: dict[float, dict]|None = None,
+    shms_by_period: dict[float, dict] | None = None,
 
     open_index_stream: bool = True,  # open a 2way stream for sample step msgs?
     sub_for_broadcasts: bool = True,  # sampler side to send step updates?
-    loglevel: str|None = None,
 
-) -> set[int]:
+) -> None:
 
-    get_console_log(
-        level=(
-            loglevel
-            or
-            tractor.current_actor().loglevel
-        ),
-        name=__name__,
-    )
+    get_console_log(tractor.current_actor().loglevel)
     incr_was_started: bool = False
 
     try:
@@ -372,12 +356,7 @@ async def register_with_sampler(
 
             # insert the base 1s period (for OHLC style sampling) into
             # the increment buffer set to update and shift every second.
-            if (
-                shms_by_period is not None
-                # and
-                # feed_is_live.is_set()
-                # ^TODO? pass it in instead?
-            ):
+            if shms_by_period is not None:
                 from ._sharedmem import (
                     attach_shm_array,
                     _Token,
@@ -391,17 +370,12 @@ async def register_with_sampler(
                         readonly=False,
                     )
                     shms_by_period[period] = shm
-                    Sampler.ohlcv_shms.setdefault(
-                        period,
-                        [],
-                    ).append(shm)
+                    Sampler.ohlcv_shms.setdefault(period, []).append(shm)
 
                 assert Sampler.ohlcv_shms
 
             # unblock caller
-            await ctx.started(
-                set(Sampler.ohlcv_shms.keys())
-            )
+            await ctx.started(set(Sampler.ohlcv_shms.keys()))
 
             if open_index_stream:
                 try:
@@ -420,8 +394,7 @@ async def register_with_sampler(
                 finally:
                     if (
                         sub_for_broadcasts
-                        and
-                        subs
+                        and subs
                     ):
                         try:
                             subs.remove(stream)
@@ -447,7 +420,7 @@ async def register_with_sampler(
 
 async def spawn_samplerd(
 
-    loglevel: str|None = None,
+    loglevel: str | None = None,
     **extra_tractor_kwargs
 
 ) -> bool:
@@ -484,7 +457,6 @@ async def spawn_samplerd(
                 register_with_sampler,
                 period_s=1,
                 sub_for_broadcasts=False,
-                loglevel=loglevel,
             )
             return True
 
@@ -493,7 +465,8 @@ async def spawn_samplerd(
 
 @acm
 async def maybe_open_samplerd(
-    loglevel: str|None = None,
+
+    loglevel: str | None = None,
     **pikerd_kwargs,
 
 ) -> tractor.Portal:  # noqa
@@ -518,13 +491,13 @@ async def maybe_open_samplerd(
 @acm
 async def open_sample_stream(
     period_s: float,
-    shms_by_period: dict[float, dict]|None = None,
+    shms_by_period: dict[float, dict] | None = None,
     open_index_stream: bool = True,
     sub_for_broadcasts: bool = True,
-    loglevel: str|None = None,
 
-    # cache_key: str|None = None,
-    # allow_new_sampler: bool = True,
+    cache_key: str | None = None,
+    allow_new_sampler: bool = True,
+
     ensure_is_active: bool = False,
 
 ) -> AsyncIterator[dict[str, float]]:
@@ -553,15 +526,11 @@ async def open_sample_stream(
     #         yield bistream
     # else:
 
-    ctx: tractor.Context
-    shm_periods: set[int]  # in `int`-seconds
     async with (
         # XXX: this should be singleton on a host,
         # a lone broker-daemon per provider should be
         # created for all practical purposes
-        maybe_open_samplerd(
-            loglevel=loglevel,
-        ) as portal,
+        maybe_open_samplerd() as portal,
 
         portal.open_context(
             register_with_sampler,
@@ -570,12 +539,11 @@ async def open_sample_stream(
                 'shms_by_period': shms_by_period,
                 'open_index_stream': open_index_stream,
                 'sub_for_broadcasts': sub_for_broadcasts,
-                'loglevel': loglevel,
             },
-        ) as (ctx, shm_periods)
+        ) as (ctx, first)
     ):
         if ensure_is_active:
-            assert len(shm_periods) > 1
+            assert len(first) > 1
 
         async with (
             ctx.open_stream(
@@ -593,7 +561,8 @@ async def open_sample_stream(
 
 
 async def sample_and_broadcast(
-    bus: _FeedsBus,
+
+    bus: _FeedsBus,  # noqa
     rt_shm: ShmArray,
     hist_shm: ShmArray,
     quote_stream: trio.abc.ReceiveChannel,
@@ -613,33 +582,11 @@ async def sample_and_broadcast(
 
     overruns = Counter()
 
-    # NOTE, only used for debugging live-data-feed issues, though
-    # this should be resolved more correctly in the future using the
-    # new typed-msgspec feats of `tractor`!
-    #
-    # XXX, a multiline nested `dict` formatter (since rn quote-msgs
-    # are just that).
-    # pfmt: Callable[[str], str] = mk_repr()
-
     # iterate stream delivered by broker
     async for quotes in quote_stream:
         # print(quotes)
 
-        # XXX WARNING XXX only enable for debugging bc ow can cost
-        # ALOT of perf with HF-feedz!!!
-        #
-        # log.info(
-        #     'Rx live quotes:\n'
-        #     f'{pfmt(quotes)}'
-        # )
-
-        # TODO,
-        # -[ ] `numba` or `cython`-nize this loop possibly?
-        #  |_alternatively could we do it in rust somehow by upacking
-        #    arrow msgs instead of using `msgspec`?
-        # -[ ] use `msgspec.Struct` support in new typed-msging from
-        #     `tractor` to ensure only allowed msgs are transmitted?
-        #
+        # TODO: ``numba`` this!
         for broker_symbol, quote in quotes.items():
             # TODO: in theory you can send the IPC msg *before* writing
             # to the sharedmem array to decrease latency, however, that
@@ -712,21 +659,6 @@ async def sample_and_broadcast(
             sub_key: str = broker_symbol.lower()
             subs: set[Sub] = bus.get_subs(sub_key)
 
-            # TODO, figure out how to make this useful whilst
-            # incoporating feed "pausing" ..
-            #
-            # if not subs:
-            #     all_bs_fqmes: list[str] = list(
-            #         bus._subscribers.keys()
-            #     )
-            #     log.warning(
-            #         f'No subscribers for {brokername!r} live-quote ??\n'
-            #         f'broker_symbol: {broker_symbol}\n\n'
-
-            #         f'Maybe the backend-sys symbol does not match one of,\n'
-            #         f'{pfmt(all_bs_fqmes)}\n'
-            #     )
-
             # NOTE: by default the broker backend doesn't append
             # it's own "name" into the fqme schema (but maybe it
             # should?) so we have to manually generate the correct
@@ -765,8 +697,8 @@ async def sample_and_broadcast(
 
                                 log.warning(
                                     f'Feed OVERRUN {sub_key}'
-                                    f'@{bus.brokername} -> \n'
-                                    f'feed @ {chan.aid.reprol()}\n'
+                                    '@{bus.brokername} -> \n'
+                                    f'feed @ {chan.uid}\n'
                                     f'throttle = {throttle} Hz'
                                 )
 
@@ -796,14 +728,18 @@ async def sample_and_broadcast(
                         if lags > 10:
                             await tractor.pause()
 
-                except Sampler.bcast_errors as ipc_err:
+                except (
+                    trio.BrokenResourceError,
+                    trio.ClosedResourceError,
+                    trio.EndOfChannel,
+                ):
                     ctx: Context = ipc._ctx
                     chan: Channel = ctx.chan
                     if ctx:
                         log.warning(
-                            f'Dropped `brokerd`-feed for {broker_symbol!r} due to,\n'
-                            f'x>) {ctx.cid}@{chan.uid}'
-                            f'|_{ipc_err!r}\n\n'
+                            'Dropped `brokerd`-quotes-feed connection:\n'
+                            f'{broker_symbol}:'
+                            f'{ctx.cid}@{chan.uid}'
                         )
                     if sub.throttle_rate:
                         assert ipc._closed
@@ -820,11 +756,12 @@ async def sample_and_broadcast(
 
 
 async def uniform_rate_send(
+
     rate: float,
     quote_stream: trio.abc.ReceiveChannel,
     stream: MsgStream,
 
-    task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
+    task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
 
 ) -> None:
     '''
@@ -842,16 +779,13 @@ async def uniform_rate_send(
     https://gist.github.com/njsmith/7ea44ec07e901cb78ebe1dd8dd846cb9
 
     '''
-    # ?TODO? dynamically compute the **actual** approx overhead latency per cycle
-    # instead of this magic # bidinezz?
-    throttle_period: float = 1/rate - 0.000616
-    left_to_sleep: float = throttle_period
+    # TODO: compute the approx overhead latency per cycle
+    left_to_sleep = throttle_period = 1/rate - 0.000616
 
     # send cycle state
-    first_quote: dict|None
     first_quote = last_quote = None
-    last_send: float = time.time()
-    diff: float = 0
+    last_send = time.time()
+    diff = 0
 
     task_status.started()
     ticks_by_type: dict[
@@ -862,28 +796,22 @@ async def uniform_rate_send(
     clear_types = _tick_groups['clears']
 
     while True:
+
         # compute the remaining time to sleep for this throttled cycle
-        left_to_sleep: float = throttle_period - diff
+        left_to_sleep = throttle_period - diff
 
         if left_to_sleep > 0:
-            cs: trio.CancelScope
             with trio.move_on_after(left_to_sleep) as cs:
-                sym: str
-                last_quote: dict
                 try:
                     sym, last_quote = await quote_stream.receive()
                 except trio.EndOfChannel:
-                    log.exception(
-                        f'Live stream for feed for ended?\n'
-                        f'<=c\n'
-                        f'  |_[{stream!r}\n'
-                    )
+                    log.exception(f"feed for {stream} ended?")
                     break
 
-                diff: float = time.time() - last_send
+                diff = time.time() - last_send
 
                 if not first_quote:
-                    first_quote: float = last_quote
+                    first_quote = last_quote
                     # first_quote['tbt'] = ticks_by_type
 
                 if (throttle_period - diff) > 0:
@@ -944,9 +872,7 @@ async def uniform_rate_send(
         # TODO: now if only we could sync this to the display
         # rate timing exactly lul
         try:
-            await stream.send({
-                sym: first_quote
-            })
+            await stream.send({sym: first_quote})
         except tractor.RemoteActorError as rme:
             if rme.type is not tractor._exceptions.StreamOverrun:
                 raise
@@ -957,28 +883,19 @@ async def uniform_rate_send(
                 f'{sym}:{ctx.cid}@{chan.uid}'
             )
 
-        # NOTE: any of these can be raised by `tractor`'s IPC
-        # transport-layer and we want to be highly resilient
-        # to consumers which crash or lose network connection.
-        # I.e. we **DO NOT** want to crash and propagate up to
-        # ``pikerd`` these kinds of errors!
         except (
+            # NOTE: any of these can be raised by ``tractor``'s IPC
+            # transport-layer and we want to be highly resilient
+            # to consumers which crash or lose network connection.
+            # I.e. we **DO NOT** want to crash and propagate up to
+            # ``pikerd`` these kinds of errors!
+            trio.ClosedResourceError,
+            trio.BrokenResourceError,
             ConnectionResetError,
-        ) + Sampler.bcast_errors as ipc_err:
-            match ipc_err:
-                case trio.EndOfChannel():
-                    log.info(
-                        f'{stream} terminated by peer,\n'
-                        f'{ipc_err!r}'
-                    )
-                case _:
-                    # if the feed consumer goes down then drop
-                    # out of this rate limiter
-                    log.warning(
-                        f'{stream} closed due to,\n'
-                        f'{ipc_err!r}'
-                    )
-
+        ):
+            # if the feed consumer goes down then drop
+            # out of this rate limiter
+            log.warning(f'{stream} closed')
             await stream.aclose()
             return
 

piker/data/_sharedmem.py

@@ -520,12 +520,9 @@ def open_shm_array(
 
     # "unlink" created shm on process teardown by
     # pushing teardown calls onto actor context stack
-    stack = tractor.current_actor(
-        err_on_no_runtime=False,
-    ).lifetime_stack
-    if stack:
-        stack.callback(shmarr.close)
-        stack.callback(shmarr.destroy)
+    stack = tractor.current_actor().lifetime_stack
+    stack.callback(shmarr.close)
+    stack.callback(shmarr.destroy)
 
     return shmarr
 
@@ -610,10 +607,7 @@ def attach_shm_array(
         _known_tokens[key] = token
 
     # "close" attached shm on actor teardown
-    if (actor := tractor.current_actor(
-        err_on_no_runtime=False,
-    )):
-        actor.lifetime_stack.callback(sha.close)
+    tractor.current_actor().lifetime_stack.callback(sha.close)
 
     return sha
 

piker/data/_symcache.py

@@ -31,7 +31,6 @@ from pathlib import Path
 from pprint import pformat
 from typing import (
     Any,
-    Callable,
     Sequence,
     Hashable,
     TYPE_CHECKING,
@@ -57,7 +56,7 @@ from piker.brokers import (
 )
 
 if TYPE_CHECKING:
-    from piker.accounting import (
+    from ..accounting import (
         Asset,
         MktPair,
     )
@@ -91,18 +90,6 @@ class SymbologyCache(Struct):
     # provided by the backend pkg.
     mktmaps: dict[str, MktPair] = field(default_factory=dict)
 
-    def pformat(self) -> str:
-        return (
-            f'<{type(self).__name__}(\n'
-            f' .mod: {self.mod!r}\n'
-            f' .assets: {len(self.assets)!r}\n'
-            f' .pairs: {len(self.pairs)!r}\n'
-            f' .mktmaps: {len(self.mktmaps)!r}\n'
-            f')>'
-        )
-
-    __repr__ = pformat
-
     def write_config(self) -> None:
 
         # put the backend's pair-struct type ref at the top
@@ -162,68 +149,57 @@ class SymbologyCache(Struct):
                     'Implement `Client.get_assets()`!'
                 )
 
-            get_mkt_pairs: Callable|None = getattr(
-                client,
-                'get_mkt_pairs',
-                None,
-            )
-            if not get_mkt_pairs:
+            if get_mkt_pairs := getattr(client, 'get_mkt_pairs', None):
+
+                pairs: dict[str, Struct] = await get_mkt_pairs()
+                for bs_fqme, pair in pairs.items():
+
+                    # NOTE: every backend defined pair should
+                    # declare it's ns path for roundtrip
+                    # serialization lookup.
+                    if not getattr(pair, 'ns_path', None):
+                        raise TypeError(
+                            f'Pair-struct for {self.mod.name} MUST define a '
+                            '`.ns_path: str`!\n'
+                            f'{pair}'
+                        )
+
+                    entry = await self.mod.get_mkt_info(pair.bs_fqme)
+                    if not entry:
+                        continue
+
+                    mkt: MktPair
+                    pair: Struct
+                    mkt, _pair = entry
+                    assert _pair is pair, (
+                        f'`{self.mod.name}` backend probably has a '
+                        'keying-symmetry problem between the pair-`Struct` '
+                        'returned from `Client.get_mkt_pairs()`and the '
+                        'module level endpoint: `.get_mkt_info()`\n\n'
+                        "Here's the struct diff:\n"
+                        f'{_pair - pair}'
+                    )
+                    # NOTE XXX: this means backends MUST implement
+                    # a `Struct.bs_mktid: str` field to provide
+                    # a native-keyed map to their own symbol
+                    # set(s).
+                    self.pairs[pair.bs_mktid] = pair
+
+                    # NOTE: `MktPair`s are keyed here using piker's
+                    # internal FQME schema so that search,
+                    # accounting and feed init can be accomplished
+                    # a sane, uniform, normalized basis.
+                    self.mktmaps[mkt.fqme] = mkt
+
+                self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
+                    pair,
+                )
+
+            else:
                 log.warning(
                     'No symbology cache `Pair` support for `{provider}`..\n'
                     'Implement `Client.get_mkt_pairs()`!'
                 )
-                return self
-
-            pairs: dict[str, Struct] = await get_mkt_pairs()
-            if not pairs:
-                log.warning(
-                    'No pairs from intial {provider!r} sym-cache request?\n\n'
-                    '`Client.get_mkt_pairs()` -> {pairs!r} ?'
-                )
-                return self
-
-            for bs_fqme, pair in pairs.items():
-                if not getattr(pair, 'ns_path', None):
-                    # XXX: every backend defined pair must declare
-                    # a `.ns_path: tractor.NamespacePath` to enable
-                    # roundtrip serialization lookup from a local
-                    # cache file.
-                    raise TypeError(
-                        f'Pair-struct for {self.mod.name} MUST define a '
-                        '`.ns_path: str`!\n\n'
-                        f'{pair!r}'
-                    )
-
-                entry = await self.mod.get_mkt_info(pair.bs_fqme)
-                if not entry:
-                    continue
-
-                mkt: MktPair
-                pair: Struct
-                mkt, _pair = entry
-                assert _pair is pair, (
-                    f'`{self.mod.name}` backend probably has a '
-                    'keying-symmetry problem between the pair-`Struct` '
-                    'returned from `Client.get_mkt_pairs()`and the '
-                    'module level endpoint: `.get_mkt_info()`\n\n'
-                    "Here's the struct diff:\n"
-                    f'{_pair - pair}'
-                )
-                # NOTE XXX: this means backends MUST implement
-                # a `Struct.bs_mktid: str` field to provide
-                # a native-keyed map to their own symbol
-                # set(s).
-                self.pairs[pair.bs_mktid] = pair
-
-                # NOTE: `MktPair`s are keyed here using piker's
-                # internal FQME schema so that search,
-                # accounting and feed init can be accomplished
-                # a sane, uniform, normalized basis.
-                self.mktmaps[mkt.fqme] = mkt
-
-            self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
-                pair,
-            )
 
         return self
 

piker/data/_util.py

@@ -26,9 +26,7 @@ from ..log import (
 )
 subsys: str = 'piker.data'
 
-log = get_logger(
-    name=subsys,
-)
+log = get_logger(subsys)
 
 get_console_log = partial(
     get_console_log,

piker/data/_web_bs.py

@@ -27,15 +27,14 @@ from functools import partial
 from types import ModuleType
 from typing import (
     Any,
+    Optional,
     Callable,
     AsyncContextManager,
     AsyncGenerator,
     Iterable,
-    Type,
 )
 import json
 
-import tractor
 import trio
 from trio_typing import TaskStatus
 from trio_websocket import (
@@ -68,7 +67,7 @@ class NoBsWs:
 
     '''
     # apparently we can QoS for all sorts of reasons..so catch em.
-    recon_errors: tuple[Type[Exception]] = (
+    recon_errors = (
         ConnectionClosed,
         DisconnectionTimeout,
         ConnectionRejected,
@@ -106,10 +105,7 @@ class NoBsWs:
     def connected(self) -> bool:
         return self._connected.is_set()
 
-    async def reset(
-        self,
-        timeout: float,
-    ) -> bool:
+    async def reset(self) -> None:
         '''
         Reset the underlying ws connection by cancelling
         the bg relay task and waiting for it to signal
@@ -118,31 +114,18 @@ class NoBsWs:
         '''
         self._connected = trio.Event()
         self._cs.cancel()
-        with trio.move_on_after(timeout) as cs:
-            await self._connected.wait()
-            return True
-
-        assert cs.cancelled_caught
-        return False
+        await self._connected.wait()
 
     async def send_msg(
         self,
         data: Any,
-        timeout: float = 3,
     ) -> None:
         while True:
             try:
                 msg: Any = self._dumps(data)
                 return await self._ws.send_message(msg)
             except self.recon_errors:
-                with trio.CancelScope(shield=True):
-                    reconnected: bool = await self.reset(
-                        timeout=timeout,
-                    )
-                    if not reconnected:
-                        log.warning(
-                            'Failed to reconnect after {timeout!r}s ??'
-                        )
+                await self.reset()
 
     async def recv_msg(self) -> Any:
         msg: Any = await self._rx.receive()
@@ -184,7 +167,7 @@ async def _reconnect_forever(
 
     async def proxy_msgs(
         ws: WebSocketConnection,
-        rent_cs: trio.CancelScope,  # parent cancel scope
+        pcs: trio.CancelScope,  # parent cancel scope
     ):
         '''
         Receive (under `timeout` deadline) all msgs from from underlying
@@ -208,10 +191,8 @@ async def _reconnect_forever(
                         f'{src_mod}\n'
                         f'{url} connection bail with:'
                     )
-                    with trio.CancelScope(shield=True):
-                        await trio.sleep(0.5)
-
-                    rent_cs.cancel()
+                    await trio.sleep(0.5)
+                    pcs.cancel()
 
                     # go back to reonnect loop in parent task
                     return
@@ -223,7 +204,7 @@ async def _reconnect_forever(
                         f'{src_mod}\n'
                         'WS feed seems down and slow af.. reconnecting\n'
                     )
-                    rent_cs.cancel()
+                    pcs.cancel()
 
                     # go back to reonnect loop in parent task
                     return
@@ -247,12 +228,7 @@ async def _reconnect_forever(
     nobsws._connected = trio.Event()
     task_status.started()
 
-    mc_state: trio._channel.MemoryChannelState = snd._state
-    while (
-        mc_state.open_receive_channels > 0
-        and
-        mc_state.open_send_channels > 0
-    ):
+    while not snd._closed:
         log.info(
             f'{src_mod}\n'
             f'{url} trying (RE)CONNECT'
@@ -261,11 +237,10 @@ async def _reconnect_forever(
         ws: WebSocketConnection
         try:
             async with (
+                trio.open_nursery() as n,
                 open_websocket_url(url) as ws,
-                tractor.trionics.collapse_eg(),
-                trio.open_nursery() as tn,
             ):
-                cs = nobsws._cs = tn.cancel_scope
+                cs = nobsws._cs = n.cancel_scope
                 nobsws._ws = ws
                 log.info(
                     f'{src_mod}\n'
@@ -273,7 +248,7 @@ async def _reconnect_forever(
                 )
 
                 # begin relay loop to forward msgs
-                tn.start_soon(
+                n.start_soon(
                     proxy_msgs,
                     ws,
                     cs,
@@ -287,7 +262,7 @@ async def _reconnect_forever(
 
                     # TODO: should we return an explicit sub-cs
                     # from this fixture task?
-                    await tn.start(
+                    await n.start(
                         open_fixture,
                         fixture,
                         nobsws,
@@ -297,24 +272,11 @@ async def _reconnect_forever(
                 # to let tasks run **inside** the ws open block above.
                 nobsws._connected.set()
                 await trio.sleep_forever()
+        except HandshakeError:
+            log.exception(f'Retrying connection')
 
-        except (
-            HandshakeError,
-            ConnectionRejected,
-        ):
-            log.exception('Retrying connection')
-            await trio.sleep(0.5)  # throttle
+        # ws & nursery block ends
 
-        except BaseException as _berr:
-            berr = _berr
-            log.exception(
-                'Reconnect-attempt failed ??\n'
-            )
-            with trio.CancelScope(shield=True):
-                await trio.sleep(0.2)  # throttle
-            raise berr
-
-        #|_ws & nursery block ends
         nobsws._connected = trio.Event()
         if cs.cancelled_caught:
             log.cancel(
@@ -362,53 +324,43 @@ async def open_autorecon_ws(
     connetivity errors, or some user defined recv timeout.
 
     You can provide a ``fixture`` async-context-manager which will be
-    entered/exitted around each connection reset; eg. for
-    (re)requesting subscriptions without requiring streaming setup
-    code to rerun.
+    entered/exitted around each connection reset; eg. for (re)requesting
+    subscriptions without requiring streaming setup code to rerun.
 
     '''
     snd: trio.MemorySendChannel
     rcv: trio.MemoryReceiveChannel
     snd, rcv = trio.open_memory_channel(616)
 
-    try:
-        async with (
-            tractor.trionics.collapse_eg(),
-            trio.open_nursery() as tn
-        ):
-            nobsws = NoBsWs(
-                url,
-                rcv,
-                msg_recv_timeout=msg_recv_timeout,
-            )
-            await tn.start(
-                partial(
-                    _reconnect_forever,
-                    url,
-                    snd,
-                    nobsws,
-                    fixture=fixture,
-                    reset_after=reset_after,
-                )
-            )
-            await nobsws._connected.wait()
-            assert nobsws._cs
-            assert nobsws.connected()
-            try:
-                yield nobsws
-            finally:
-                tn.cancel_scope.cancel()
-
-    except NoBsWs.recon_errors as con_err:
-        log.warning(
-            f'Entire ws-channel disconnect due to,\n'
-            f'con_err: {con_err!r}\n'
+    async with trio.open_nursery() as n:
+        nobsws = NoBsWs(
+            url,
+            rcv,
+            msg_recv_timeout=msg_recv_timeout,
         )
+        await n.start(
+            partial(
+                _reconnect_forever,
+                url,
+                snd,
+                nobsws,
+                fixture=fixture,
+                reset_after=reset_after,
+            )
+        )
+        await nobsws._connected.wait()
+        assert nobsws._cs
+        assert nobsws.connected()
+
+        try:
+            yield nobsws
+        finally:
+            n.cancel_scope.cancel()
 
 
 '''
-JSONRPC response-request style machinery for transparent multiplexing
-of msgs over a `NoBsWs`.
+JSONRPC response-request style machinery for transparent multiplexing of msgs
+over a NoBsWs.
 
 '''
 
@@ -416,8 +368,8 @@ of msgs over a `NoBsWs`.
 class JSONRPCResult(Struct):
     id: int
     jsonrpc: str = '2.0'
-    result: dict|None = None
-    error: dict|None = None
+    result: Optional[dict] = None
+    error: Optional[dict] = None
 
 
 @acm
@@ -425,82 +377,43 @@ async def open_jsonrpc_session(
     url: str,
     start_id: int = 0,
     response_type: type = JSONRPCResult,
-    msg_recv_timeout: float = float('inf'),
-    # ^NOTE, since only `deribit` is using this jsonrpc stuff atm
-    # and options mkts are generally "slow moving"..
-    #
-    # FURTHER if we break the underlying ws connection then since we
-    # don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
-    # `_reconnect_forever()`, the jsonrpc "transport pipe" get's
-    # broken and never restored with wtv init sequence is required to
-    # re-establish a working req-resp session.
-
+    request_type: Optional[type] = None,
+    request_hook: Optional[Callable] = None,
+    error_hook: Optional[Callable] = None,
 ) -> Callable[[str, dict], dict]:
-    '''
-    Init a json-RPC-over-websocket connection to the provided `url`.
-
-    A `json_rpc: Callable[[str, dict], dict` is delivered to the
-    caller for sending requests and a bg-`trio.Task` handles
-    processing of response msgs including error reporting/raising in
-    the parent/caller task.
-
-    '''
-    # NOTE, store all request msgs so we can raise errors on the
-    # caller side!
-    req_msgs: dict[int, dict] = {}
 
     async with (
-        trio.open_nursery() as tn,
-        open_autorecon_ws(
-            url=url,
-            msg_recv_timeout=msg_recv_timeout,
-        ) as ws
+        trio.open_nursery() as n,
+        open_autorecon_ws(url) as ws
     ):
-        rpc_id: Iterable[int] = count(start_id)
+        rpc_id: Iterable = count(start_id)
         rpc_results: dict[int, dict] = {}
 
-        async def json_rpc(
-            method: str,
-            params: dict,
-        ) -> dict:
+        async def json_rpc(method: str, params: dict) -> dict:
             '''
             perform a json rpc call and wait for the result, raise exception in
             case of error field present on response
             '''
-            nonlocal req_msgs
-
-            req_id: int = next(rpc_id)
             msg = {
                 'jsonrpc': '2.0',
-                'id': req_id,
+                'id': next(rpc_id),
                 'method': method,
                 'params': params
             }
             _id = msg['id']
 
-            result = rpc_results[_id] = {
+            rpc_results[_id] = {
                 'result': None,
-                'error': None,
-                'event': trio.Event(),  # signal caller resp arrived
+                'event': trio.Event()
             }
-            req_msgs[_id] = msg
 
             await ws.send_msg(msg)
 
-            # wait for reponse before unblocking requester code
             await rpc_results[_id]['event'].wait()
 
-            if (maybe_result := result['result']):
-                ret = maybe_result
-                del rpc_results[_id]
+            ret = rpc_results[_id]['result']
 
-            else:
-                err = result['error']
-                raise Exception(
-                    f'JSONRPC request failed\n'
-                    f'req: {msg}\n'
-                    f'resp: {err}\n'
-                )
+            del rpc_results[_id]
 
             if ret.error is not None:
                 raise Exception(json.dumps(ret.error, indent=4))
@@ -515,7 +428,6 @@ async def open_jsonrpc_session(
             the server side.
 
             '''
-            nonlocal req_msgs
             async for msg in ws:
                 match msg:
                     case {
@@ -539,28 +451,19 @@ async def open_jsonrpc_session(
                         'params': _,
                     }:
                         log.debug(f'Recieved\n{msg}')
+                        if request_hook:
+                            await request_hook(request_type(**msg))
 
                     case {
                         'error': error
                     }:
-                        # retreive orig request msg, set error
-                        # response in original "result" msg,
-                        # THEN FINALLY set the event to signal caller
-                        # to raise the error in the parent task.
-                        req_id: int = error['id']
-                        req_msg: dict = req_msgs[req_id]
-                        result: dict = rpc_results[req_id]
-                        result['error'] = error
-                        result['event'].set()
-                        log.error(
-                            f'JSONRPC request failed\n'
-                            f'req: {req_msg}\n'
-                            f'resp: {error}\n'
-                        )
+                        log.warning(f'Recieved\n{error}')
+                        if error_hook:
+                            await error_hook(response_type(**msg))
 
                     case _:
                         log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
 
-        tn.start_soon(recv_task)
+        n.start_soon(recv_task)
         yield json_rpc
-        tn.cancel_scope.cancel()
+        n.cancel_scope.cancel()

piker/data/feed.py

@@ -39,7 +39,6 @@ from typing import (
     AsyncContextManager,
     Awaitable,
     Sequence,
-    TYPE_CHECKING,
 )
 
 import trio
@@ -62,6 +61,7 @@ from ._util import (
     log,
     get_console_log,
 )
+from .flows import Flume
 from .validate import (
     FeedInit,
     validate_backend,
@@ -75,11 +75,6 @@ from ._sampling import (
     uniform_rate_send,
 )
 
-if TYPE_CHECKING:
-    from .flows import Flume
-    from tractor._addr import Address
-    from tractor.msg.types import Aid
-
 
 class Sub(Struct, frozen=True):
     '''
@@ -239,6 +234,7 @@ async def allocate_persistent_feed(
 
     brokername: str,
     symstr: str,
+
     loglevel: str,
     start_stream: bool = True,
     init_timeout: float = 616,
@@ -277,7 +273,7 @@ async def allocate_persistent_feed(
     # ``stream_quotes()``, a required broker backend endpoint.
     init_msgs: (
         list[FeedInit]  # new
-        |dict[str, dict[str, str]]  # legacy / deprecated
+        | dict[str, dict[str, str]]  # legacy / deprecated
     )
 
     # TODO: probably make a struct msg type for this as well
@@ -347,25 +343,18 @@ async def allocate_persistent_feed(
         izero_rt,
         rt_shm,
     ) = await bus.nursery.start(
-        partial(
-            manage_history,
-            mod=mod,
-            mkt=mkt,
-            some_data_ready=some_data_ready,
-            feed_is_live=feed_is_live,
-            loglevel=loglevel,
-        )
+        manage_history,
+        mod,
+        mkt,
+        some_data_ready,
+        feed_is_live,
     )
 
     # yield back control to starting nursery once we receive either
     # some history or a real-time quote.
-    log.info(
-        f'loading OHLCV history: {fqme!r}\n'
-    )
+    log.info(f'loading OHLCV history: {fqme}')
     await some_data_ready.wait()
 
-    # XXX, avoid cycle; it imports this mod.
-    from .flows import Flume
     flume = Flume(
 
         # TODO: we have to use this for now since currently the
@@ -462,6 +451,7 @@ async def allocate_persistent_feed(
 
 @tractor.context
 async def open_feed_bus(
+
     ctx: tractor.Context,
     brokername: str,
     symbols: list[str],  # normally expected to the broker-specific fqme
@@ -482,16 +472,13 @@ async def open_feed_bus(
 
     '''
     if loglevel is None:
-        loglevel: str = tractor.current_actor().loglevel
+        loglevel = tractor.current_actor().loglevel
 
     # XXX: required to propagate ``tractor`` loglevel to piker
     # logging
     get_console_log(
-        level=(loglevel
-            or
-            tractor.current_actor().loglevel
-        ),
-        name=__name__,
+        loglevel
+        or tractor.current_actor().loglevel
     )
 
     # local state sanity checks
@@ -506,6 +493,7 @@ async def open_feed_bus(
     sub_registered = trio.Event()
 
     flumes: dict[str, Flume] = {}
+
     for symbol in symbols:
 
         # if no cached feed for this symbol has been created for this
@@ -689,7 +677,6 @@ class Feed(Struct):
     '''
     mods: dict[str, ModuleType] = {}
     portals: dict[ModuleType, tractor.Portal] = {}
-
     flumes: dict[
         str,  # FQME
         Flume,
@@ -736,10 +723,7 @@ class Feed(Struct):
                 async for msg in stream:
                     await tx.send(msg)
 
-        async with (
-            tractor.trionics.collapse_eg(),
-            trio.open_nursery() as nurse
-        ):
+        async with trio.open_nursery() as nurse:
             # spawn a relay task for each stream so that they all
             # multiplex to a common channel.
             for brokername in mods:
@@ -802,8 +786,9 @@ async def install_brokerd_search(
 
 @acm
 async def maybe_open_feed(
+
     fqmes: list[str],
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
     **kwargs,
 
@@ -855,12 +840,13 @@ async def maybe_open_feed(
 
 @acm
 async def open_feed(
+
     fqmes: list[str],
 
-    loglevel: str|None = None,
+    loglevel: str | None = None,
     allow_overruns: bool = True,
     start_stream: bool = True,
-    tick_throttle: float|None = None,  # Hz
+    tick_throttle: float | None = None,  # Hz
 
     allow_remote_ctl_ui: bool = False,
 
@@ -887,6 +873,7 @@ async def open_feed(
 
     # one actor per brokerd for now
     brokerd_ctxs = []
+
     for brokermod, bfqmes in providers.items():
 
         # if no `brokerd` for this backend exists yet we spawn
@@ -912,19 +899,19 @@ async def open_feed(
             feed.portals[brokermod] = portal
 
             # fill out "status info" that the UI can show
-            chan: tractor.Channel = portal.chan
-            raddr: Address = chan.raddr
-            aid: Aid = chan.aid
-            # TAG_feed_status_update
+            host, port = portal.channel.raddr
+            if host == '127.0.0.1':
+                host = 'localhost'
+
             feed.status.update({
-                'actor_id': aid,
-                'actor_short_id': f'{aid.name}@{aid.pid}',
-                'ipc': chan.raddr.proto_key,
-                'ipc_addr': raddr,
+                'actor_name': portal.channel.uid[0],
+                'host': host,
+                'port': port,
                 'hist_shm': 'NA',
                 'rt_shm': 'NA',
-                'throttle_hz': tick_throttle,
+                'throttle_rate': tick_throttle,
             })
+            # feed.status.update(init_msg.pop('status', {}))
 
             # (allocate and) connect to any feed bus for this broker
             bus_ctxs.append(
@@ -956,8 +943,6 @@ async def open_feed(
 
         assert len(feed.mods) == len(feed.portals)
 
-        # XXX, avoid cycle; it imports this mod.
-        from .flows import Flume
         async with (
             trionics.gather_contexts(bus_ctxs) as ctxs,
         ):

piker/data/flows.py

@@ -36,10 +36,10 @@ from ._sharedmem import (
     ShmArray,
     _Token,
 )
-from piker.accounting import MktPair
 
 if TYPE_CHECKING:
-    from piker.data.feed import Feed
+    from ..accounting import MktPair
+    from .feed import Feed
 
 
 class Flume(Struct):
@@ -82,7 +82,7 @@ class Flume(Struct):
 
     # TODO: do we need this really if we can pull the `Portal` from
     # ``tractor``'s internals?
-    feed: Feed|None = None
+    feed: Feed | None = None
 
     @property
     def rt_shm(self) -> ShmArray:

piker/data/validate.py

@@ -113,9 +113,9 @@ def validate_backend(
             )
             if ep is None:
                 log.warning(
-                    f'Provider backend {mod.name!r} is missing '
-                    f'{daemon_name!r} support?\n'
-                    f'|_module endpoint-func missing: {name!r}\n'
+                    f'Provider backend {mod.name} is missing '
+                    f'{daemon_name} support :(\n'
+                    f'The following endpoint is missing: {name}'
                 )
 
     inits: list[

piker/fsp/_engine.py

@@ -24,7 +24,6 @@ from functools import partial
 from typing import (
     AsyncIterator,
     Callable,
-    TYPE_CHECKING,
 )
 
 import numpy as np
@@ -34,12 +33,12 @@ import tractor
 from tractor.msg import NamespacePath
 
 from piker.types import Struct
-from ..log import (
-    get_logger,
-    get_console_log,
-)
+from ..log import get_logger, get_console_log
 from .. import data
-from ..data.flows import Flume
+from ..data.feed import (
+    Flume,
+    Feed,
+)
 from ..data._sharedmem import ShmArray
 from ..data._sampling import (
     _default_delay_s,
@@ -53,9 +52,6 @@ from ._api import (
 )
 from ..toolz import Profiler
 
-if TYPE_CHECKING:
-    from ..data.feed import Feed
-
 log = get_logger(__name__)
 
 
@@ -173,10 +169,8 @@ class Cascade(Struct):
         if not synced:
             fsp: Fsp = self.fsp
             log.warning(
-                f'***DESYNCED fsp***\n'
-                f'------------------\n'
-                f'ns-path: {fsp.ns_path!r}\n'
-                f'shm-token: {src_shm.token}\n'
+                '***DESYNCED FSP***\n'
+                f'{fsp.ns_path}@{src_shm.token}\n'
                 f'step_diff: {step_diff}\n'
                 f'len_diff: {len_diff}\n'
             )
@@ -404,6 +398,7 @@ async def connect_streams(
 
 @tractor.context
 async def cascade(
+
     ctx: tractor.Context,
 
     # data feed key
@@ -417,7 +412,7 @@ async def cascade(
     shm_registry: dict[str, _Token],
 
     zero_on_step: bool = False,
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
 ) -> None:
     '''
@@ -431,17 +426,7 @@ async def cascade(
     )
 
     if loglevel:
-        log = get_console_log(
-            loglevel,
-            name=__name__,
-        )
-        # XXX TODO!
-        # figure out why this writes a dict to,
-        # `tractor._state._runtime_vars['_root_mailbox']`
-        # XD .. wtf
-        # TODO, solve this as reported in,
-        # https://www.pikers.dev/pikers/piker/issues/70
-        # await tractor.pause()
+        get_console_log(loglevel)
 
     src: Flume = Flume.from_msg(src_flume_addr)
     dst: Flume = Flume.from_msg(
@@ -484,8 +469,7 @@ async def cascade(
     # open a data feed stream with requested broker
     feed: Feed
     async with data.feed.maybe_open_feed(
-        fqmes=[fqme],
-        loglevel=loglevel,
+        [fqme],
 
         # TODO throttle tick outputs from *this* daemon since
         # it'll emit tons of ticks due to the throttle only
@@ -514,7 +498,6 @@ async def cascade(
 
         func_name: str = func.__name__
         async with (
-            tractor.trionics.collapse_eg(),  # avoid multi-taskc tb in console
             trio.open_nursery() as tn,
         ):
             # TODO: might be better to just make a "restart" method where
@@ -583,8 +566,7 @@ async def cascade(
                     # on every step msg received from the global `samplerd`
                     # service.
                     async with open_sample_stream(
-                        period_s=float(delay_s),
-                        loglevel=loglevel,
+                        float(delay_s)
                     ) as istream:
 
                         profiler(f'{func_name}: sample stream up')

piker/log.py

@@ -19,10 +19,6 @@ Log like a forester!
 """
 import logging
 import json
-import reprlib
-from typing import (
-    Callable,
-)
 
 import tractor
 from pygments import (
@@ -37,84 +33,35 @@ _proj_name: str = 'piker'
 
 
 def get_logger(
-    name: str|None = None,
-    **tractor_log_kwargs,
+    name: str = None,
+
 ) -> logging.Logger:
     '''
-    Return the package log or a sub-logger if a `name=` is provided,
-    which defaults to the calling module's pkg-namespace path.
-
-    See `tractor.log.get_logger()` for details.
+    Return the package log or a sub-log for `name` if provided.
 
     '''
-    pkg_name: str = _proj_name
-    if (
-        name
-        and
-        pkg_name in name
-    ):
-        name: str = name.lstrip(f'{_proj_name}.')
-
     return tractor.log.get_logger(
         name=name,
-        pkg_name=pkg_name,
-        **tractor_log_kwargs,
+        _root_name=_proj_name,
     )
 
 
 def get_console_log(
-    level: str|None = None,
-    name: str|None = None,
-    pkg_name: str|None = None,
-    with_tractor_log: bool = False,
-    # ?TODO, support a "log-spec" style `str|dict[str, str]` which
-    # dictates both the sublogger-key and a level?
-    # -> see similar idea in `modden`'s usage.
-    **tractor_log_kwargs,
+    level: str | None = None,
+    name: str | None = None,
 
 ) -> logging.Logger:
     '''
-    Get the package logger and enable a handler which writes to
-    stderr.
+    Get the package logger and enable a handler which writes to stderr.
 
-    Yeah yeah, i know we can use `DictConfig`.
-    You do it.. Bp
+    Yeah yeah, i know we can use ``DictConfig``. You do it...
 
     '''
-    pkg_name: str = _proj_name
-    if (
-        name
-        and
-        pkg_name in name
-    ):
-        name: str = name.lstrip(f'{_proj_name}.')
-
-    tll: str|None = None
-    if (
-        with_tractor_log is not False
-    ):
-        tll = level
-
-    elif maybe_actor := tractor.current_actor(
-        err_on_no_runtime=False,
-    ):
-        tll = maybe_actor.loglevel
-
-    if tll:
-        t_log = tractor.log.get_console_log(
-            level=tll,
-            name='tractor',  # <- XXX, force root tractor log!
-            **tractor_log_kwargs,
-        )
-        # TODO/ allow only enabling certain tractor sub-logs?
-        assert t_log.name == 'tractor'
-
     return tractor.log.get_console_log(
-        level=level,
+        level,
         name=name,
-        pkg_name=pkg_name,
-        **tractor_log_kwargs,
-    )
+        _root_name=_proj_name,
+    )  # our root logger
 
 
 def colorize_json(
@@ -137,29 +84,3 @@ def colorize_json(
         # likeable styles: algol_nu, tango, monokai
         formatters.TerminalTrueColorFormatter(style=style)
     )
-
-
-# TODO, eventually defer to the version in `modden` once
-# it becomes a dep!
-def mk_repr(
-    **repr_kws,
-) -> Callable[[str], str]:
-    '''
-    Allocate and deliver a `repr.Repr` instance with provided input
-    settings using the std-lib's `reprlib` mod,
-     * https://docs.python.org/3/library/reprlib.html
-
-    ------ Ex. ------
-    An up to 6-layer-nested `dict` as multi-line:
-    - https://stackoverflow.com/a/79102479
-    - https://docs.python.org/3/library/reprlib.html#reprlib.Repr.maxlevel
-
-    '''
-    def_kws: dict[str, int] = dict(
-        indent=2,
-        maxlevel=6,  # recursion levels
-        maxstring=66,  # match editor line-len limit
-    )
-    def_kws |= repr_kws
-    reprr = reprlib.Repr(**def_kws)
-    return reprr.repr

piker/service/_actor_runtime.py

@@ -21,6 +21,7 @@
 from __future__ import annotations
 import os
 from typing import (
+    Optional,
     Any,
     ClassVar,
 )
@@ -31,11 +32,8 @@ from contextlib import (
 import tractor
 import trio
 
-from piker.log import (
-    get_console_log,
-)
 from ._util import (
-    subsys,
+    get_console_log,
 )
 from ._mngr import (
     Services,
@@ -61,7 +59,7 @@ async def open_piker_runtime(
     registry_addrs: list[tuple[str, int]] = [],
 
     enable_modules: list[str] = [],
-    loglevel: str|None = None,
+    loglevel: Optional[str] = None,
 
     # XXX NOTE XXX: you should pretty much never want debug mode
     # for data daemons when running in production.
@@ -71,7 +69,7 @@ async def open_piker_runtime(
     # and spawn the service tree distributed per that.
     start_method: str = 'trio',
 
-    tractor_runtime_overrides: dict|None = None,
+    tractor_runtime_overrides: dict | None = None,
     **tractor_kwargs,
 
 ) -> tuple[
@@ -99,8 +97,7 @@ async def open_piker_runtime(
         # setting it as the root actor on localhost.
         registry_addrs = (
             registry_addrs
-            or
-            [_default_reg_addr]
+            or [_default_reg_addr]
         )
 
         if ems := tractor_kwargs.pop('enable_modules', None):
@@ -110,22 +107,17 @@ async def open_piker_runtime(
         async with (
             tractor.open_root_actor(
 
-                # passed through to `open_root_actor`
+                # passed through to ``open_root_actor``
                 registry_addrs=registry_addrs,
                 name=name,
-                start_method=start_method,
                 loglevel=loglevel,
                 debug_mode=debug_mode,
-
-                # XXX NOTE MEMBER DAT der's a perf hit yo!!
-                # https://greenback.readthedocs.io/en/latest/principle.html#performance
-                maybe_enable_greenback=True,
+                start_method=start_method,
 
                 # TODO: eventually we should be able to avoid
                 # having the root have more then permissions to
                 # spawn other specialized daemons I think?
                 enable_modules=enable_modules,
-                hide_tb=False,
 
                 **tractor_kwargs,
             ) as actor,
@@ -166,7 +158,8 @@ _root_modules: list[str] = [
 @acm
 async def open_pikerd(
     registry_addrs: list[tuple[str, int]],
-    loglevel: str|None = None,
+
+    loglevel: str | None = None,
 
     # XXX: you should pretty much never want debug mode
     # for data daemons when running in production.
@@ -194,6 +187,7 @@ async def open_pikerd(
 
     async with (
         open_piker_runtime(
+
             name=_root_dname,
             loglevel=loglevel,
             debug_mode=debug_mode,
@@ -206,8 +200,7 @@ async def open_pikerd(
             reg_addrs,
         ),
         tractor.open_nursery() as actor_nursery,
-        tractor.trionics.collapse_eg(),
-        trio.open_nursery() as service_tn,
+        trio.open_nursery() as service_nursery,
     ):
         for addr in reg_addrs:
             if addr not in root_actor.accept_addrs:
@@ -218,7 +211,7 @@ async def open_pikerd(
 
         # assign globally for future daemon/task creation
         Services.actor_n = actor_nursery
-        Services.service_n = service_tn
+        Services.service_n = service_nursery
         Services.debug_mode = debug_mode
 
         try:
@@ -228,7 +221,7 @@ async def open_pikerd(
             # TODO: is this more clever/efficient?
             # if 'samplerd' in Services.service_tasks:
             #     await Services.cancel_service('samplerd')
-            service_tn.cancel_scope.cancel()
+            service_nursery.cancel_scope.cancel()
 
 
 # TODO: do we even need this?
@@ -263,10 +256,7 @@ async def maybe_open_pikerd(
     loglevel: str | None = None,
     **kwargs,
 
-) -> (
-    tractor._portal.Portal
-    |ClassVar[Services]
-):
+) -> tractor._portal.Portal | ClassVar[Services]:
     '''
     If no ``pikerd`` daemon-root-actor can be found start it and
     yield up (we should probably figure out returning a portal to self
@@ -274,10 +264,7 @@ async def maybe_open_pikerd(
 
     '''
     if loglevel:
-        get_console_log(
-            name=subsys,
-            level=loglevel
-        )
+        get_console_log(loglevel)
 
     # subtle, we must have the runtime up here or portal lookup will fail
     query_name = kwargs.pop(
@@ -294,11 +281,10 @@ async def maybe_open_pikerd(
 
     registry_addrs: list[tuple[str, int]] = (
         registry_addrs
-        or
-        [_default_reg_addr]
+        or [_default_reg_addr]
     )
 
-    pikerd_portal: tractor.Portal|None
+    pikerd_portal: tractor.Portal | None
     async with (
         open_piker_runtime(
             name=query_name,

piker/service/_ahab.py

@@ -49,15 +49,13 @@ from requests.exceptions import (
     ReadTimeout,
 )
 
-from piker.log import (
-    get_console_log,
-    get_logger,
-)
 from ._mngr import Services
+from ._util import (
+    log,  # sub-sys logger
+    get_console_log,
+)
 from .. import config
 
-log = get_logger(name=__name__)
-
 
 class DockerNotStarted(Exception):
     'Prolly you dint start da daemon bruh'
@@ -338,16 +336,13 @@ class Container:
 async def open_ahabd(
     ctx: tractor.Context,
     endpoint: str,  # ns-pointer str-msg-type
-    loglevel: str = 'cancel',
+    loglevel: str | None = None,
 
     **ep_kwargs,
 
 ) -> None:
 
-    log = get_console_log(
-        level=loglevel,
-        name='piker.service',
-    )
+    log = get_console_log(loglevel or 'cancel')
 
     async with open_docker() as client:
 

piker/service/_daemon.py

@@ -28,11 +28,9 @@ from contextlib import (
 )
 
 import tractor
-from trio.lowlevel import current_task
 
-from piker.log import (
-    get_console_log,
-    get_logger,
+from ._util import (
+    log,  # sub-sys logger
 )
 from ._mngr import (
     Services,
@@ -40,17 +38,16 @@ from ._mngr import (
 from ._actor_runtime import maybe_open_pikerd
 from ._registry import find_service
 
-log = get_logger(name=__name__)
-
 
 @acm
 async def maybe_spawn_daemon(
+
     service_name: str,
     service_task_target: Callable,
 
     spawn_args: dict[str, Any],
 
-    loglevel: str|None = None,
+    loglevel: str | None = None,
     singleton: bool = False,
 
     **pikerd_kwargs,
@@ -68,99 +65,79 @@ async def maybe_spawn_daemon(
     clients.
 
     '''
-    log = get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
-    assert log.name == 'piker.service'
-
     # serialize access to this section to avoid
     # 2 or more tasks racing to create a daemon
     lock = Services.locks[service_name]
     await lock.acquire()
 
-    try:
-        async with find_service(
-            service_name,
-            registry_addrs=[('127.0.0.1', 6116)],
-        ) as portal:
-            if portal is not None:
-                lock.release()
-                yield portal
-                return
-
-        log.warning(
-            f"Couldn't find any existing {service_name}\n"
-            'Attempting to spawn new daemon-service..'
-        )
-
-        # ask root ``pikerd`` daemon to spawn the daemon we need if
-        # pikerd is not live we now become the root of the
-        # process tree
-        async with maybe_open_pikerd(
-            loglevel=loglevel,
-            **pikerd_kwargs,
-
-        ) as pikerd_portal:
-
-            # we are the root and thus are `pikerd`
-            # so spawn the target service directly by calling
-            # the provided target routine.
-            # XXX: this assumes that the target is well formed and will
-            # do the right things to setup both a sub-actor **and** call
-            # the ``_Services`` api from above to start the top level
-            # service task for that actor.
-            started: bool
-            if pikerd_portal is None:
-                started = await service_task_target(
-                    loglevel=loglevel,
-                    **spawn_args,
-                )
-
-            else:
-                # request a remote `pikerd` (service manager) to start the
-                # target daemon-task, the target can't return
-                # a non-serializable value since it is expected that service
-                # starting is non-blocking and the target task will persist
-                # running "under" or "within" the `pikerd` actor tree after
-                # the questing client disconnects. in other words this
-                # spawns a persistent daemon actor that continues to live
-                # for the lifespan of whatever the service manager inside
-                # `pikerd` says it should.
-                started = await pikerd_portal.run(
-                    service_task_target,
-                    loglevel=loglevel,
-                    **spawn_args,
-                )
-
-            if started:
-                log.info(f'Service {service_name} started!')
-
-            # block until we can discover (by IPC connection) to the newly
-            # spawned daemon-actor and then deliver the portal to the
-            # caller.
-            async with tractor.wait_for_actor(service_name) as portal:
-                lock.release()
-                yield portal
-                await portal.cancel_actor()
-
-    except BaseException as _err:
-        err = _err
-        if (
-            lock.locked()
-            and
-            lock.statistics().owner is current_task()
-        ):
-            log.exception(
-                f'Releasing stale lock after crash..?'
-                f'{err!r}\n'
-            )
+    async with find_service(
+        service_name,
+        registry_addrs=[('127.0.0.1', 6116)],
+    ) as portal:
+        if portal is not None:
             lock.release()
-            raise err
+            yield portal
+            return
+
+    log.warning(
+        f"Couldn't find any existing {service_name}\n"
+        'Attempting to spawn new daemon-service..'
+    )
+
+    # ask root ``pikerd`` daemon to spawn the daemon we need if
+    # pikerd is not live we now become the root of the
+    # process tree
+    async with maybe_open_pikerd(
+        loglevel=loglevel,
+        **pikerd_kwargs,
+
+    ) as pikerd_portal:
+
+        # we are the root and thus are `pikerd`
+        # so spawn the target service directly by calling
+        # the provided target routine.
+        # XXX: this assumes that the target is well formed and will
+        # do the right things to setup both a sub-actor **and** call
+        # the ``_Services`` api from above to start the top level
+        # service task for that actor.
+        started: bool
+        if pikerd_portal is None:
+            started = await service_task_target(
+                loglevel=loglevel,
+                **spawn_args,
+            )
+
+        else:
+            # request a remote `pikerd` (service manager) to start the
+            # target daemon-task, the target can't return
+            # a non-serializable value since it is expected that service
+            # starting is non-blocking and the target task will persist
+            # running "under" or "within" the `pikerd` actor tree after
+            # the questing client disconnects. in other words this
+            # spawns a persistent daemon actor that continues to live
+            # for the lifespan of whatever the service manager inside
+            # `pikerd` says it should.
+            started = await pikerd_portal.run(
+                service_task_target,
+                loglevel=loglevel,
+                **spawn_args,
+            )
+
+        if started:
+            log.info(f'Service {service_name} started!')
+
+        # block until we can discover (by IPC connection) to the newly
+        # spawned daemon-actor and then deliver the portal to the
+        # caller.
+        async with tractor.wait_for_actor(service_name) as portal:
+            lock.release()
+            yield portal
+            await portal.cancel_actor()
 
 
 async def spawn_emsd(
-    loglevel: str|None = None,
+
+    loglevel: str | None = None,
     **extra_tractor_kwargs
 
 ) -> bool:
@@ -197,8 +174,9 @@ async def spawn_emsd(
 
 @acm
 async def maybe_open_emsd(
+
     brokername: str,
-    loglevel: str|None = None,
+    loglevel: str | None = None,
 
     **pikerd_kwargs,
 

piker/service/_mngr.py

@@ -34,9 +34,9 @@ from tractor import (
     Portal,
 )
 
-from piker.log import get_logger
-
-log = get_logger(name=__name__)
+from ._util import (
+    log,  # sub-sys logger
+)
 
 
 # TODO: we need remote wrapping and a general soln:
@@ -109,7 +109,7 @@ class Services:
                         # wait on any context's return value
                         # and any final portal result from the
                         # sub-actor.
-                        ctx_res: Any = await ctx.wait_for_result()
+                        ctx_res: Any = await ctx.result()
 
                         # NOTE: blocks indefinitely until cancelled
                         # either by error from the target context

piker/service/_registry.py

@@ -27,29 +27,15 @@ from typing import (
 )
 
 import tractor
-from tractor import (
-    msg,
-    Actor,
-    Portal,
+from tractor import Portal
+
+from ._util import (
+    log,  # sub-sys logger
 )
 
-from piker.log import get_logger
-
-log = get_logger(name=__name__)
-
-# TODO? default path-space for UDS registry?
-# [ ] needs to be Xplatform tho!
-# _default_registry_path: Path = (
-#     Path(os.environ['XDG_RUNTIME_DIR'])
-#     /'piker'
-# )
-
 _default_registry_host: str = '127.0.0.1'
 _default_registry_port: int = 6116
-_default_reg_addr: tuple[
-    str,
-    int, # |str TODO, once we support UDS, see above.
-] = (
+_default_reg_addr: tuple[str, int] = (
     _default_registry_host,
     _default_registry_port,
 )
@@ -89,22 +75,16 @@ async def open_registry(
 
     '''
     global _tractor_kwargs
-    actor: Actor = tractor.current_actor()
-    aid: msg.Aid = actor.aid
-    uid: tuple[str, str] = aid.uid
-    preset_reg_addrs: list[
-        tuple[str, int]
-    ] = Registry.addrs
+    actor = tractor.current_actor()
+    uid = actor.uid
+    preset_reg_addrs: list[tuple[str, int]] = Registry.addrs
     if (
         preset_reg_addrs
-        and
-        addrs
+        and addrs
     ):
         if preset_reg_addrs != addrs:
             # if any(addr in preset_reg_addrs for addr in addrs):
-            diff: set[
-                tuple[str, int]
-            ] = set(preset_reg_addrs) - set(addrs)
+            diff: set[tuple[str, int]] = set(preset_reg_addrs) - set(addrs)
             if diff:
                 log.warning(
                     f'`{uid}` requested only subset of registrars: {addrs}\n'
@@ -118,39 +98,31 @@ async def open_registry(
                 )
 
     was_set: bool = False
+
     if (
         not tractor.is_root_process()
-        and
-        not Registry.addrs
+        and not Registry.addrs
     ):
         Registry.addrs.extend(actor.reg_addrs)
 
     if (
         ensure_exists
-        and
-        not Registry.addrs
+        and not Registry.addrs
     ):
         raise RuntimeError(
             f"`{uid}` registry should already exist but doesn't?"
         )
 
-    if not Registry.addrs:
+    if (
+        not Registry.addrs
+    ):
         was_set = True
-        Registry.addrs = (
-            addrs
-            or
-            [_default_reg_addr]
-        )
+        Registry.addrs = addrs or [_default_reg_addr]
 
     # NOTE: only spot this seems currently used is inside
     # `.ui._exec` which is the (eventual qtloops) bootstrapping
     # with guest mode.
-    reg_addrs: list[tuple[str, str|int]] = Registry.addrs
-    # !TODO, a struct-API to stringently allow this only in special
-    # cases?
-    # -> better would be to have some way to (atomically) rewrite
-    #    and entire `RuntimeVars`?? ideas welcome obvi..
-    _tractor_kwargs['registry_addrs'] = reg_addrs
+    _tractor_kwargs['registry_addrs'] = Registry.addrs
 
     try:
         yield Registry.addrs
@@ -174,8 +146,8 @@ async def find_service(
     | list[Portal]
     | None
 ):
-    # try:
-    reg_addrs: list[tuple[str, int|str]]
+
+    reg_addrs: list[tuple[str, int]]
     async with open_registry(
         addrs=(
             registry_addrs
@@ -185,41 +157,27 @@ async def find_service(
             or Registry.addrs
         ),
     ) as reg_addrs:
+        log.info(f'Scanning for service `{service_name}`')
 
-        log.info(
-            f'Scanning for service {service_name!r}'
-        )
+        maybe_portals: list[Portal] | Portal | None
 
         # attach to existing daemon by name if possible
-        maybe_portals: list[Portal]|Portal|None
         async with tractor.find_actor(
             service_name,
             registry_addrs=reg_addrs,
             only_first=first_only,  # if set only returns single ref
         ) as maybe_portals:
             if not maybe_portals:
-                log.info(
-                    f'Could NOT find service {service_name!r} -> {maybe_portals!r}'
-                )
                 yield None
                 return
 
-            log.info(
-                f'Found service {service_name!r} -> {maybe_portals}'
-            )
             yield maybe_portals
 
-    # except BaseException as _berr:
-    #     berr = _berr
-    #     log.exception(
-    #         'tractor.find_actor() failed with,\n'
-    #     )
-    #     raise berr
-
 
 async def check_for_service(
     service_name: str,
-) -> None|tuple[str, int]:
+
+) -> None | tuple[str, int]:
     '''
     Service daemon "liveness" predicate.
 

piker/service/_util.py

@@ -14,12 +14,20 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-Sub-sys module commons (if any ?? Bp).
+Sub-sys module commons.
 
 """
+from functools import partial
+
+from ..log import (
+    get_logger,
+    get_console_log,
+)
 subsys: str = 'piker.service'
 
-# ?TODO, if we were going to keep a `get_console_log()` in here to be
-# invoked at `import`-time, how do we dynamically hand in the
-# `level=` value? seems too early in the runtime to be injected
-# right?
+log = get_logger(subsys)
+
+get_console_log = partial(
+    get_console_log,
+    name=subsys,
+)

piker/service/elastic.py

@@ -16,7 +16,6 @@
 
 from __future__ import annotations
 from contextlib import asynccontextmanager as acm
-from pprint import pformat
 from typing import (
     Any,
     TYPE_CHECKING,
@@ -27,17 +26,12 @@ import asks
 if TYPE_CHECKING:
     import docker
     from ._ahab import DockerContainer
-    from . import (
-        Services,
-    )
 
-from piker.log import (
+from ._util import log  # sub-sys logger
+from ._util import (
     get_console_log,
-    get_logger,
 )
 
-log = get_logger(name=__name__)
-
 
 # container level config
 _config = {
@@ -73,10 +67,7 @@ def start_elasticsearch(
             elastic
 
     '''
-    get_console_log(
-        level='info',
-        name=__name__,
-    )
+    get_console_log('info', name=__name__)
 
     dcntr: DockerContainer = client.containers.run(
         'piker:elastic',

piker/service/marketstore.py

@@ -52,18 +52,17 @@ import pendulum
 # TODO: import this for specific error set expected by mkts client
 # import purerpc
 
-from piker.data.feed import maybe_open_feed
+from ..data.feed import maybe_open_feed
 from . import Services
-from piker.log import (
+from ._util import (
+    log,  # sub-sys logger
     get_console_log,
-    get_logger,
 )
 
 if TYPE_CHECKING:
     import docker
     from ._ahab import DockerContainer
 
-log = get_logger(name=__name__)
 
 
 # ahabd-supervisor and container level config

piker/storage/__init__.py

@@ -43,6 +43,7 @@ from typing import (
 
 import numpy as np
 
+
 from .. import config
 from ..service import (
     check_for_service,
@@ -151,10 +152,7 @@ class StorageConnectionError(ConnectionError):
 
     '''
 
-def get_storagemod(
-    name: str,
-
-) -> ModuleType:
+def get_storagemod(name: str) -> ModuleType:
     mod: ModuleType = import_module(
         '.' + name,
         'piker.storage',
@@ -167,12 +165,9 @@ def get_storagemod(
 
 @acm
 async def open_storage_client(
-    backend: str|None = None,
+    backend: str | None = None,
 
-) -> tuple[
-    ModuleType,
-    StorageClient,
-]:
+) -> tuple[ModuleType, StorageClient]:
     '''
     Load the ``StorageClient`` for named backend.
 
@@ -272,10 +267,7 @@ async def open_tsdb_client(
     from ..data.feed import maybe_open_feed
 
     async with (
-        open_storage_client() as (
-            _,
-            storage,
-        ),
+        open_storage_client() as (_, storage),
 
         maybe_open_feed(
             [fqme],
@@ -283,7 +275,7 @@ async def open_tsdb_client(
 
         ) as feed,
     ):
-        profiler(f'opened feed for {fqme!r}')
+        profiler(f'opened feed for {fqme}')
 
         # to_append = feed.hist_shm.array
         # to_prepend = None

piker/storage/cli.py

@@ -19,10 +19,16 @@ Storage middle-ware CLIs.
 
 """
 from __future__ import annotations
+# from datetime import datetime
+# from contextlib import (
+#     AsyncExitStack,
+# )
 from pathlib import Path
+from math import copysign
 import time
 from types import ModuleType
 from typing import (
+    Any,
     TYPE_CHECKING,
 )
 
@@ -41,6 +47,7 @@ from piker.data import (
     ShmArray,
 )
 from piker import tsp
+from piker.data._formatters import BGM
 from . import log
 from . import (
     __tsdbs__,
@@ -235,12 +242,122 @@ def anal(
     trio.run(main)
 
 
+async def markup_gaps(
+    fqme: str,
+    timeframe: float,
+    actl: AnnotCtl,
+    wdts: pl.DataFrame,
+    gaps: pl.DataFrame,
+
+) -> dict[int, dict]:
+    '''
+    Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
+    with rectangles.
+
+    '''
+    aids: dict[int] = {}
+    for i in range(gaps.height):
+
+        row: pl.DataFrame = gaps[i]
+
+        # the gap's RIGHT-most bar's OPEN value
+        # at that time (sample) step.
+        iend: int = row['index'][0]
+        # dt: datetime = row['dt'][0]
+        # dt_prev: datetime = row['dt_prev'][0]
+        # dt_end_t: float = dt.timestamp()
+
+
+        # TODO: can we eventually remove this
+        # once we figure out why the epoch cols
+        # don't match?
+        # TODO: FIX HOW/WHY these aren't matching
+        # and are instead off by 4hours (EST
+        # vs. UTC?!?!)
+        # end_t: float = row['time']
+        # assert (
+        #     dt.timestamp()
+        #     ==
+        #     end_t
+        # )
+
+        # the gap's LEFT-most bar's CLOSE value
+        # at that time (sample) step.
+        prev_r: pl.DataFrame = wdts.filter(
+            pl.col('index') == iend - 1
+        )
+        # XXX: probably a gap in the (newly sorted or de-duplicated)
+        # dt-df, so we might need to re-index first..
+        if prev_r.is_empty():
+            await tractor.pause()
+
+        istart: int = prev_r['index'][0]
+        # dt_start_t: float = dt_prev.timestamp()
+
+        # start_t: float = prev_r['time']
+        # assert (
+        #     dt_start_t
+        #     ==
+        #     start_t
+        # )
+
+        # TODO: implement px-col width measure
+        # and ensure at least as many px-cols
+        # shown per rect as configured by user.
+        # gap_w: float = abs((iend - istart))
+        # if gap_w < 6:
+        #     margin: float = 6
+        #     iend += margin
+        #     istart -= margin
+
+        rect_gap: float = BGM*3/8
+        opn: float = row['open'][0]
+        ro: tuple[float, float] = (
+            # dt_end_t,
+            iend + rect_gap + 1,
+            opn,
+        )
+        cls: float = prev_r['close'][0]
+        lc: tuple[float, float] = (
+            # dt_start_t,
+            istart - rect_gap, # + 1 ,
+            cls,
+        )
+
+        color: str = 'dad_blue'
+        diff: float = cls - opn
+        sgn: float = copysign(1, diff)
+        color: str = {
+            -1: 'buy_green',
+            1: 'sell_red',
+        }[sgn]
+
+        rect_kwargs: dict[str, Any] = dict(
+            fqme=fqme,
+            timeframe=timeframe,
+            start_pos=lc,
+            end_pos=ro,
+            color=color,
+        )
+
+        aid: int = await actl.add_rect(**rect_kwargs)
+        assert aid
+        aids[aid] = rect_kwargs
+
+    # tell chart to redraw all its
+    # graphics view layers Bo
+    await actl.redraw(
+        fqme=fqme,
+        timeframe=timeframe,
+    )
+    return aids
+
+
 @store.command()
 def ldshm(
     fqme: str,
     write_parquet: bool = True,
     reload_parquet_to_shm: bool = True,
-    pdb: bool = False,  # --pdb passed?
 
 ) -> None:
     '''
@@ -260,7 +377,7 @@ def ldshm(
             open_piker_runtime(
                 'polars_boi',
                 enable_modules=['piker.data._sharedmem'],
-                debug_mode=pdb,
+                debug_mode=True,
             ),
             open_storage_client() as (
                 mod,
@@ -269,8 +386,6 @@ def ldshm(
             open_annot_ctl() as actl,
         ):
             shm_df: pl.DataFrame | None = None
-            tf2aids: dict[float, dict] = {}
-
             for (
                 shmfile,
                 shm,
@@ -280,19 +395,17 @@ def ldshm(
 
                 times: np.ndarray = shm.array['time']
                 d1: float = float(times[-1] - times[-2])
-                d2: float = 0
-                # XXX, take a median sample rate if sufficient data
-                if times.size > 2:
-                    d2: float = float(times[-2] - times[-3])
-                    med: float = np.median(np.diff(times))
-                    if (
-                        d1 < 1.
-                        and d2 < 1.
-                        and med < 1.
-                    ):
-                        raise ValueError(
-                            f'Something is wrong with time period for {shm}:\n{times}'
-                        )
+                d2: float = float(times[-2] - times[-3])
+                med: float = np.median(np.diff(times))
+                if (
+                    d1 < 1.
+                    and d2 < 1.
+                    and med < 1.
+                ):
+                    raise ValueError(
+                        f'Something is wrong with time period for {shm}:\n{times}'
+                    )
+
                 period_s: float = float(max(d1, d2, med))
 
                 null_segs: tuple = tsp.get_null_segs(
@@ -302,9 +415,7 @@ def ldshm(
 
                 # TODO: call null-seg fixer somehow?
                 if null_segs:
-
-                    if tractor._state.is_debug_mode():
-                        await tractor.pause()
+                    await tractor.pause()
                 #     async with (
                 #         trio.open_nursery() as tn,
                 #         mod.open_history_client(
@@ -328,37 +439,11 @@ def ldshm(
                     wdts,
                     deduped,
                     diff,
-                    valid_races,
-                    dq_issues,
-                ) = tsp.dedupe_ohlcv_smart(
+                ) = tsp.dedupe(
                     shm_df,
+                    period=period_s,
                 )
 
-                # Report duplicate analysis
-                if diff > 0:
-                    log.info(
-                        f'Removed {diff} duplicate timestamp(s)\n'
-                    )
-                    if valid_races is not None:
-                        identical: int = (
-                            valid_races
-                            .filter(pl.col('identical_bars'))
-                            .height
-                        )
-                        monotonic: int = valid_races.height - identical
-                        log.info(
-                            f'Valid race conditions: {valid_races.height}\n'
-                            f'  - Identical bars: {identical}\n'
-                            f'  - Volume monotonic: {monotonic}\n'
-                        )
-
-                    if dq_issues is not None:
-                        log.warning(
-                            f'DATA QUALITY ISSUES from provider: '
-                            f'{dq_issues.height} timestamp(s)\n'
-                            f'{dq_issues}\n'
-                        )
-
                 # detect gaps from in expected (uniform OHLC) sample period
                 step_gaps: pl.DataFrame = tsp.detect_time_gaps(
                     deduped,
@@ -373,8 +458,7 @@ def ldshm(
 
                     # TODO: actually pull the exact duration
                     # expected for each venue operational period?
-                    # gap_dt_unit='day',
-                    gap_dt_unit='day',
+                    gap_dt_unit='days',
                     gap_thresh=1,
                 )
 
@@ -385,11 +469,8 @@ def ldshm(
                 if (
                     not venue_gaps.is_empty()
                     or (
-                        not step_gaps.is_empty()
-                        # XXX, i presume i put this bc i was guarding
-                        # for ib venue gaps?
-                        # and
-                        # period_s < 60
+                        period_s < 60
+                        and not step_gaps.is_empty()
                     )
                 ):
                     # write repaired ts to parquet-file?
@@ -438,35 +519,23 @@ def ldshm(
                     do_markup_gaps: bool = True
                     if do_markup_gaps:
                         new_df: pl.DataFrame = tsp.np2pl(new)
-                        aids: dict = await tsp._annotate.markup_gaps(
+                        aids: dict = await markup_gaps(
                             fqme,
                             period_s,
                             actl,
                             new_df,
                             step_gaps,
                         )
+
                         # last chance manual overwrites in REPL
-                        # await tractor.pause()
-                        if not aids:
-                            log.warning(
-                                f'No gaps were found !?\n'
-                                f'fqme: {fqme!r}\n'
-                                f'timeframe: {period_s!r}\n'
-                                f"WELL THAT'S GOOD NOOZ!\n"
-                            )
-                        tf2aids[period_s] = aids
+                        await tractor.pause()
+                        assert aids
 
                 else:
-                    # No significant gaps to handle, but may have had
-                    # duplicates removed (valid race conditions are ok)
-                    if diff > 0 and dq_issues is not None:
-                        log.warning(
-                            'Found duplicates with data quality issues '
-                            'but no significant time gaps!\n'
-                        )
+                    # allow interaction even when no ts problems.
+                    await tractor.pause()
+                    # assert not diff
 
-            await tractor.pause()
-            log.info('Exiting TSP shm anal-izer!')
 
             if shm_df is None:
                 log.error(

piker/storage/nativedb.py

@@ -161,13 +161,7 @@ class NativeStorageClient:
 
     def index_files(self):
         for path in self._datadir.iterdir():
-            if (
-                path.is_dir()
-                or
-                '.parquet' not in str(path)
-                # or
-                # path.name in {'borked', 'expired',}
-            ):
+            if path.name in {'borked', 'expired',}:
                 continue
 
             key: str = path.name.rstrip('.parquet')

piker/tsp/_anal.py

@@ -54,10 +54,10 @@ from ..log import (
 # for "time series processing"
 subsys: str = 'piker.tsp'
 
-log = get_logger(name=__name__)
+log = get_logger(subsys)
 get_console_log = partial(
     get_console_log,
-    name=subsys,  # activate for subsys-pkg "downward"
+    name=subsys,
 )
 
 # NOTE: union type-defs to handle generic `numpy` and `polars` types
@@ -275,18 +275,6 @@ def get_null_segs(
         # diff of abs index steps between each zeroed row
         absi_zdiff: np.ndarray = np.diff(absi_zeros)
 
-        if zero_t.size < 2:
-            try:
-                breakpoint()
-            except RuntimeError:
-                # XXX, if greenback not active from
-                # piker store ldshm cmd..
-                log.exception(
-                    "Can't debug single-sample null!\n"
-                )
-
-            return None
-
         # scan for all frame-indices where the
         # zeroed-row-abs-index-step-diff is greater then the
         # expected increment of 1.
@@ -446,8 +434,8 @@ def get_null_segs(
 
 def iter_null_segs(
     timeframe: float,
-    frame: Frame|None = None,
-    null_segs: tuple|None = None,
+    frame: Frame | None = None,
+    null_segs: tuple | None = None,
 
 ) -> Generator[
     tuple[
@@ -499,8 +487,7 @@ def iter_null_segs(
         start_dt = None
         if (
             absi_start is not None
-            and
-            start_t != 0
+            and start_t != 0
         ):
             fi_start: int = absi_start - absi_first
             start_row: Seq = frame[fi_start]
@@ -514,8 +501,8 @@ def iter_null_segs(
         yield (
             absi_start, absi_end,  # abs indices
             fi_start, fi_end,  # relative "frame" indices
-            start_t, end_t,  # epoch times
-            start_dt, end_dt,  # dts
+            start_t, end_t,
+            start_dt, end_dt,
         )
 
 
@@ -530,7 +517,7 @@ def with_dts(
 
     '''
     return df.with_columns([
-        pl.col(time_col).shift(1).name.suffix('_prev'),
+        pl.col(time_col).shift(1).suffix('_prev'),
         pl.col(time_col).diff().alias('s_diff'),
         pl.from_epoch(pl.col(time_col)).alias('dt'),
     ]).with_columns([
@@ -591,22 +578,11 @@ def detect_time_gaps(
     # NOTE: this flag is to indicate that on this (sampling) time
     # scale we expect to only be filtering against larger venue
     # closures-scale time gaps.
-    #
-    # Map to total_ method since `dt_diff` is a duration type,
-    # not datetime - modern polars requires `total_*` methods
-    # for duration types (e.g. `total_days()` not `day()`)
-    # Ensure plural form for polars API (e.g. 'day' -> 'days')
-    unit_plural: str = (
-        gap_dt_unit
-        if gap_dt_unit.endswith('s')
-        else f'{gap_dt_unit}s'
-    )
-    duration_method: str = f'total_{unit_plural}'
     return step_gaps.filter(
         # Second by an arbitrary dt-unit step size
         getattr(
             pl.col('dt_diff').dt,
-            duration_method,
+            gap_dt_unit,
         )().abs() > gap_thresh
     )
 
@@ -640,18 +616,6 @@ def detect_price_gaps(
     # ])
     ...
 
-# TODO: probably just use the null_segs impl above?
-def detect_vlm_gaps(
-    df: pl.DataFrame,
-    col: str = 'volume',
-
-) -> pl.DataFrame:
-
-    vnull: pl.DataFrame = df.filter(
-        pl.col(col) == 0
-    )
-    return vnull
-
 
 def dedupe(
     src_df: pl.DataFrame,
@@ -662,6 +626,7 @@ def dedupe(
 
 ) -> tuple[
     pl.DataFrame,  # with dts
+    pl.DataFrame,  # gaps
     pl.DataFrame,  # with deduplicated dts (aka gap/repeat removal)
     int,  # len diff between input and deduped
 ]:
@@ -674,22 +639,19 @@ def dedupe(
     '''
     wdts: pl.DataFrame = with_dts(src_df)
 
-    deduped = wdts
-
-    # remove duplicated datetime samples/sections
-    deduped: pl.DataFrame = wdts.unique(
-        # subset=['dt'],
-        subset=['time'],
-        maintain_order=True,
-    )
-
     # maybe sort on any time field
     if sort:
-        deduped = deduped.sort(by='time')
+        wdts = wdts.sort(by='time')
         # TODO: detect out-of-order segments which were corrected!
         # -[ ] report in log msg
         # -[ ] possibly return segment sections which were moved?
 
+    # remove duplicated datetime samples/sections
+    deduped: pl.DataFrame = wdts.unique(
+        subset=['dt'],
+        maintain_order=True,
+    )
+
     diff: int = (
         wdts.height
         -

piker/tsp/_annotate.py

@@ -1,306 +0,0 @@
-# piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-Time-series (remote) annotation APIs.
-
-"""
-from __future__ import annotations
-from math import copysign
-from typing import (
-    Any,
-    TYPE_CHECKING,
-)
-
-import polars as pl
-import tractor
-
-from piker.data._formatters import BGM
-from piker.storage import log
-from piker.ui._style import get_fonts
-
-if TYPE_CHECKING:
-    from piker.ui._remote_ctl import AnnotCtl
-
-
-def humanize_duration(
-    seconds: float,
-) -> str:
-    '''
-    Convert duration in seconds to short human-readable form.
-
-    Uses smallest appropriate time unit:
-    - d: days
-    - h: hours
-    - m: minutes
-    - s: seconds
-
-    Examples:
-    - 86400 -> "1d"
-    - 28800 -> "8h"
-    - 180 -> "3m"
-    - 45 -> "45s"
-
-    '''
-    abs_secs: float = abs(seconds)
-
-    if abs_secs >= 86400:
-        days: float = abs_secs / 86400
-        if days >= 10 or days == int(days):
-            return f'{int(days)}d'
-        return f'{days:.1f}d'
-
-    elif abs_secs >= 3600:
-        hours: float = abs_secs / 3600
-        if hours >= 10 or hours == int(hours):
-            return f'{int(hours)}h'
-        return f'{hours:.1f}h'
-
-    elif abs_secs >= 60:
-        mins: float = abs_secs / 60
-        if mins >= 10 or mins == int(mins):
-            return f'{int(mins)}m'
-        return f'{mins:.1f}m'
-
-    else:
-        if abs_secs >= 10 or abs_secs == int(abs_secs):
-            return f'{int(abs_secs)}s'
-        return f'{abs_secs:.1f}s'
-
-
-async def markup_gaps(
-    fqme: str,
-    timeframe: float,
-    actl: AnnotCtl,
-    wdts: pl.DataFrame,
-    gaps: pl.DataFrame,
-
-    # XXX, switch on to see txt showing a "humanized" label of each
-    # gap's duration.
-    show_txt: bool = False,
-
-) -> dict[int, dict]:
-    '''
-    Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
-    with rectangles.
-
-    '''
-    # XXX: force chart redraw FIRST to ensure PlotItem coordinate
-    # system is properly initialized before we position annotations!
-    # Without this, annotations may be misaligned on first creation
-    # due to Qt/pyqtgraph initialization race conditions.
-    await actl.redraw(
-        fqme=fqme,
-        timeframe=timeframe,
-    )
-
-    aids: dict[int] = {}
-    for i in range(gaps.height):
-        row: pl.DataFrame = gaps[i]
-
-        # the gap's RIGHT-most bar's OPEN value
-        # at that time (sample) step.
-        iend: int = row['index'][0]
-
-        # dt: datetime = row['dt'][0]
-        # dt_prev: datetime = row['dt_prev'][0]
-        # dt_end_t: float = dt.timestamp()
-
-
-        # TODO: can we eventually remove this
-        # once we figure out why the epoch cols
-        # don't match?
-        # TODO: FIX HOW/WHY these aren't matching
-        # and are instead off by 4hours (EST
-        # vs. UTC?!?!)
-        # end_t: float = row['time']
-        # assert (
-        #     dt.timestamp()
-        #     ==
-        #     end_t
-        # )
-
-        # the gap's LEFT-most bar's CLOSE value
-        # at that time (sample) step.
-        prev_r: pl.DataFrame = wdts.filter(
-            pl.col('index') == iend - 1
-        )
-        # XXX: probably a gap in the (newly sorted or de-duplicated)
-        # dt-df, so we might need to re-index first..
-        dt: pl.Series = row['dt']
-        dt_prev: pl.Series = row['dt_prev']
-        if prev_r.is_empty():
-
-            # XXX, filter out any special ignore cases,
-            # - UNIX-epoch stamped datums
-            # - first row
-            if (
-                dt_prev.dt.epoch()[0] == 0
-                or
-                dt.dt.epoch()[0] == 0
-            ):
-                log.warning('Skipping row with UNIX epoch timestamp ??')
-                continue
-
-            if wdts[0]['index'][0] == iend:  # first row
-                log.warning('Skipping first-row (has no previous obvi) !!')
-                continue
-
-            # XXX, if the previous-row by shm-index is missing,
-            # meaning there is a missing sample (set), get the prior
-            # row by df index and attempt to use it?
-            i_wdts: pl.DataFrame = wdts.with_row_index(name='i')
-            i_row: int = i_wdts.filter(pl.col('index') == iend)['i'][0]
-            prev_row_by_i = wdts[i_row]
-            prev_r: pl.DataFrame = prev_row_by_i
-
-            # debug any missing pre-row
-            if tractor._state.is_debug_mode():
-                await tractor.pause()
-
-        istart: int = prev_r['index'][0]
-        # TODO: implement px-col width measure
-        # and ensure at least as many px-cols
-        # shown per rect as configured by user.
-        # gap_w: float = abs((iend - istart))
-        # if gap_w < 6:
-        #     margin: float = 6
-        #     iend += margin
-        #     istart -= margin
-
-        opn: float = row['open'][0]
-        cls: float = prev_r['close'][0]
-
-        # get gap duration for humanized label
-        gap_dur_s: float = row['s_diff'][0]
-        gap_label: str = humanize_duration(gap_dur_s)
-
-        # XXX: get timestamps for server-side index lookup
-        start_time: float = prev_r['time'][0]
-        end_time: float = row['time'][0]
-
-        # BGM=0.16 is the normal diff from overlap between bars, SO
-        # just go slightly "in" from that "between them".
-        from_idx: int = BGM - .06  # = .10
-        lc: tuple[float, float] = (
-            istart + 1 - from_idx,
-            cls,
-        )
-        ro: tuple[float, float] = (
-            iend + from_idx,
-            opn,
-        )
-
-        diff: float = cls - opn
-        sgn: float = copysign(1, diff)
-        up_gap: bool = sgn == -1
-        down_gap: bool = sgn == 1
-        flat: bool = sgn == 0
-
-        color: str = 'dad_blue'
-        # TODO? mks more sense to have up/down coloring?
-        # color: str = {
-        #     -1: 'lilypad_green',  # up-gap
-        #     1: 'wine',  # down-gap
-        # }[sgn]
-
-        rect_kwargs: dict[str, Any] = dict(
-            fqme=fqme,
-            timeframe=timeframe,
-            start_pos=lc,
-            end_pos=ro,
-            color=color,
-            start_time=start_time,
-            end_time=end_time,
-        )
-
-        # add up/down rects
-        aid: int|None = await actl.add_rect(**rect_kwargs)
-        if aid is None:
-            log.error(
-                f'Failed to add rect for,\n'
-                f'{rect_kwargs!r}\n'
-                f'\n'
-                f'Skipping to next gap!\n'
-            )
-            continue
-
-        assert aid
-        aids[aid] = rect_kwargs
-        direction: str = (
-            'down' if down_gap
-            else 'up'
-        )
-        # TODO! mk this a `msgspec.Struct` which we deserialize
-        # on the server side!
-        # XXX: send timestamp for server-side index lookup
-        # to ensure alignment with current shm state
-        gap_time: float = row['time'][0]
-        arrow_kwargs: dict[str, Any] = dict(
-            fqme=fqme,
-            timeframe=timeframe,
-            x=iend,  # fallback if timestamp lookup fails
-            y=cls,
-            time=gap_time,  # for server-side index lookup
-            color=color,
-            alpha=169,
-            pointing=direction,
-            # TODO: expose these as params to markup_gaps()?
-            headLen=10,
-            headWidth=2.222,
-            pxMode=True,
-        )
-
-        aid: int = await actl.add_arrow(
-            **arrow_kwargs
-        )
-
-        # add duration label to RHS of arrow
-        if up_gap:
-            anchor = (0, 0)
-            # ^XXX? i dun get dese dims.. XD
-        elif down_gap:
-            anchor = (0, 1)  # XXX y, x?
-        else:  # no-gap?
-            assert flat
-            anchor = (0, 0)  # up from bottom
-
-        # use a slightly smaller font for gap label txt.
-        font, small_font = get_fonts()
-        font_size: int = small_font.px_size - 1
-        assert isinstance(font_size, int)
-
-        if show_txt:
-            text_aid: int = await actl.add_text(
-                fqme=fqme,
-                timeframe=timeframe,
-                text=gap_label,
-                x=iend + 1,  # fallback if timestamp lookup fails
-                y=cls,
-                time=gap_time,  # server-side index lookup
-                color=color,
-                anchor=anchor,
-                font_size=font_size,
-            )
-            aids[text_aid] = {'text': gap_label}
-
-    # tell chart to redraw all its
-    # graphics view layers Bo
-    await actl.redraw(
-        fqme=fqme,
-        timeframe=timeframe,
-    )
-    return aids

piker/tsp/_dedupe_smart.py

@@ -1,206 +0,0 @@
-'''
-Smart OHLCV deduplication with data quality validation.
-
-Handles concurrent write conflicts by keeping the most complete bar
-(highest volume) while detecting data quality anomalies.
-
-'''
-import polars as pl
-
-from ._anal import with_dts
-
-
-def dedupe_ohlcv_smart(
-    src_df: pl.DataFrame,
-    time_col: str = 'time',
-    volume_col: str = 'volume',
-    sort: bool = True,
-
-) -> tuple[
-    pl.DataFrame,  # with dts
-    pl.DataFrame,  # deduped (keeping higher volume bars)
-    int,  # count of dupes removed
-    pl.DataFrame|None,  # valid race conditions
-    pl.DataFrame|None,  # data quality violations
-]:
-    '''
-    Smart OHLCV deduplication keeping most complete bars.
-
-    For duplicate timestamps, keeps bar with highest volume under
-    the assumption that higher volume indicates more complete/final
-    data from backfill vs partial live updates.
-
-    Returns
-    -------
-    Tuple of:
-    - wdts: original dataframe with datetime columns added
-    - deduped: deduplicated frame keeping highest-volume bars
-    - diff: number of duplicate rows removed
-    - valid_races: duplicates meeting expected race condition pattern
-      (volume monotonic, OHLC ranges valid)
-    - data_quality_issues: duplicates violating expected relationships
-      indicating provider data problems
-
-    '''
-    wdts: pl.DataFrame = with_dts(src_df)
-
-    # Find duplicate timestamps
-    dupes: pl.DataFrame = wdts.filter(
-        pl.col(time_col).is_duplicated()
-    )
-
-    if dupes.is_empty():
-        # No duplicates, return as-is
-        return (wdts, wdts, 0, None, None)
-
-    # Analyze duplicate groups for validation
-    dupe_analysis: pl.DataFrame = (
-        dupes
-        .sort([time_col, 'index'])
-        .group_by(time_col, maintain_order=True)
-        .agg([
-            pl.col('index').alias('indices'),
-            pl.col('volume').alias('volumes'),
-            pl.col('high').alias('highs'),
-            pl.col('low').alias('lows'),
-            pl.col('open').alias('opens'),
-            pl.col('close').alias('closes'),
-            pl.col('dt').first().alias('dt'),
-            pl.len().alias('count'),
-        ])
-    )
-
-    # Validate OHLCV monotonicity for each duplicate group
-    def check_ohlcv_validity(row) -> dict[str, bool]:
-        '''
-        Check if duplicate bars follow expected race condition pattern.
-
-        For a valid live-update → backfill race:
-        - volume should be monotonically increasing
-        - high should be monotonically non-decreasing
-        - low should be monotonically non-increasing
-        - open should be identical (fixed at bar start)
-
-        Returns dict of violation flags.
-
-        '''
-        vols: list = row['volumes']
-        highs: list = row['highs']
-        lows: list = row['lows']
-        opens: list = row['opens']
-
-        violations: dict[str, bool] = {
-            'volume_non_monotonic': False,
-            'high_decreased': False,
-            'low_increased': False,
-            'open_mismatch': False,
-            'identical_bars': False,
-        }
-
-        # Check if all bars are identical (pure duplicate)
-        if (
-            len(set(vols)) == 1
-            and len(set(highs)) == 1
-            and len(set(lows)) == 1
-            and len(set(opens)) == 1
-        ):
-            violations['identical_bars'] = True
-            return violations
-
-        # Check volume monotonicity
-        for i in range(1, len(vols)):
-            if vols[i] < vols[i-1]:
-                violations['volume_non_monotonic'] = True
-                break
-
-        # Check high monotonicity (can only increase or stay same)
-        for i in range(1, len(highs)):
-            if highs[i] < highs[i-1]:
-                violations['high_decreased'] = True
-                break
-
-        # Check low monotonicity (can only decrease or stay same)
-        for i in range(1, len(lows)):
-            if lows[i] > lows[i-1]:
-                violations['low_increased'] = True
-                break
-
-        # Check open consistency (should be fixed)
-        if len(set(opens)) > 1:
-            violations['open_mismatch'] = True
-
-        return violations
-
-    # Apply validation
-    dupe_analysis = dupe_analysis.with_columns([
-        pl.struct(['volumes', 'highs', 'lows', 'opens'])
-        .map_elements(
-            check_ohlcv_validity,
-            return_dtype=pl.Struct([
-                pl.Field('volume_non_monotonic', pl.Boolean),
-                pl.Field('high_decreased', pl.Boolean),
-                pl.Field('low_increased', pl.Boolean),
-                pl.Field('open_mismatch', pl.Boolean),
-                pl.Field('identical_bars', pl.Boolean),
-            ])
-        )
-        .alias('validity')
-    ])
-
-    # Unnest validity struct
-    dupe_analysis = dupe_analysis.unnest('validity')
-
-    # Separate valid races from data quality issues
-    valid_races: pl.DataFrame|None = (
-        dupe_analysis
-        .filter(
-            # Valid if no violations OR just identical bars
-            ~pl.col('volume_non_monotonic')
-            & ~pl.col('high_decreased')
-            & ~pl.col('low_increased')
-            & ~pl.col('open_mismatch')
-        )
-    )
-    if valid_races.is_empty():
-        valid_races = None
-
-    data_quality_issues: pl.DataFrame|None = (
-        dupe_analysis
-        .filter(
-            # Issues if any non-identical violation exists
-            (
-                pl.col('volume_non_monotonic')
-                | pl.col('high_decreased')
-                | pl.col('low_increased')
-                | pl.col('open_mismatch')
-            )
-            & ~pl.col('identical_bars')
-        )
-    )
-    if data_quality_issues.is_empty():
-        data_quality_issues = None
-
-    # Deduplicate: keep highest volume bar for each timestamp
-    deduped: pl.DataFrame = (
-        wdts
-        .sort([time_col, volume_col])
-        .unique(
-            subset=[time_col],
-            keep='last',
-            maintain_order=False,
-        )
-    )
-
-    # Re-sort by time or index
-    if sort:
-        deduped = deduped.sort(by=time_col)
-
-    diff: int = wdts.height - deduped.height
-
-    return (
-        wdts,
-        deduped,
-        diff,
-        valid_races,
-        data_quality_issues,
-    )

piker/types.py

@@ -21,6 +21,230 @@ Extensions to built-in or (heavily used but 3rd party) friend-lib
 types.
 
 '''
-from tractor.msg.pretty_struct import (
-    Struct as Struct,
+from __future__ import annotations
+from collections import UserList
+from pprint import (
+    saferepr,
 )
+from typing import Any
+
+from msgspec import (
+    msgpack,
+    Struct as _Struct,
+    structs,
+)
+
+
+class DiffDump(UserList):
+    '''
+    Very simple list delegator that repr() dumps (presumed) tuple
+    elements of the form `tuple[str, Any, Any]` in a nice
+    multi-line readable form for analyzing `Struct` diffs.
+
+    '''
+    def __repr__(self) -> str:
+        if not len(self):
+            return super().__repr__()
+
+        # format by displaying item pair's ``repr()`` on multiple,
+        # indented lines such that they are more easily visually
+        # comparable when printed to console when printed to
+        # console.
+        repstr: str = '[\n'
+        for k, left, right in self:
+            repstr += (
+                f'({k},\n'
+                f'\t{repr(left)},\n'
+                f'\t{repr(right)},\n'
+                ')\n'
+            )
+        repstr += ']\n'
+        return repstr
+
+
+class Struct(
+    _Struct,
+
+    # https://jcristharif.com/msgspec/structs.html#tagged-unions
+    # tag='pikerstruct',
+    # tag=True,
+):
+    '''
+    A "human friendlier" (aka repl buddy) struct subtype.
+
+    '''
+    def _sin_props(self) -> Iterator[
+        tuple[
+            structs.FieldIinfo,
+            str,
+            Any,
+        ]
+    ]:
+        '''
+        Iterate over all non-@property fields of this struct.
+
+        '''
+        fi: structs.FieldInfo
+        for fi in structs.fields(self):
+            key: str = fi.name
+            val: Any = getattr(self, key)
+            yield fi, key, val
+
+    def to_dict(
+        self,
+        include_non_members: bool = True,
+
+    ) -> dict:
+        '''
+        Like it sounds.. direct delegation to:
+        https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
+
+        BUT, by default we pop all non-member (aka not defined as
+        struct fields) fields by default.
+
+        '''
+        asdict: dict = structs.asdict(self)
+        if include_non_members:
+            return asdict
+
+        # only return a dict of the struct members
+        # which were provided as input, NOT anything
+        # added as type-defined `@property` methods!
+        sin_props: dict = {}
+        fi: structs.FieldInfo
+        for fi, k, v in self._sin_props():
+            sin_props[k] = asdict[k]
+
+        return sin_props
+
+    def pformat(
+        self,
+        field_indent: int = 2,
+        indent: int = 0,
+
+    ) -> str:
+        '''
+        Recursion-safe `pprint.pformat()` style formatting of
+        a `msgspec.Struct` for sane reading by a human using a REPL.
+
+        '''
+        # global whitespace indent
+        ws: str = ' '*indent
+
+        # field whitespace indent
+        field_ws: str = ' '*(field_indent + indent)
+
+        # qtn: str = ws + self.__class__.__qualname__
+        qtn: str = self.__class__.__qualname__
+
+        obj_str: str = ''  # accumulator
+        fi: structs.FieldInfo
+        k: str
+        v: Any
+        for fi, k, v in self._sin_props():
+
+            # TODO: how can we prefer `Literal['option1',  'option2,
+            # ..]` over .__name__ == `Literal` but still get only the
+            # latter for simple types like `str | int | None` etc..?
+            ft: type = fi.type
+            typ_name: str = getattr(ft, '__name__', str(ft))
+
+            # recurse to get sub-struct's `.pformat()` output Bo
+            if isinstance(v, Struct):
+                val_str: str =  v.pformat(
+                    indent=field_indent + indent,
+                    field_indent=indent + field_indent,
+                )
+
+            else:  # the `pprint` recursion-safe format:
+                # https://docs.python.org/3.11/library/pprint.html#pprint.saferepr
+                val_str: str = saferepr(v)
+
+            obj_str += (field_ws + f'{k}: {typ_name} = {val_str},\n')
+
+        return (
+            f'{qtn}(\n'
+            f'{obj_str}'
+            f'{ws})'
+        )
+
+    # TODO: use a pprint.PrettyPrinter instance around ONLY rendering
+    # inside a known tty?
+    # def __repr__(self) -> str:
+    #     ...
+
+    # __str__ = __repr__ = pformat
+    __repr__ = pformat
+
+    def copy(
+        self,
+        update: dict | None = None,
+
+    ) -> Struct:
+        '''
+        Validate-typecast all self defined fields, return a copy of
+        us with all such fields.
+
+        NOTE: This is kinda like the default behaviour in
+        `pydantic.BaseModel` except a copy of the object is
+        returned making it compat with `frozen=True`.
+
+        '''
+        if update:
+            for k, v in update.items():
+                setattr(self, k, v)
+
+        # NOTE: roundtrip serialize to validate
+        # - enode to msgpack binary format,
+        # - decode that back to a struct.
+        return msgpack.Decoder(type=type(self)).decode(
+            msgpack.Encoder().encode(self)
+        )
+
+    def typecast(
+        self,
+
+        # TODO: allow only casting a named subset?
+        # fields: set[str] | None = None,
+
+    ) -> None:
+        '''
+        Cast all fields using their declared type annotations
+        (kinda like what `pydantic` does by default).
+
+        NOTE: this of course won't work on frozen types, use
+        ``.copy()`` above in such cases.
+
+        '''
+        # https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
+        fi: structs.FieldInfo
+        for fi in structs.fields(self):
+            setattr(
+                self,
+                fi.name,
+                fi.type(getattr(self, fi.name)),
+            )
+
+    def __sub__(
+        self,
+        other: Struct,
+
+    ) -> DiffDump[tuple[str, Any, Any]]:
+        '''
+        Compare fields/items key-wise and return a ``DiffDump``
+        for easy visual REPL comparison B)
+
+        '''
+        diffs: DiffDump[tuple[str, Any, Any]] = DiffDump()
+        for fi in structs.fields(self):
+            attr_name: str = fi.name
+            ours: Any = getattr(self, attr_name)
+            theirs: Any = getattr(other, attr_name)
+            if ours != theirs:
+                diffs.append((
+                    attr_name,
+                    ours,
+                    theirs,
+                ))
+
+        return diffs

piker/ui/__init__.py

@@ -14,8 +14,9 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-'''
-UI components built using `Qt` with major versions swapped in via
-the import indirection in the `.qt` sub-mod.
+"""
+Stuff for your eyes, aka super hawt Qt UI components.
 
-'''
+Currently we only support PyQt5 due to this issue in Pyside2:
+https://bugreports.qt.io/projects/PYSIDE/issues/PYSIDE-1313
+"""

piker/ui/_anchors.py

@@ -21,10 +21,8 @@ Anchor funtions for UI placement of annotions.
 from __future__ import annotations
 from typing import Callable, TYPE_CHECKING
 
-from piker.ui.qt import (
-    QPointF,
-    QGraphicsPathItem,
-)
+from PyQt5.QtCore import QPointF
+from PyQt5.QtWidgets import QGraphicsPathItem
 
 if TYPE_CHECKING:
     from ._chart import ChartPlotWidget

piker/ui/_annotate.py

@@ -20,22 +20,12 @@ Annotations for ur faces.
 """
 from typing import Callable
 
-from pyqtgraph import (
-    Point,
-    functions as fn,
-    Color,
-)
+from PyQt5 import QtCore, QtGui, QtWidgets
+from PyQt5.QtCore import QPointF, QRectF
+from PyQt5.QtWidgets import QGraphicsPathItem
+from pyqtgraph import Point, functions as fn, Color
 import numpy as np
 
-from piker.ui.qt import (
-    QtCore,
-    QtGui,
-    QtWidgets,
-    QPointF,
-    QRectF,
-    QGraphicsPathItem,
-)
-
 
 def mk_marker_path(
 

piker/ui/_app.py

@@ -21,24 +21,18 @@ Main app startup and run.
 from functools import partial
 from types import ModuleType
 
-import tractor
+from PyQt5.QtCore import QEvent
 import trio
 
-from piker.ui.qt import (
-    QEvent,
-)
-from . import _chart
-from . import _event
-from . import _search
-from ..accounting import unpack_fqme
-from ..data._symcache import open_symcache
-from ..data.feed import install_brokerd_search
-from ..log import (
-    get_logger,
-    get_console_log,
-)
 from ..service import maybe_spawn_brokerd
+from . import _event
 from ._exec import run_qtractor
+from ..data.feed import install_brokerd_search
+from ..data._symcache import open_symcache
+from ..accounting import unpack_fqme
+from . import _search
+from ._chart import GodWidget
+from ..log import get_logger
 
 log = get_logger(__name__)
 
@@ -76,8 +70,8 @@ async def load_provider_search(
 
 async def _async_main(
 
-    # implicit required argument provided by `qtractor_run()`
-    main_widget: _chart.GodWidget,
+    # implicit required argument provided by ``qtractor_run()``
+    main_widget: GodWidget,
 
     syms: list[str],
     brokers: dict[str, ModuleType],
@@ -90,16 +84,6 @@ async def _async_main(
     Provision the "main" widget with initial symbol data and root nursery.
 
     """
-    # enable chart's console logging
-    if loglevel:
-        get_console_log(
-            level=loglevel,
-            name=__name__,
-        )
-
-    # set as singleton
-    _chart._godw = main_widget
-
     from . import _display
     from ._pg_overrides import _do_overrides
     _do_overrides()
@@ -130,7 +114,6 @@ async def _async_main(
         needed_brokermods[brokername] = brokers[brokername]
 
     async with (
-        tractor.trionics.collapse_eg(),
         trio.open_nursery() as root_n,
     ):
         # set root nursery and task stack for spawning other charts/feeds
@@ -214,6 +197,6 @@ def _main(
             brokermods,
             piker_loglevel,
         ),
-        main_widget_type=_chart.GodWidget,
+        main_widget_type=GodWidget,
         tractor_kwargs=tractor_kwargs,
     )

piker/ui/_axes.py

@@ -25,16 +25,9 @@ from math import floor
 
 import polars as pl
 import pyqtgraph as pg
+from PyQt5 import QtCore, QtGui, QtWidgets
+from PyQt5.QtCore import QPointF
 
-from piker.ui.qt import (
-    QtCore,
-    QtGui,
-    QtWidgets,
-    QPointF,
-    txt_flag,
-    align_flag,
-    px_cache_mode,
-)
 from . import _pg_overrides as pgo
 from ..accounting._mktinfo import float_digits
 from ._label import Label
@@ -421,15 +414,11 @@ class AxisLabel(pg.GraphicsObject):
         super().__init__()
         self.setParentItem(parent)
 
-        self.setFlag(
-            self.GraphicsItemFlag.ItemIgnoresTransformations
-        )
+        self.setFlag(self.ItemIgnoresTransformations)
         self.setZValue(100)
 
         # XXX: pretty sure this is faster
-        self.setCacheMode(
-            px_cache_mode.DeviceCoordinateCache
-        )
+        self.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
 
         self._parent = parent
 
@@ -566,14 +555,21 @@ class AxisLabel(pg.GraphicsObject):
 
         return (self.rect.width(), self.rect.height())
 
+# _common_text_flags = (
+#     QtCore.Qt.TextDontClip |
+#     QtCore.Qt.AlignCenter |
+#     QtCore.Qt.AlignTop |
+#     QtCore.Qt.AlignHCenter |
+#     QtCore.Qt.AlignVCenter
+# )
 
 
 class XAxisLabel(AxisLabel):
     _x_margin = 8
 
     text_flags = (
-        align_flag.AlignCenter
-        | txt_flag.TextDontClip
+        QtCore.Qt.TextDontClip
+        | QtCore.Qt.AlignCenter
     )
 
     def size_hint(self) -> tuple[float, float]:
@@ -630,10 +626,10 @@ class YAxisLabel(AxisLabel):
     _y_margin: int = 4
 
     text_flags = (
-        align_flag.AlignLeft
-        | align_flag.AlignVCenter
-        # | align_flag.AlignHCenter
-        | txt_flag.TextDontClip
+        QtCore.Qt.AlignLeft
+        # QtCore.Qt.AlignHCenter
+        | QtCore.Qt.AlignVCenter
+        | QtCore.Qt.TextDontClip
     )
 
     def __init__(

piker/ui/_chart.py

@@ -28,19 +28,22 @@ from typing import (
     TYPE_CHECKING,
 )
 
-import pyqtgraph as pg
-
-from piker.ui.qt import (
-    QtCore,
+from PyQt5 import QtCore, QtWidgets
+from PyQt5.QtCore import (
     Qt,
     QLineF,
+    # QPointF,
+)
+from PyQt5.QtWidgets import (
     QFrame,
     QWidget,
     QHBoxLayout,
     QVBoxLayout,
     QSplitter,
 )
-from ._widget import GodWidget
+import pyqtgraph as pg
+import trio
+
 from ._axes import (
     DynamicDateAxis,
     PriceAxis,
@@ -61,6 +64,10 @@ from ._style import (
     _xaxis_at,
     # _min_points_to_show,
 )
+from ..data.feed import (
+    Feed,
+    Flume,
+)
 from ..accounting import (
     MktPair,
 )
@@ -74,12 +81,286 @@ from . import _pg_overrides as pgo
 
 if TYPE_CHECKING:
     from ._display import DisplayState
-    from ..data.flows import Flume
-    from ..data.feed import Feed
 
 log = get_logger(__name__)
 
 
+class GodWidget(QWidget):
+    '''
+    "Our lord and savior, the holy child of window-shua, there is no
+    widget above thee." - 6|6
+
+    The highest level composed widget which contains layouts for
+    organizing charts as well as other sub-widgets used to control or
+    modify them.
+
+    '''
+    search: SearchWidget
+    mode_name: str = 'god'
+
+    def __init__(
+
+        self,
+        parent=None,
+
+    ) -> None:
+
+        super().__init__(parent)
+
+        self.search: SearchWidget | None = None
+
+        self.hbox = QHBoxLayout(self)
+        self.hbox.setContentsMargins(0, 0, 0, 0)
+        self.hbox.setSpacing(6)
+        self.hbox.setAlignment(Qt.AlignTop)
+
+        self.vbox = QVBoxLayout()
+        self.vbox.setContentsMargins(0, 0, 0, 0)
+        self.vbox.setSpacing(2)
+        self.vbox.setAlignment(Qt.AlignTop)
+
+        self.hbox.addLayout(self.vbox)
+
+        self._chart_cache: dict[
+            str,
+            tuple[LinkedSplits, LinkedSplits],
+        ] = {}
+
+        self.hist_linked: LinkedSplits | None = None
+        self.rt_linked: LinkedSplits | None = None
+        self._active_cursor: Cursor | None = None
+
+        # assigned in the startup func `_async_main()`
+        self._root_n: trio.Nursery = None
+
+        self._widgets: dict[str, QWidget] = {}
+        self._resizing: bool = False
+
+        # TODO: do we need this, when would god get resized
+        # and the window does not? Never right?!
+        # self.reg_for_resize(self)
+
+    # TODO: strat loader/saver that we don't need yet.
+    # def init_strategy_ui(self):
+    #     self.toolbar_layout = QHBoxLayout()
+    #     self.toolbar_layout.setContentsMargins(0, 0, 0, 0)
+    #     self.vbox.addLayout(self.toolbar_layout)
+    #     self.strategy_box = StrategyBoxWidget(self)
+    #     self.toolbar_layout.addWidget(self.strategy_box)
+
+    @property
+    def linkedsplits(self) -> LinkedSplits:
+        return self.rt_linked
+
+    def set_chart_symbols(
+        self,
+        group_key: tuple[str],  # of form <fqme>.<providername>
+        all_linked: tuple[LinkedSplits, LinkedSplits],  # type: ignore
+
+    ) -> None:
+        # re-sort org cache symbol list in LIFO order
+        cache = self._chart_cache
+        cache.pop(group_key, None)
+        cache[group_key] = all_linked
+
+    def get_chart_symbols(
+        self,
+        symbol_key: str,
+
+    ) -> tuple[LinkedSplits, LinkedSplits]:  # type: ignore
+        return self._chart_cache.get(symbol_key)
+
+    async def load_symbols(
+        self,
+        fqmes: list[str],
+        loglevel: str,
+        reset: bool = False,
+
+    ) -> trio.Event:
+        '''
+        Load a new contract into the charting app.
+
+        Expects a ``numpy`` structured array containing all the ohlcv fields.
+
+        '''
+        # NOTE: for now we use the first symbol in the set as the "key"
+        # for the overlay of feeds on the chart.
+        group_key: tuple[str] = tuple(fqmes)
+
+        all_linked = self.get_chart_symbols(group_key)
+        order_mode_started = trio.Event()
+
+        if not self.vbox.isEmpty():
+
+            # XXX: seems to make switching slower?
+            # qframe = self.hist_linked.chart.qframe
+            # if qframe.sidepane is self.search:
+            #     qframe.hbox.removeWidget(self.search)
+
+            for linked in [self.rt_linked, self.hist_linked]:
+                # XXX: this is CRITICAL especially with pixel buffer caching
+                linked.hide()
+                linked.unfocus()
+
+                # XXX: pretty sure we don't need this
+                # remove any existing plots?
+                # XXX: ahh we might want to support cache unloading..
+                # self.vbox.removeWidget(linked)
+
+        # switching to a new viewable chart
+        if all_linked is None or reset:
+            from ._display import display_symbol_data
+
+            # we must load a fresh linked charts set
+            self.rt_linked = rt_charts = LinkedSplits(self)
+            self.hist_linked = hist_charts = LinkedSplits(self)
+
+            # spawn new task to start up and update new sub-chart instances
+            self._root_n.start_soon(
+                display_symbol_data,
+                self,
+                fqmes,
+                loglevel,
+                order_mode_started,
+            )
+
+            # self.vbox.addWidget(hist_charts)
+            self.vbox.addWidget(rt_charts)
+            self.set_chart_symbols(
+                group_key,
+                (hist_charts, rt_charts),
+            )
+
+            for linked in [hist_charts, rt_charts]:
+                linked.show()
+                linked.focus()
+
+            await trio.sleep(0)
+
+        else:
+            # symbol is already loaded and ems ready
+            order_mode_started.set()
+
+            self.hist_linked, self.rt_linked = all_linked
+
+            for linked in all_linked:
+                # TODO:
+                # - we'll probably want per-instrument/provider state here?
+                #   change the order config form over to the new chart
+
+                # chart is already in memory so just focus it
+                linked.show()
+                linked.focus()
+                linked.graphics_cycle()
+                await trio.sleep(0)
+
+                # resume feeds *after* rendering chart view asap
+                chart = linked.chart
+                if chart:
+                    chart.resume_all_feeds()
+
+            # TODO: we need a check to see if the chart
+            # last had the xlast in view, if so then shift so it's
+            # still in view, if the user was viewing history then
+            # do nothing yah?
+            self.rt_linked.chart.main_viz.default_view(
+                do_min_bars=True,
+            )
+
+        # if a history chart instance is already up then
+        # set the search widget as its sidepane.
+        hist_chart = self.hist_linked.chart
+        if hist_chart:
+            hist_chart.qframe.set_sidepane(self.search)
+
+            # NOTE: this is really stupid/hard to follow.
+            # we have to reposition the active position nav
+            # **AFTER** applying the search bar as a sidepane
+            # to the newly switched to symbol.
+            await trio.sleep(0)
+
+            # TODO: probably stick this in some kinda `LooknFeel` API?
+            for tracker in self.rt_linked.mode.trackers.values():
+                pp_nav = tracker.nav
+                if tracker.live_pp.cumsize:
+                    pp_nav.show()
+                    pp_nav.hide_info()
+                else:
+                    pp_nav.hide()
+
+        # set window titlebar info
+        symbol = self.rt_linked.mkt
+        if symbol is not None:
+            self.window.setWindowTitle(
+                f'{symbol.fqme} '
+                f'tick:{symbol.size_tick}'
+            )
+
+        return order_mode_started
+
+    def focus(self) -> None:
+        '''
+        Focus the top level widget which in turn focusses the chart
+        ala "view mode".
+
+        '''
+        # go back to view-mode focus (aka chart focus)
+        self.clearFocus()
+        chart = self.rt_linked.chart
+        if chart:
+            chart.setFocus()
+
+    def reg_for_resize(
+        self,
+        widget: QWidget,
+    ) -> None:
+        getattr(widget, 'on_resize')
+        self._widgets[widget.mode_name] = widget
+
+    def on_win_resize(self, event: QtCore.QEvent) -> None:
+        '''
+        Top level god widget handler from window (the real yaweh) resize
+        events such that any registered widgets which wish to be
+        notified are invoked using our pythonic `.on_resize()` method
+        api.
+
+        Where we do UX magic to make things not suck B)
+
+        '''
+        if self._resizing:
+            return
+
+        self._resizing = True
+
+        log.info('God widget resize')
+        for name, widget in self._widgets.items():
+            widget.on_resize()
+
+        self._resizing = False
+
+    # on_resize = on_win_resize
+
+    def get_cursor(self) -> Cursor:
+        return self._active_cursor
+
+    def iter_linked(self) -> Iterator[LinkedSplits]:
+        for linked in [self.hist_linked, self.rt_linked]:
+            yield linked
+
+    def resize_all(self) -> None:
+        '''
+        Dynamic resize sequence: adjusts all sub-widgets/charts to
+        sensible default ratios of what space is detected as available
+        on the display / window.
+
+        '''
+        rt_linked = self.rt_linked
+        rt_linked.set_split_sizes()
+        self.rt_linked.resize_sidepanes()
+        self.hist_linked.resize_sidepanes(from_linked=rt_linked)
+        self.search.on_resize()
+
+
 class ChartnPane(QFrame):
     '''
     One-off ``QFrame`` composite which pairs a chart
@@ -91,9 +372,9 @@ class ChartnPane(QFrame):
     https://doc.qt.io/qt-5/qwidget.html#composite-widgets
 
     '''
-    sidepane: FieldsForm|SearchWidget
+    sidepane: FieldsForm | SearchWidget
     hbox: QHBoxLayout
-    chart: ChartPlotWidget|None = None
+    chart: ChartPlotWidget | None = None
 
     def __init__(
         self,
@@ -109,13 +390,13 @@ class ChartnPane(QFrame):
         self.chart = None
 
         hbox = self.hbox = QHBoxLayout(self)
-        hbox.setAlignment(Qt.AlignTop|Qt.AlignLeft)
+        hbox.setAlignment(Qt.AlignTop | Qt.AlignLeft)
         hbox.setContentsMargins(0, 0, 0, 0)
         hbox.setSpacing(3)
 
     def set_sidepane(
         self,
-        sidepane: FieldsForm|SearchWidget,
+        sidepane: FieldsForm | SearchWidget,
     ) -> None:
 
         # add sidepane **after** chart; place it on axis side
@@ -126,7 +407,7 @@ class ChartnPane(QFrame):
         self._sidepane = sidepane
 
     @property
-    def sidepane(self) -> FieldsForm|SearchWidget:
+    def sidepane(self) -> FieldsForm | SearchWidget:
         return self._sidepane
 
 
@@ -141,6 +422,7 @@ class LinkedSplits(QWidget):
 
     '''
     def __init__(
+
         self,
         godwidget: GodWidget,
 
@@ -171,7 +453,7 @@ class LinkedSplits(QWidget):
         # chart-local graphics state that can be passed to
         # a ``graphic_update_cycle()`` call by any task wishing to
         # update the UI for a given "chart instance".
-        self.display_state: DisplayState|None = None
+        self.display_state: DisplayState | None = None
 
         self._mkt: MktPair = None
 
@@ -207,7 +489,7 @@ class LinkedSplits(QWidget):
 
     def set_split_sizes(
         self,
-        prop: float|None = None,
+        prop: float | None = None,
 
     ) -> None:
         '''
@@ -288,8 +570,8 @@ class LinkedSplits(QWidget):
 
         # style?
         self.chart.setFrameStyle(
-            QFrame.Shape.StyledPanel
-            |QFrame.Shadow.Plain
+            QFrame.StyledPanel |
+            QFrame.Plain
         )
 
         return self.chart
@@ -301,11 +583,11 @@ class LinkedSplits(QWidget):
         shm: ShmArray,
         flume: Flume,
 
-        array_key: str|None = None,
+        array_key: str | None = None,
         style: str = 'line',
         _is_main: bool = False,
 
-        sidepane: QWidget|None = None,
+        sidepane: QWidget | None = None,
         draw_kwargs: dict = {},
 
         **cpw_kwargs,
@@ -407,8 +689,8 @@ class LinkedSplits(QWidget):
 
         cpw.plotItem.vb.linked = self
         cpw.setFrameStyle(
-            QFrame.Shape.StyledPanel
-            # |QFrame.Shadow.Plain
+            QtWidgets.QFrame.StyledPanel
+            # | QtWidgets.QFrame.Plain
         )
 
         # don't show the little "autoscale" A label.
@@ -521,7 +803,7 @@ class LinkedSplits(QWidget):
 
     def resize_sidepanes(
         self,
-        from_linked: LinkedSplits|None = None,
+        from_linked: LinkedSplits | None = None,
 
     ) -> None:
         '''
@@ -595,7 +877,7 @@ class ChartPlotWidget(pg.PlotWidget):
         # TODO: load from config
         use_open_gl: bool = False,
 
-        static_yrange: tuple[float, float]|None = None,
+        static_yrange: tuple[float, float] | None = None,
 
         parent=None,
         **kwargs,
@@ -610,7 +892,7 @@ class ChartPlotWidget(pg.PlotWidget):
 
         # NOTE: must be set bfore calling ``.mk_vb()``
         self.linked = linkedsplits
-        self.sidepane: FieldsForm|None = None
+        self.sidepane: FieldsForm | None = None
 
         # source of our custom interactions
         self.cv = self.mk_vb(name)
@@ -644,7 +926,7 @@ class ChartPlotWidget(pg.PlotWidget):
         self.useOpenGL(use_open_gl)
         self.name = name
         self.data_key = data_key or name
-        self.qframe: ChartnPane|None = None
+        self.qframe: ChartnPane | None = None
 
         # scene-local placeholder for book graphics
         # sizing to avoid overlap with data contents
@@ -655,7 +937,7 @@ class ChartPlotWidget(pg.PlotWidget):
         # registry of overlay curve names
         self._vizs: dict[str, Viz] = {}
 
-        self.feed: Feed|None = None
+        self.feed: Feed | None = None
 
         self._labels = {}  # registry of underlying graphics
         self._ysticks = {}  # registry of underlying graphics
@@ -748,11 +1030,11 @@ class ChartPlotWidget(pg.PlotWidget):
     def increment_view(
         self,
         datums: int = 1,
-        vb: ChartView|None = None,
+        vb: ChartView | None = None,
 
     ) -> None:
         '''
-        Increment the data view `datums`` steps toward y-axis thus
+        Increment the data view ``datums``` steps toward y-axis thus
         "following" the current time slot/step/bar.
 
         '''
@@ -762,7 +1044,7 @@ class ChartPlotWidget(pg.PlotWidget):
         x_shift = viz.index_step() * datums
 
         if datums >= 300:
-            log.warning('FUCKING FIX THE GLOBAL STEP BULLSHIT')
+            print("FUCKING FIX THE GLOBAL STEP BULLSHIT")
             # breakpoint()
             return
 
@@ -779,8 +1061,8 @@ class ChartPlotWidget(pg.PlotWidget):
     def overlay_plotitem(
         self,
         name: str,
-        index: int|None = None,
-        axis_title: str|None = None,
+        index: int | None = None,
+        axis_title: str | None = None,
         axis_side: str = 'right',
         axis_kwargs: dict = {},
 
@@ -868,14 +1150,14 @@ class ChartPlotWidget(pg.PlotWidget):
         shm: ShmArray,
         flume: Flume,
 
-        array_key: str|None = None,
+        array_key: str | None = None,
         overlay: bool = False,
-        color: str|None = None,
+        color: str | None = None,
         add_label: bool = True,
-        pi: pg.PlotItem|None = None,
+        pi: pg.PlotItem | None = None,
         step_mode: bool = False,
         is_ohlc: bool = False,
-        add_sticky: None|str = 'right',
+        add_sticky: None | str = 'right',
 
         **graphics_kwargs,
 
@@ -973,7 +1255,7 @@ class ChartPlotWidget(pg.PlotWidget):
                 # use the tick size precision for display
                 name = name or pi.name
                 mkt: MktPair = self.linked.mkt
-                digits: int|None = None
+                digits: int | None = None
                 if name in mkt.fqme:
                     digits = mkt.price_tick_digits
 
@@ -1007,7 +1289,7 @@ class ChartPlotWidget(pg.PlotWidget):
         shm: ShmArray,
         flume: Flume,
 
-        array_key: str|None = None,
+        array_key: str | None = None,
         **draw_curve_kwargs,
 
     ) -> Viz:

piker/ui/_cursor.py

@@ -28,14 +28,9 @@ from typing import (
 import inspect
 import numpy as np
 import pyqtgraph as pg
+from PyQt5 import QtCore, QtWidgets
+from PyQt5.QtCore import QPointF, QRectF
 
-from piker.ui.qt import (
-    QPointF,
-    QRectF,
-    QtCore,
-    QtWidgets,
-    px_cache_mode,
-)
 from ._style import (
     _xaxis_at,
     hcolor,
@@ -109,9 +104,7 @@ class LineDot(pg.CurvePoint):
         dot.setParentItem(self)
 
         # keep a static size
-        self.setFlag(
-            self.GraphicsItemFlag.ItemIgnoresTransformations
-        )
+        self.setFlag(self.ItemIgnoresTransformations)
 
     def event(
         self,
@@ -413,18 +406,9 @@ class Cursor(pg.GraphicsObject):
         self,
         item: pg.GraphicsObject,
     ) -> None:
-        assert getattr(
-            item,
-            'delete',
-        ), f"{item} must define a ``.delete()``"
+        assert getattr(item, 'delete'), f"{item} must define a ``.delete()``"
         self._hovered.add(item)
 
-    def is_hovered(
-        self,
-        item: pg.GraphicsObject,
-    ) -> bool:
-        return item in self._hovered
-
     def add_plot(
         self,
         plot: ChartPlotWidget,  # noqa
@@ -440,10 +424,10 @@ class Cursor(pg.GraphicsObject):
         # vertical and horizonal lines and a y-axis label
 
         vl = plot.addLine(x=0, pen=self.lines_pen, movable=False)
-        vl.setCacheMode(px_cache_mode.DeviceCoordinateCache)
+        vl.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
 
         hl = plot.addLine(y=0, pen=self.lines_pen, movable=False)
-        hl.setCacheMode(px_cache_mode.DeviceCoordinateCache)
+        hl.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
         hl.hide()
 
         yl = YAxisLabel(
@@ -527,10 +511,7 @@ class Cursor(pg.GraphicsObject):
             plot=chart
         )
         chart.addItem(cursor)
-        self.graphics[chart].setdefault(
-            'cursors',
-            [],
-        ).append(cursor)
+        self.graphics[chart].setdefault('cursors', []).append(cursor)
         return cursor
 
     def mouseAction(

piker/ui/_curve.py

@@ -19,20 +19,20 @@ Fast, smooth, sexy curves.
 
 """
 from contextlib import contextmanager as cm
-from enum import EnumType
 from typing import Callable
 
 import numpy as np
 import pyqtgraph as pg
-
-from piker.ui.qt import (
-    QtWidgets,
+from PyQt5 import QtWidgets
+from PyQt5.QtWidgets import QGraphicsItem
+from PyQt5.QtCore import (
     Qt,
     QLineF,
     QRectF,
+)
+from PyQt5.QtGui import (
     QPainter,
     QPainterPath,
-    px_cache_mode,
 )
 from ._style import hcolor
 from ..log import get_logger
@@ -42,16 +42,15 @@ from ..toolz.profile import (
     ms_slower_then,
 )
 
+
 log = get_logger(__name__)
 
 
-pen_style: EnumType = Qt.PenStyle
-
 _line_styles: dict[str, int] = {
-    'solid': pen_style.SolidLine,
-    'dash': pen_style.DashLine,
-    'dot': pen_style.DotLine,
-    'dashdot': pen_style.DashDotLine,
+    'solid': Qt.PenStyle.SolidLine,
+    'dash': Qt.PenStyle.DashLine,
+    'dot': Qt.PenStyle.DotLine,
+    'dashdot': Qt.PenStyle.DashDotLine,
 }
 
 
@@ -70,12 +69,12 @@ class FlowGraphic(pg.GraphicsObject):
     # XXX-NOTE-XXX: graphics caching B)
     # see explanation for different caching modes:
     # https://stackoverflow.com/a/39410081
-    cache_mode: int = px_cache_mode.DeviceCoordinateCache
+    cache_mode: int = QGraphicsItem.DeviceCoordinateCache
     # XXX: WARNING item caching seems to only be useful
     # if we don't re-generate the entire QPainterPath every time
     # don't ever use this - it's a colossal nightmare of artefacts
     # and is disastrous for performance.
-    # cache_mode.ItemCoordinateCache
+    # QGraphicsItem.ItemCoordinateCache
     # TODO: still questions todo with coord-cacheing that we should
     # probably talk to a core dev about:
     # - if this makes trasform interactions slower (such as zooming)
@@ -177,7 +176,7 @@ class FlowGraphic(pg.GraphicsObject):
     @cm
     def reset_cache(self) -> None:
         try:
-            none = px_cache_mode.NoCache
+            none = QGraphicsItem.NoCache
             log.debug(
                 f'{self._name} -> CACHE DISABLE: {none}'
             )

piker/ui/_dataviz.py

@@ -40,12 +40,12 @@ from numpy import (
     ndarray,
 )
 import pyqtgraph as pg
+from PyQt5.QtCore import QLineF
 
-from piker.ui.qt import QLineF
 from ..data._sharedmem import (
     ShmArray,
 )
-from ..data.flows import Flume
+from ..data.feed import Flume
 from ..data._formatters import (
     IncrementalFormatter,
     OHLCBarsFmtr,  # Plain OHLC renderer

piker/ui/_display.py

@@ -21,7 +21,6 @@ this module ties together quote and computational (fsp) streams with
 graphics update methods via our custom ``pyqtgraph`` charting api.
 
 '''
-from functools import partial
 import itertools
 from math import floor
 import time
@@ -58,7 +57,6 @@ from piker.toolz import (
     Profiler,
 )
 from piker.log import get_logger
-from piker import config
 # from ..data._source import tf_in_1s
 from ._axes import YAxisLabel
 from ._chart import (
@@ -209,7 +207,6 @@ class DisplayState(Struct):
 async def increment_history_view(
     # min_istream: tractor.MsgStream,
     ds: DisplayState,
-    loglevel: str = 'warning',
 ):
     hist_chart: ChartPlotWidget = ds.hist_chart
     hist_viz: Viz = ds.hist_viz
@@ -231,10 +228,7 @@ async def increment_history_view(
         hist_viz.reset_graphics()
         # hist_viz.update_graphics(force_redraw=True)
 
-    async with open_sample_stream(
-        period_s=1.,
-        loglevel=loglevel,
-    ) as min_istream:
+    async with open_sample_stream(1.) as min_istream:
         async for msg in min_istream:
 
             profiler = Profiler(
@@ -315,6 +309,7 @@ async def increment_history_view(
 
 
 async def graphics_update_loop(
+
     dss: dict[str, DisplayState],
     nurse: trio.Nursery,
     godwidget: GodWidget,
@@ -323,7 +318,6 @@ async def graphics_update_loop(
 
     pis: dict[str, list[pgo.PlotItem, pgo.PlotItem]] = {},
     vlm_charts: dict[str, ChartPlotWidget] = {},
-    loglevel: str = 'warning',
 
 ) -> None:
     '''
@@ -467,12 +461,9 @@ async def graphics_update_loop(
         # })
 
         nurse.start_soon(
-            partial(
-                increment_history_view,
-                # min_istream,
-                ds=ds,
-                loglevel=loglevel,
-            ),
+            increment_history_view,
+            # min_istream,
+            ds,
         )
         await trio.sleep(0)
 
@@ -519,19 +510,14 @@ async def graphics_update_loop(
                     fast_chart.linked.isHidden()
                     or not rt_pi.isVisible()
                 ):
-                    log.debug(
-                        f'{fqme} skipping update for HIDDEN CHART'
-                    )
+                    print(f'{fqme} skipping update for HIDDEN CHART')
                     fast_chart.pause_all_feeds()
                     continue
 
                 ic = fast_chart.view._in_interact
                 if ic:
                     fast_chart.pause_all_feeds()
-                    log.debug(
-                        f'Pausing chart updaates during interaction\n'
-                        f'fqme: {fqme!r}'
-                    )
+                    print(f'{fqme} PAUSING DURING INTERACTION')
                     await ic.wait()
                     fast_chart.resume_all_feeds()
 
@@ -1245,8 +1231,6 @@ async def link_views_with_region(
     # region.sigRegionChangeFinished.connect(update_pi_from_region)
 
 
-# NOTE: default is set to 60 FPS until the runtime delivers the
-# discoverd hw value below.
 _quote_throttle_rate: int = 60 - 6
 
 
@@ -1288,54 +1272,26 @@ async def display_symbol_data(
     # TODO: ctl over update loop's maximum frequency.
     # - load this from a config.toml!
     # - allow dyanmic configuration from chart UI?
-    (
-        conf,
-        path,
-    ) = config.load()
-    ui_conf: dict = conf['ui']
-
     global _quote_throttle_rate
     from ._window import main_window
-
-    display_rate: int = floor(
-        main_window().current_screen().refreshRate()
-    ) - 6
-
-    mx_redraw_rate: int = ui_conf.get(
-        'max_redraw_rate',
-        _quote_throttle_rate,
-    )
-
-    if mx_redraw_rate < display_rate:
-        log.info(
-            'Down-throttling redraw rate to config setting\n'
-            f'display FPS: {display_rate}\n'
-            'max_redraw_rate: {max_redraw_rate}\n'
-        )
-    else:
-        _quote_throttle_rate = display_rate
+    display_rate = main_window().current_screen().refreshRate()
+    _quote_throttle_rate = floor(display_rate) - 6
 
     # TODO: we should be able to increase this if we use some
     # `mypyc` speedups elsewhere? 22ish seems to be the sweet
     # spot for single-feed chart.
     num_of_feeds = len(fqmes)
-    # if num_of_feeds > 1:
-
-    # there will be more ctx switches with more than 1 feed so we
-    # max throttle down a bit more.
-    mx_per_feed: int = (
-        ui_conf.get(
-        'per_feed_redraw_rate',
-        mx_redraw_rate,
-        )
-        or 16
-    )
+    mx: int = 22
+    if num_of_feeds > 1:
+        # there will be more ctx switches with more than 1 feed so we
+        # max throttle down a bit more.
+        mx = 16
 
     # limit to at least display's FPS
     # avoiding needless Qt-in-guest-mode context switches
     cycles_per_feed = min(
         round(_quote_throttle_rate/num_of_feeds),
-        mx_per_feed,
+        mx,
     )
 
     feed: Feed
@@ -1458,10 +1414,7 @@ async def display_symbol_data(
         # for pause/resume on mouse interaction
         rt_chart.feed = feed
 
-        async with (
-            tractor.trionics.collapse_eg(),
-            trio.open_nursery() as ln,
-        ):
+        async with trio.open_nursery() as ln:
             # if available load volume related built-in display(s)
             vlm_charts: dict[
                 str,
@@ -1604,18 +1557,15 @@ async def display_symbol_data(
             # start update loop task
             dss: dict[str, DisplayState] = {}
             ln.start_soon(
-                partial(
-                    graphics_update_loop,
-                    dss=dss,
-                    nurse=ln,
-                    godwidget=godwidget,
-                    feed=feed,
-                    # min_istream,
+                graphics_update_loop,
+                dss,
+                ln,
+                godwidget,
+                feed,
+                # min_istream,
 
-                    pis=pis,
-                    vlm_charts=vlm_charts,
-                    loglevel=loglevel,
-                )
+                pis,
+                vlm_charts,
             )
 
             # boot order-mode

piker/ui/_editors.py

@@ -21,7 +21,6 @@ Higher level annotation editors.
 from __future__ import annotations
 from collections import defaultdict
 from typing import (
-    Literal,
     Sequence,
     TYPE_CHECKING,
 )
@@ -33,21 +32,24 @@ from pyqtgraph import (
     QtCore,
     QtWidgets,
 )
+from PyQt5.QtCore import (
+    QPointF,
+    QRectF,
+)
+from PyQt5.QtGui import (
+    QColor,
+    QTransform,
+)
+from PyQt5.QtWidgets import (
+    QGraphicsProxyWidget,
+    QGraphicsScene,
+    QLabel,
+)
 
 from pyqtgraph import functions as fn
 import numpy as np
 
 from piker.types import Struct
-from piker.ui.qt import (
-    Qt,
-    QPointF,
-    QRectF,
-    QGraphicsProxyWidget,
-    QGraphicsScene,
-    QLabel,
-    QColor,
-    QTransform,
-)
 from ._style import (
     hcolor,
     _font,
@@ -55,11 +57,6 @@ from ._style import (
 from ._lines import LevelLine
 from ..log import get_logger
 
-# TODO, rm the cycle here!
-from ._widget import (
-    GodWidget,
-)
-
 if TYPE_CHECKING:
     from ._chart import (
         GodWidget,
@@ -72,18 +69,9 @@ log = get_logger(__name__)
 
 
 class ArrowEditor(Struct):
-    '''
-    Annotate a chart-view with arrows most often used for indicating,
-    - order txns/clears,
-    - positions directions,
-    - general points-of-interest like nooz events.
 
-    '''
     godw: GodWidget = None  # type: ignore # noqa
-    _arrows: dict[
-        str,
-        list[pg.ArrowItem]
-    ] = {}
+    _arrows: dict[str, list[pg.ArrowItem]] = {}
 
     def add(
         self,
@@ -91,19 +79,8 @@ class ArrowEditor(Struct):
         uid: str,
         x: float,
         y: float,
-        color: str|None = None,
-        pointing: Literal[
-            'up',
-            'down',
-            None,
-        ] = None,
-        alpha: int = 255,
-        zval: float = 1e9,
-        headLen: float|None = None,
-        headWidth: float|None = None,
-        tailLen: float|None = None,
-        tailWidth: float|None = None,
-        pxMode: bool = True,
+        color: str = 'default',
+        pointing: str | None = None,
 
     ) -> pg.ArrowItem:
         '''
@@ -119,83 +96,29 @@ class ArrowEditor(Struct):
         # scale arrow sizing to dpi-aware font
         size = _font.font.pixelSize() * 0.8
 
-        # allow caller override of head dimensions
-        if headLen is None:
-            headLen = size
-        if headWidth is None:
-            headWidth = size/2
-        # tail params default to None (no tail)
-        if tailWidth is None:
-            tailWidth = 3
-
-        color = color or 'default'
-        color = QColor(hcolor(color))
-        color.setAlpha(alpha)
-        pen = fn.mkPen(color, width=1)
-        brush = fn.mkBrush(color)
         arrow = pg.ArrowItem(
             angle=angle,
             baseAngle=0,
-            headLen=headLen,
-            headWidth=headWidth,
-            tailLen=tailLen,
-            tailWidth=tailWidth,
-            pxMode=pxMode,
-            # coloring
-            pen=pen,
-            brush=brush,
-        )
-        arrow.setZValue(zval)
-        arrow.setPos(x, y)
-        plot.addItem(arrow)  # render to view
+            headLen=size,
+            headWidth=size/2,
+            tailLen=None,
+            pxMode=True,
 
-        # register for removal
-        arrow._uid = uid
-        self._arrows.setdefault(
-            uid, []
-        ).append(arrow)
+            # coloring
+            pen=pg.mkPen(hcolor('papas_special')),
+            brush=pg.mkBrush(hcolor(color)),
+        )
+        arrow.setPos(x, y)
+        self._arrows.setdefault(uid, []).append(arrow)
+
+        # render to view
+        plot.addItem(arrow)
 
         return arrow
 
-    def remove(
-        self,
-        arrow: pg.ArrowItem,
-    ) -> None:
-        '''
-        Remove a *single arrow* from all chart views to which it was
-        added.
-
-        '''
-        uid: str = arrow._uid
-        arrows: list[pg.ArrowItem] = self._arrows[uid]
-        log.info(
-            f'Removing arrow from views\n'
-            f'uid: {uid!r}\n'
-            f'{arrow!r}\n'
-        )
+    def remove(self, arrow) -> bool:
         for linked in self.godw.iter_linked():
-            if not (chart := linked.chart):
-                continue
-
-            chart.plotItem.removeItem(arrow)
-        try:
-            arrows.remove(arrow)
-        except ValueError:
-            log.warning(
-                f'Arrow was already removed?\n'
-                f'uid: {uid!r}\n'
-                f'{arrow!r}\n'
-            )
-
-    def remove_all(self) -> set[pg.ArrowItem]:
-        '''
-        Remove all arrows added by this editor from all
-        chart-views.
-
-        '''
-        for uid, arrows in self._arrows.items():
-            for arrow in arrows:
-                self.remove(arrow)
+            linked.chart.plotItem.removeItem(arrow)
 
 
 class LineEditor(Struct):
@@ -341,9 +264,6 @@ class LineEditor(Struct):
 
         return lines
 
-    # compat with ArrowEditor
-    remove = remove_line
-
 
 def as_point(
     pair: Sequence[float, float] | QPointF,
@@ -376,7 +296,7 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
     def __init__(
         self,
         viewbox: ViewBox,
-        color: str|None = None,
+        color: str | None = None,
     ) -> None:
         super().__init__(0, 0, 1, 1)
 
@@ -396,9 +316,7 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
         self.setZValue(1e9)
 
         label = self._label = QLabel()
-        label.setTextFormat(
-            Qt.TextFormat.MarkdownText
-        )
+        label.setTextFormat(0)  # markdown
         label.setFont(_font.font)
         label.setMargin(0)
         label.setAlignment(
@@ -692,6 +610,3 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
 
         ):
             scen.removeItem(self._label_proxy)
-
-    # compat with ArrowEditor
-    remove = delete

piker/ui/_event.py

@@ -22,33 +22,29 @@ from contextlib import asynccontextmanager as acm
 from typing import Callable
 
 import trio
-from tractor.trionics import (
-    gather_contexts,
-    collapse_eg,
+from tractor.trionics import gather_contexts
+from PyQt5 import QtCore
+from PyQt5.QtCore import QEvent, pyqtBoundSignal
+from PyQt5.QtWidgets import QWidget
+from PyQt5.QtWidgets import (
+    QGraphicsSceneMouseEvent as gs_mouse,
 )
 
-from piker.ui.qt import (
-    QtCore,
-    QWidget,
-    QEvent,
-    keys,
-    gs_keys,
-    pyqtBoundSignal,
-)
 from piker.types import Struct
 
 
 MOUSE_EVENTS = {
-    gs_keys.GraphicsSceneMousePress,
-    gs_keys.GraphicsSceneMouseRelease,
-    keys.MouseButtonPress,
-    keys.MouseButtonRelease,
+    gs_mouse.GraphicsSceneMousePress,
+    gs_mouse.GraphicsSceneMouseRelease,
+    QEvent.MouseButtonPress,
+    QEvent.MouseButtonRelease,
     # QtGui.QMouseEvent,
 }
 
 
 # TODO: maybe consider some constrained ints down the road?
 # https://pydantic-docs.helpmanual.io/usage/types/#constrained-types
+
 class KeyboardMsg(Struct):
     '''Unpacked Qt keyboard event data.
 
@@ -118,10 +114,7 @@ class EventRelay(QtCore.QObject):
         # something to do with Qt internals and calling the
         # parent handler?
 
-        if etype in {
-            QEvent.Type.KeyPress,
-            QEvent.Type.KeyRelease,
-        }:
+        if etype in {QEvent.KeyPress, QEvent.KeyRelease}:
 
             msg = KeyboardMsg(
                 event=ev,
@@ -167,9 +160,7 @@ class EventRelay(QtCore.QObject):
 async def open_event_stream(
 
     source_widget: QWidget,
-    event_types: set[QEvent] = {
-        QEvent.Type.KeyPress,
-    },
+    event_types: set[QEvent] = {QEvent.KeyPress},
     filter_auto_repeats: bool = True,
 
 ) -> trio.abc.ReceiveChannel:
@@ -210,10 +201,7 @@ async def open_signal_handler(
         async for args in recv:
             await async_handler(*args)
 
-    async with (
-        collapse_eg(),
-        trio.open_nursery() as tn
-    ):
+    async with trio.open_nursery() as tn:
         tn.start_soon(proxy_to_handler)
         async with send:
             yield
@@ -248,7 +236,6 @@ async def open_handlers(
     widget: QWidget
     streams: list[trio.abc.ReceiveChannel]
     async with (
-        collapse_eg(),
         trio.open_nursery() as tn,
         gather_contexts([
             open_event_stream(

piker/ui/_exec.py

@@ -30,22 +30,25 @@ from typing import (
 import platform
 import traceback
 
+# Qt specific
+import PyQt5  # noqa
+from PyQt5.QtWidgets import (
+    QWidget,
+    QMainWindow,
+    QApplication,
+)
+from PyQt5 import QtCore
+from PyQt5.QtCore import (
+    pyqtRemoveInputHook,
+    Qt,
+    QCoreApplication,
+)
 import qdarkstyle
 from qdarkstyle import DarkPalette
 # import qdarkgraystyle  # TODO: play with it
 import trio
 from outcome import Error
 
-# Qt version-agnostic
-from .qt import (
-    QWidget,
-    QMainWindow,
-    QApplication,
-    QtCore,
-    pyqtRemoveInputHook,
-    Qt,
-    QCoreApplication,
-)
 from ..service import (
     maybe_open_pikerd,
     get_runtime_vars,
@@ -56,7 +59,7 @@ from . import _style
 
 
 if TYPE_CHECKING:
-    from ._widget import GodWidget
+    from ._chart import GodWidget
 
 
 log = get_logger(__name__)
@@ -91,10 +94,6 @@ def run_qtractor(
     window_type: QMainWindow = None,
 
 ) -> None:
-    '''
-    Run the Qt event loop and embed `trio` via guest mode on it.
-
-    '''
     # avoids annoying message when entering debugger from qt loop
     pyqtRemoveInputHook()
 
@@ -151,7 +150,7 @@ def run_qtractor(
 
     # load dark theme
     stylesheet = qdarkstyle.load_stylesheet(
-        qt_api='pyqt6',
+        qt_api='pyqt5',
         palette=DarkPalette,
     )
     app.setStyleSheet(stylesheet)
@@ -174,7 +173,7 @@ def run_qtractor(
     # hook into app focus change events
     app.focusChanged.connect(window.on_focus_change)
 
-    instance: GodWidget = main_widget_type()
+    instance = main_widget_type()
     instance.window = window
 
     # override tractor's defaults

piker/ui/_feedstatus.py

@@ -18,11 +18,10 @@
 Feed status and controls widget(s) for embedding in a UI-pane.
 
 """
+
 from __future__ import annotations
-from typing import (
-    Any,
-    TYPE_CHECKING,
-)
+from textwrap import dedent
+from typing import TYPE_CHECKING
 
 # from PyQt5.QtCore import Qt
 
@@ -50,55 +49,35 @@ def mk_feed_label(
     a feed control protocol.
 
     '''
-    status: dict[str, Any] = feed.status
+    status = feed.status
     assert status
 
-    # SO tips on ws/nls,
-    # https://stackoverflow.com/a/15721400
-    ws: str = ' '
-    # nl: str = '<br>'  # dun work?
-    actor_info_repr: str = (
-        f')> **{status["actor_short_id"]}**\n'
-        '\n'  # bc md?
-    )
+    msg = dedent("""
+        actor: **{actor_name}**\n
+        |_ @**{host}:{port}**\n
+    """)
 
-    # fields to select *IN* for display
-    # (see `.data.feed.open_feed()` status
-    #  update -> TAG_feed_status_update)
-    for key in [
-        'ipc',
-        'hist_shm',
-        'rt_shm',
-        'throttle_hz',
-    ]:
-        # NOTE, the 2nd key is filled via `.format()` updates.
-        actor_info_repr += (
-            f'\n'  # bc md?
-            f'{ws}|_{key}: **{{{key}}}**\n'
-        )
-        # ^TODO? formatting and content..
-        # -[ ] showing which fqme is "forward" on the
-        #    chart/fsp/order-mode?
-        #  '|_ flows: **{symbols}**\n'
-        #
-        # -[x] why isn't the indent working?
-        #  => markdown, now solved..
+    for key, val in status.items():
+        if key in ('host', 'port', 'actor_name'):
+            continue
+        msg += f'\n|_ {key}: **{{{key}}}**\n'
 
     feed_label = FormatLabel(
-        fmt_str=actor_info_repr,
+        fmt_str=msg,
+        # |_ streams: **{symbols}**\n
         font=_font.font,
         font_size=_font_small.px_size,
         font_color='default_lightest',
     )
 
-    # ?TODO, remove this?
     # form.vbox.setAlignment(feed_label, Qt.AlignBottom)
     # form.vbox.setAlignment(Qt.AlignBottom)
-    # _ = chart.height() - (
-    #     form.height() +
-    #     form.fill_bar.height()
-    #     # feed_label.height()
-    # )
+    _ = chart.height() - (
+        form.height() +
+        form.fill_bar.height()
+        # feed_label.height()
+    )
 
     feed_label.format(**feed.status)
+
     return feed_label

piker/ui/_forms.py

@@ -28,15 +28,9 @@ from typing import (
 )
 
 import trio
-
-from piker.ui.qt import (
-    keys,
-    size_policy,
-    QtGui,
-    QSize,
-    QModelIndex,
-    Qt,
-    QEvent,
+from PyQt5 import QtGui
+from PyQt5.QtCore import QSize, QModelIndex, Qt, QEvent
+from PyQt5.QtWidgets import (
     QWidget,
     QLabel,
     QComboBox,
@@ -45,6 +39,7 @@ from piker.ui.qt import (
     QVBoxLayout,
     QFormLayout,
     QProgressBar,
+    QSizePolicy,
     QStyledItemDelegate,
     QStyleOptionViewItem,
 )
@@ -76,14 +71,14 @@ class Edit(QLineEdit):
 
         if width_in_chars:
             self._chars = int(width_in_chars)
-            x_size_policy = size_policy.Fixed
+            x_size_policy = QSizePolicy.Fixed
 
         else:
             # chart count which will be used to calculate
             # width of input field.
             self._chars: int = 6
             # fit to surroundingn frame width
-            x_size_policy = size_policy.Expanding
+            x_size_policy = QSizePolicy.Expanding
 
         super().__init__(parent)
 
@@ -91,7 +86,7 @@ class Edit(QLineEdit):
         # https://doc.qt.io/qt-5/qsizepolicy.html#Policy-enum
         self.setSizePolicy(
             x_size_policy,
-            size_policy.Fixed,
+            QSizePolicy.Fixed,
         )
         self.setFont(font.font)
 
@@ -185,13 +180,11 @@ class Selection(QComboBox):
 
         self._items: dict[str, int] = {}
         super().__init__(parent=parent)
-        self.setSizeAdjustPolicy(
-            QComboBox.SizeAdjustPolicy.AdjustToContents,
-        )
+        self.setSizeAdjustPolicy(QComboBox.AdjustToContents)
         # make line edit expand to surrounding frame
         self.setSizePolicy(
-            size_policy.Expanding,
-            size_policy.Fixed,
+            QSizePolicy.Expanding,
+            QSizePolicy.Fixed,
         )
         view = self.view()
         view.setUniformItemSizes(True)
@@ -315,8 +308,8 @@ class FieldsForm(QWidget):
 
         # size it as we specify
         self.setSizePolicy(
-            size_policy.Expanding,
-            size_policy.Expanding,
+            QSizePolicy.Expanding,
+            QSizePolicy.Expanding,
         )
 
         # XXX: not sure why we have to create this here exactly
@@ -423,8 +416,8 @@ class FieldsForm(QWidget):
         select.set_items(values)
 
         self.setSizePolicy(
-            size_policy.Fixed,
-            size_policy.Fixed,
+            QSizePolicy.Fixed,
+            QSizePolicy.Fixed,
         )
         select.show()
         self.form.addRow(label, select)
@@ -444,10 +437,7 @@ async def handle_field_input(
 
     async for kbmsg in recv_chan:
 
-        if kbmsg.etype in {
-            keys.KeyPress,
-            keys.KeyRelease,
-        }:
+        if kbmsg.etype in {QEvent.KeyPress, QEvent.KeyRelease}:
             event, etype, key, mods, txt = kbmsg.to_tuple()
             print(f'key: {kbmsg.key}, mods: {kbmsg.mods}, txt: {kbmsg.txt}')
 
@@ -713,8 +703,7 @@ def mk_fill_status_bar(
     )
 
     bottom_label = form.add_field_label(
-        # 'x: {step_size}',
-       '{unit_prefix}: {step_size}',
+        'x: {step_size}',
         font_size=bar_label_font_size,
         font_color='gunmetal',
     )

piker/ui/_fsp.py

@@ -73,7 +73,7 @@ log = get_logger(__name__)
 def update_fsp_chart(
     viz,
     graphics_name: str,
-    array_key: str|None,
+    array_key: str | None,
     **kwargs,
 
 ) -> None:
@@ -87,11 +87,7 @@ def update_fsp_chart(
 
     # guard against unreadable case
     if not last_row:
-        log.warning(
-            f'Read-race on shm array,\n'
-            f'graphics_name: {graphics_name!r}\n'
-            f'shm.token: {shm.token}\n'
-        )
+        log.warning(f'Read-race on shm array: {graphics_name}@{shm.token}')
         return
 
     # update graphics
@@ -183,17 +179,13 @@ async def open_fsp_sidepane(
 
 @acm
 async def open_fsp_actor_cluster(
-    names: list[str] = [
-        'fsp_0',
-        'fsp_1',
-    ],
+    names: list[str] = ['fsp_0', 'fsp_1'],
 
 ) -> AsyncGenerator[
     int,
     dict[str, tractor.Portal]
 ]:
 
-    # TODO! change to .experimental!
     from tractor._clustering import open_actor_cluster
 
     # profiler = Profiler(
@@ -201,7 +193,7 @@ async def open_fsp_actor_cluster(
     #     disabled=False
     # )
     async with open_actor_cluster(
-        count=len(names),
+        count=2,
         names=names,
         modules=['piker.fsp._engine'],
 
@@ -211,6 +203,7 @@ async def open_fsp_actor_cluster(
 
 
 async def run_fsp_ui(
+
     linkedsplits: LinkedSplits,
     flume: Flume,
     started: trio.Event,
@@ -478,7 +471,7 @@ class FspAdmin:
         target: Fsp,
         conf: dict[str, dict[str, Any]],
 
-        worker_name: str|None = None,
+        worker_name: str | None = None,
         loglevel: str = 'info',
 
     ) -> (Flume, trio.Event):
@@ -501,8 +494,7 @@ class FspAdmin:
 
         portal: tractor.Portal = (
             self.cluster.get(worker_name)
-            or
-            self.rr_next_portal()
+            or self.rr_next_portal()
         )
 
         # TODO: this should probably be turned into a
@@ -608,7 +600,6 @@ async def open_fsp_admin(
             kwargs=kwargs,
         ) as (cache_hit, cluster_map),
 
-        tractor.trionics.collapse_eg(),
         trio.open_nursery() as tn,
     ):
         if cache_hit:
@@ -622,8 +613,6 @@ async def open_fsp_admin(
         )
         try:
             yield admin
-
-        # ??TODO, does this *need* to be inside a finally?
         finally:
             # terminate all tasks via signals
             for key, entry in admin._registry.items():
@@ -631,10 +620,8 @@ async def open_fsp_admin(
                 event.set()
 
 
-# TODO, passing in `pikerd` related settings here!
-# [ ] read in the `tractor` setting for `enable_transports: list`
-#    from the root `conf.toml`!
 async def open_vlm_displays(
+
     linked: LinkedSplits,
     flume: Flume,
     dvlm: bool = True,
@@ -644,12 +631,12 @@ async def open_vlm_displays(
 
 ) -> None:
     '''
-    Vlm (volume) subchart displays.
+    Volume subchart displays.
 
     Since "volume" is often included directly alongside OHLCV price
-    data, we don't really need a separate FSP-actor + shm array for
-    it since it's likely already directly adjacent to OHLC samples
-    from the data provider.
+    data, we don't really need a separate FSP-actor + shm array for it
+    since it's likely already directly adjacent to OHLC samples from the
+    data provider.
 
     Further only if volume data is detected (it sometimes isn't provided
     eg. forex, certain commodities markets) will volume dependent FSPs

piker/ui/_icons.py

@@ -15,18 +15,15 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-`QIcon` hackery.
-
-Mostly dynamically loading pixmaps for use with `QGraphicsScene`.
+``QIcon`` hackery.
 
 '''
-from piker.ui.qt import (
-    QSize,
-    QStyle,
-    QIcon,
-    QPixmap,
-    QColor,
+from PyQt5.QtWidgets import QStyle
+from PyQt5.QtGui import (
+    QIcon, QPixmap, QColor
 )
+from PyQt5.QtCore import QSize
+
 from ._style import hcolor
 
 # https://www.pythonguis.com/faq/built-in-qicons-pyqt/
@@ -47,8 +44,7 @@ def mk_icons(
     size: QSize,
 
 ) -> dict[str, QIcon]:
-    '''
-    This helper is indempotent.
+    '''This helper is indempotent.
 
     '''
     global _icons, _icon_names
@@ -60,11 +56,7 @@ def mk_icons(
     # load account selection using current style
     for name, icon_name in _icon_names.items():
 
-        stdpixmap = getattr(
-            # https://www.pythonguis.com/faq/built-in-qicons-pyqt/
-            QStyle.StandardPixmap,  # pyqt/pyside6
-            icon_name,
-        )
+        stdpixmap = getattr(QStyle, icon_name)
         stdicon = style.standardIcon(stdpixmap)
         pixmap = stdicon.pixmap(size)
 

piker/ui/_interaction.py

@@ -36,22 +36,23 @@ import pyqtgraph as pg
 # this down the road.. Bo
 from pyqtgraph.GraphicsScene import mouseEvents as mevs
 # from pyqtgraph.GraphicsScene.mouseEvents import MouseDragEvent
+from PyQt5.QtWidgets import QGraphicsSceneMouseEvent as gs_mouse
+from PyQt5.QtGui import (
+    QWheelEvent,
+)
+from PyQt5.QtCore import (
+    Qt,
+    QEvent,
+)
 from pyqtgraph import (
     ViewBox,
     Point,
     QtCore,
-    functions as fn,
 )
+from pyqtgraph import functions as fn
 import numpy as np
-import tractor
 import trio
 
-from piker.ui.qt import (
-    QWheelEvent,
-    QGraphicsSceneMouseEvent as gs_mouse,
-    Qt,
-    QEvent,
-)
 from ..log import get_logger
 from ..toolz import (
     Profiler,
@@ -73,32 +74,29 @@ if TYPE_CHECKING:
         GodWidget,
     )
     from ._dataviz import Viz
-    from .order_mode import (
-        OrderMode,
-        Dialog,
-    )
+    from .order_mode import OrderMode
     from ._display import DisplayState
 
 
 log = get_logger(__name__)
 
 NUMBER_LINE = {
-    Qt.Key.Key_1,
-    Qt.Key.Key_2,
-    Qt.Key.Key_3,
-    Qt.Key.Key_4,
-    Qt.Key.Key_5,
-    Qt.Key.Key_6,
-    Qt.Key.Key_7,
-    Qt.Key.Key_8,
-    Qt.Key.Key_9,
-    Qt.Key.Key_0,
+    Qt.Key_1,
+    Qt.Key_2,
+    Qt.Key_3,
+    Qt.Key_4,
+    Qt.Key_5,
+    Qt.Key_6,
+    Qt.Key_7,
+    Qt.Key_8,
+    Qt.Key_9,
+    Qt.Key_0,
 }
 
 ORDER_MODE = {
-    Qt.Key.Key_A,
-    Qt.Key.Key_F,
-    Qt.Key.Key_D,
+    Qt.Key_A,
+    Qt.Key_F,
+    Qt.Key_D,
 }
 
 
@@ -134,12 +132,7 @@ async def handle_viewmode_kb_inputs(
 
     async for kbmsg in recv_chan:
         event, etype, key, mods, text = kbmsg.to_tuple()
-        log.debug(
-            f'View-mode kb-msg received,\n'
-            f'mods: {mods!r}\n'
-            f'key: {key!r}\n'
-            f'text: {text!r}\n'
-        )
+        log.debug(f'key: {key}, mods: {mods}, text: {text}')
         now = time.time()
         period = now - last
 
@@ -167,12 +160,8 @@ async def handle_viewmode_kb_inputs(
             # have no previous keys or we do and the min_tap period is
             # met
             if (
-                not fast_key_seq
-                or (
-                    period <= min_tap
-                    and
-                    fast_key_seq
-                )
+                not fast_key_seq or
+                period <= min_tap and fast_key_seq
             ):
                 fast_key_seq.append(text)
                 log.debug(f'fast keys seqs {fast_key_seq}')
@@ -187,8 +176,7 @@ async def handle_viewmode_kb_inputs(
             # UI REPL-shell, with ctrl-p (for "pause")
             if (
                 ctrl
-                and
-                key in {
+                and key in {
                     Qt.Key_P,
                 }
             ):
@@ -198,6 +186,7 @@ async def handle_viewmode_kb_inputs(
                 vlm_chart = chart.linked.subplots['volume']  # noqa
                 vlm_viz = vlm_chart.main_viz  # noqa
                 dvlm_pi = vlm_chart._vizs['dolla_vlm'].plot  # noqa
+                import tractor
                 await tractor.pause()
                 view.interact_graphics_cycle()
 
@@ -205,8 +194,7 @@ async def handle_viewmode_kb_inputs(
             # shown data `Viz`s for the current chart app.
             if (
                 ctrl
-                and
-                key in {
+                and key in {
                     Qt.Key_R,
                 }
             ):
@@ -245,8 +233,7 @@ async def handle_viewmode_kb_inputs(
                 key == Qt.Key_Escape
                 or (
                     ctrl
-                    and
-                    key == Qt.Key_C
+                    and key == Qt.Key_C
                 )
             ):
                 # ctrl-c as cancel
@@ -257,35 +244,17 @@ async def handle_viewmode_kb_inputs(
             # cancel order or clear graphics
             if (
                 key == Qt.Key_C
-                or
-                key == Qt.Key_Delete
+                or key == Qt.Key_Delete
             ):
-                # log.info('Handling <c> hotkey!')
-                try:
-                    dialogs: list[Dialog] = order_mode.cancel_orders_under_cursor()
-                except BaseException:
-                    log.exception('Failed to cancel orders !?\n')
-                    await tractor.pause()
 
-                if not dialogs:
-                    log.warning(
-                        'No orders were cancelled?\n'
-                        'Is there an order-line under the cursor?\n'
-                        'If you think there IS your DE might be "hiding the mouse" before '
-                        'we rx the keyboard input via Qt..\n'
-                        '=> Check your DE and/or TWM settings to be sure! <=\n'
-                    )
-                    # ^TODO?, some way to detect if there's lines and
-                    # the DE is cuckin with things?
-                    # await tractor.pause()
+                order_mode.cancel_orders_under_cursor()
 
             # View modes
             if (
                 ctrl
                 and (
                     key == Qt.Key_Equal
-                    or
-                    key == Qt.Key_I
+                    or key == Qt.Key_I
                 )
             ):
                 view.wheelEvent(
@@ -297,8 +266,7 @@ async def handle_viewmode_kb_inputs(
                 ctrl
                 and (
                     key == Qt.Key_Minus
-                    or
-                    key == Qt.Key_O
+                    or key == Qt.Key_O
                 )
             ):
                 view.wheelEvent(
@@ -309,8 +277,7 @@ async def handle_viewmode_kb_inputs(
 
             elif (
                 not ctrl
-                and
-                key == Qt.Key_R
+                and key == Qt.Key_R
             ):
                 # NOTE: seems that if we don't yield a Qt render
                 # cycle then the m4 downsampled curves will show here
@@ -512,8 +479,7 @@ async def handle_viewmode_mouse(
         #     view.raiseContextMenu(event)
 
         if (
-            view.order_mode.active
-            and
+            view.order_mode.active and
             button == QtCore.Qt.LeftButton
         ):
             # when in order mode, submit execution
@@ -817,8 +783,7 @@ class ChartView(ViewBox):
 
         # Scale or translate based on mouse button
         if btn & (
-            QtCore.Qt.LeftButton
-            | QtCore.Qt.MidButton
+            QtCore.Qt.LeftButton | QtCore.Qt.MidButton
         ):
             # zoom y-axis ONLY when click-n-drag on it
             # if axis == 1:

piker/ui/_l1.py

@@ -21,12 +21,9 @@ Double auction top-of-book (L1) graphics.
 from typing import Tuple
 
 import pyqtgraph as pg
+from PyQt5 import QtCore, QtGui
+from PyQt5.QtCore import QPointF
 
-from piker.ui.qt import (
-    QPointF,
-    QtCore,
-    QtGui,
-)
 from ._axes import YAxisLabel
 from ._style import hcolor
 from ._pg_overrides import PlotItem
@@ -237,8 +234,8 @@ class LevelLabel(YAxisLabel):
 class L1Label(LevelLabel):
 
     text_flags = (
-        QtCore.Qt.TextFlag.TextDontClip
-        | QtCore.Qt.AlignmentFlag.AlignLeft
+        QtCore.Qt.TextDontClip
+        | QtCore.Qt.AlignLeft
     )
 
     def set_label_str(

piker/ui/_label.py

@@ -25,17 +25,10 @@ from typing import (
 )
 
 import pyqtgraph as pg
+from PyQt5 import QtGui, QtWidgets
+from PyQt5.QtWidgets import QLabel, QSizePolicy
+from PyQt5.QtCore import QPointF, QRectF, Qt
 
-from piker.ui.qt import (
-    px_cache_mode,
-    QtGui,
-    QtWidgets,
-    QLabel,
-    size_policy,
-    QPointF,
-    QRectF,
-    Qt,
-)
 from ._style import (
     DpiAwareFont,
     hcolor,
@@ -85,7 +78,7 @@ class Label:
         self._x_offset = x_offset
 
         txt = self.txt = QtWidgets.QGraphicsTextItem(parent=parent)
-        txt.setCacheMode(px_cache_mode.DeviceCoordinateCache)
+        txt.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
 
         vb.scene().addItem(txt)
 
@@ -110,7 +103,7 @@ class Label:
         self._anchor_func = self.txt.pos().x
 
         # not sure if this makes a diff
-        self.txt.setCacheMode(px_cache_mode.DeviceCoordinateCache)
+        self.txt.setCacheMode(QtWidgets.QGraphicsItem.DeviceCoordinateCache)
 
         # TODO: edit and selection support
         # https://doc.qt.io/qt-5/qt.html#TextInteractionFlag-enum
@@ -285,20 +278,18 @@ class FormatLabel(QLabel):
         font_size: int,
         font_color: str,
 
-        use_md: bool = True,
-
         parent=None,
 
     ) -> None:
 
         super().__init__(parent)
 
-        # by default set the format string verbatim and expect user
-        # to call ``.format()`` later (presumably they'll notice the
+        # by default set the format string verbatim and expect user to
+        # call ``.format()`` later (presumably they'll notice the
         # unformatted content if ``fmt_str`` isn't meant to be
         # unformatted).
         self.fmt_str = fmt_str
-        # self.setText(fmt_str)  # ?TODO, why here?
+        self.setText(fmt_str)
 
         self.setStyleSheet(
             f"""QLabel {{
@@ -308,21 +299,15 @@ class FormatLabel(QLabel):
             """
         )
         self.setFont(_font.font)
-        if use_md:
-            self.setTextFormat(
-                Qt.TextFormat.MarkdownText
-            )
+        self.setTextFormat(Qt.MarkdownText)  # markdown
         self.setMargin(0)
 
         self.setSizePolicy(
-            size_policy.Expanding,
-            size_policy.Expanding,
+            QSizePolicy.Expanding,
+            QSizePolicy.Expanding,
         )
         self.setAlignment(
-            Qt.AlignLeft
-            |
-            Qt.AlignBottom
-            # Qt.AlignVCenter
+            Qt.AlignVCenter | Qt.AlignLeft
         )
         self.setText(self.fmt_str)
 

piker/ui/_lines.py

@@ -27,22 +27,20 @@ from typing import (
 )
 
 import pyqtgraph as pg
-from pyqtgraph import (
-    Point,
-    functions as fn,
-)
-
-from piker.ui.qt import (
-    px_cache_mode,
+from pyqtgraph import Point, functions as fn
+from PyQt5 import (
     QtCore,
     QtGui,
+)
+from PyQt5.QtWidgets import (
     QGraphicsPathItem,
     QStyleOptionGraphicsItem,
     QGraphicsItem,
     QGraphicsScene,
     QWidget,
-    QPointF,
 )
+from PyQt5.QtCore import QPointF
+
 from ._annotate import LevelMarker
 from ._anchors import (
     vbr_left,
@@ -52,13 +50,10 @@ from ._anchors import (
 from ..calc import humanize
 from ._label import Label
 from ._style import hcolor, _font
-from ..log import get_logger
 
 if TYPE_CHECKING:
     from ._cursor import Cursor
 
-log = get_logger(__name__)
-
 
 # TODO: probably worth investigating if we can
 # make .boundingRect() faster:
@@ -145,9 +140,7 @@ class LevelLine(pg.InfiniteLine):
         self._right_end_sc: float = 0
 
         # use px caching
-        self.setCacheMode(
-            px_cache_mode.DeviceCoordinateCache
-        )
+        self.setCacheMode(QGraphicsItem.DeviceCoordinateCache)
 
     def txt_offsets(self) -> tuple[int, int]:
         return 0, 0
@@ -218,7 +211,7 @@ class LevelLine(pg.InfiniteLine):
     ) -> None:
 
         if not called_from_on_pos_change:
-            last: float = self.value()
+            last = self.value()
 
             # if the position hasn't changed then ``.update_labels()``
             # will not be called by a non-triggered `.on_pos_change()`,
@@ -350,7 +343,7 @@ class LevelLine(pg.InfiniteLine):
 
     ) -> None:
         # TODO: enter labels edit mode
-        log.debug(f'double click {ev}')
+        print(f'double click {ev}')
 
     def paint(
         self,
@@ -464,19 +457,10 @@ class LevelLine(pg.InfiniteLine):
         # hovered
         if (
             not ev.isExit()
-            and
-            ev.acceptDrags(QtCore.Qt.LeftButton)
+            and ev.acceptDrags(QtCore.Qt.LeftButton)
         ):
             # if already hovered we don't need to run again
-            if (
-                self.mouseHovering is True
-                and
-                cur.is_hovered(self)
-            ):
-                log.debug(
-                    f'Already hovering ??\n'
-                    f'cur._hovered: {cur._hovered!r}\n'
-                )
+            if self.mouseHovering is True:
                 return
 
             if self.only_show_markers_on_hover:
@@ -493,7 +477,6 @@ class LevelLine(pg.InfiniteLine):
                     cur._y_label_update = False
 
             # add us to cursor state
-            log.debug(f'Adding line {self!r}\n')
             cur.add_hovered(self)
 
             if self._hide_xhair_on_hover:
@@ -521,7 +504,6 @@ class LevelLine(pg.InfiniteLine):
 
             self.currentPen = self.pen
 
-            log.debug(f'Removing line {self!r}\n')
             cur._hovered.remove(self)
 
             if self.only_show_markers_on_hover:

piker/ui/_ohlc.py

@@ -20,14 +20,16 @@ Super fast OHLC sampling graphics types.
 from __future__ import annotations
 
 import numpy as np
-
-from piker.ui.qt import (
+from PyQt5 import (
     QtGui,
     QtWidgets,
-    QPainterPath,
+)
+from PyQt5.QtCore import (
     QLineF,
     QRectF,
 )
+from PyQt5.QtGui import QPainterPath
+
 from ._curve import FlowGraphic
 from ..toolz import (
     Profiler,

piker/ui/_position.py

@@ -344,10 +344,7 @@ class SettingsPane:
         dsize = tracker.live_pp.dsize
 
         # READ out settings and update the status UI / settings widgets
-        unit_char: str = {
-            'currency': '$',
-            'units': 'u',
-        }[alloc.size_unit]
+        suffix = {'currency': ' $', 'units': ' u'}[alloc.size_unit]
         size_unit, limit = alloc.limit_info()
 
         step_size, currency_per_slot = alloc.step_sizes()
@@ -361,11 +358,10 @@ class SettingsPane:
             self.apply_setting('limit', limit)
 
         self.step_label.format(
-            unit_prefix=unit_char,
-            step_size=str(humanize(step_size))
+            step_size=str(humanize(step_size)) + suffix
         )
         self.limit_label.format(
-            limit=f'{unit_char}: {str(humanize(limit))}'
+            limit=str(humanize(limit)) + suffix
         )
 
         # update size unit in UI

piker/ui/_remote_ctl.py

@@ -15,8 +15,8 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-Remote control tasks for sending annotations (and maybe more cmds) to
-a chart from some other actor.
+Remote control tasks for sending annotations (and maybe more cmds)
+to a chart from some other actor.
 
 '''
 from __future__ import annotations
@@ -27,38 +27,31 @@ from contextlib import (
 from functools import partial
 from pprint import pformat
 from typing import (
+    # Any,
     AsyncContextManager,
-    Literal,
 )
-from uuid import uuid4
 
-import pyqtgraph as pg
 import tractor
-import trio
 from tractor import trionics
 from tractor import (
     Portal,
     Context,
     MsgStream,
 )
+from PyQt5.QtWidgets import (
+    QGraphicsItem,
+)
 
 from piker.log import get_logger
 from piker.types import Struct
 from piker.service import find_service
 from piker.brokers import SymbolNotFound
-from piker.ui.qt import (
-    QGraphicsItem,
-)
-from PyQt6.QtGui import QFont
 from ._display import DisplayState
 from ._interaction import ChartView
-from ._editors import (
-    SelectRect,
-    ArrowEditor,
-)
+from ._editors import SelectRect
 from ._chart import ChartPlotWidget
 from ._dataviz import Viz
-from ._style import hcolor
+
 
 log = get_logger(__name__)
 
@@ -89,40 +82,8 @@ _ctxs: IpcCtxTable = {}
 # the "annotations server" which actually renders to a Qt canvas).
 # type AnnotsTable = dict[int, QGraphicsItem]
 AnnotsTable = dict[int, QGraphicsItem]
-EditorsTable = dict[int, ArrowEditor]
 
 _annots: AnnotsTable  = {}
-_editors: EditorsTable = {}
-
-def rm_annot(
-    annot: ArrowEditor|SelectRect|pg.TextItem
-) -> bool:
-    global _editors
-    match annot:
-        case pg.ArrowItem():
-            editor = _editors[annot._uid]
-            editor.remove(annot)
-            # ^TODO? only remove each arrow or all?
-            # if editor._arrows:
-            #     editor.remove_all()
-            # else:
-            #     log.warning(
-            #         f'Annot already removed!\n'
-            #         f'{annot!r}\n'
-            #     )
-            return True
-
-        case SelectRect():
-            annot.delete()
-            return True
-
-        case pg.TextItem():
-            scene = annot.scene()
-            if scene:
-                scene.removeItem(annot)
-            return True
-
-    return False
 
 
 async def serve_rc_annots(
@@ -133,12 +94,6 @@ async def serve_rc_annots(
     annots: AnnotsTable,
 
 ) -> None:
-    '''
-    A small viz(ualization) server for remote ctl of chart
-    annotations.
-
-    '''
-    global _editors
     async for msg in annot_req_stream:
         match msg:
             case {
@@ -148,77 +103,14 @@ async def serve_rc_annots(
                 'meth': str(meth),
                 'kwargs': dict(kwargs),
             }:
+
                 ds: DisplayState = _dss[fqme]
-                try:
-                    chart: ChartPlotWidget = {
-                        60: ds.hist_chart,
-                        1: ds.chart,
-                    }[timeframe]
-                except KeyError:
-                    msg: str = (
-                        f'No chart for timeframe={timeframe}s, '
-                        f'skipping rect annotation'
-                    )
-                    log.exeception(msg)
-                    await annot_req_stream.send({'error': msg})
-                    continue
-
+                chart: ChartPlotWidget = {
+                    60: ds.hist_chart,
+                    1: ds.chart,
+                }[timeframe]
                 cv: ChartView = chart.cv
 
-                # NEW: if timestamps provided, lookup current indices
-                # from shm to ensure alignment with current buffer
-                # state
-                start_time = kwargs.pop('start_time', None)
-                end_time = kwargs.pop('end_time', None)
-                if (
-                    start_time is not None
-                    and end_time is not None
-                ):
-                    viz: Viz = chart.get_viz(fqme)
-                    shm = viz.shm
-                    arr = shm.array
-
-                    # lookup start index
-                    start_matches = arr[arr['time'] == start_time]
-                    if len(start_matches) == 0:
-                        msg: str = (
-                            f'No shm entry for start_time={start_time}, '
-                            f'skipping rect'
-                        )
-                        log.error(msg)
-                        await annot_req_stream.send({'error': msg})
-                        continue
-
-                    # lookup end index
-                    end_matches = arr[arr['time'] == end_time]
-                    if len(end_matches) == 0:
-                        msg: str = (
-                            f'No shm entry for end_time={end_time}, '
-                            f'skipping rect'
-                        )
-                        log.error(msg)
-                        await annot_req_stream.send({'error': msg})
-                        continue
-
-                    # get close price from start bar, open from end
-                    # bar
-                    start_idx = float(start_matches[0]['index'])
-                    end_idx = float(end_matches[0]['index'])
-                    start_close = float(start_matches[0]['close'])
-                    end_open = float(end_matches[0]['open'])
-
-                    # reconstruct start_pos and end_pos with
-                    # looked-up indices
-                    from_idx: float = 0.16 - 0.06  # BGM offset
-                    kwargs['start_pos'] = (
-                        start_idx + 1 - from_idx,
-                        start_close,
-                    )
-                    kwargs['end_pos'] = (
-                        end_idx + from_idx,
-                        end_open,
-                    )
-
                 # annot type lookup from cmd
                 rect = SelectRect(
                     viewbox=cv,
@@ -237,207 +129,21 @@ async def serve_rc_annots(
                 # delegate generically to the requested method
                 getattr(rect, meth)(**kwargs)
                 rect.show()
-
-                # XXX: store absolute coords for repositioning
-                # during viz redraws (eg backfill updates)
-                rect._meth = meth
-                rect._kwargs = kwargs
-
                 aid: int = id(rect)
                 annots[aid] = rect
                 aids: set[int] = ctxs[ipc_key][1]
                 aids.add(aid)
                 await annot_req_stream.send(aid)
 
-            case {
-                'cmd': 'ArrowEditor',
-                'fqme': fqme,
-                'timeframe': timeframe,
-                'meth': 'add'|'remove' as meth,
-                'kwargs': {
-                    'x': float(x),
-                    'y': float(y),
-                    'pointing': pointing,
-                    'color': color,
-                    'aid': str()|None as aid,
-                    'alpha': int(alpha),
-                    'headLen': int()|float()|None as headLen,
-                    'headWidth': int()|float()|None as headWidth,
-                    'tailLen': int()|float()|None as tailLen,
-                    'tailWidth': int()|float()|None as tailWidth,
-                    'pxMode': bool(pxMode),
-                    'time': int()|float()|None as timestamp,
-                },
-                # ?TODO? split based on method fn-sigs?
-                # 'pointing',
-            }:
-                ds: DisplayState = _dss[fqme]
-                try:
-                    chart: ChartPlotWidget = {
-                        60: ds.hist_chart,
-                        1: ds.chart,
-                    }[timeframe]
-                except KeyError:
-                    log.warning(
-                        f'No chart for timeframe={timeframe}s, '
-                        f'skipping arrow annotation'
-                    )
-                    # return -1 to indicate failure
-                    await annot_req_stream.send(-1)
-                    continue
-                cv: ChartView = chart.cv
-                godw = chart.linked.godwidget
-
-                # NEW: if timestamp provided, lookup current index
-                # from shm to ensure alignment with current buffer
-                # state
-                if timestamp is not None:
-                    viz: Viz = chart.get_viz(fqme)
-                    shm = viz.shm
-                    arr = shm.array
-                    # find index where time matches timestamp
-                    matches = arr[arr['time'] == timestamp]
-                    if len(matches) == 0:
-                        log.error(
-                            f'No shm entry for timestamp={timestamp}, '
-                            f'skipping arrow annotation'
-                        )
-                        await annot_req_stream.send(-1)
-                        continue
-                    # use the matched row's index as x
-                    x = float(matches[0]['index'])
-
-                arrows = ArrowEditor(godw=godw)
-                # `.add/.remove()` API
-                if meth != 'add':
-                    # await tractor.pause()
-                    raise ValueError(
-                        f'Invalid arrow-edit request ?\n'
-                        f'{msg!r}\n'
-                    )
-
-                aid: str = str(uuid4())
-                arrow: pg.ArrowItem = arrows.add(
-                    plot=chart.plotItem,
-                    uid=aid,
-                    x=x,
-                    y=y,
-                    pointing=pointing,
-                    color=color,
-                    alpha=alpha,
-                    headLen=headLen,
-                    headWidth=headWidth,
-                    tailLen=tailLen,
-                    tailWidth=tailWidth,
-                    pxMode=pxMode,
-                )
-                # XXX: store absolute coords for repositioning
-                # during viz redraws (eg backfill updates)
-                arrow._abs_x = x
-                arrow._abs_y = y
-
-                annots[aid] = arrow
-                _editors[aid] = arrows
-                aids: set[int] = ctxs[ipc_key][1]
-                aids.add(aid)
-                await annot_req_stream.send(aid)
-
-            case {
-                'cmd': 'TextItem',
-                'fqme': fqme,
-                'timeframe': timeframe,
-                'kwargs': {
-                    'text': str(text),
-                    'x': int()|float() as x,
-                    'y': int()|float() as y,
-                    'color': color,
-                    'anchor': list(anchor),
-                    'font_size': int()|None as font_size,
-                    'time': int()|float()|None as timestamp,
-                },
-            }:
-                ds: DisplayState = _dss[fqme]
-                try:
-                    chart: ChartPlotWidget = {
-                        60: ds.hist_chart,
-                        1: ds.chart,
-                    }[timeframe]
-                except KeyError:
-                    log.warning(
-                        f'No chart for timeframe={timeframe}s, '
-                        f'skipping text annotation'
-                    )
-                    await annot_req_stream.send(-1)
-                    continue
-
-                # NEW: if timestamp provided, lookup current index
-                # from shm to ensure alignment with current buffer
-                # state
-                if timestamp is not None:
-                    viz: Viz = chart.get_viz(fqme)
-                    shm = viz.shm
-                    arr = shm.array
-                    # find index where time matches timestamp
-                    matches = arr[arr['time'] == timestamp]
-                    if len(matches) == 0:
-                        log.error(
-                            f'No shm entry for timestamp={timestamp}, '
-                            f'skipping text annotation'
-                        )
-                        await annot_req_stream.send(-1)
-                        continue
-                    # use the matched row's index as x, +1 for text
-                    # offset
-                    x = float(matches[0]['index']) + 1
-
-                # convert named color to hex
-                color_hex: str = hcolor(color)
-
-                # create text item
-                text_item: pg.TextItem = pg.TextItem(
-                    text=text,
-                    color=color_hex,
-                    anchor=anchor,
-
-                    # ?TODO, pin to github:main for this?
-                    # legacy, can have scaling ish?
-                    # ensureInBounds=True,
-                )
-
-                # apply font size (default to DpiAwareFont if not
-                # provided)
-                if font_size is None:
-                    from ._style import get_fonts
-                    font, font_small = get_fonts()
-                    font_size =  font_small.px_size - 1
-
-                qfont: QFont = text_item.textItem.font()
-                qfont.setPixelSize(font_size)
-                text_item.setFont(qfont)
-
-                text_item.setPos(x, y)
-                chart.plotItem.addItem(text_item)
-
-                # XXX: store absolute coords for repositioning
-                # during viz redraws (eg backfill updates)
-                text_item._abs_x = x
-                text_item._abs_y = y
-
-                aid: str = str(uuid4())
-                annots[aid] = text_item
-                aids: set[int] = ctxs[ipc_key][1]
-                aids.add(aid)
-                await annot_req_stream.send(aid)
-
             case {
                 'cmd': 'remove',
-                'aid': int(aid)|str(aid),
+                'aid': int(aid),
             }:
                 # NOTE: this is normally entered on
                 # a client's annotation de-alloc normally
                 # prior to detach or modify.
                 annot: QGraphicsItem = annots[aid]
-                assert rm_annot(annot)
+                annot.delete()
 
                 # respond to client indicating annot
                 # was indeed deleted.
@@ -468,38 +174,6 @@ async def serve_rc_annots(
                 )
                 viz.reset_graphics()
 
-                # XXX: reposition all annotations to ensure they
-                # stay aligned with viz data after reset (eg during
-                # backfill when abs-index range changes)
-                n_repositioned: int = 0
-                for aid, annot in annots.items():
-                    # arrows and text items use abs x,y coords
-                    if (
-                        hasattr(annot, '_abs_x')
-                        and
-                        hasattr(annot, '_abs_y')
-                    ):
-                        annot.setPos(
-                            annot._abs_x,
-                            annot._abs_y,
-                        )
-                        n_repositioned += 1
-
-                    # rects use method + kwargs
-                    elif (
-                        hasattr(annot, '_meth')
-                        and
-                        hasattr(annot, '_kwargs')
-                    ):
-                        getattr(annot, annot._meth)(**annot._kwargs)
-                        n_repositioned += 1
-
-                if n_repositioned:
-                    log.info(
-                        f'Repositioned {n_repositioned} annotation(s) '
-                        f'after viz redraw'
-                    )
-
             case _:
                 log.error(
                     'Unknown remote annotation cmd:\n'
@@ -513,12 +187,6 @@ async def remote_annotate(
 ) -> None:
 
     global _dss, _ctxs
-    if not _dss:
-        raise RuntimeError(
-            'Race condition on chart-init state ??\n'
-            'Anoter actor is trying to annoate this chart '
-            'before it has fully spawned.\n'
-        )
     assert _dss
 
     _ctxs[ctx.cid] = (ctx, set())
@@ -543,7 +211,7 @@ async def remote_annotate(
             assert _ctx is ctx
             for aid in aids:
                 annot: QGraphicsItem = _annots[aid]
-                assert rm_annot(annot)
+                annot.delete()
 
 
 class AnnotCtl(Struct):
@@ -588,47 +256,36 @@ class AnnotCtl(Struct):
 
         from_acm: bool = False,
 
-        # NEW: optional timestamps for server-side index lookup
-        start_time: float|None = None,
-        end_time: float|None = None,
-
-    ) -> int|None:
+    ) -> int:
         '''
         Add a `SelectRect` annotation to the target view, return
         the instances `id(obj)` from the remote UI actor.
 
         '''
         ipc: MsgStream = self._get_ipc(fqme)
-        with trio.fail_after(3):
-            await ipc.send({
-                'fqme': fqme,
-                'cmd': 'SelectRect',
-                'timeframe': timeframe,
-                # 'meth': str(meth),
-                'meth': 'set_view_pos' if domain == 'view' else 'set_scene_pos',
-                'kwargs': {
-                    'start_pos': tuple(start_pos),
-                    'end_pos': tuple(end_pos),
-                    'color': color,
-                    'update_label': False,
-                    'start_time': start_time,
-                    'end_time': end_time,
-                },
-            })
-            aid: int|dict = await ipc.receive()
-            match aid:
-                case {'error': str(msg)}:
-                    log.error(msg)
-                    return None
-            self._ipcs[aid] = ipc
-            if not from_acm:
-                self._annot_stack.push_async_callback(
-                    partial(
-                        self.remove,
-                        aid,
-                    )
+        await ipc.send({
+            'fqme': fqme,
+            'cmd': 'SelectRect',
+            'timeframe': timeframe,
+            # 'meth': str(meth),
+            'meth': 'set_view_pos' if domain == 'view' else 'set_scene_pos',
+            'kwargs': {
+                'start_pos': tuple(start_pos),
+                'end_pos': tuple(end_pos),
+                'color': color,
+                'update_label': False,
+            },
+        })
+        aid: int = await ipc.receive()
+        self._ipcs[aid] = ipc
+        if not from_acm:
+            self._annot_stack.push_async_callback(
+                partial(
+                    self.remove,
+                    aid,
                 )
-            return aid
+            )
+        return aid
 
     async def remove(
         self,
@@ -659,9 +316,7 @@ class AnnotCtl(Struct):
             )
             yield aid
         finally:
-            # async ipc send op
-            with trio.CancelScope(shield=True):
-                await self.remove(aid)
+            await self.remove(aid)
 
     async def redraw(
         self,
@@ -676,130 +331,20 @@ class AnnotCtl(Struct):
             'timeframe': timeframe,
         })
 
-    async def add_arrow(
-        self,
-        fqme: str,
-        timeframe: float,
-        x: float,
-        y: float,
-        pointing: Literal[
-            'up',
-            'down',
-        ],
-        # TODO: a `Literal['view', 'scene']` for this?
-        # domain: str = 'view',  # or 'scene'
-        color: str = 'dad_blue',
-        alpha: int = 116,
-        headLen: float|None = None,
-        headWidth: float|None = None,
-        tailLen: float|None = None,
-        tailWidth: float|None = None,
-        pxMode: bool = True,
+    # TODO: do we even need this?
+    # async def modify(
+    #     self,
+    #     aid: int,  # annotation id
+    #     meth: str,  # far end graphics object method to invoke
+    #     params: dict[str, Any],  # far end `meth(**kwargs)`
+    # ) -> bool:
+    #     '''
+    #     Modify an existing (remote) annotation's graphics
+    #     paramters, thus changing it's appearance / state in real
+    #     time.
 
-        from_acm: bool = False,
-
-        # NEW: optional timestamp for server-side index lookup
-        time: float|None = None,
-
-    ) -> int|None:
-        '''
-        Add a `SelectRect` annotation to the target view, return
-        the instances `id(obj)` from the remote UI actor.
-
-        '''
-        ipc: MsgStream = self._get_ipc(fqme)
-        with trio.fail_after(3):
-            await ipc.send({
-                'fqme': fqme,
-                'cmd': 'ArrowEditor',
-                'timeframe': timeframe,
-                # 'meth': str(meth),
-                'meth': 'add',
-                'kwargs': {
-                    'x': float(x),
-                    'y': float(y),
-                    'color': color,
-                    'pointing': pointing,  # up|down
-                    'alpha': alpha,
-                    'aid': None,
-                    'headLen': headLen,
-                    'headWidth': headWidth,
-                    'tailLen': tailLen,
-                    'tailWidth': tailWidth,
-                    'pxMode': pxMode,
-                    'time': time,  # for server-side index lookup
-                },
-            })
-            aid: int|dict = await ipc.receive()
-            match aid:
-                case {'error': str(msg)}:
-                    log.error(msg)
-                    return None
-
-            self._ipcs[aid] = ipc
-            if not from_acm:
-                self._annot_stack.push_async_callback(
-                    partial(
-                        self.remove,
-                        aid,
-                    )
-                )
-            return aid
-
-    async def add_text(
-        self,
-        fqme: str,
-        timeframe: float,
-        text: str,
-        x: float,
-        y: float,
-        color: str|tuple = 'dad_blue',
-        anchor: tuple[float, float] = (0, 1),
-        font_size: int|None = None,
-
-        from_acm: bool = False,
-
-        # NEW: optional timestamp for server-side index lookup
-        time: float|None = None,
-
-    ) -> int|None:
-        '''
-        Add a `pg.TextItem` annotation to the target view.
-
-        anchor: (x, y) where (0,0) is upper-left, (1,1) is lower-right
-        font_size: pixel size for font, defaults to `_font.font.pixelSize()`
-
-        '''
-        ipc: MsgStream = self._get_ipc(fqme)
-        with trio.fail_after(3):
-            await ipc.send({
-                'fqme': fqme,
-                'cmd': 'TextItem',
-                'timeframe': timeframe,
-                'kwargs': {
-                    'text': text,
-                    'x': float(x),
-                    'y': float(y),
-                    'color': color,
-                    'anchor': tuple(anchor),
-                    'font_size': font_size,
-                    'time': time,  # for server-side index lookup
-                },
-            })
-            aid: int|dict = await ipc.receive()
-            match aid:
-                case {'error': str(msg)}:
-                    log.error(msg)
-                    return None
-            self._ipcs[aid] = ipc
-            if not from_acm:
-                self._annot_stack.push_async_callback(
-                    partial(
-                        self.remove,
-                        aid,
-                    )
-                )
-            return aid
+    #     '''
+    #     raise NotImplementedError
 
 
 @acm
@@ -826,9 +371,7 @@ async def open_annot_ctl(
         # TODO: print the current discoverable actor UID set
         # here as well?
         if not maybe_portals:
-            raise RuntimeError(
-                'No chart actors found in service domain?'
-            )
+            raise RuntimeError('No chart UI actors found in service domain?')
 
         for portal in maybe_portals:
             ctx_mngrs.append(