Compare commits
14 Commits
main
...
nix_shell_
| Author | SHA1 | Date |
|---|---|---|
|
|
08811f9a61 | |
|
|
b577180773 | |
|
|
f12c452d96 | |
|
|
3531c2edc1 | |
|
|
97dd7e766a | |
|
|
ab1463d942 | |
|
|
5314cb79d4 | |
|
|
0c0b7116e3 | |
|
|
19c343e8b2 | |
|
|
b7883325a9 | |
|
|
37ca081555 | |
|
|
44b8c70521 | |
|
|
e6af97c596 | |
|
|
95ace5acb8 |
278
README.rst
278
README.rst
|
|
@ -1,199 +1,162 @@
|
|||
piker
|
||||
-----
|
||||
trading gear for hackers
|
||||
trading gear for hackers.
|
||||
|
||||
|gh_actions|
|
||||
|
||||
.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
|
||||
:target: https://actions-badge.atrox.dev/piker/pikers/goto
|
||||
|
||||
``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
|
||||
real-time computational trading targeted at `hardcore Linux users
|
||||
<comp_trader>`_ .
|
||||
``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
|
||||
computational trading targeted at `hardcore Linux users <comp_trader>`_ .
|
||||
|
||||
we use much bleeding edge tech including (but not limited to):
|
||||
we use as much bleeding edge tech as possible including (but not limited to):
|
||||
|
||||
- latest python for glue_
|
||||
- uv_ for packaging and distribution
|
||||
- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
|
||||
- Qt_ for pristine low latency UIs
|
||||
- pyqtgraph_ (which we've extended) for real-time charting and graphics
|
||||
- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
|
||||
- `apache arrow and parquet`_ for time-series storage
|
||||
- trio_ & tractor_ for our distributed, multi-core, real-time streaming
|
||||
`structured concurrency`_ runtime B)
|
||||
- Qt_ for pristine high performance UIs
|
||||
- pyqtgraph_ for real-time charting
|
||||
- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
|
||||
- `apache arrow and parquet`_ for time series history management
|
||||
persistence and sharing
|
||||
- (prototyped) techtonicdb_ for L2 book storage
|
||||
|
||||
potential projects we might integrate with soon,
|
||||
|
||||
- (already prototyped in ) techtonicdb_ for L2 book storage
|
||||
|
||||
.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
|
||||
.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
|
||||
.. _uv: https://docs.astral.sh/uv/
|
||||
.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
|
||||
:target: https://travis-ci.org/pikers/piker
|
||||
.. _trio: https://github.com/python-trio/trio
|
||||
.. _tractor: https://github.com/goodboy/tractor
|
||||
.. _structured concurrency: https://trio.discourse.group/
|
||||
.. _marketstore: https://github.com/alpacahq/marketstore
|
||||
.. _techtonicdb: https://github.com/0b01/tectonicdb
|
||||
.. _Qt: https://www.qt.io/
|
||||
.. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
|
||||
.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
|
||||
.. _apache arrow and parquet: https://arrow.apache.org/faq/
|
||||
.. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
|
||||
.. _techtonicdb: https://github.com/0b01/tectonicdb
|
||||
.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
|
||||
|
||||
|
||||
focus and feats:
|
||||
****************
|
||||
fitting with these tenets, we're always open to new
|
||||
framework/lib/service interop suggestions and ideas!
|
||||
focus and features:
|
||||
*******************
|
||||
- 100% federated: your code, your hardware, your data feeds, your broker fills.
|
||||
- zero web: low latency, native software that doesn't try to re-invent the OS
|
||||
- maximal **privacy**: prevent brokers and mms from knowing your
|
||||
planz; smack their spreads with dark volume.
|
||||
- zero clutter: modal, context oriented UIs that echew minimalism, reduce
|
||||
thought noise and encourage un-emotion.
|
||||
- first class parallelism: built from the ground up on next-gen structured concurrency
|
||||
primitives.
|
||||
- traders first: broker/exchange/asset-class agnostic
|
||||
- systems grounded: real-time financial signal processing that will
|
||||
make any queuing or DSP eng juice their shorts.
|
||||
- non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
|
||||
- data collaboration: every process and protocol is multi-host scalable.
|
||||
- fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
|
||||
|
||||
- **100% federated**:
|
||||
your code, your hardware, your data feeds, your broker fills.
|
||||
fitting with these tenets, we're always open to new framework suggestions and ideas.
|
||||
|
||||
- **zero web**:
|
||||
low latency as a prime objective, native UIs and modern IPC
|
||||
protocols without trying to re-invent the "OS-as-an-app"..
|
||||
|
||||
- **maximal privacy**:
|
||||
prevent brokers and mms from knowing your planz; smack their
|
||||
spreads with dark volume from a VPN tunnel.
|
||||
|
||||
- **zero clutter**:
|
||||
modal, context oriented UIs that echew minimalism, reduce thought
|
||||
noise and encourage un-emotion.
|
||||
|
||||
- **first class parallelism**:
|
||||
built from the ground up on a next-gen structured concurrency
|
||||
supervision sys.
|
||||
|
||||
- **traders first**:
|
||||
broker/exchange/venue/asset-class/money-sys agnostic
|
||||
|
||||
- **systems grounded**:
|
||||
real-time financial signal processing (fsp) that will make any
|
||||
queuing or DSP eng juice their shorts.
|
||||
|
||||
- **non-tina UX**:
|
||||
sleek, powerful keyboard driven interaction with expected use in
|
||||
tiling wms (or maybe even a DDE).
|
||||
|
||||
- **data collab at scale**:
|
||||
every actor-process and protocol is multi-host aware.
|
||||
|
||||
- **fight club ready**:
|
||||
zero interest in adoption by suits; no corporate friendly license,
|
||||
ever.
|
||||
|
||||
building the hottest looking, fastest, most reliable, keyboard
|
||||
friendly FOSS trading platform is the dream; join the cause.
|
||||
building the best looking, most reliable, keyboard friendly trading
|
||||
platform is the dream; join the cause.
|
||||
|
||||
|
||||
a sane install with `uv`
|
||||
************************
|
||||
bc why install with `python` when you can faster with `rust` ::
|
||||
|
||||
uv sync
|
||||
|
||||
# ^ astral's docs,
|
||||
# https://docs.astral.sh/uv/concepts/projects/sync/
|
||||
|
||||
include all GUIs (ex. for charting)::
|
||||
|
||||
uv sync --group uis
|
||||
|
||||
AND with **all** our normal hacking tools::
|
||||
|
||||
uv sync --dev
|
||||
|
||||
AND if you want to try WIP integrations::
|
||||
|
||||
uv sync --all-groups
|
||||
|
||||
Ensure you can run the root-daemon::
|
||||
|
||||
uv run pikerd [-l info --pdb]
|
||||
sane install with `poetry`
|
||||
**************************
|
||||
TODO!
|
||||
|
||||
|
||||
install on nix(os)
|
||||
******************
|
||||
``NixOS`` is our core devs' distro of choice for which we offer
|
||||
a stringently defined development shell envoirment that can currently
|
||||
be applied in one of 2 ways::
|
||||
|
||||
# ONLY if running on X11
|
||||
nix-shell default.nix
|
||||
|
||||
Or if you prefer flakes style and a modern DE::
|
||||
|
||||
# ONLY if also running on Wayland
|
||||
nix develop # for default bash
|
||||
nix develop -c uv run xonsh # for @goodboy's preferred sh B)
|
||||
rigorous install on ``nixos`` using ``poetry2nix``
|
||||
**************************************************
|
||||
TODO!
|
||||
|
||||
|
||||
start a chart
|
||||
*************
|
||||
run a realtime OHLCV chart stand-alone::
|
||||
hacky install on nixos
|
||||
**********************
|
||||
`NixOS` is our core devs' distro of choice for which we offer
|
||||
a stringently defined development shell envoirment that can be loaded with::
|
||||
|
||||
[uv run] piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
|
||||
nix-shell develop.nix
|
||||
|
||||
# ^^^ iff you haven't activated the py-env,
|
||||
# - https://docs.astral.sh/uv/concepts/projects/run/
|
||||
#
|
||||
# in order to create an explicit virt-env see,
|
||||
# - https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment
|
||||
# - https://docs.astral.sh/uv/pip/environments/
|
||||
#
|
||||
# use $UV_PROJECT_ENVIRONMENT to select any non-`.venv/`
|
||||
# as the venv sudir in the repo's root.
|
||||
# - https://docs.astral.sh/uv/reference/environment/#uv_project_environment
|
||||
this will setup the required python environment to run piker, make sure to
|
||||
run::
|
||||
|
||||
this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
|
||||
overlayed on the same graph. Use of `piker` without first starting
|
||||
a daemon (`pikerd` - see below) means there is an implicit spawning of the
|
||||
multi-actor-runtime (implemented as a `tractor` app).
|
||||
pip install -r requirements.txt -e .
|
||||
|
||||
For additional subsystem feats available through our chart UI see the
|
||||
various sub-readmes:
|
||||
|
||||
- order control using a mouse-n-keyboard UX B)
|
||||
- cross venue market-pair (what most call "symbol") search, select, overlay Bo
|
||||
- financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
|
||||
- src-asset derivatives scan for anal, like the infamous "max pain" XO
|
||||
once after loading the shell
|
||||
|
||||
|
||||
spawn a daemon standalone
|
||||
*************************
|
||||
we call the root actor-process the ``pikerd``. it can be (and is
|
||||
recommended normally to be) started separately from the ``piker
|
||||
chart`` program::
|
||||
install wild-west style via `pip`
|
||||
*********************************
|
||||
``piker`` is currently under heavy pre-alpha development and as such
|
||||
should be cloned from this repo and hacked on directly.
|
||||
|
||||
for a development install::
|
||||
|
||||
git clone git@github.com:pikers/piker.git
|
||||
cd piker
|
||||
virtualenv env
|
||||
source ./env/bin/activate
|
||||
pip install -r requirements.txt -e .
|
||||
|
||||
|
||||
check out our charts
|
||||
********************
|
||||
bet you weren't expecting this from the foss::
|
||||
|
||||
piker -l info -b kraken -b binance chart btcusdt.binance --pdb
|
||||
|
||||
|
||||
this runs the main chart (currently with 1m sampled OHLC) in in debug
|
||||
mode and you can practice paper trading using the following
|
||||
micro-manual:
|
||||
|
||||
``order_mode`` (
|
||||
edge triggered activation by any of the following keys,
|
||||
``mouse-click`` on y-level to submit at that price
|
||||
):
|
||||
|
||||
- ``f``/ ``ctl-f`` to stage buy
|
||||
- ``d``/ ``ctl-d`` to stage sell
|
||||
- ``a`` to stage alert
|
||||
|
||||
|
||||
``search_mode`` (
|
||||
``ctl-l`` or ``ctl-space`` to open,
|
||||
``ctl-c`` or ``ctl-space`` to close
|
||||
) :
|
||||
|
||||
- begin typing to have symbol search automatically lookup
|
||||
symbols from all loaded backend (broker) providers
|
||||
- arrow keys and mouse click to navigate selection
|
||||
- vi-like ``ctl-[hjkl]`` for navigation
|
||||
|
||||
|
||||
you can also configure your position allocation limits from the
|
||||
sidepane.
|
||||
|
||||
|
||||
run in distributed mode
|
||||
***********************
|
||||
start the service manager and data feed daemon in the background and
|
||||
connect to it::
|
||||
|
||||
pikerd -l info --pdb
|
||||
|
||||
the daemon does nothing until a ``piker``-client (like ``piker
|
||||
chart``) connects and requests some particular sub-system. for
|
||||
a connecting chart ``pikerd`` will spawn and manage at least,
|
||||
|
||||
- a data-feed daemon: ``datad`` which does all the work of comms with
|
||||
the backend provider (in this case the ``binance`` cex).
|
||||
- a paper-trading engine instance, ``paperboi.binance``, (if no live
|
||||
account has been configured) which allows for auto/manual order
|
||||
control against the live quote stream.
|
||||
connect your chart::
|
||||
|
||||
*using* an actor-service (aka micro-daemon) manager which dynamically
|
||||
supervises various sub-subsystems-as-services throughout the ``piker``
|
||||
runtime-stack.
|
||||
piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
|
||||
|
||||
now you can (implicitly) connect your chart::
|
||||
|
||||
piker chart btcusdt.spot.binance
|
||||
|
||||
since ``pikerd`` was started separately you can now enjoy a persistent
|
||||
real-time data stream tied to the daemon-tree's lifetime. i.e. the next
|
||||
time you spawn a chart it will obviously not only load much faster
|
||||
(since the underlying ``datad.binance`` is left running with its
|
||||
in-memory IPC data structures) but also the data-feed and any order
|
||||
mgmt states should be persistent until you finally cancel ``pikerd``.
|
||||
enjoy persistent real-time data feeds tied to daemon lifetime. the next
|
||||
time you spawn a chart it will load much faster since the data feed has
|
||||
been cached and is now always running live in the background until you
|
||||
kill ``pikerd``.
|
||||
|
||||
|
||||
if anyone asks you what this project is about
|
||||
*********************************************
|
||||
you don't talk about it; just use it.
|
||||
you don't talk about it.
|
||||
|
||||
|
||||
how do i get involved?
|
||||
|
|
@ -203,15 +166,6 @@ enter the matrix.
|
|||
|
||||
how come there ain't that many docs
|
||||
***********************************
|
||||
i mean we want/need them but building the core right has been higher
|
||||
prio then marketting (and likely will stay that way Bp).
|
||||
|
||||
soo, suck it up bc,
|
||||
|
||||
- no one is trying to sell you on anything
|
||||
- learning the code base is prolly way more valuable
|
||||
- the UI/UXs are intended to be "intuitive" for any hacker..
|
||||
|
||||
we obviously need tonz help so if you want to start somewhere and
|
||||
can't necessarily write "advanced" concurrent python/rust code, this
|
||||
helping document literally anything might be the place for you!
|
||||
suck it up, learn the code; no one is trying to sell you on anything.
|
||||
also, we need lotsa help so if you want to start somewhere and can't
|
||||
necessarily write serious code, this might be the place for you!
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
################
|
||||
# ---- CEXY ----
|
||||
|
||||
################
|
||||
[binance]
|
||||
accounts.paper = 'paper'
|
||||
|
||||
|
|
@ -12,41 +13,28 @@ accounts.spot = 'spot'
|
|||
spot.use_testnet = false
|
||||
spot.api_key = ''
|
||||
spot.api_secret = ''
|
||||
# ------ binance ------
|
||||
|
||||
|
||||
[deribit]
|
||||
# std assets
|
||||
key_id = ''
|
||||
key_secret = ''
|
||||
# options
|
||||
accounts.option = 'option'
|
||||
option.use_testnet = false
|
||||
option.key_id = ''
|
||||
option.key_secret = ''
|
||||
# aux logging from `cryptofeed`
|
||||
option.log.filename = 'cryptofeed.log'
|
||||
option.log.level = 'DEBUG'
|
||||
option.log.disabled = true
|
||||
# ------ deribit ------
|
||||
|
||||
|
||||
[kraken]
|
||||
key_descr = ''
|
||||
api_key = ''
|
||||
secret = ''
|
||||
# ------ kraken ------
|
||||
|
||||
|
||||
[kucoin]
|
||||
key_id = ''
|
||||
key_secret = ''
|
||||
key_passphrase = ''
|
||||
# ------ kucoin ------
|
||||
|
||||
|
||||
################
|
||||
# -- BROKERZ ---
|
||||
|
||||
################
|
||||
[questrade]
|
||||
refresh_token = ''
|
||||
access_token = ''
|
||||
|
|
@ -54,55 +42,44 @@ api_server = 'https://api06.iq.questrade.com/'
|
|||
expires_in = 1800
|
||||
token_type = 'Bearer'
|
||||
expires_at = 1616095326.355846
|
||||
# ------ questrade ------
|
||||
|
||||
|
||||
[ib]
|
||||
# define the (set of) host-port socketaddrs that
|
||||
# brokerd.ib will scan to connect to an API endpoint
|
||||
# (ib-gw or ib-tws listening instances)
|
||||
hosts = [
|
||||
'127.0.0.1',
|
||||
]
|
||||
# XXX: the order in which ports will be scanned
|
||||
# (by the `brokerd` daemon-actor)
|
||||
# is determined # by the line order here.
|
||||
# TODO: when we eventually spawn gateways in our
|
||||
# container, we can just dynamically allocate these
|
||||
# using IBC.
|
||||
ports = [
|
||||
4002, # gw
|
||||
7497, # tws
|
||||
]
|
||||
|
||||
# When API endpoints are being scanned durin startup, the order
|
||||
# of user-defined-account "names" (as defined below) here
|
||||
# determines which py-client connection is given priority to be
|
||||
# used for data-feed-requests by according to whichever client
|
||||
# connected to an API endpoing which reported the equivalent
|
||||
# account number for that name.
|
||||
# XXX: for a paper account the flex web query service
|
||||
# is not supported so you have to manually download
|
||||
# and XML report and put it in a location that can be
|
||||
# accessed by the ``brokerd.ib`` backend code for parsing.
|
||||
flex_token = ''
|
||||
flex_trades_query_id = '' # live account
|
||||
|
||||
# when clients are being scanned this determines
|
||||
# which clients are preferred to be used for data
|
||||
# feeds based on the order of account names, if
|
||||
# detected as active on an API client.
|
||||
prefer_data_account = [
|
||||
'paper',
|
||||
'margin',
|
||||
'ira',
|
||||
]
|
||||
|
||||
# For long-term trades txn (transaction) history
|
||||
# processing (i.e your txn ledger with IB) you can
|
||||
# (automatically for live accounts) query the FLEX
|
||||
# report system for past history.
|
||||
#
|
||||
# (For paper accounts the web query service
|
||||
# is not supported so you have to manually download
|
||||
# an XML report and put it in a location that can be
|
||||
# accessed by our `brokerd.ib` backend code for parsing).
|
||||
#
|
||||
flex_token = ''
|
||||
flex_trades_query_id = '' # live account
|
||||
|
||||
# define "aliases" (names) for each account number
|
||||
# such that the names can be reffed and logged throughout
|
||||
# `piker.accounting` subsys and more easily
|
||||
# referred to by the user.
|
||||
#
|
||||
# These keys will be the set exposed through the order-mode
|
||||
# account-selection UI so that numbers are never shown.
|
||||
[ib.accounts]
|
||||
paper = 'DU0000000' # <- literal account #
|
||||
margin = 'U0000000'
|
||||
ira = 'U0000000'
|
||||
# ------ ib ------
|
||||
# the order in which accounts will be selectable
|
||||
# in the order mode UI (if found via clients during
|
||||
# API-app scanning)when a new symbol is loaded.
|
||||
paper = 'XX0000000'
|
||||
margin = 'X0000000'
|
||||
ira = 'X0000000'
|
||||
|
|
|
|||
|
|
@ -1,9 +1,7 @@
|
|||
[network]
|
||||
pikerd = [
|
||||
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
||||
# '/uds/6116', # TODO std uds socket file
|
||||
]
|
||||
|
||||
tsdb.backend = 'marketstore'
|
||||
tsdb.host = 'localhost'
|
||||
tsdb.grpc_port = 5995
|
||||
|
||||
[ui]
|
||||
# set custom font + size which will scale entire UI
|
||||
|
|
|
|||
49
default.nix
49
default.nix
|
|
@ -11,12 +11,11 @@ let
|
|||
libxkbcommonStorePath = lib.getLib libxkbcommon;
|
||||
xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
|
||||
|
||||
pypkgs = python313Packages;
|
||||
qtpyStorePath = lib.getLib pypkgs.qtpy;
|
||||
pyqt6StorePath = lib.getLib pypkgs.pyqt6;
|
||||
pyqt6SipStorePath = lib.getLib pypkgs.pyqt6-sip;
|
||||
rapidfuzzStorePath = lib.getLib pypkgs.rapidfuzz;
|
||||
qdarkstyleStorePath = lib.getLib pypkgs.qdarkstyle;
|
||||
qtpyStorePath = lib.getLib python312Packages.qtpy;
|
||||
pyqt6StorePath = lib.getLib python312Packages.pyqt6;
|
||||
pyqt6SipStorePath = lib.getLib python312Packages.pyqt6-sip;
|
||||
rapidfuzzStorePath = lib.getLib python312Packages.rapidfuzz;
|
||||
qdarkstyleStorePath = lib.getLib python312Packages.qdarkstyle;
|
||||
|
||||
xorgLibX11StorePath = lib.getLib xorg.libX11;
|
||||
xorgLibxcbStorePath = lib.getLib xorg.libxcb;
|
||||
|
|
@ -27,7 +26,7 @@ let
|
|||
xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
|
||||
in
|
||||
stdenv.mkDerivation {
|
||||
name = "piker-qt6-uv";
|
||||
name = "piker-qt6-poetry-shell";
|
||||
buildInputs = [
|
||||
# System requirements.
|
||||
glib
|
||||
|
|
@ -52,12 +51,12 @@ stdenv.mkDerivation {
|
|||
xorg.xcbutilrenderutil
|
||||
|
||||
# Python requirements.
|
||||
python313
|
||||
uv
|
||||
pypkgs.qdarkstyle
|
||||
pypkgs.rapidfuzz
|
||||
pypkgs.pyqt6
|
||||
pypkgs.qtpy
|
||||
python312Full
|
||||
python312Packages.poetry-core
|
||||
python312Packages.qdarkstyle
|
||||
python312Packages.rapidfuzz
|
||||
python312Packages.pyqt6
|
||||
python312Packages.qtpy
|
||||
];
|
||||
src = null;
|
||||
shellHook = ''
|
||||
|
|
@ -114,11 +113,11 @@ stdenv.mkDerivation {
|
|||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.13/site-packages"
|
||||
QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.13/site-packages"
|
||||
QTPY_PATH="${qtpyStorePath}/lib/python3.13/site-packages"
|
||||
PYQT6_PATH="${pyqt6StorePath}/lib/python3.13/site-packages"
|
||||
PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.13/site-packages"
|
||||
RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.12/site-packages"
|
||||
QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.12/site-packages"
|
||||
QTPY_PATH="${qtpyStorePath}/lib/python3.12/site-packages"
|
||||
PYQT6_PATH="${pyqt6StorePath}/lib/python3.12/site-packages"
|
||||
PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.12/site-packages"
|
||||
|
||||
PATCH="$PATCH:$RPDFUZZ_PATH"
|
||||
PATCH="$PATCH:$QDRKSTYLE_PATH"
|
||||
|
|
@ -128,8 +127,18 @@ stdenv.mkDerivation {
|
|||
|
||||
export PATCH
|
||||
|
||||
# install all dev and extras
|
||||
uv sync --dev --all-extras
|
||||
# Maybe create venv & install deps
|
||||
poetry install --with uis
|
||||
|
||||
# Use pyqt6 from System, patch activate script
|
||||
ACTIVATE_SCRIPT_PATH="$(poetry env info --path)/bin/activate"
|
||||
|
||||
if grep -q "$PATCH" "$ACTIVATE_SCRIPT_PATH"; then
|
||||
echo "venv is already patched."
|
||||
else
|
||||
echo "patching $ACTIVATE_SCRIPT_PATH to use pyqt6 from nixos..."
|
||||
sed -i "\$i$PATCH" $ACTIVATE_SCRIPT_PATH
|
||||
fi
|
||||
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,138 +1,30 @@
|
|||
running ``ib`` gateway in ``docker``
|
||||
------------------------------------
|
||||
We have a config based on a well maintained community
|
||||
image from `@gnzsnz`:
|
||||
We have a config based on the (now defunct)
|
||||
image from "waytrade":
|
||||
|
||||
https://github.com/gnzsnz/ib-gateway-docker
|
||||
https://github.com/waytrade/ib-gateway-docker
|
||||
|
||||
|
||||
To startup this image simply run the command::
|
||||
To startup this image with our custom settings
|
||||
simply run the command::
|
||||
|
||||
docker compose up
|
||||
|
||||
(For further usage^ see the official `docker-compose`_ docs)
|
||||
And you should have the following socket-available services:
|
||||
|
||||
- ``x11vnc1@127.0.0.1:3003``
|
||||
- ``ib-gw@127.0.0.1:4002``
|
||||
|
||||
And you should have the following socket-available services by
|
||||
default:
|
||||
You can attach to the container via a VNC client
|
||||
without password auth.
|
||||
|
||||
- ``x11vnc1 @ 127.0.0.1:5900``
|
||||
- ``ib-gw @ 127.0.0.1:4002``
|
||||
|
||||
You can now attach to the container via a VNC client with password-auth;
|
||||
here is an example using ``vncclient`` on ``linux``::
|
||||
|
||||
vncviewer localhost:5900
|
||||
|
||||
now enter the pw (password) you set via an (see second code blob)
|
||||
`.env file`_ or pw-file according to the `credentials section`_.
|
||||
|
||||
If you want to change away from their default config see the example
|
||||
`docker-compose.yml`-config issue and config-section of the readme,
|
||||
|
||||
- https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
|
||||
- https://github.com/gnzsnz/ib-gateway-docker/discussions/103
|
||||
|
||||
.. _.env file: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#how-to-use-it
|
||||
.. _docker-compose: https://docs.docker.com/compose/
|
||||
.. _credentials section: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#credentials
|
||||
|
||||
|
||||
Connecting to the API from `piker`
|
||||
---------------------------------
|
||||
In order to expose the container's API endpoint to the
|
||||
`brokerd/datad/ib` actor, we need to add a section to the user's
|
||||
`brokers.toml` config (note the below is similar to the repo-shipped
|
||||
template file),
|
||||
|
||||
.. code:: toml
|
||||
|
||||
[ib]
|
||||
# define the (set of) host-port socketaddrs that
|
||||
# brokerd.ib will scan to connect to an API endpoint
|
||||
# (ib-gw or ib-tws listening instances)
|
||||
hosts = [
|
||||
'127.0.0.1',
|
||||
]
|
||||
ports = [
|
||||
4002, # gw
|
||||
7497, # tws
|
||||
]
|
||||
|
||||
# When API endpoints are being scanned durin startup, the order
|
||||
# of user-defined-account "names" (as defined below) here
|
||||
# determines which py-client connection is given priority to be
|
||||
# used for data-feed-requests by according to whichever client
|
||||
# connected to an API endpoing which reported the equivalent
|
||||
# account number for that name.
|
||||
prefer_data_account = [
|
||||
'paper',
|
||||
'margin',
|
||||
'ira',
|
||||
]
|
||||
|
||||
# define "aliases" (names) for each account number
|
||||
# such that the names can be reffed and logged throughout
|
||||
# `piker.accounting` subsys and more easily
|
||||
# referred to by the user.
|
||||
#
|
||||
# These keys will be the set exposed through the order-mode
|
||||
# account-selection UI so that numbers are never shown.
|
||||
[ib.accounts]
|
||||
paper = 'XX0000000'
|
||||
margin = 'X0000000'
|
||||
ira = 'X0000000'
|
||||
|
||||
|
||||
the broker daemon can also connect to the container's VNC server for
|
||||
added functionalies including,
|
||||
|
||||
- viewing the API endpoint program's GUI for manual interventions,
|
||||
- workarounds for historical data throttling using hotkey hacks,
|
||||
|
||||
Add a further section to `brokers.toml` which maps each API-ep's
|
||||
port to a table of VNC server connection info like,
|
||||
|
||||
.. code:: toml
|
||||
|
||||
[ib.vnc_addrs]
|
||||
4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
|
||||
|
||||
The `pw = 'doggy'` here ^ should the same value as the particular
|
||||
container instances `.env` file setting (when it was run),
|
||||
|
||||
.. code:: ini
|
||||
|
||||
VNC_SERVER_PASSWORD='doggy'
|
||||
|
||||
|
||||
IF you also want to run ``TWS``
|
||||
-------------------------------
|
||||
You can also run it containerized,
|
||||
|
||||
https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#using-tws
|
||||
|
||||
|
||||
SECURITY stuff (advanced, only if you're paranoid)
|
||||
--------------------------------------------------
|
||||
First and foremost if doing a "distributed" container setup where you
|
||||
run the ``ib-gw`` docker container and your connecting API client
|
||||
(likely ``ib_async`` from python) on **different hosts** be sure to
|
||||
read the `security considerations`_ section!
|
||||
|
||||
And for a further (somewhat paranoid) perspective from
|
||||
a long-time-ago serious devops eng..
|
||||
|
||||
Though "``ib``" claims they filter remote host connections outside
|
||||
``localhost`` (aka ``127.0.0.1`` on ipv4) it's prolly justified if
|
||||
you'd like to filter the socket at the *OS level* using a stateless
|
||||
firewall rule::
|
||||
SECURITY STUFF!?!?!
|
||||
-------------------
|
||||
Though "``ib``" claims they host filter connections outside
|
||||
localhost (aka ``127.0.0.1``) it's probably better if you filter
|
||||
the socket at the OS level using a stateless firewall rule::
|
||||
|
||||
ip rule add not unicast iif lo to 0.0.0.0/0 dport 4002
|
||||
|
||||
|
||||
We will soon have this either baked into our own custom derivative
|
||||
image (or patched into the current upstream one after further testin)
|
||||
but for now you'll have to do it urself, diggity dawg.
|
||||
|
||||
.. _security considerations: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#security-considerations
|
||||
We will soon have this baked into our own custom image but for
|
||||
now you'll have to do it urself dawgy.
|
||||
|
|
|
|||
|
|
@ -1,15 +1,10 @@
|
|||
# a community maintained IB API container!
|
||||
#
|
||||
# https://github.com/gnzsnz/ib-gateway-docker
|
||||
#
|
||||
# For piker we (currently) include some minor deviations
|
||||
# for some config files in the `volumes` section.
|
||||
#
|
||||
# See full configuration settings @
|
||||
# - https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
|
||||
# - https://github.com/gnzsnz/ib-gateway-docker/discussions/103
|
||||
# rework from the original @
|
||||
# https://github.com/waytrade/ib-gateway-docker/blob/master/docker-compose.yml
|
||||
version: "3.5"
|
||||
|
||||
|
||||
services:
|
||||
|
||||
ib_gw_paper:
|
||||
|
||||
# apparently java is a mega cukc:
|
||||
|
|
@ -55,22 +50,16 @@ services:
|
|||
target: /root/scripts/run_x11_vnc.sh
|
||||
read_only: true
|
||||
|
||||
# NOTE: an alt method to fill these out is to
|
||||
# define an `.env` file in the same dir as
|
||||
# this compose file.
|
||||
# NOTE:to fill these out, define an `.env` file in the same dir as
|
||||
# this compose file which looks something like:
|
||||
# TWS_USERID='myuser'
|
||||
# TWS_PASSWORD='guest'
|
||||
environment:
|
||||
TWS_USERID: ${TWS_USERID}
|
||||
# TWS_USERID: 'myuser'
|
||||
TWS_PASSWORD: ${TWS_PASSWORD}
|
||||
# TWS_PASSWORD: 'guest'
|
||||
TRADING_MODE: ${TRADING_MODE}
|
||||
# TRADING_MODE: 'paper'
|
||||
VNC_SERVER_PASSWORD: ${VNC_SERVER_PASSWORD}
|
||||
# VNC_SERVER_PASSWORD: 'doggy'
|
||||
|
||||
# TODO, see if we can get this supported like it
|
||||
# was on the old `waytrade` image?
|
||||
# VNC_SERVER_PORT: '3003'
|
||||
TRADING_MODE: 'paper'
|
||||
VNC_SERVER_PASSWORD: 'doggy'
|
||||
VNC_SERVER_PORT: '3003'
|
||||
|
||||
# ports:
|
||||
# - target: 4002
|
||||
|
|
@ -87,9 +76,6 @@ services:
|
|||
# - "127.0.0.1:4002:4002"
|
||||
# - "127.0.0.1:5900:5900"
|
||||
|
||||
# TODO, a masked but working example of dual paper + live
|
||||
# ib-gw instances running in a single app run!
|
||||
#
|
||||
# ib_gw_live:
|
||||
# image: waytrade/ib-gateway:1012.2i
|
||||
# restart: no
|
||||
|
|
|
|||
|
|
@ -121,7 +121,6 @@ async def bot_main():
|
|||
# tick_throttle=10,
|
||||
) as feed,
|
||||
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
assert accounts
|
||||
|
|
|
|||
127
flake.lock
127
flake.lock
|
|
@ -1,24 +1,135 @@
|
|||
{
|
||||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"flake-utils": {
|
||||
"inputs": {
|
||||
"systems": "systems"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1765779637,
|
||||
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
||||
"owner": "nixos",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
||||
"lastModified": 1689068808,
|
||||
"narHash": "sha256-6ixXo3wt24N/melDWjq70UuHQLxGV8jZvooRanIHXw0=",
|
||||
"owner": "numtide",
|
||||
"repo": "flake-utils",
|
||||
"rev": "919d646de7be200f3bf08cb76ae1f09402b6f9b4",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nixos",
|
||||
"owner": "numtide",
|
||||
"repo": "flake-utils",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"flake-utils_2": {
|
||||
"inputs": {
|
||||
"systems": "systems_2"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1689068808,
|
||||
"narHash": "sha256-6ixXo3wt24N/melDWjq70UuHQLxGV8jZvooRanIHXw0=",
|
||||
"owner": "numtide",
|
||||
"repo": "flake-utils",
|
||||
"rev": "919d646de7be200f3bf08cb76ae1f09402b6f9b4",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "numtide",
|
||||
"repo": "flake-utils",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nix-github-actions": {
|
||||
"inputs": {
|
||||
"nixpkgs": [
|
||||
"poetry2nix",
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1688870561,
|
||||
"narHash": "sha256-4UYkifnPEw1nAzqqPOTL2MvWtm3sNGw1UTYTalkTcGY=",
|
||||
"owner": "nix-community",
|
||||
"repo": "nix-github-actions",
|
||||
"rev": "165b1650b753316aa7f1787f3005a8d2da0f5301",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-community",
|
||||
"repo": "nix-github-actions",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1692174805,
|
||||
"narHash": "sha256-xmNPFDi/AUMIxwgOH/IVom55Dks34u1g7sFKKebxUm0=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "caac0eb6bdcad0b32cb2522e03e4002c8975c62e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"poetry2nix": {
|
||||
"inputs": {
|
||||
"flake-utils": "flake-utils_2",
|
||||
"nix-github-actions": "nix-github-actions",
|
||||
"nixpkgs": [
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1692048894,
|
||||
"narHash": "sha256-cDw03rso2V4CDc3Mll0cHN+ztzysAvdI8pJ7ybbz714=",
|
||||
"ref": "refs/heads/pyqt6",
|
||||
"rev": "b059ad4c3051f45d6c912e17747aae37a9ec1544",
|
||||
"revCount": 2276,
|
||||
"type": "git",
|
||||
"url": "file:///home/lord_fomo/repos/poetry2nix"
|
||||
},
|
||||
"original": {
|
||||
"type": "git",
|
||||
"url": "file:///home/lord_fomo/repos/poetry2nix"
|
||||
}
|
||||
},
|
||||
"root": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs"
|
||||
"flake-utils": "flake-utils",
|
||||
"nixpkgs": "nixpkgs",
|
||||
"poetry2nix": "poetry2nix"
|
||||
}
|
||||
},
|
||||
"systems": {
|
||||
"locked": {
|
||||
"lastModified": 1681028828,
|
||||
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"systems_2": {
|
||||
"locked": {
|
||||
"lastModified": 1681028828,
|
||||
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"type": "github"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
|
|||
251
flake.nix
251
flake.nix
|
|
@ -1,103 +1,180 @@
|
|||
# An "impure" template thx to `pyproject.nix`,
|
||||
# https://pyproject-nix.github.io/pyproject.nix/templates.html#impure
|
||||
# https://github.com/pyproject-nix/pyproject.nix/blob/master/templates/impure/flake.nix
|
||||
{
|
||||
description = "An impure `piker` overlay using `uv` with Nix(OS)";
|
||||
# NOTE: to convert to a poetry2nix env like this here are the
|
||||
# steps:
|
||||
# - install poetry in your system nix config
|
||||
# - convert the repo to use poetry using `poetry init`:
|
||||
# https://python-poetry.org/docs/basic-usage/#initialising-a-pre-existing-project
|
||||
# - then manually ensuring all deps are converted over:
|
||||
# - add this file to the repo and commit it
|
||||
# -
|
||||
|
||||
inputs = {
|
||||
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
|
||||
# GROKin tips:
|
||||
# - CLI eps are (ostensibly) added via an `entry_points.txt`:
|
||||
# - https://packaging.python.org/en/latest/specifications/entry-points/#file-format
|
||||
# - https://github.com/nix-community/poetry2nix/blob/master/editable.nix#L49
|
||||
{
|
||||
description = "piker: trading gear for hackers (pkged with poetry2nix)";
|
||||
|
||||
inputs.flake-utils.url = "github:numtide/flake-utils";
|
||||
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
|
||||
|
||||
# see https://github.com/nix-community/poetry2nix/tree/master#api
|
||||
inputs.poetry2nix = {
|
||||
# url = "github:nix-community/poetry2nix";
|
||||
# url = "github:K900/poetry2nix/qt5-explicit-deps";
|
||||
url = "/home/lord_fomo/repos/poetry2nix";
|
||||
|
||||
inputs.nixpkgs.follows = "nixpkgs";
|
||||
};
|
||||
|
||||
outputs =
|
||||
{ nixpkgs, ... }:
|
||||
let
|
||||
inherit (nixpkgs) lib;
|
||||
forAllSystems = lib.genAttrs lib.systems.flakeExposed;
|
||||
in
|
||||
{
|
||||
devShells = forAllSystems (
|
||||
system:
|
||||
let
|
||||
pkgs = nixpkgs.legacyPackages.${system};
|
||||
outputs = {
|
||||
self,
|
||||
nixpkgs,
|
||||
flake-utils,
|
||||
poetry2nix,
|
||||
}:
|
||||
# TODO: build cross-OS and use the `${system}` var thingy..
|
||||
flake-utils.lib.eachDefaultSystem (system:
|
||||
let
|
||||
# use PWD as sources
|
||||
projectDir = ./.;
|
||||
pyproject = ./pyproject.toml;
|
||||
poetrylock = ./poetry.lock;
|
||||
|
||||
# do store-path extractions
|
||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||
# ?TODO? can remove below since manual linking not needed?
|
||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||
# TODO: port to 3.11 and support both versions?
|
||||
python = "python3.10";
|
||||
|
||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||
# a major-version-pinned-`cpython`
|
||||
cpython = "python313";
|
||||
pypkgs = pkgs."${cpython}Packages";
|
||||
in
|
||||
{
|
||||
default = pkgs.mkShell {
|
||||
# for more functions and examples.
|
||||
# inherit
|
||||
# (poetry2nix.legacyPackages.${system})
|
||||
# mkPoetryApplication;
|
||||
# pkgs = nixpkgs.legacyPackages.${system};
|
||||
|
||||
packages = with pkgs; [
|
||||
# XXX, ensure sh completions active!
|
||||
bashInteractive
|
||||
bash-completion
|
||||
pkgs = nixpkgs.legacyPackages.x86_64-linux;
|
||||
lib = pkgs.lib;
|
||||
p2npkgs = poetry2nix.legacyPackages.x86_64-linux;
|
||||
|
||||
# dev utils
|
||||
ruff
|
||||
pypkgs.ruff
|
||||
# define all pkg overrides per dep, see edgecases.md:
|
||||
# https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md
|
||||
# TODO: add these into the json file:
|
||||
# https://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json
|
||||
pypkgs-build-requirements = {
|
||||
asyncvnc = [ "setuptools" ];
|
||||
eventkit = [ "setuptools" ];
|
||||
ib-insync = [ "setuptools" "flake8" ];
|
||||
msgspec = [ "setuptools"];
|
||||
pdbp = [ "setuptools" ];
|
||||
pyqt6-sip = [ "setuptools" ];
|
||||
tabcompleter = [ "setuptools" ];
|
||||
tractor = [ "setuptools" ];
|
||||
tricycle = [ "setuptools" ];
|
||||
trio-typing = [ "setuptools" ];
|
||||
trio-util = [ "setuptools" ];
|
||||
xonsh = [ "setuptools" ];
|
||||
};
|
||||
|
||||
qt6.qtwayland
|
||||
qt6.qtbase
|
||||
# auto-generate override entries
|
||||
p2n-overrides = p2npkgs.defaultPoetryOverrides.extend (self: super:
|
||||
builtins.mapAttrs (package: build-requirements:
|
||||
(builtins.getAttr package super).overridePythonAttrs (old: {
|
||||
buildInputs = (
|
||||
old.buildInputs or [ ]
|
||||
) ++ (
|
||||
builtins.map (
|
||||
pkg: if builtins.isString pkg then builtins.getAttr pkg super else pkg
|
||||
) build-requirements
|
||||
);
|
||||
})
|
||||
) pypkgs-build-requirements
|
||||
);
|
||||
|
||||
uv
|
||||
python313 # ?TODO^ how to set from `cpython` above?
|
||||
pypkgs.pyqt6
|
||||
pypkgs.pyqt6-sip
|
||||
pypkgs.qtpy
|
||||
pypkgs.qdarkstyle
|
||||
pypkgs.rapidfuzz
|
||||
];
|
||||
# override some ahead-of-time compiled extensions
|
||||
# to be built with their wheels.
|
||||
ahot_overrides = p2n-overrides.extend(
|
||||
final: prev: {
|
||||
|
||||
shellHook = ''
|
||||
# unmask to debug **this** dev-shell-hook
|
||||
# set -e
|
||||
# llvmlite = prev.llvmlite.override {
|
||||
# preferWheel = false;
|
||||
# };
|
||||
|
||||
# set qt-base/plugin path(s)
|
||||
QTBASE_PATH="${qt6baseStorePath}/lib"
|
||||
QT_PLUGIN_PATH="${qt6baseStorePath}/lib/qt-6/plugins"
|
||||
QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
|
||||
# TODO: get this workin with p2n and nixpkgs..
|
||||
# pyqt6 = prev.pyqt6.override {
|
||||
# preferWheel = true;
|
||||
# };
|
||||
|
||||
# link in Qt cc lib paths from <nixpkgs>
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
|
||||
# NOTE: this DOESN'T work atm but after a fix
|
||||
# to poetry2nix, it will and actually this line
|
||||
# won't be needed - thanks @k900:
|
||||
# https://github.com/nix-community/poetry2nix/pull/1257
|
||||
pyqt5 = prev.pyqt5.override {
|
||||
# withWebkit = false;
|
||||
preferWheel = true;
|
||||
};
|
||||
|
||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||
# see PR from @k900:
|
||||
# https://github.com/nix-community/poetry2nix/pull/1257
|
||||
# pyqt5-qt5 = prev.pyqt5-qt5.override {
|
||||
# withWebkit = false;
|
||||
# preferWheel = true;
|
||||
# };
|
||||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
# RUNTIME-SETTINGS
|
||||
#
|
||||
# ------ Qt ------
|
||||
# XXX, unmask to debug qt .so linking/loading deats
|
||||
# export QT_DEBUG_PLUGINS=1
|
||||
#
|
||||
# ALSO, for *modern linux* DEs,
|
||||
# - maybe set wayland-mode (TODO, parametrtize this!)
|
||||
# * a chosen wayland-mode shell-integration
|
||||
export QT_QPA_PLATFORM="wayland"
|
||||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||
|
||||
# ------ uv ------
|
||||
# - always use the ./py313/ venv-subdir
|
||||
export UV_PROJECT_ENVIRONMENT="py313"
|
||||
# sync project-env with all extras
|
||||
uv sync --dev --all-extras --no-group lint
|
||||
|
||||
# ------ TIPS ------
|
||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||
# run the `nix develop` cmd with,
|
||||
# >> nix develop -c uv run xonsh
|
||||
'';
|
||||
};
|
||||
}
|
||||
# TODO: patch in an override for polars to build
|
||||
# from src! See the details likely needed from
|
||||
# the cryptography entry:
|
||||
# https://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix#L426-L435
|
||||
polars = prev.polars.override {
|
||||
preferWheel = true;
|
||||
};
|
||||
}
|
||||
);
|
||||
};
|
||||
|
||||
# WHY!? -> output-attrs that `nix develop` scans for:
|
||||
# https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html#flake-output-attributes
|
||||
in
|
||||
rec {
|
||||
packages = {
|
||||
# piker = poetry2nix.legacyPackages.x86_64-linux.mkPoetryEditablePackage {
|
||||
# editablePackageSources = { piker = ./piker; };
|
||||
|
||||
piker = p2npkgs.mkPoetryApplication {
|
||||
projectDir = projectDir;
|
||||
|
||||
# SEE ABOVE for auto-genned input set, override
|
||||
# buncha deps with extras.. like `setuptools` mostly.
|
||||
# TODO: maybe propose a patch to p2n to show that you
|
||||
# can even do this in the edgecases docs?
|
||||
overrides = ahot_overrides;
|
||||
|
||||
# XXX: won't work on llvmlite..
|
||||
# preferWheels = true;
|
||||
};
|
||||
};
|
||||
|
||||
# devShells.default = pkgs.mkShell {
|
||||
# projectDir = projectDir;
|
||||
# python = "python3.10";
|
||||
# overrides = ahot_overrides;
|
||||
# inputsFrom = [ self.packages.x86_64-linux.piker ];
|
||||
# packages = packages;
|
||||
# # packages = [ poetry2nix.packages.${system}.poetry ];
|
||||
# };
|
||||
|
||||
# TODO: grok the difference here..
|
||||
# - avoid re-cloning git repos on every develop entry..
|
||||
# - ideally allow hacking on the src code of some deps
|
||||
# (tractor, pyqtgraph, tomlkit, etc.) WITHOUT having to
|
||||
# re-install them every time a change is made.
|
||||
# - boot a usable xonsh inside the poetry virtualenv when
|
||||
# defined via a custom entry point?
|
||||
devShells.default = p2npkgs.mkPoetryEnv {
|
||||
# env = p2npkgs.mkPoetryEnv {
|
||||
projectDir = projectDir;
|
||||
python = pkgs.python310;
|
||||
overrides = ahot_overrides;
|
||||
editablePackageSources = packages;
|
||||
# piker = "./";
|
||||
# tractor = "../tractor/";
|
||||
# }; # wut?
|
||||
};
|
||||
}
|
||||
); # end of .outputs scope
|
||||
}
|
||||
|
|
|
|||
|
|
@ -19,10 +19,8 @@
|
|||
for tendiez.
|
||||
|
||||
'''
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ..log import get_logger
|
||||
|
||||
from .calc import (
|
||||
iter_by_dt,
|
||||
)
|
||||
|
|
@ -35,6 +33,7 @@ from ._pos import (
|
|||
Account,
|
||||
load_account,
|
||||
load_account_from_ledger,
|
||||
open_pps,
|
||||
open_account,
|
||||
Position,
|
||||
)
|
||||
|
|
@ -43,6 +42,7 @@ from ._mktinfo import (
|
|||
dec_digits,
|
||||
digits_to_dec,
|
||||
MktPair,
|
||||
Symbol,
|
||||
unpack_fqme,
|
||||
_derivs as DerivTypes,
|
||||
)
|
||||
|
|
@ -53,23 +53,14 @@ from ._allocate import (
|
|||
|
||||
|
||||
log = get_logger(__name__)
|
||||
# ?TODO, enable console on import
|
||||
# [ ] necessary? or `open_brokerd_dialog()` doing it is sufficient?
|
||||
#
|
||||
# bc might as well enable whenev imported by
|
||||
# other sub-sys code (namely `.clearing`).
|
||||
get_console_log(
|
||||
level='warning',
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# TODO, the `as <samename>` style?
|
||||
__all__ = [
|
||||
'Account',
|
||||
'Allocator',
|
||||
'Asset',
|
||||
'MktPair',
|
||||
'Position',
|
||||
'Symbol',
|
||||
'Transaction',
|
||||
'TransactionLedger',
|
||||
'dec_digits',
|
||||
|
|
@ -79,6 +70,7 @@ __all__ = [
|
|||
'load_account_from_ledger',
|
||||
'mk_allocator',
|
||||
'open_account',
|
||||
'open_pps',
|
||||
'open_trade_ledger',
|
||||
'unpack_fqme',
|
||||
'DerivTypes',
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ import tomli_w # for fast ledger writing
|
|||
|
||||
from piker.types import Struct
|
||||
from piker import config
|
||||
from piker.log import get_logger
|
||||
from ..log import get_logger
|
||||
from .calc import (
|
||||
iter_by_dt,
|
||||
)
|
||||
|
|
@ -239,9 +239,7 @@ class TransactionLedger(UserDict):
|
|||
|
||||
symcache: SymbologyCache = self._symcache
|
||||
towrite: dict[str, Any] = {}
|
||||
for tid, txdict in self.tx_sort(
|
||||
self.data.copy()
|
||||
):
|
||||
for tid, txdict in self.tx_sort(self.data.copy()):
|
||||
# write blank-str expiry for non-expiring assets
|
||||
if (
|
||||
'expiry' in txdict
|
||||
|
|
@ -379,7 +377,7 @@ def open_trade_ledger(
|
|||
account,
|
||||
dirpath=_fp,
|
||||
)
|
||||
cpy: dict = ledger_dict.copy()
|
||||
cpy = ledger_dict.copy()
|
||||
|
||||
# XXX NOTE: if not provided presume we are being called from
|
||||
# sync code and need to maybe run `trio` to generate..
|
||||
|
|
@ -408,13 +406,7 @@ def open_trade_ledger(
|
|||
account=account,
|
||||
mod=mod,
|
||||
symcache=symcache,
|
||||
|
||||
# NOTE: allow backends to provide custom ledger sorting
|
||||
tx_sort=getattr(
|
||||
mod,
|
||||
'tx_sort',
|
||||
tx_sort,
|
||||
),
|
||||
tx_sort=getattr(mod, 'tx_sort', tx_sort),
|
||||
)
|
||||
try:
|
||||
yield ledger
|
||||
|
|
|
|||
|
|
@ -305,8 +305,8 @@ class MktPair(Struct, frozen=True):
|
|||
# config right?
|
||||
# src_type: AssetTypeName
|
||||
|
||||
# for derivs, info describing contract, egs. strike price, call
|
||||
# or put, swap type, exercise model, etc.
|
||||
# for derivs, info describing contract, egs.
|
||||
# strike price, call or put, swap type, exercise model, etc.
|
||||
contract_info: list[str] | None = None
|
||||
|
||||
# TODO: rename to sectype since all of these can
|
||||
|
|
@ -390,8 +390,8 @@ class MktPair(Struct, frozen=True):
|
|||
cls,
|
||||
fqme: str,
|
||||
|
||||
price_tick: float|str,
|
||||
size_tick: float|str,
|
||||
price_tick: float | str,
|
||||
size_tick: float | str,
|
||||
bs_mktid: str,
|
||||
|
||||
broker: str | None = None,
|
||||
|
|
@ -677,3 +677,90 @@ def unpack_fqme(
|
|||
# '.'.join([mkt_ep, venue]),
|
||||
suffix,
|
||||
)
|
||||
|
||||
|
||||
class Symbol(Struct):
|
||||
'''
|
||||
I guess this is some kinda container thing for dealing with
|
||||
all the different meta-data formats from brokers?
|
||||
|
||||
'''
|
||||
key: str
|
||||
|
||||
broker: str = ''
|
||||
venue: str = ''
|
||||
|
||||
# precision descriptors for price and vlm
|
||||
tick_size: Decimal = Decimal('0.01')
|
||||
lot_tick_size: Decimal = Decimal('0.0')
|
||||
|
||||
suffix: str = ''
|
||||
broker_info: dict[str, dict[str, Any]] = {}
|
||||
|
||||
@classmethod
|
||||
def from_fqme(
|
||||
cls,
|
||||
fqsn: str,
|
||||
info: dict[str, Any],
|
||||
|
||||
) -> Symbol:
|
||||
broker, mktep, venue, suffix = unpack_fqme(fqsn)
|
||||
tick_size = info.get('price_tick_size', 0.01)
|
||||
lot_size = info.get('lot_tick_size', 0.0)
|
||||
|
||||
return Symbol(
|
||||
broker=broker,
|
||||
key=mktep,
|
||||
tick_size=tick_size,
|
||||
lot_tick_size=lot_size,
|
||||
venue=venue,
|
||||
suffix=suffix,
|
||||
broker_info={broker: info},
|
||||
)
|
||||
|
||||
@property
|
||||
def type_key(self) -> str:
|
||||
return list(self.broker_info.values())[0]['asset_type']
|
||||
|
||||
@property
|
||||
def tick_size_digits(self) -> int:
|
||||
return float_digits(self.tick_size)
|
||||
|
||||
@property
|
||||
def lot_size_digits(self) -> int:
|
||||
return float_digits(self.lot_tick_size)
|
||||
|
||||
@property
|
||||
def price_tick(self) -> Decimal:
|
||||
return Decimal(str(self.tick_size))
|
||||
|
||||
@property
|
||||
def size_tick(self) -> Decimal:
|
||||
return Decimal(str(self.lot_tick_size))
|
||||
|
||||
@property
|
||||
def broker(self) -> str:
|
||||
return list(self.broker_info.keys())[0]
|
||||
|
||||
@property
|
||||
def fqme(self) -> str:
|
||||
return maybe_cons_tokens([
|
||||
self.key, # final "pair name" (eg. qqq[/usd], btcusdt)
|
||||
self.venue,
|
||||
self.suffix, # includes expiry and other con info
|
||||
self.broker,
|
||||
])
|
||||
|
||||
def quantize(
|
||||
self,
|
||||
size: float,
|
||||
) -> Decimal:
|
||||
digits = float_digits(self.lot_tick_size)
|
||||
return Decimal(size).quantize(
|
||||
Decimal(f'1.{"0".ljust(digits, "0")}'),
|
||||
rounding=ROUND_HALF_EVEN
|
||||
)
|
||||
|
||||
# NOTE: when cast to `str` return fqme
|
||||
def __str__(self) -> str:
|
||||
return self.fqme
|
||||
|
|
|
|||
|
|
@ -30,8 +30,7 @@ from types import ModuleType
|
|||
from typing import (
|
||||
Any,
|
||||
Iterator,
|
||||
Generator,
|
||||
TYPE_CHECKING,
|
||||
Generator
|
||||
)
|
||||
|
||||
import pendulum
|
||||
|
|
@ -60,16 +59,10 @@ from ..clearing._messages import (
|
|||
BrokerdPosition,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
from piker.data._symcache import SymbologyCache
|
||||
from ..log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.data._symcache import SymbologyCache
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
class Position(Struct):
|
||||
|
|
@ -360,20 +353,17 @@ class Position(Struct):
|
|||
) -> bool:
|
||||
'''
|
||||
Update clearing table by calculating the rolling ppu and
|
||||
(accumulative) size in both the clears entry and local attrs
|
||||
state.
|
||||
(accumulative) size in both the clears entry and local
|
||||
attrs state.
|
||||
|
||||
Inserts are always done in datetime sorted order.
|
||||
|
||||
'''
|
||||
# added: bool = False
|
||||
tid: str = t.tid
|
||||
if tid in self._events:
|
||||
log.debug(
|
||||
f'Txn is already added?\n'
|
||||
f'\n'
|
||||
f'{t}\n'
|
||||
)
|
||||
return False
|
||||
log.warning(f'{t} is already added?!')
|
||||
# return added
|
||||
|
||||
# TODO: apparently this IS possible with a dict but not
|
||||
# common and probably not that beneficial unless we're also
|
||||
|
|
@ -454,12 +444,6 @@ class Position(Struct):
|
|||
# def suggest_split(self) -> float:
|
||||
# ...
|
||||
|
||||
# ?TODO, for sending rendered state over the wire?
|
||||
# def summary(self) -> PositionSummary:
|
||||
# do minimal conversion to a subset of fields
|
||||
# currently defined in `.clearing._messages.BrokerdPosition`
|
||||
|
||||
|
||||
|
||||
class Account(Struct):
|
||||
'''
|
||||
|
|
@ -503,23 +487,12 @@ class Account(Struct):
|
|||
|
||||
def update_from_ledger(
|
||||
self,
|
||||
ledger: TransactionLedger|dict[str, Transaction],
|
||||
ledger: TransactionLedger | dict[str, Transaction],
|
||||
cost_scalar: float = 2,
|
||||
symcache: SymbologyCache|None = None,
|
||||
symcache: SymbologyCache | None = None,
|
||||
|
||||
_mktmap_table: dict[str, MktPair] | None = None,
|
||||
|
||||
only_require: list[str]|True = True,
|
||||
# ^list of fqmes that are "required" to be processed from
|
||||
# this ledger pass; we often don't care about others and
|
||||
# definitely shouldn't always error in such cases.
|
||||
# (eg. broker backend loaded that doesn't yet supsport the
|
||||
# symcache but also, inside the paper engine we don't ad-hoc
|
||||
# request `get_mkt_info()` for every symbol in the ledger,
|
||||
# only the one for which we're simulating against).
|
||||
# TODO, not sure if there's a better soln for this, ideally
|
||||
# all backends get symcache support afap i guess..
|
||||
|
||||
) -> dict[str, Position]:
|
||||
'''
|
||||
Update the internal `.pps[str, Position]` table from input
|
||||
|
|
@ -562,32 +535,11 @@ class Account(Struct):
|
|||
if _mktmap_table is None:
|
||||
raise
|
||||
|
||||
required: bool = (
|
||||
only_require is True
|
||||
or (
|
||||
only_require is not True
|
||||
and
|
||||
fqme in only_require
|
||||
)
|
||||
)
|
||||
# XXX: caller is allowed to provide a fallback
|
||||
# mktmap table for the case where a new position is
|
||||
# being added and the preloaded symcache didn't
|
||||
# have this entry prior (eg. with frickin IB..)
|
||||
if (
|
||||
not (mkt := _mktmap_table.get(fqme))
|
||||
and
|
||||
required
|
||||
):
|
||||
raise
|
||||
|
||||
elif not required:
|
||||
continue
|
||||
|
||||
else:
|
||||
# should be an entry retreived somewhere
|
||||
assert mkt
|
||||
|
||||
mkt = _mktmap_table[fqme]
|
||||
|
||||
if not (pos := pps.get(bs_mktid)):
|
||||
|
||||
|
|
@ -704,7 +656,7 @@ class Account(Struct):
|
|||
def write_config(self) -> None:
|
||||
'''
|
||||
Write the current account state to the user's account TOML file, normally
|
||||
something like `pps.toml`.
|
||||
something like ``pps.toml``.
|
||||
|
||||
'''
|
||||
# TODO: show diff output?
|
||||
|
|
@ -744,7 +696,7 @@ class Account(Struct):
|
|||
else:
|
||||
# TODO: we reallly need a diff set of
|
||||
# loglevels/colors per subsys.
|
||||
log.debug(
|
||||
log.warning(
|
||||
f'Recent position for {fqme} was closed!'
|
||||
)
|
||||
|
||||
|
|
@ -758,7 +710,7 @@ class Account(Struct):
|
|||
# XXX WTF: if we use a tomlkit.Integer here we get this
|
||||
# super weird --1 thing going on for cumsize!?1!
|
||||
# NOTE: the fix was to always float() the size value loaded
|
||||
# in open_account() below!
|
||||
# in open_pps() below!
|
||||
config.write(
|
||||
config=self.conf,
|
||||
path=self.conf_path,
|
||||
|
|
@ -942,6 +894,7 @@ def open_account(
|
|||
clears_table['dt'] = dt
|
||||
trans.append(Transaction(
|
||||
fqme=bs_mktid,
|
||||
# sym=mkt,
|
||||
bs_mktid=bs_mktid,
|
||||
tid=tid,
|
||||
# XXX: not sure why sometimes these are loaded as
|
||||
|
|
@ -964,22 +917,11 @@ def open_account(
|
|||
):
|
||||
expiry: pendulum.DateTime = pendulum.parse(expiry)
|
||||
|
||||
# !XXX, should never be duplicates over
|
||||
# a backend-(broker)-system's unique market-IDs!
|
||||
if pos := pp_objs.get(bs_mktid):
|
||||
if mkt != pos.mkt:
|
||||
log.warning(
|
||||
f'Duplicated position but diff `MktPair.fqme` ??\n'
|
||||
f'bs_mktid: {bs_mktid!r}\n'
|
||||
f'pos.mkt: {pos.mkt}\n'
|
||||
f'mkt: {mkt}\n'
|
||||
)
|
||||
else:
|
||||
pos = pp_objs[bs_mktid] = Position(
|
||||
mkt,
|
||||
split_ratio=split_ratio,
|
||||
bs_mktid=bs_mktid,
|
||||
)
|
||||
pp = pp_objs[bs_mktid] = Position(
|
||||
mkt,
|
||||
split_ratio=split_ratio,
|
||||
bs_mktid=bs_mktid,
|
||||
)
|
||||
|
||||
# XXX: super critical, we need to be sure to include
|
||||
# all pps.toml clears to avoid reusing clears that were
|
||||
|
|
@ -987,13 +929,8 @@ def open_account(
|
|||
# state, since today's records may have already been
|
||||
# processed!
|
||||
for t in trans:
|
||||
added: bool = pos.add_clear(t)
|
||||
if not added:
|
||||
log.warning(
|
||||
f'Txn already recorded in pp ??\n'
|
||||
f'\n'
|
||||
f'{t}\n'
|
||||
)
|
||||
pp.add_clear(t)
|
||||
|
||||
try:
|
||||
yield acnt
|
||||
finally:
|
||||
|
|
@ -1001,6 +938,20 @@ def open_account(
|
|||
acnt.write_config()
|
||||
|
||||
|
||||
# TODO: drop the old name and THIS!
|
||||
@cm
|
||||
def open_pps(
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> Generator[Account, None, None]:
|
||||
log.warning(
|
||||
'`open_pps()` is now deprecated!\n'
|
||||
'Please use `with open_account() as cnt:`'
|
||||
)
|
||||
with open_account(*args, **kwargs) as acnt:
|
||||
yield acnt
|
||||
|
||||
|
||||
def load_account_from_ledger(
|
||||
|
||||
brokername: str,
|
||||
|
|
|
|||
|
|
@ -22,9 +22,7 @@ you know when you're losing money (if possible) XD
|
|||
from __future__ import annotations
|
||||
from collections.abc import ValuesView
|
||||
from contextlib import contextmanager as cm
|
||||
from functools import partial
|
||||
from math import copysign
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
|
|
@ -32,7 +30,6 @@ from typing import (
|
|||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
from tractor.devx import maybe_open_crash_handler
|
||||
import polars as pl
|
||||
from pendulum import (
|
||||
DateTime,
|
||||
|
|
@ -40,16 +37,12 @@ from pendulum import (
|
|||
parse,
|
||||
)
|
||||
|
||||
from ..log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._ledger import (
|
||||
Transaction,
|
||||
TransactionLedger,
|
||||
)
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def ppu(
|
||||
clears: Iterator[Transaction],
|
||||
|
|
@ -245,9 +238,6 @@ def iter_by_dt(
|
|||
|
||||
def dyn_parse_to_dt(
|
||||
tx: tuple[str, dict[str, Any]] | Transaction,
|
||||
|
||||
debug: bool = False,
|
||||
_invalid: list|None = None,
|
||||
) -> DateTime:
|
||||
|
||||
# handle `.items()` inputs
|
||||
|
|
@ -260,90 +250,33 @@ def iter_by_dt(
|
|||
# get best parser for this record..
|
||||
for k in parsers:
|
||||
if (
|
||||
(v := getattr(tx, k, None))
|
||||
or
|
||||
(
|
||||
isdict
|
||||
and
|
||||
(v := tx.get(k))
|
||||
)
|
||||
isdict and k in tx
|
||||
or getattr(tx, k, None)
|
||||
):
|
||||
v = tx[k] if isdict else tx.dt
|
||||
assert v is not None, f'No valid value for `{k}`!?'
|
||||
|
||||
# only call parser on the value if not None from
|
||||
# the `parsers` table above (when NOT using
|
||||
# `.get()`), otherwise pass through the value and
|
||||
# sort on it directly
|
||||
if (
|
||||
not isinstance(v, DateTime)
|
||||
and
|
||||
(parser := parsers.get(k))
|
||||
and (parser := parsers.get(k))
|
||||
):
|
||||
ret = parser(v)
|
||||
return parser(v)
|
||||
else:
|
||||
ret = v
|
||||
return v
|
||||
|
||||
return ret
|
||||
|
||||
else:
|
||||
log.debug(
|
||||
f'Parser-field not found in txn\n'
|
||||
f'\n'
|
||||
f'parser-field: {k!r}\n'
|
||||
f'txn: {tx!r}\n'
|
||||
f'\n'
|
||||
f'Trying next..\n'
|
||||
)
|
||||
continue
|
||||
|
||||
# XXX: we should never really get here bc it means some kinda
|
||||
# bad txn-record (field) data..
|
||||
#
|
||||
# -> set the `debug_mode = True` if you want to trace such
|
||||
# cases from REPL ;)
|
||||
else:
|
||||
# XXX: we should really never get here..
|
||||
# only if a ledger record has no expected sort(able)
|
||||
# field will we likely hit this.. like with ze IB.
|
||||
# if no sortable field just deliver epoch?
|
||||
log.warning(
|
||||
'No (time) sortable field for TXN:\n'
|
||||
f'{tx!r}\n'
|
||||
)
|
||||
report: str = (
|
||||
f'No supported time-field found in txn !?\n'
|
||||
f'\n'
|
||||
f'supported-time-fields: {parsers!r}\n'
|
||||
f'\n'
|
||||
f'txn: {tx!r}\n'
|
||||
)
|
||||
if debug:
|
||||
with maybe_open_crash_handler(
|
||||
pdb=debug,
|
||||
raise_on_exit=False,
|
||||
):
|
||||
raise ValueError(report)
|
||||
else:
|
||||
log.error(report)
|
||||
# XXX: should never get here..
|
||||
breakpoint()
|
||||
|
||||
if _invalid is not None:
|
||||
_invalid.append(tx)
|
||||
return from_timestamp(0.)
|
||||
|
||||
entry: tuple[str, dict]|Transaction
|
||||
invalid: list = []
|
||||
entry: tuple[str, dict] | Transaction
|
||||
for entry in sorted(
|
||||
records,
|
||||
key=key or partial(
|
||||
dyn_parse_to_dt,
|
||||
_invalid=invalid,
|
||||
),
|
||||
key=key or dyn_parse_to_dt,
|
||||
):
|
||||
if entry in invalid:
|
||||
log.warning(
|
||||
f'Ignoring txn w invalid timestamp ??\n'
|
||||
f'{pformat(entry)}\n'
|
||||
)
|
||||
continue
|
||||
|
||||
# NOTE the type sig above; either pairs or txns B)
|
||||
yield entry
|
||||
|
||||
|
|
@ -406,7 +339,6 @@ def open_ledger_dfs(
|
|||
acctname: str,
|
||||
|
||||
ledger: TransactionLedger | None = None,
|
||||
debug_mode: bool = False,
|
||||
|
||||
**kwargs,
|
||||
|
||||
|
|
@ -421,10 +353,8 @@ def open_ledger_dfs(
|
|||
can update the ledger on exit.
|
||||
|
||||
'''
|
||||
with maybe_open_crash_handler(
|
||||
pdb=debug_mode,
|
||||
# raise_on_exit=False,
|
||||
):
|
||||
from piker.toolz import open_crash_handler
|
||||
with open_crash_handler():
|
||||
if not ledger:
|
||||
import time
|
||||
from ._ledger import open_trade_ledger
|
||||
|
|
@ -516,7 +446,7 @@ def ledger_to_dfs(
|
|||
|
||||
df = dfs[key] = ldf.with_columns([
|
||||
|
||||
pl.cum_sum('size').alias('cumsize'),
|
||||
pl.cumsum('size').alias('cumsize'),
|
||||
|
||||
# amount of source asset "sent" (via buy txns in
|
||||
# the market) to acquire the dst asset, PER txn.
|
||||
|
|
@ -531,7 +461,7 @@ def ledger_to_dfs(
|
|||
]).with_columns([
|
||||
|
||||
# rolling balance in src asset units
|
||||
(pl.col('dst_bot').cum_sum() * -1).alias('src_balance'),
|
||||
(pl.col('dst_bot').cumsum() * -1).alias('src_balance'),
|
||||
|
||||
# "position operation type" in terms of increasing the
|
||||
# amount in the dst asset (entering) or decreasing the
|
||||
|
|
@ -673,7 +603,7 @@ def ledger_to_dfs(
|
|||
# cost that was included in the least-recently
|
||||
# entered txn that is still part of the current CSi
|
||||
# set.
|
||||
# => we look up the cost-per-unit cum_sum and apply
|
||||
# => we look up the cost-per-unit cumsum and apply
|
||||
# if over the current txn size (by multiplication)
|
||||
# and then reverse that previusly applied cost on
|
||||
# the txn_cost for this record.
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ CLI front end for trades ledger and position tracking management.
|
|||
from __future__ import annotations
|
||||
from pprint import pformat
|
||||
|
||||
|
||||
from rich.console import Console
|
||||
from rich.markdown import Markdown
|
||||
import polars as pl
|
||||
|
|
@ -28,10 +29,7 @@ import tractor
|
|||
import trio
|
||||
import typer
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ..log import get_logger
|
||||
from ..service import (
|
||||
open_piker_runtime,
|
||||
)
|
||||
|
|
@ -47,7 +45,6 @@ from .calc import (
|
|||
open_ledger_dfs,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
ledger = typer.Typer()
|
||||
|
||||
|
|
@ -82,10 +79,7 @@ def sync(
|
|||
"-l",
|
||||
),
|
||||
):
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
log = get_logger(loglevel)
|
||||
console = Console()
|
||||
|
||||
pair: tuple[str, str]
|
||||
|
|
@ -306,8 +300,7 @@ def disect(
|
|||
assert not df.is_empty()
|
||||
|
||||
# muck around in pdbp REPL
|
||||
# tractor.devx.mk_pdb().set_trace()
|
||||
# breakpoint()
|
||||
breakpoint()
|
||||
|
||||
# TODO: we REALLY need a better console REPL for this
|
||||
# kinda thing..
|
||||
|
|
|
|||
|
|
@ -25,16 +25,15 @@ from types import ModuleType
|
|||
|
||||
from tractor.trionics import maybe_open_context
|
||||
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
BrokerError,
|
||||
SymbolNotFound,
|
||||
NoData,
|
||||
DataUnavailable,
|
||||
DataThrottle,
|
||||
resproc,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
__all__: list[str] = [
|
||||
|
|
@ -44,6 +43,7 @@ __all__: list[str] = [
|
|||
'DataUnavailable',
|
||||
'DataThrottle',
|
||||
'resproc',
|
||||
'get_logger',
|
||||
]
|
||||
|
||||
__brokers__: list[str] = [
|
||||
|
|
@ -65,10 +65,6 @@ __brokers__: list[str] = [
|
|||
# bitso
|
||||
]
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
|
||||
def get_brokermod(brokername: str) -> ModuleType:
|
||||
'''
|
||||
|
|
@ -102,14 +98,13 @@ async def open_cached_client(
|
|||
If one has not been setup do it and cache it.
|
||||
|
||||
'''
|
||||
brokermod: ModuleType = get_brokermod(brokername)
|
||||
|
||||
# TODO: make abstract or `typing.Protocol`
|
||||
# client: Client
|
||||
brokermod = get_brokermod(brokername)
|
||||
async with maybe_open_context(
|
||||
acm_func=brokermod.get_client,
|
||||
kwargs=kwargs,
|
||||
|
||||
) as (cache_hit, client):
|
||||
|
||||
if cache_hit:
|
||||
log.runtime(f'Reusing existing {client}')
|
||||
|
||||
|
|
|
|||
|
|
@ -33,18 +33,12 @@ import exceptiongroup as eg
|
|||
import tractor
|
||||
import trio
|
||||
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from . import _util
|
||||
from . import get_brokermod
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..data import _FeedsBus
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
# `brokerd` enabled modules
|
||||
# TODO: move this def to the `.data` subpkg..
|
||||
# NOTE: keeping this list as small as possible is part of our caps-sec
|
||||
|
|
@ -65,7 +59,7 @@ _data_mods: str = [
|
|||
async def _setup_persistent_brokerd(
|
||||
ctx: tractor.Context,
|
||||
brokername: str,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -78,14 +72,13 @@ async def _setup_persistent_brokerd(
|
|||
# since all hosted daemon tasks will reference this same
|
||||
# log instance's (actor local) state and thus don't require
|
||||
# any further (level) configuration on their own B)
|
||||
actor: tractor.Actor = tractor.current_actor()
|
||||
tll: str = actor.loglevel
|
||||
log = get_console_log(
|
||||
level=loglevel or tll,
|
||||
log = _util.get_console_log(
|
||||
loglevel or tractor.current_actor().loglevel,
|
||||
name=f'{_util.subsys}.{brokername}',
|
||||
with_tractor_log=bool(tll),
|
||||
)
|
||||
assert log.name == _util.subsys
|
||||
|
||||
# set global for this actor to this new process-wide instance B)
|
||||
_util.log = log
|
||||
|
||||
# further, set the log level on any broker broker specific
|
||||
# logger instance.
|
||||
|
|
@ -103,10 +96,7 @@ async def _setup_persistent_brokerd(
|
|||
# - `open_symbol_search()`
|
||||
# NOTE: see ep invocation details inside `.data.feed`.
|
||||
try:
|
||||
async with (
|
||||
# tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as service_nursery
|
||||
):
|
||||
async with trio.open_nursery() as service_nursery:
|
||||
bus: _FeedsBus = feed.get_feed_bus(
|
||||
brokername,
|
||||
service_nursery,
|
||||
|
|
@ -200,6 +190,7 @@ def broker_init(
|
|||
|
||||
|
||||
async def spawn_brokerd(
|
||||
|
||||
brokername: str,
|
||||
loglevel: str | None = None,
|
||||
|
||||
|
|
@ -207,10 +198,8 @@ async def spawn_brokerd(
|
|||
|
||||
) -> bool:
|
||||
|
||||
log.info(
|
||||
f'Spawning broker-daemon,\n'
|
||||
f'backend: {brokername!r}'
|
||||
)
|
||||
from piker.service._util import log # use service mngr log
|
||||
log.info(f'Spawning {brokername} broker daemon')
|
||||
|
||||
(
|
||||
brokermode,
|
||||
|
|
@ -257,7 +246,7 @@ async def spawn_brokerd(
|
|||
async def maybe_spawn_brokerd(
|
||||
|
||||
brokername: str,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
**pikerd_kwargs,
|
||||
|
||||
|
|
@ -273,7 +262,8 @@ async def maybe_spawn_brokerd(
|
|||
from piker.service import maybe_spawn_daemon
|
||||
|
||||
async with maybe_spawn_daemon(
|
||||
service_name=f'brokerd.{brokername}',
|
||||
|
||||
f'brokerd.{brokername}',
|
||||
service_task_target=spawn_brokerd,
|
||||
spawn_args={
|
||||
'brokername': brokername,
|
||||
|
|
|
|||
|
|
@ -19,13 +19,15 @@ Handy cross-broker utils.
|
|||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
# from functools import partial
|
||||
from functools import partial
|
||||
|
||||
import json
|
||||
import httpx
|
||||
import logging
|
||||
|
||||
from piker.log import (
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
colorize_json,
|
||||
)
|
||||
subsys: str = 'piker.brokers'
|
||||
|
|
@ -33,22 +35,12 @@ subsys: str = 'piker.brokers'
|
|||
# NOTE: level should be reset by any actor that is spawned
|
||||
# as well as given a (more) explicit name/key such
|
||||
# as `piker.brokers.binance` matching the subpkg.
|
||||
# log = get_logger(subsys)
|
||||
log = get_logger(subsys)
|
||||
|
||||
# ?TODO?? we could use this approach, but we need to be able
|
||||
# to pass multiple `name=` values so for example we can include the
|
||||
# emissions in `.accounting._pos` and others!
|
||||
# [ ] maybe we could do the `log = get_logger()` above,
|
||||
# then cycle through the list of subsys mods we depend on
|
||||
# and then get all their loggers and pass them to
|
||||
# `get_console_log(logger=)`??
|
||||
# [ ] OR just write THIS `get_console_log()` as a hook which does
|
||||
# that based on who calls it?.. i dunno
|
||||
#
|
||||
# get_console_log = partial(
|
||||
# get_console_log,
|
||||
# name=subsys,
|
||||
# )
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name=subsys,
|
||||
)
|
||||
|
||||
|
||||
class BrokerError(Exception):
|
||||
|
|
|
|||
|
|
@ -374,14 +374,9 @@ class Client:
|
|||
pair: Pair = pair_type(**item)
|
||||
except Exception as e:
|
||||
e.add_note(
|
||||
f'\n'
|
||||
f'New or removed field we need to codify!\n'
|
||||
f'pair-type: {pair_type!r}\n'
|
||||
f'\n'
|
||||
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||
f'Check out their API docs here:\n'
|
||||
f'\n'
|
||||
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
||||
"\nDon't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||
'Check out their API docs here:\n\n'
|
||||
'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
|
||||
)
|
||||
raise
|
||||
pair_table[pair.symbol.upper()] = pair
|
||||
|
|
@ -572,7 +567,6 @@ class Client:
|
|||
) -> str:
|
||||
return {
|
||||
'USDTM': 'usdtm_futes',
|
||||
'SPOT': 'spot',
|
||||
# 'COINM': 'coin_futes',
|
||||
# ^-TODO-^ bc someone might want it..?
|
||||
}[pair.venue]
|
||||
|
|
|
|||
|
|
@ -37,9 +37,8 @@ import trio
|
|||
from piker.accounting import (
|
||||
Asset,
|
||||
)
|
||||
from piker.log import (
|
||||
from piker.brokers._util import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from piker.data._web_bs import (
|
||||
open_autorecon_ws,
|
||||
|
|
@ -70,9 +69,7 @@ from .venues import (
|
|||
)
|
||||
from .api import Client
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
log = get_logger('piker.brokers.binance')
|
||||
|
||||
|
||||
# Fee schedule template, mostly for paper engine fees modelling.
|
||||
|
|
@ -248,16 +245,9 @@ async def handle_order_requests(
|
|||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
# enable piker.clearing console log for *this* `brokerd` subactor
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# TODO: how do we set this from the EMS such that
|
||||
# positions are loaded from the correct venue on the user
|
||||
# stream at startup? (that is in an attempt to support both
|
||||
|
|
@ -450,7 +440,6 @@ async def open_trade_dialog(
|
|||
# - ledger: TransactionLedger
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
ctx.open_stream() as ems_stream,
|
||||
):
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@ from trio_typing import TaskStatus
|
|||
from pendulum import (
|
||||
from_timestamp,
|
||||
)
|
||||
from rapidfuzz import process as fuzzy
|
||||
import numpy as np
|
||||
import tractor
|
||||
|
||||
|
|
@ -64,9 +65,9 @@ from piker.data._web_bs import (
|
|||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
)
|
||||
from piker.log import get_logger
|
||||
from piker.brokers._util import (
|
||||
DataUnavailable,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
from .api import (
|
||||
|
|
@ -78,7 +79,7 @@ from .venues import (
|
|||
get_api_eps,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
log = get_logger('piker.brokers.binance')
|
||||
|
||||
|
||||
class L1(Struct):
|
||||
|
|
@ -94,26 +95,22 @@ class L1(Struct):
|
|||
|
||||
|
||||
# validation type
|
||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Aggregate-Trade-Streams#response-example
|
||||
class AggTrade(Struct, frozen=True):
|
||||
e: str # Event type
|
||||
E: int # Event time
|
||||
s: str # Symbol
|
||||
a: int # Aggregate trade ID
|
||||
p: float # Price
|
||||
q: float # Quantity with all the market trades
|
||||
q: float # Quantity
|
||||
f: int # First trade ID
|
||||
l: int # noqa Last trade ID
|
||||
T: int # Trade time
|
||||
m: bool # Is the buyer the market maker?
|
||||
M: bool|None = None # Ignore
|
||||
nq: float|None = None # Normal quantity without the trades involving RPI orders
|
||||
# ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||
M: bool | None = None # Ignore
|
||||
|
||||
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
|
||||
) -> AsyncGenerator[NoBsWs, dict]:
|
||||
|
||||
# TODO: match syntax here!
|
||||
|
|
@ -224,8 +221,6 @@ def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
|||
}
|
||||
|
||||
|
||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||
# code?!
|
||||
@acm
|
||||
async def open_history_client(
|
||||
mkt: MktPair,
|
||||
|
|
@ -237,8 +232,8 @@ async def open_history_client(
|
|||
|
||||
async def get_ohlc(
|
||||
timeframe: float,
|
||||
end_dt: datetime|None = None,
|
||||
start_dt: datetime|None = None,
|
||||
end_dt: datetime | None = None,
|
||||
start_dt: datetime | None = None,
|
||||
|
||||
) -> tuple[
|
||||
np.ndarray,
|
||||
|
|
@ -275,15 +270,9 @@ async def open_history_client(
|
|||
f'{times}'
|
||||
)
|
||||
|
||||
# XXX, debug any case where the latest 1m bar we get is
|
||||
# already another "sample's-step-old"..
|
||||
if end_dt is None:
|
||||
inow: int = round(time.time())
|
||||
if (
|
||||
_time_step := (inow - times[-1])
|
||||
>
|
||||
timeframe * 2
|
||||
):
|
||||
if (inow - times[-1]) > 60:
|
||||
await tractor.pause()
|
||||
|
||||
start_dt = from_timestamp(times[0])
|
||||
|
|
@ -297,7 +286,7 @@ async def open_history_client(
|
|||
async def get_mkt_info(
|
||||
fqme: str,
|
||||
|
||||
) -> tuple[MktPair, Pair]|None:
|
||||
) -> tuple[MktPair, Pair] | None:
|
||||
|
||||
# uppercase since kraken bs_mktid is always upper
|
||||
if 'binance' not in fqme.lower():
|
||||
|
|
@ -374,7 +363,7 @@ async def get_mkt_info(
|
|||
if 'futes' in mkt_mode:
|
||||
assert isinstance(pair, FutesPair)
|
||||
|
||||
dst: Asset|None = assets.get(pair.bs_dst_asset)
|
||||
dst: Asset | None = assets.get(pair.bs_dst_asset)
|
||||
if (
|
||||
not dst
|
||||
# TODO: a known asset DNE list?
|
||||
|
|
@ -433,7 +422,7 @@ async def subscribe(
|
|||
# might get ack from ws server, or maybe some
|
||||
# other msg still in transit..
|
||||
res = await ws.recv_msg()
|
||||
subid: str|None = res.get('id')
|
||||
subid: str | None = res.get('id')
|
||||
if subid:
|
||||
assert res['id'] == subid
|
||||
|
||||
|
|
@ -457,6 +446,7 @@ async def subscribe(
|
|||
|
||||
|
||||
async def stream_quotes(
|
||||
|
||||
send_chan: trio.abc.SendChannel,
|
||||
symbols: list[str],
|
||||
feed_is_live: trio.Event,
|
||||
|
|
@ -468,14 +458,11 @@ async def stream_quotes(
|
|||
) -> None:
|
||||
|
||||
async with (
|
||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||
send_chan as send_chan,
|
||||
open_cached_client('binance') as client,
|
||||
):
|
||||
init_msgs: list[FeedInit] = []
|
||||
for sym in symbols:
|
||||
mkt: MktPair
|
||||
pair: Pair
|
||||
mkt, pair = await get_mkt_info(sym)
|
||||
|
||||
# build out init msgs according to latest spec
|
||||
|
|
@ -524,6 +511,7 @@ async def stream_quotes(
|
|||
|
||||
# start streaming
|
||||
async for typ, quote in msg_gen:
|
||||
|
||||
# period = time.time() - last
|
||||
# hz = 1/period if period else float('inf')
|
||||
# if hz > 60:
|
||||
|
|
@ -559,7 +547,7 @@ async def open_symbol_search(
|
|||
)
|
||||
|
||||
# repack in fqme-keyed table
|
||||
byfqme: dict[str, Pair] = {}
|
||||
byfqme: dict[start, Pair] = {}
|
||||
for pair in pairs.values():
|
||||
byfqme[pair.bs_fqme] = pair
|
||||
|
||||
|
|
|
|||
|
|
@ -97,16 +97,6 @@ class Pair(Struct, frozen=True, kw_only=True):
|
|||
baseAsset: str
|
||||
baseAssetPrecision: int
|
||||
|
||||
permissionSets: list[list[str]]
|
||||
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs#2025-08-26
|
||||
# will become non-optional 2025-08-28?
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs#future-changes
|
||||
pegInstructionsAllowed: bool = False
|
||||
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs#2025-12-02
|
||||
opoAllowed: bool = False
|
||||
|
||||
filters: dict[
|
||||
str,
|
||||
str | int | float,
|
||||
|
|
@ -152,11 +142,7 @@ class SpotPair(Pair, frozen=True):
|
|||
defaultSelfTradePreventionMode: str
|
||||
allowedSelfTradePreventionModes: list[str]
|
||||
permissions: list[str]
|
||||
|
||||
# can the paint botz creat liq gaps even easier on this asset?
|
||||
# Bp
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs/faqs/order_amend_keep_priority
|
||||
amendAllowed: bool
|
||||
permissionSets: list[list[str]]
|
||||
|
||||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||
ns_path: str = 'piker.brokers.binance:SpotPair'
|
||||
|
|
@ -195,6 +181,7 @@ class FutesPair(Pair):
|
|||
quoteAsset: str # 'USDT',
|
||||
quotePrecision: int # 8,
|
||||
requiredMarginPercent: float # '5.0000',
|
||||
settlePlan: int # 0,
|
||||
timeInForce: list[str] # ['GTC', 'IOC', 'FOK', 'GTX'],
|
||||
triggerProtect: float # '0.0500',
|
||||
underlyingSubType: list[str] # ['PoW'],
|
||||
|
|
@ -223,10 +210,7 @@ class FutesPair(Pair):
|
|||
assert pair == self.pair # sanity
|
||||
return f'{expiry}'
|
||||
|
||||
case (
|
||||
'PERPETUAL'
|
||||
| 'TRADIFI_PERPETUAL'
|
||||
):
|
||||
case 'PERPETUAL':
|
||||
return 'PERP'
|
||||
|
||||
case '':
|
||||
|
|
@ -255,10 +239,7 @@ class FutesPair(Pair):
|
|||
margin: str = self.marginAsset
|
||||
|
||||
match ctype:
|
||||
case (
|
||||
'PERPETUAL'
|
||||
| 'TRADIFI_PERPETUAL'
|
||||
):
|
||||
case 'PERPETUAL':
|
||||
return f'{margin}M'
|
||||
|
||||
case (
|
||||
|
|
|
|||
|
|
@ -27,12 +27,14 @@ import click
|
|||
import trio
|
||||
import tractor
|
||||
|
||||
from piker.cli import cli
|
||||
from piker import watchlists as wl
|
||||
from piker.log import (
|
||||
from ..cli import cli
|
||||
from .. import watchlists as wl
|
||||
from ..log import (
|
||||
colorize_json,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ..service import (
|
||||
maybe_spawn_brokerd,
|
||||
|
|
@ -43,15 +45,12 @@ from ..brokers import (
|
|||
get_brokermod,
|
||||
data,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
DEFAULT_BROKER = 'binance'
|
||||
|
||||
_config_dir = click.get_app_dir('piker')
|
||||
_watchlists_data_path = os.path.join(_config_dir, 'watchlists.json')
|
||||
|
||||
|
||||
OK = '\033[92m'
|
||||
WARNING = '\033[93m'
|
||||
FAIL = '\033[91m'
|
||||
|
|
@ -346,10 +345,7 @@ def contracts(ctx, loglevel, broker, symbol, ids):
|
|||
|
||||
'''
|
||||
brokermod = get_brokermod(broker)
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(loglevel)
|
||||
|
||||
contracts = trio.run(partial(core.contracts, brokermod, symbol))
|
||||
if not ids:
|
||||
|
|
@ -475,18 +471,13 @@ def search(
|
|||
|
||||
'''
|
||||
# global opts
|
||||
brokermods: list[ModuleType] = list(config['brokermods'].values())
|
||||
|
||||
# TODO: this is coming from the `search --pdb` NOT from
|
||||
# the `piker --pdb` XD ..
|
||||
# -[ ] pull from the parent click ctx's values..dumdum
|
||||
# assert pdb
|
||||
loglevel: str = config['loglevel']
|
||||
brokermods = list(config['brokermods'].values())
|
||||
|
||||
# define tractor entrypoint
|
||||
async def main(func):
|
||||
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
loglevel=config['loglevel'],
|
||||
debug_mode=pdb,
|
||||
):
|
||||
return await func()
|
||||
|
|
@ -499,7 +490,6 @@ def search(
|
|||
core.symbol_search,
|
||||
brokermods,
|
||||
pattern,
|
||||
loglevel=loglevel,
|
||||
),
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -22,26 +22,20 @@ routines should be primitive data types where possible.
|
|||
"""
|
||||
import inspect
|
||||
from types import ModuleType
|
||||
from typing import (
|
||||
Any,
|
||||
)
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
import trio
|
||||
|
||||
from piker.log import get_logger
|
||||
from ._util import log
|
||||
from . import get_brokermod
|
||||
from ..service import maybe_spawn_brokerd
|
||||
from . import open_cached_client
|
||||
from ..accounting import MktPair
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
async def api(brokername: str, methname: str, **kwargs) -> dict:
|
||||
'''
|
||||
Make (proxy through) a broker API call by name and return its result.
|
||||
|
||||
'''
|
||||
"""Make (proxy through) a broker API call by name and return its result.
|
||||
"""
|
||||
brokermod = get_brokermod(brokername)
|
||||
async with brokermod.get_client() as client:
|
||||
meth = getattr(client, methname, None)
|
||||
|
|
@ -68,14 +62,10 @@ async def api(brokername: str, methname: str, **kwargs) -> dict:
|
|||
|
||||
async def stocks_quote(
|
||||
brokermod: ModuleType,
|
||||
tickers: list[str]
|
||||
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
'''
|
||||
Return a `dict` of snapshot quotes for the provided input
|
||||
`tickers`: a `list` of fqmes.
|
||||
|
||||
'''
|
||||
tickers: List[str]
|
||||
) -> Dict[str, Dict[str, Any]]:
|
||||
"""Return quotes dict for ``tickers``.
|
||||
"""
|
||||
async with brokermod.get_client() as client:
|
||||
return await client.quote(tickers)
|
||||
|
||||
|
|
@ -84,15 +74,13 @@ async def stocks_quote(
|
|||
async def option_chain(
|
||||
brokermod: ModuleType,
|
||||
symbol: str,
|
||||
date: str|None = None,
|
||||
) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
'''
|
||||
Return option chain for ``symbol`` for ``date``.
|
||||
date: Optional[str] = None,
|
||||
) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
"""Return option chain for ``symbol`` for ``date``.
|
||||
|
||||
By default all expiries are returned. If ``date`` is provided
|
||||
then contract quotes for that single expiry are returned.
|
||||
|
||||
'''
|
||||
"""
|
||||
async with brokermod.get_client() as client:
|
||||
if date:
|
||||
id = int((await client.tickers2ids([symbol]))[symbol])
|
||||
|
|
@ -110,7 +98,7 @@ async def option_chain(
|
|||
# async def contracts(
|
||||
# brokermod: ModuleType,
|
||||
# symbol: str,
|
||||
# ) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
# ) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
# """Return option contracts (all expiries) for ``symbol``.
|
||||
# """
|
||||
# async with brokermod.get_client() as client:
|
||||
|
|
@ -122,24 +110,15 @@ async def bars(
|
|||
brokermod: ModuleType,
|
||||
symbol: str,
|
||||
**kwargs,
|
||||
) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
'''
|
||||
Return option contracts (all expiries) for ``symbol``.
|
||||
|
||||
'''
|
||||
) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
"""Return option contracts (all expiries) for ``symbol``.
|
||||
"""
|
||||
async with brokermod.get_client() as client:
|
||||
return await client.bars(symbol, **kwargs)
|
||||
|
||||
|
||||
async def search_w_brokerd(
|
||||
name: str,
|
||||
pattern: str,
|
||||
) -> dict:
|
||||
async def search_w_brokerd(name: str, pattern: str) -> dict:
|
||||
|
||||
# TODO: WHY NOT WORK!?!
|
||||
# when we `step` through the next block?
|
||||
# import tractor
|
||||
# await tractor.pause()
|
||||
async with open_cached_client(name) as client:
|
||||
|
||||
# TODO: support multiple asset type concurrent searches.
|
||||
|
|
@ -149,15 +128,14 @@ async def search_w_brokerd(
|
|||
async def symbol_search(
|
||||
brokermods: list[ModuleType],
|
||||
pattern: str,
|
||||
loglevel: str = 'warning',
|
||||
**kwargs,
|
||||
|
||||
) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
'''
|
||||
Return symbol info from broker.
|
||||
|
||||
'''
|
||||
results: list[str] = []
|
||||
results = []
|
||||
|
||||
async def search_backend(
|
||||
brokermod: ModuleType
|
||||
|
|
@ -165,13 +143,6 @@ async def symbol_search(
|
|||
|
||||
brokername: str = mod.name
|
||||
|
||||
# TODO: figure this the FUCK OUT
|
||||
# -> ok so obvi in the root actor any async task that's
|
||||
# spawned outside the main tractor-root-actor task needs to
|
||||
# call this..
|
||||
# await tractor.devx._debug.maybe_init_greenback()
|
||||
# tractor.pause_from_sync()
|
||||
|
||||
async with maybe_spawn_brokerd(
|
||||
mod.name,
|
||||
infect_asyncio=getattr(
|
||||
|
|
@ -179,7 +150,6 @@ async def symbol_search(
|
|||
'_infect_asyncio',
|
||||
False,
|
||||
),
|
||||
loglevel=loglevel
|
||||
) as portal:
|
||||
|
||||
results.append((
|
||||
|
|
@ -192,6 +162,7 @@ async def symbol_search(
|
|||
))
|
||||
|
||||
async with trio.open_nursery() as n:
|
||||
|
||||
for mod in brokermods:
|
||||
n.start_soon(search_backend, mod.name)
|
||||
|
||||
|
|
@ -201,13 +172,11 @@ async def symbol_search(
|
|||
async def mkt_info(
|
||||
brokermod: ModuleType,
|
||||
fqme: str,
|
||||
|
||||
**kwargs,
|
||||
|
||||
) -> MktPair:
|
||||
'''
|
||||
Return the `piker.accounting.MktPair` info struct from a given
|
||||
backend broker tradable src/dst asset pair.
|
||||
Return MktPair info from broker including src and dst assets.
|
||||
|
||||
'''
|
||||
async with open_cached_client(brokermod.name) as client:
|
||||
|
|
|
|||
|
|
@ -41,15 +41,12 @@ import tractor
|
|||
from tractor.experimental import msgpub
|
||||
from async_generator import asynccontextmanager
|
||||
|
||||
from piker.log import(
|
||||
get_logger,
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
)
|
||||
from . import get_brokermod
|
||||
|
||||
log = get_logger(
|
||||
name='piker.brokers.binance',
|
||||
)
|
||||
|
||||
async def wait_for_network(
|
||||
net_func: Callable,
|
||||
|
|
@ -246,10 +243,7 @@ async def start_quote_stream(
|
|||
|
||||
'''
|
||||
# XXX: why do we need this again?
|
||||
get_console_log(
|
||||
level=tractor.current_actor().loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(tractor.current_actor().loglevel)
|
||||
|
||||
# pull global vars from local actor
|
||||
symbols = list(symbols)
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ from typing import (
|
|||
Callable,
|
||||
)
|
||||
|
||||
from pendulum import now
|
||||
import pendulum
|
||||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
from rapidfuzz import process as fuzzy
|
||||
|
|
@ -39,7 +39,6 @@ import numpy as np
|
|||
from tractor.trionics import (
|
||||
broadcast_receiver,
|
||||
maybe_open_context
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor import to_asyncio
|
||||
# XXX WOOPS XD
|
||||
|
|
@ -433,7 +432,6 @@ async def get_client(
|
|||
) -> Client:
|
||||
|
||||
async with (
|
||||
collapse_eg(),
|
||||
trio.open_nursery() as n,
|
||||
open_jsonrpc_session(
|
||||
_testnet_ws_url, dtype=JSONRPCResult) as json_rpc
|
||||
|
|
|
|||
|
|
@ -20,11 +20,6 @@ runnable script-programs.
|
|||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
from datetime import ( # noqa
|
||||
datetime,
|
||||
date,
|
||||
tzinfo as TzInfo,
|
||||
)
|
||||
from functools import partial
|
||||
from typing import (
|
||||
Literal,
|
||||
|
|
@ -34,13 +29,13 @@ import subprocess
|
|||
|
||||
import tractor
|
||||
|
||||
from piker.log import get_logger
|
||||
from piker.brokers._util import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .api import Client
|
||||
import i3ipc
|
||||
from ib_insync import IB
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
log = get_logger('piker.brokers.ib')
|
||||
|
||||
_reset_tech: Literal[
|
||||
'vnc',
|
||||
|
|
@ -53,39 +48,8 @@ _reset_tech: Literal[
|
|||
] = 'vnc'
|
||||
|
||||
|
||||
no_setup_msg:str = (
|
||||
'No data reset hack test setup for {vnc_sockaddr}!\n'
|
||||
'See config setup tips @\n'
|
||||
'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
|
||||
)
|
||||
|
||||
|
||||
def try_xdo_manual(
|
||||
client: Client,
|
||||
):
|
||||
'''
|
||||
Do the "manual" `xdo`-based screen switch + click
|
||||
combo since apparently the `asyncvnc` client ain't workin..
|
||||
|
||||
Note this is only meant as a backup method for Xorg users,
|
||||
ideally you can use a real vnc client and the `vnc_click_hack()`
|
||||
impl!
|
||||
|
||||
'''
|
||||
global _reset_tech
|
||||
try:
|
||||
i3ipc_xdotool_manual_click_hack()
|
||||
_reset_tech = 'i3ipc_xdotool'
|
||||
return True
|
||||
except OSError:
|
||||
vnc_sockaddr: str = client.conf.vnc_addrs
|
||||
log.exception(
|
||||
no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
async def data_reset_hack(
|
||||
# vnc_host: str,
|
||||
client: Client,
|
||||
reset_type: Literal['data', 'connection'],
|
||||
|
||||
|
|
@ -117,60 +81,65 @@ async def data_reset_hack(
|
|||
that need to be wrangle.
|
||||
|
||||
'''
|
||||
ib_client: IB = client.ib
|
||||
|
||||
# look up any user defined vnc socket address mapped from
|
||||
# a particular API socket port.
|
||||
vnc_addrs: tuple[str]|None = client.conf.get('vnc_addrs')
|
||||
if not vnc_addrs:
|
||||
api_port: str = str(ib_client.client.port)
|
||||
vnc_host: str
|
||||
vnc_port: int
|
||||
vnc_sockaddr: tuple[str] | None = client.conf.get('vnc_addrs')
|
||||
|
||||
no_setup_msg:str = (
|
||||
f'No data reset hack test setup for {vnc_sockaddr}!\n'
|
||||
'See config setup tips @\n'
|
||||
'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
|
||||
)
|
||||
|
||||
if not vnc_sockaddr:
|
||||
log.warning(
|
||||
no_setup_msg.format(vnc_sockaddr=client.conf)
|
||||
no_setup_msg
|
||||
+
|
||||
'REQUIRES A `vnc_addrs: array` ENTRY'
|
||||
)
|
||||
|
||||
vnc_host, vnc_port = vnc_sockaddr.get(
|
||||
api_port,
|
||||
('localhost', 3003)
|
||||
)
|
||||
global _reset_tech
|
||||
|
||||
match _reset_tech:
|
||||
case 'vnc':
|
||||
try:
|
||||
await tractor.to_asyncio.run_task(
|
||||
partial(
|
||||
vnc_click_hack,
|
||||
client=client,
|
||||
host=vnc_host,
|
||||
port=vnc_port,
|
||||
)
|
||||
)
|
||||
except (
|
||||
OSError, # no VNC server avail..
|
||||
PermissionError, # asyncvnc pw fail..
|
||||
):
|
||||
except OSError:
|
||||
if vnc_host != 'localhost':
|
||||
log.warning(no_setup_msg)
|
||||
return False
|
||||
|
||||
try:
|
||||
import i3ipc # noqa (since a deps dynamic check)
|
||||
except ModuleNotFoundError:
|
||||
log.warning(
|
||||
no_setup_msg.format(vnc_sockaddr=client.conf)
|
||||
)
|
||||
log.warning(no_setup_msg)
|
||||
return False
|
||||
|
||||
# XXX, Xorg only workaround..
|
||||
# TODO? remove now that we have `pyvnc`?
|
||||
# if vnc_host not in {
|
||||
# 'localhost',
|
||||
# '127.0.0.1',
|
||||
# }:
|
||||
# focussed, matches = i3ipc_fin_wins_titled()
|
||||
# if not matches:
|
||||
# log.warning(
|
||||
# no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
|
||||
# )
|
||||
# return False
|
||||
# else:
|
||||
# try_xdo_manual(vnc_sockaddr)
|
||||
|
||||
# localhost but no vnc-client or it borked..
|
||||
else:
|
||||
try_xdo_manual(client)
|
||||
try:
|
||||
i3ipc_xdotool_manual_click_hack()
|
||||
_reset_tech = 'i3ipc_xdotool'
|
||||
return True
|
||||
except OSError:
|
||||
log.exception(no_setup_msg)
|
||||
return False
|
||||
|
||||
case 'i3ipc_xdotool':
|
||||
try_xdo_manual(client)
|
||||
# i3ipc_xdotool_manual_click_hack()
|
||||
i3ipc_xdotool_manual_click_hack()
|
||||
|
||||
case _ as tech:
|
||||
raise RuntimeError(f'{tech} is not supported for reset tech!?')
|
||||
|
|
@ -180,66 +149,21 @@ async def data_reset_hack(
|
|||
|
||||
|
||||
async def vnc_click_hack(
|
||||
client: Client,
|
||||
reset_type: str = 'data',
|
||||
pw: str|None = None,
|
||||
|
||||
host: str,
|
||||
port: int,
|
||||
reset_type: str = 'data'
|
||||
) -> None:
|
||||
'''
|
||||
Reset the data or network connection for the VNC attached
|
||||
ib-gateway using a (magic) keybinding combo.
|
||||
|
||||
A vnc-server password can be set either by an input `pw` param or
|
||||
set in the client's config with the latter loaded from the user's
|
||||
`brokers.toml` in a vnc-addrs-port-mapping section,
|
||||
|
||||
.. code:: toml
|
||||
|
||||
[ib.vnc_addrs]
|
||||
4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
|
||||
ib gateway using magic combos.
|
||||
|
||||
'''
|
||||
api_port: str = str(client.ib.client.port)
|
||||
conf: dict = client.conf
|
||||
vnc_addrs: dict[int, tuple] = conf.get('vnc_addrs')
|
||||
if not vnc_addrs:
|
||||
return None
|
||||
|
||||
addr_entry: dict|tuple = vnc_addrs.get(
|
||||
api_port,
|
||||
('localhost', 5900) # a typical default
|
||||
)
|
||||
if pw is None:
|
||||
match addr_entry:
|
||||
case (
|
||||
host,
|
||||
port,
|
||||
):
|
||||
pass
|
||||
|
||||
case {
|
||||
'host': host,
|
||||
'port': port,
|
||||
'pw': pw
|
||||
}:
|
||||
pass
|
||||
|
||||
case _:
|
||||
raise ValueError(
|
||||
f'Invalid `ib.vnc_addrs` entry ?\n'
|
||||
f'{addr_entry!r}\n'
|
||||
)
|
||||
try:
|
||||
from pyvnc import (
|
||||
AsyncVNCClient,
|
||||
VNCConfig,
|
||||
Point,
|
||||
MOUSE_BUTTON_LEFT,
|
||||
)
|
||||
import asyncvnc
|
||||
except ModuleNotFoundError:
|
||||
log.warning(
|
||||
"In order to leverage `piker`'s built-in data reset hacks, install "
|
||||
"the `pyvnc` project: https://github.com/regulad/pyvnc.git"
|
||||
"the `asyncvnc` project: https://github.com/barneygale/asyncvnc"
|
||||
)
|
||||
return
|
||||
|
||||
|
|
@ -250,80 +174,24 @@ async def vnc_click_hack(
|
|||
'connection': 'r'
|
||||
}[reset_type]
|
||||
|
||||
with tractor.devx.open_crash_handler(
|
||||
ignore={TimeoutError,},
|
||||
):
|
||||
client = await AsyncVNCClient.connect(
|
||||
VNCConfig(
|
||||
host=host,
|
||||
port=port,
|
||||
password=pw,
|
||||
)
|
||||
async with asyncvnc.connect(
|
||||
host,
|
||||
port=port,
|
||||
|
||||
# TODO: doesn't work see:
|
||||
# https://github.com/barneygale/asyncvnc/issues/7
|
||||
# password='ibcansmbz',
|
||||
|
||||
) as client:
|
||||
|
||||
# move to middle of screen
|
||||
# 640x1800
|
||||
client.mouse.move(
|
||||
x=500,
|
||||
y=500,
|
||||
)
|
||||
async with client:
|
||||
# move to middle of screen
|
||||
# 640x1800
|
||||
await client.move(
|
||||
Point(
|
||||
500,
|
||||
500,
|
||||
)
|
||||
)
|
||||
# ensure the ib-gw window is active
|
||||
await client.click(MOUSE_BUTTON_LEFT)
|
||||
# send the hotkeys combo B)
|
||||
await client.press('Ctrl', 'Alt', key) # keys are stacked
|
||||
|
||||
|
||||
def i3ipc_fin_wins_titled(
|
||||
titles: list[str] = [
|
||||
'Interactive Brokers', # tws running in i3
|
||||
'IB Gateway', # gw running in i3
|
||||
# 'IB', # gw running in i3 (newer version?)
|
||||
|
||||
# !TODO, remote vnc instance
|
||||
# -[ ] something in title (or other Con-props) that indicates
|
||||
# this is explicitly for ibrk sw?
|
||||
# |_[ ] !can use modden spawn eventually!
|
||||
'TigerVNC',
|
||||
# 'vncviewer', # the terminal..
|
||||
],
|
||||
) -> tuple[
|
||||
i3ipc.Con, # orig focussed win
|
||||
list[tuple[str, i3ipc.Con]], # matching wins by title
|
||||
]:
|
||||
'''
|
||||
Attempt to find a local-DE window titled with an entry in
|
||||
`titles`.
|
||||
|
||||
If found deliver the current focussed window and all matching
|
||||
`i3ipc.Con`s in a list.
|
||||
|
||||
'''
|
||||
import i3ipc
|
||||
ipc = i3ipc.Connection()
|
||||
|
||||
# TODO: might be worth offering some kinda api for grabbing
|
||||
# the window id from the pid?
|
||||
# https://stackoverflow.com/a/2250879
|
||||
tree = ipc.get_tree()
|
||||
focussed: i3ipc.Con = tree.find_focused()
|
||||
|
||||
matches: list[i3ipc.Con] = []
|
||||
for name in titles:
|
||||
results = tree.find_titled(name)
|
||||
print(f'results for {name}: {results}')
|
||||
if results:
|
||||
con = results[0]
|
||||
matches.append((
|
||||
name,
|
||||
con,
|
||||
))
|
||||
|
||||
return (
|
||||
focussed,
|
||||
matches,
|
||||
)
|
||||
client.mouse.click()
|
||||
client.keyboard.press('Ctrl', 'Alt', key) # keys are stacked
|
||||
|
||||
|
||||
def i3ipc_xdotool_manual_click_hack() -> None:
|
||||
|
|
@ -331,53 +199,65 @@ def i3ipc_xdotool_manual_click_hack() -> None:
|
|||
Do the data reset hack but expecting a local X-window using `xdotool`.
|
||||
|
||||
'''
|
||||
focussed, matches = i3ipc_fin_wins_titled()
|
||||
try:
|
||||
orig_win_id = focussed.window
|
||||
except AttributeError:
|
||||
# XXX if .window cucks we prolly aren't intending to
|
||||
# use this and/or just woke up from suspend..
|
||||
log.exception('xdotool invalid usage ya ??\n')
|
||||
return
|
||||
import i3ipc
|
||||
i3 = i3ipc.Connection()
|
||||
|
||||
# TODO: might be worth offering some kinda api for grabbing
|
||||
# the window id from the pid?
|
||||
# https://stackoverflow.com/a/2250879
|
||||
t = i3.get_tree()
|
||||
|
||||
orig_win_id = t.find_focused().window
|
||||
|
||||
# for tws
|
||||
win_names: list[str] = [
|
||||
'Interactive Brokers', # tws running in i3
|
||||
'IB Gateway', # gw running in i3
|
||||
# 'IB', # gw running in i3 (newer version?)
|
||||
]
|
||||
|
||||
try:
|
||||
for name, con in matches:
|
||||
print(f'Resetting data feed for {name}')
|
||||
win_id = str(con.window)
|
||||
w, h = con.rect.width, con.rect.height
|
||||
for name in win_names:
|
||||
results = t.find_titled(name)
|
||||
print(f'results for {name}: {results}')
|
||||
if results:
|
||||
con = results[0]
|
||||
print(f'Resetting data feed for {name}')
|
||||
win_id = str(con.window)
|
||||
w, h = con.rect.width, con.rect.height
|
||||
|
||||
# TODO: seems to be a few libs for python but not sure
|
||||
# if they support all the sub commands we need, order of
|
||||
# most recent commit history:
|
||||
# https://github.com/rr-/pyxdotool
|
||||
# https://github.com/ShaneHutter/pyxdotool
|
||||
# https://github.com/cphyc/pyxdotool
|
||||
# TODO: seems to be a few libs for python but not sure
|
||||
# if they support all the sub commands we need, order of
|
||||
# most recent commit history:
|
||||
# https://github.com/rr-/pyxdotool
|
||||
# https://github.com/ShaneHutter/pyxdotool
|
||||
# https://github.com/cphyc/pyxdotool
|
||||
|
||||
# TODO: only run the reconnect (2nd) kc on a detected
|
||||
# disconnect?
|
||||
for key_combo, timeout in [
|
||||
# only required if we need a connection reset.
|
||||
# ('ctrl+alt+r', 12),
|
||||
# data feed reset.
|
||||
('ctrl+alt+f', 6)
|
||||
]:
|
||||
subprocess.call([
|
||||
'xdotool',
|
||||
'windowactivate', '--sync', win_id,
|
||||
# TODO: only run the reconnect (2nd) kc on a detected
|
||||
# disconnect?
|
||||
for key_combo, timeout in [
|
||||
# only required if we need a connection reset.
|
||||
# ('ctrl+alt+r', 12),
|
||||
# data feed reset.
|
||||
('ctrl+alt+f', 6)
|
||||
]:
|
||||
subprocess.call([
|
||||
'xdotool',
|
||||
'windowactivate', '--sync', win_id,
|
||||
|
||||
# move mouse to bottom left of window (where
|
||||
# there should be nothing to click).
|
||||
'mousemove_relative', '--sync', str(w-4), str(h-4),
|
||||
# move mouse to bottom left of window (where
|
||||
# there should be nothing to click).
|
||||
'mousemove_relative', '--sync', str(w-4), str(h-4),
|
||||
|
||||
# NOTE: we may need to stick a `--retry 3` in here..
|
||||
'click', '--window', win_id,
|
||||
'--repeat', '3', '1',
|
||||
# NOTE: we may need to stick a `--retry 3` in here..
|
||||
'click', '--window', win_id,
|
||||
'--repeat', '3', '1',
|
||||
|
||||
# hackzorzes
|
||||
'key', key_combo,
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
# hackzorzes
|
||||
'key', key_combo,
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
# re-activate and focus original window
|
||||
subprocess.call([
|
||||
|
|
|
|||
|
|
@ -48,13 +48,11 @@ from bidict import bidict
|
|||
import trio
|
||||
import tractor
|
||||
from tractor import to_asyncio
|
||||
from tractor import trionics
|
||||
from pendulum import (
|
||||
from_timestamp,
|
||||
DateTime,
|
||||
Duration,
|
||||
duration as mk_duration,
|
||||
from_timestamp,
|
||||
Interval,
|
||||
)
|
||||
from eventkit import Event
|
||||
from ib_insync import (
|
||||
|
|
@ -92,15 +90,10 @@ from .symbols import (
|
|||
_exch_skip_list,
|
||||
_futes_venues,
|
||||
)
|
||||
from ...log import get_logger
|
||||
from .venues import (
|
||||
is_venue_open,
|
||||
sesh_times,
|
||||
is_venue_closure,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
from ._util import (
|
||||
log,
|
||||
# only for the ib_sync internal logging
|
||||
get_logger,
|
||||
)
|
||||
|
||||
_bar_load_dtype: list[tuple[str, type]] = [
|
||||
|
|
@ -186,7 +179,7 @@ class NonShittyIB(IB):
|
|||
# override `ib_insync` internal loggers so we can see wtf
|
||||
# it's doing..
|
||||
self._logger = get_logger(
|
||||
name=__name__,
|
||||
'ib_insync.ib',
|
||||
)
|
||||
self._createEvents()
|
||||
|
||||
|
|
@ -194,7 +187,7 @@ class NonShittyIB(IB):
|
|||
self.wrapper = NonShittyWrapper(self)
|
||||
self.client = ib_client.Client(self.wrapper)
|
||||
self.client._logger = get_logger(
|
||||
name='ib_insync.client',
|
||||
'ib_insync.client',
|
||||
)
|
||||
|
||||
# self.errorEvent += self._onError
|
||||
|
|
@ -266,16 +259,6 @@ def remove_handler_on_err(
|
|||
event.disconnect(handler)
|
||||
|
||||
|
||||
# (originally?) i thot that,
|
||||
# > "EST in ISO 8601 format is required.."
|
||||
#
|
||||
# XXX, but see `ib_async`'s impl,
|
||||
# - `ib_async.ib.IB.reqHistoricalDataAsync()`
|
||||
# - `ib_async.util.formatIBDatetime()`
|
||||
# below is EPOCH.
|
||||
_iso8601_epoch_in_est: str = "1970-01-01T00:00:00.000000-05:00"
|
||||
|
||||
|
||||
class Client:
|
||||
'''
|
||||
IB wrapped for our broker backend API.
|
||||
|
|
@ -349,32 +332,22 @@ class Client:
|
|||
self,
|
||||
fqme: str,
|
||||
|
||||
# EST in ISO 8601 format is required..
|
||||
# XXX, see `ib_async.ib.IB.reqHistoricalDataAsync()`
|
||||
# below is EPOCH.
|
||||
start_dt: datetime|None = None, # _iso8601_epoch_in_est,
|
||||
end_dt: datetime|None = None,
|
||||
# EST in ISO 8601 format is required... below is EPOCH
|
||||
start_dt: datetime | str = "1970-01-01T00:00:00.000000-05:00",
|
||||
end_dt: datetime | str = "",
|
||||
|
||||
# ohlc sample period in seconds
|
||||
sample_period_s: int = 1,
|
||||
|
||||
# optional "duration of time" equal to the
|
||||
# length of the returned history frame.
|
||||
duration: str|None = None,
|
||||
duration: str | None = None,
|
||||
|
||||
**kwargs,
|
||||
|
||||
) -> tuple[
|
||||
BarDataList,
|
||||
np.ndarray,
|
||||
Duration,
|
||||
]:
|
||||
) -> tuple[BarDataList, np.ndarray, Duration]:
|
||||
'''
|
||||
Retreive the `fqme`'s OHLCV-bars for the time-range "until `end_dt`".
|
||||
|
||||
Notes:
|
||||
- IB's api doesn't support a `start_dt` (which is why default
|
||||
is null) so we only use it for bar-frame duration checking.
|
||||
Retreive OHLCV bars for a fqme over a range to the present.
|
||||
|
||||
'''
|
||||
# See API docs here:
|
||||
|
|
@ -389,19 +362,13 @@ class Client:
|
|||
|
||||
dt_duration: Duration = (
|
||||
duration
|
||||
or
|
||||
default_dt_duration
|
||||
or default_dt_duration
|
||||
)
|
||||
|
||||
# TODO: maybe remove all this?
|
||||
global _enters
|
||||
if end_dt is None:
|
||||
end_dt: str = ''
|
||||
|
||||
else:
|
||||
est_end_dt = end_dt.in_tz('EST')
|
||||
if est_end_dt != end_dt:
|
||||
breakpoint()
|
||||
if not end_dt:
|
||||
end_dt = ''
|
||||
|
||||
_enters += 1
|
||||
|
||||
|
|
@ -470,116 +437,58 @@ class Client:
|
|||
+ query_info
|
||||
)
|
||||
|
||||
# TODO: we could maybe raise `NoData` instead if we
|
||||
# TODO: we could maybe raise ``NoData`` instead if we
|
||||
# rewrite the method in the first case?
|
||||
# right now there's no way to detect a timeout..
|
||||
return [], np.empty(0), dt_duration
|
||||
|
||||
log.info(query_info)
|
||||
|
||||
# ------ GAP-DETECTION ------
|
||||
# NOTE XXX: ensure minimum duration in bars?
|
||||
# => recursively call this method until we get at least as
|
||||
# many bars such that they sum in aggregate to the the
|
||||
# desired total time (duration) at most.
|
||||
# - if you query over a gap and get no data
|
||||
# that may short circuit the history
|
||||
if end_dt:
|
||||
if (
|
||||
# XXX XXX XXX
|
||||
# => WHY DID WE EVEN NEED THIS ORIGINALLY!? <=
|
||||
# XXX XXX XXX
|
||||
False
|
||||
and end_dt
|
||||
):
|
||||
nparr: np.ndarray = bars_to_np(bars)
|
||||
times: np.ndarray = nparr['time']
|
||||
first: float = times[0]
|
||||
last: float = times[-1]
|
||||
# frame_dur: float = times[-1] - first
|
||||
|
||||
details: ContractDetails = (
|
||||
await self.ib.reqContractDetailsAsync(contract)
|
||||
)[0]
|
||||
# convert to makt-native tz
|
||||
tz: str = details.timeZoneId
|
||||
end_dt = end_dt.in_tz(tz)
|
||||
first_dt: DateTime = from_timestamp(first).in_tz(tz)
|
||||
last_dt: DateTime = from_timestamp(last).in_tz(tz)
|
||||
tdiff: int = (
|
||||
last_dt
|
||||
-
|
||||
first_dt
|
||||
).in_seconds() + sample_period_s
|
||||
_open_now: bool = is_venue_open(
|
||||
con_deats=details,
|
||||
)
|
||||
|
||||
# XXX, do gap detections.
|
||||
has_closure_gap: bool = False
|
||||
if (
|
||||
last_dt.add(seconds=sample_period_s)
|
||||
<
|
||||
end_dt
|
||||
):
|
||||
open_time, close_time = sesh_times(details)
|
||||
# XXX, always calc gap in mkt-venue-local timezone
|
||||
gap: Interval = end_dt - last_dt
|
||||
if not (
|
||||
has_closure_gap := is_venue_closure(
|
||||
gap=gap,
|
||||
con_deats=details,
|
||||
time_step_s=sample_period_s,
|
||||
)):
|
||||
log.warning(
|
||||
f'Invalid non-closure gap for {fqme!r} ?!?\n'
|
||||
f'is-open-now: {_open_now}\n'
|
||||
f'\n'
|
||||
f'{gap}\n'
|
||||
)
|
||||
log.warning(
|
||||
f'Detected NON venue-closure GAP ??\n'
|
||||
f'{gap}\n'
|
||||
)
|
||||
breakpoint()
|
||||
else:
|
||||
assert has_closure_gap
|
||||
log.debug(
|
||||
f'Detected venue closure gap (weekend),\n'
|
||||
f'{gap}\n'
|
||||
)
|
||||
tdiff: float = times[-1] - first
|
||||
|
||||
if (
|
||||
start_dt is None
|
||||
and (
|
||||
tdiff
|
||||
<
|
||||
dt_duration.in_seconds()
|
||||
)
|
||||
and
|
||||
not has_closure_gap
|
||||
# len(bars) * sample_period_s) < dt_duration.in_seconds()
|
||||
tdiff < dt_duration.in_seconds()
|
||||
# and False
|
||||
):
|
||||
log.error(
|
||||
end_dt: DateTime = from_timestamp(first)
|
||||
log.warning(
|
||||
f'Frame result was shorter then {dt_duration}!?\n'
|
||||
'Recursing for more bars:\n'
|
||||
f'end_dt: {end_dt}\n'
|
||||
f'dt_duration: {dt_duration}\n'
|
||||
# f'\n'
|
||||
# f'Recursing for more bars:\n'
|
||||
)
|
||||
# XXX, debug!
|
||||
breakpoint()
|
||||
# XXX ? TODO? recursively try to re-request?
|
||||
# => i think *NO* right?
|
||||
#
|
||||
# (
|
||||
# r_bars,
|
||||
# r_arr,
|
||||
# r_duration,
|
||||
# ) = await self.bars(
|
||||
# fqme,
|
||||
# start_dt=start_dt,
|
||||
# end_dt=end_dt,
|
||||
# sample_period_s=sample_period_s,
|
||||
(
|
||||
r_bars,
|
||||
r_arr,
|
||||
r_duration,
|
||||
) = await self.bars(
|
||||
fqme,
|
||||
start_dt=start_dt,
|
||||
end_dt=end_dt,
|
||||
sample_period_s=sample_period_s,
|
||||
|
||||
# # TODO: make a table for Duration to
|
||||
# # the ib str values in order to use this?
|
||||
# # duration=duration,
|
||||
# )
|
||||
# r_bars.extend(bars)
|
||||
# bars = r_bars
|
||||
# TODO: make a table for Duration to
|
||||
# the ib str values in order to use this?
|
||||
# duration=duration,
|
||||
)
|
||||
r_bars.extend(bars)
|
||||
bars = r_bars
|
||||
|
||||
nparr: np.ndarray = bars_to_np(bars)
|
||||
|
||||
|
|
@ -806,8 +715,8 @@ class Client:
|
|||
|
||||
async def find_contracts(
|
||||
self,
|
||||
pattern: str|None = None,
|
||||
contract: Contract|None = None,
|
||||
pattern: str | None = None,
|
||||
contract: Contract | None = None,
|
||||
qualify: bool = True,
|
||||
err_on_qualify: bool = True,
|
||||
|
||||
|
|
@ -874,16 +783,9 @@ class Client:
|
|||
# crypto$
|
||||
elif exch == 'PAXOS': # btc.paxos
|
||||
con = Crypto(
|
||||
symbol=symbol.upper(),
|
||||
currency='USD',
|
||||
exchange='PAXOS',
|
||||
symbol=symbol,
|
||||
currency=currency,
|
||||
)
|
||||
# XXX, on `ib_insync` when first tried this,
|
||||
# > Error 10299, reqId 141: Expected what to show is
|
||||
# > AGGTRADES, please use that instead of TRADES.,
|
||||
# > contract: Crypto(conId=479624278, symbol='BTC',
|
||||
# > exchange='PAXOS', currency='USD',
|
||||
# > localSymbol='BTC.USD', tradingClass='BTC')
|
||||
|
||||
# stonks
|
||||
else:
|
||||
|
|
@ -959,7 +861,7 @@ class Client:
|
|||
self,
|
||||
fqme: str,
|
||||
|
||||
) -> datetime|None:
|
||||
) -> datetime | None:
|
||||
'''
|
||||
Return the first datetime stamp for `fqme` or `None`
|
||||
on request failure.
|
||||
|
|
@ -1015,7 +917,7 @@ class Client:
|
|||
tries: int = 100,
|
||||
raise_on_timeout: bool = False,
|
||||
|
||||
) -> Ticker|None:
|
||||
) -> Ticker | None:
|
||||
'''
|
||||
Return a single (snap) quote for symbol.
|
||||
|
||||
|
|
@ -1027,7 +929,7 @@ class Client:
|
|||
ready: ticker.TickerUpdateEvent = ticker.updateEvent
|
||||
|
||||
# ensure a last price gets filled in before we deliver quote
|
||||
timeouterr: Exception|None = None
|
||||
timeouterr: Exception | None = None
|
||||
warnset: bool = False
|
||||
for _ in range(tries):
|
||||
|
||||
|
|
@ -1041,7 +943,6 @@ class Client:
|
|||
)
|
||||
if tkr:
|
||||
break
|
||||
|
||||
except TimeoutError as err:
|
||||
timeouterr = err
|
||||
await asyncio.sleep(0.01)
|
||||
|
|
@ -1050,9 +951,7 @@ class Client:
|
|||
else:
|
||||
if not warnset:
|
||||
log.warning(
|
||||
f'Quote req timed out..\n'
|
||||
f'Maybe the venue is closed?\n'
|
||||
f'\n'
|
||||
f'Quote req timed out..maybe venue is closed?\n'
|
||||
f'{asdict(contract)}'
|
||||
)
|
||||
warnset = True
|
||||
|
|
@ -1064,11 +963,9 @@ class Client:
|
|||
)
|
||||
break
|
||||
else:
|
||||
if (
|
||||
timeouterr
|
||||
and
|
||||
raise_on_timeout
|
||||
):
|
||||
if timeouterr and raise_on_timeout:
|
||||
import pdbp
|
||||
pdbp.set_trace()
|
||||
raise timeouterr
|
||||
|
||||
if not warnset:
|
||||
|
|
@ -1284,7 +1181,7 @@ async def load_aio_clients(
|
|||
# the API TCP in `ib_insync` connection can be flaky af so instead
|
||||
# retry a few times to get the client going..
|
||||
connect_retries: int = 3,
|
||||
connect_timeout: float = 30, # in case a remote-host
|
||||
connect_timeout: float = 10,
|
||||
disconnect_on_exit: bool = True,
|
||||
|
||||
) -> dict[str, Client]:
|
||||
|
|
@ -1353,12 +1250,6 @@ async def load_aio_clients(
|
|||
|
||||
for i in range(connect_retries):
|
||||
try:
|
||||
log.info(
|
||||
'Trying `ib_async` connect\n'
|
||||
f'{host}: {port}\n'
|
||||
f'clientId: {client_id}\n'
|
||||
f'timeout: {connect_timeout}\n'
|
||||
)
|
||||
await ib.connectAsync(
|
||||
host,
|
||||
port,
|
||||
|
|
@ -1465,20 +1356,21 @@ async def load_aio_clients(
|
|||
|
||||
|
||||
async def load_clients_for_trio(
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Pure async mngr proxy to ``load_aio_clients()``.
|
||||
|
||||
This is a bootstrap entrypoint to call from
|
||||
a `tractor.to_asyncio.open_channel_from()`.
|
||||
This is a bootstrap entrypoing to call from
|
||||
a ``tractor.to_asyncio.open_channel_from()``.
|
||||
|
||||
'''
|
||||
async with load_aio_clients(
|
||||
disconnect_on_exit=False,
|
||||
) as accts2clients:
|
||||
async with load_aio_clients() as accts2clients:
|
||||
|
||||
to_trio.send_nowait(accts2clients)
|
||||
|
||||
chan.started_nowait(accts2clients)
|
||||
# TODO: maybe a sync event to wait on instead?
|
||||
await asyncio.sleep(float('inf'))
|
||||
|
||||
|
|
@ -1491,10 +1383,7 @@ async def open_client_proxies() -> tuple[
|
|||
async with (
|
||||
tractor.trionics.maybe_open_context(
|
||||
acm_func=tractor.to_asyncio.open_channel_from,
|
||||
kwargs={
|
||||
'target': load_clients_for_trio,
|
||||
# ^XXX, kwarg to `open_channel_from()`
|
||||
},
|
||||
kwargs={'target': load_clients_for_trio},
|
||||
|
||||
# lock around current actor task access
|
||||
# TODO: maybe this should be the default in tractor?
|
||||
|
|
@ -1604,7 +1493,7 @@ class MethodProxy:
|
|||
self,
|
||||
pattern: str,
|
||||
|
||||
) -> dict[str, Any]|trio.Event:
|
||||
) -> dict[str, Any] | trio.Event:
|
||||
|
||||
ev = self.event_table.get(pattern)
|
||||
|
||||
|
|
@ -1625,22 +1514,23 @@ class MethodProxy:
|
|||
|
||||
|
||||
async def open_aio_client_method_relay(
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
client: Client,
|
||||
event_consumers: dict[str, trio.Event],
|
||||
|
||||
) -> None:
|
||||
|
||||
# sync with `open_client_proxy()` caller
|
||||
chan.started_nowait(client)
|
||||
to_trio.send_nowait(client)
|
||||
|
||||
# TODO: separate channel for error handling?
|
||||
client.inline_errors(chan)
|
||||
client.inline_errors(to_trio)
|
||||
|
||||
# relay all method requests to ``asyncio``-side client and deliver
|
||||
# back results
|
||||
while not chan._to_trio._closed: # <- TODO, better check like `._web_bs`?
|
||||
msg: tuple[str, dict]|dict|None = await chan.get()
|
||||
while not to_trio._closed:
|
||||
msg: tuple[str, dict] | dict | None = await from_trio.get()
|
||||
match msg:
|
||||
case None: # termination sentinel
|
||||
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
||||
|
|
@ -1653,7 +1543,7 @@ async def open_aio_client_method_relay(
|
|||
try:
|
||||
resp = await meth(**kwargs)
|
||||
# echo the msg back
|
||||
chan.send_nowait({'result': resp})
|
||||
to_trio.send_nowait({'result': resp})
|
||||
|
||||
except (
|
||||
RequestError,
|
||||
|
|
@ -1661,10 +1551,10 @@ async def open_aio_client_method_relay(
|
|||
# TODO: relay all errors to trio?
|
||||
# BaseException,
|
||||
) as err:
|
||||
chan.send_nowait({'exception': err})
|
||||
to_trio.send_nowait({'exception': err})
|
||||
|
||||
case {'error': content}:
|
||||
chan.send_nowait({'exception': content})
|
||||
to_trio.send_nowait({'exception': content})
|
||||
|
||||
case _:
|
||||
raise ValueError(f'Unhandled msg {msg}')
|
||||
|
|
@ -1686,8 +1576,7 @@ async def open_client_proxy(
|
|||
event_consumers=event_table,
|
||||
) as (first, chan),
|
||||
|
||||
trionics.collapse_eg(), # loose-ify
|
||||
trio.open_nursery() as relay_tn,
|
||||
trio.open_nursery() as relay_n,
|
||||
):
|
||||
|
||||
assert isinstance(first, Client)
|
||||
|
|
@ -1727,7 +1616,7 @@ async def open_client_proxy(
|
|||
|
||||
continue
|
||||
|
||||
relay_tn.start_soon(relay_events)
|
||||
relay_n.start_soon(relay_events)
|
||||
|
||||
yield proxy
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,6 @@ import trio
|
|||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from tractor.to_asyncio import LinkedTaskChannel
|
||||
from tractor import trionics
|
||||
from ib_insync.contract import (
|
||||
Contract,
|
||||
)
|
||||
|
|
@ -50,10 +49,6 @@ from ib_insync.objects import (
|
|||
)
|
||||
|
||||
from piker import config
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.accounting import (
|
||||
Position,
|
||||
|
|
@ -81,6 +76,7 @@ from piker.clearing._messages import (
|
|||
BrokerdFill,
|
||||
BrokerdError,
|
||||
)
|
||||
from ._util import log
|
||||
from .api import (
|
||||
_accounts2clients,
|
||||
get_config,
|
||||
|
|
@ -98,10 +94,6 @@ from .ledger import (
|
|||
update_ledger_from_api_trades,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
|
||||
def pack_position(
|
||||
pos: IbPosition,
|
||||
|
|
@ -124,11 +116,7 @@ def pack_position(
|
|||
symbol=fqme,
|
||||
currency=con.currency,
|
||||
size=float(pos.position),
|
||||
avg_price=(
|
||||
float(pos.avgCost)
|
||||
/
|
||||
float(con.multiplier or 1.0)
|
||||
),
|
||||
avg_price=float(pos.avgCost) / float(con.multiplier or 1.0),
|
||||
),
|
||||
)
|
||||
|
||||
|
|
@ -369,10 +357,6 @@ async def update_and_audit_pos_msg(
|
|||
size=ibpos.position,
|
||||
|
||||
avg_price=pikerpos.ppu,
|
||||
|
||||
# XXX ensures matching even if multiple venue-names
|
||||
# in `.bs_fqme`, likely from txn records..
|
||||
bs_mktid=mkt.bs_mktid,
|
||||
)
|
||||
|
||||
ibfmtmsg: str = pformat(ibpos._asdict())
|
||||
|
|
@ -423,7 +407,7 @@ async def update_and_audit_pos_msg(
|
|||
|
||||
# TODO: make this a "propaganda" log level?
|
||||
if ibpos.avgCost != msg.avg_price:
|
||||
log.debug(
|
||||
log.warning(
|
||||
f'IB "FIFO" avg price for {msg.symbol} is DIFF:\n'
|
||||
f'ib: {ibfmtmsg}\n'
|
||||
'---------------------------\n'
|
||||
|
|
@ -441,8 +425,7 @@ async def aggr_open_orders(
|
|||
|
||||
) -> None:
|
||||
'''
|
||||
Collect all open orders from client and fill in `order_msgs:
|
||||
list`.
|
||||
Collect all open orders from client and fill in `order_msgs: list`.
|
||||
|
||||
'''
|
||||
trades: list[Trade] = client.ib.openTrades()
|
||||
|
|
@ -543,15 +526,9 @@ class IbAcnt(Struct):
|
|||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# task local msg dialog tracking
|
||||
flows = OrderDialogs()
|
||||
accounts_def = config.load_accounts(['ib'])
|
||||
|
|
@ -569,10 +546,7 @@ async def open_trade_dialog(
|
|||
),
|
||||
|
||||
# TODO: do this as part of `open_account()`!?
|
||||
open_symcache(
|
||||
'ib',
|
||||
only_from_memcache=True,
|
||||
) as symcache,
|
||||
open_symcache('ib', only_from_memcache=True) as symcache,
|
||||
):
|
||||
# Open a trade ledgers stack for appending trade records over
|
||||
# multiple accounts.
|
||||
|
|
@ -580,10 +554,8 @@ async def open_trade_dialog(
|
|||
ledgers: dict[str, TransactionLedger] = {}
|
||||
tables: dict[str, Account] = {}
|
||||
order_msgs: list[Status] = []
|
||||
conf: dict = get_config()
|
||||
accounts_def_inv: bidict[str, str] = bidict(
|
||||
conf['accounts']
|
||||
).inverse
|
||||
conf = get_config()
|
||||
accounts_def_inv: bidict[str, str] = bidict(conf['accounts']).inverse
|
||||
|
||||
with (
|
||||
ExitStack() as lstack,
|
||||
|
|
@ -733,11 +705,7 @@ async def open_trade_dialog(
|
|||
# client-account and build out position msgs to deliver to
|
||||
# EMS.
|
||||
for acctid, acnt in tables.items():
|
||||
active_pps: dict[str, Position]
|
||||
(
|
||||
active_pps,
|
||||
closed_pps,
|
||||
) = acnt.dump_active()
|
||||
active_pps, closed_pps = acnt.dump_active()
|
||||
|
||||
for pps in [active_pps, closed_pps]:
|
||||
piker_pps: list[Position] = list(pps.values())
|
||||
|
|
@ -753,7 +721,6 @@ async def open_trade_dialog(
|
|||
)
|
||||
if ibpos:
|
||||
bs_mktid: str = str(ibpos.contract.conId)
|
||||
|
||||
msg = await update_and_audit_pos_msg(
|
||||
acctid,
|
||||
pikerpos,
|
||||
|
|
@ -771,7 +738,7 @@ async def open_trade_dialog(
|
|||
f'UNEXPECTED POSITION says IB => {msg.symbol}\n'
|
||||
'Maybe they LIQUIDATED YOU or your ledger is wrong?\n'
|
||||
)
|
||||
log.debug(logmsg)
|
||||
log.error(logmsg)
|
||||
|
||||
await ctx.started((
|
||||
all_positions,
|
||||
|
|
@ -780,22 +747,21 @@ async def open_trade_dialog(
|
|||
|
||||
async with (
|
||||
ctx.open_stream() as ems_stream,
|
||||
trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
trio.open_nursery() as n,
|
||||
):
|
||||
# relay existing open orders to ems
|
||||
for msg in order_msgs:
|
||||
await ems_stream.send(msg)
|
||||
|
||||
for client in set(aioclients.values()):
|
||||
trade_event_stream: LinkedTaskChannel = await tn.start(
|
||||
trade_event_stream: LinkedTaskChannel = await n.start(
|
||||
open_trade_event_stream,
|
||||
client,
|
||||
)
|
||||
|
||||
# start order request handler **before** local trades
|
||||
# event loop
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
handle_order_requests,
|
||||
ems_stream,
|
||||
accounts_def,
|
||||
|
|
@ -803,7 +769,7 @@ async def open_trade_dialog(
|
|||
)
|
||||
|
||||
# allocate event relay tasks for each client connection
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
deliver_trade_events,
|
||||
|
||||
trade_event_stream,
|
||||
|
|
@ -1275,47 +1241,32 @@ async def deliver_trade_events(
|
|||
# never relay errors for non-broker related issues
|
||||
# https://interactivebrokers.github.io/tws-api/message_codes.html
|
||||
code: int = err['error_code']
|
||||
reason: str = err['reason']
|
||||
reqid: str = str(err['reqid'])
|
||||
|
||||
# "Warning:" msg codes,
|
||||
# https://interactivebrokers.github.io/tws-api/message_codes.html#warning_codes
|
||||
# - 2109: 'Outside Regular Trading Hours'
|
||||
if 'Warning:' in reason:
|
||||
log.warning(
|
||||
f'Order-API-warning: {code!r}\n'
|
||||
f'reqid: {reqid!r}\n'
|
||||
f'\n'
|
||||
f'{pformat(err)}\n'
|
||||
# ^TODO? should we just print the `reason`
|
||||
# not the full `err`-dict?
|
||||
)
|
||||
continue
|
||||
|
||||
# XXX known special (ignore) cases
|
||||
elif code in {
|
||||
200, # uhh.. ni idea
|
||||
if code in {
|
||||
200, # uhh
|
||||
|
||||
# hist pacing / connectivity
|
||||
162,
|
||||
165,
|
||||
|
||||
# WARNING codes:
|
||||
# https://interactivebrokers.github.io/tws-api/message_codes.html#warning_codes
|
||||
# Attribute 'Outside Regular Trading Hours' is
|
||||
# " 'ignored based on the order type and
|
||||
# destination. PlaceOrder is now ' 'being
|
||||
# processed.',
|
||||
2109,
|
||||
|
||||
# XXX: lol this isn't even documented..
|
||||
# 'No market data during competing live session'
|
||||
1669,
|
||||
}:
|
||||
log.error(
|
||||
f'Order-API-error which is non-cancel-causing ?!\n'
|
||||
f'\n'
|
||||
f'{pformat(err)}\n'
|
||||
)
|
||||
continue
|
||||
|
||||
reqid: str = str(err['reqid'])
|
||||
reason: str = err['reason']
|
||||
|
||||
if err['reqid'] == -1:
|
||||
log.error(
|
||||
f'TWS external order error ??\n'
|
||||
f'{pformat(err)}\n'
|
||||
)
|
||||
log.error(f'TWS external order error:\n{pformat(err)}')
|
||||
|
||||
flow: dict = dict(
|
||||
flows.get(reqid)
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) 2018-forever Tyler Goodlet (in stewardship for pikers)
|
||||
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
|
|
@ -13,12 +13,10 @@
|
|||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
"""
|
||||
Data feed endpoints pre-wrapped and ready for use with ``tractor``/``trio``.
|
||||
|
||||
'''
|
||||
Data feed endpoints pre-wrapped and ready for use with `tractor`/`trio`
|
||||
via "infected-asyncio-mode".
|
||||
|
||||
'''
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import asyncio
|
||||
from contextlib import (
|
||||
|
|
@ -28,6 +26,7 @@ from dataclasses import asdict
|
|||
from datetime import datetime
|
||||
from functools import partial
|
||||
from pprint import pformat
|
||||
from math import isnan
|
||||
import time
|
||||
from typing import (
|
||||
Any,
|
||||
|
|
@ -41,6 +40,7 @@ import numpy as np
|
|||
from pendulum import (
|
||||
now,
|
||||
from_timestamp,
|
||||
# DateTime,
|
||||
Duration,
|
||||
duration as mk_duration,
|
||||
)
|
||||
|
|
@ -56,11 +56,11 @@ from piker.brokers._util import (
|
|||
NoData,
|
||||
DataUnavailable,
|
||||
)
|
||||
from piker.log import get_logger
|
||||
from .api import (
|
||||
# _adhoc_futes_set,
|
||||
Client,
|
||||
con2fqme,
|
||||
log,
|
||||
load_aio_clients,
|
||||
MethodProxy,
|
||||
open_client_proxies,
|
||||
|
|
@ -69,18 +69,12 @@ from .api import (
|
|||
Contract,
|
||||
RequestError,
|
||||
)
|
||||
from .venues import is_venue_open
|
||||
from ._util import (
|
||||
data_reset_hack,
|
||||
)
|
||||
from ._util import data_reset_hack
|
||||
from .symbols import get_mkt_info
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from trio._core._run import Task
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# XXX NOTE: See available types table docs:
|
||||
# https://interactivebrokers.github.io/tws-api/tick_types.html
|
||||
|
|
@ -181,8 +175,8 @@ async def open_history_client(
|
|||
|
||||
async def get_hist(
|
||||
timeframe: float,
|
||||
end_dt: datetime|None = None,
|
||||
start_dt: datetime|None = None,
|
||||
end_dt: datetime | None = None,
|
||||
start_dt: datetime | None = None,
|
||||
|
||||
) -> tuple[np.ndarray, str]:
|
||||
|
||||
|
|
@ -190,8 +184,7 @@ async def open_history_client(
|
|||
|
||||
if (
|
||||
start_dt
|
||||
and
|
||||
start_dt.timestamp() == 0
|
||||
and start_dt.timestamp() == 0
|
||||
):
|
||||
await tractor.pause()
|
||||
|
||||
|
|
@ -206,27 +199,22 @@ async def open_history_client(
|
|||
latency = time.time() - query_start
|
||||
if (
|
||||
not timedout
|
||||
# and
|
||||
# latency <= max_timeout
|
||||
# and latency <= max_timeout
|
||||
):
|
||||
count += 1
|
||||
mean += latency / count
|
||||
log.debug(
|
||||
print(
|
||||
f'HISTORY FRAME QUERY LATENCY: {latency}\n'
|
||||
f'mean: {mean}'
|
||||
)
|
||||
|
||||
# could be trying to retreive bars over weekend
|
||||
if out is None:
|
||||
log.error(
|
||||
f"No bars starting at {end_dt!r} !?!?"
|
||||
)
|
||||
log.error(f"Can't grab bars starting at {end_dt}!?!?")
|
||||
if (
|
||||
end_dt
|
||||
and
|
||||
head_dt
|
||||
and
|
||||
end_dt <= head_dt
|
||||
and head_dt
|
||||
and end_dt <= head_dt
|
||||
):
|
||||
raise DataUnavailable(
|
||||
f'First timestamp is {head_dt}\n'
|
||||
|
|
@ -268,43 +256,12 @@ async def open_history_client(
|
|||
vlm = bars_array['volume']
|
||||
vlm[vlm < 0] = 0
|
||||
|
||||
# XXX, if a start-limit was passed ensure we only
|
||||
# return history that far back!
|
||||
if (
|
||||
start_dt
|
||||
and
|
||||
first_dt < start_dt
|
||||
):
|
||||
trimmed_bars = bars_array[
|
||||
bars_array['time'] >= start_dt.timestamp()
|
||||
]
|
||||
if (
|
||||
trimmed_first_dt := from_timestamp(trimmed_bars['time'][0])
|
||||
!=
|
||||
start_dt
|
||||
):
|
||||
# TODO! rm this once we're more confident it never hits!
|
||||
# breakpoint()
|
||||
raise RuntimeError(
|
||||
f'OHLC-bars array start is gt `start_dt` limit !!\n'
|
||||
f'start_dt: {start_dt}\n'
|
||||
f'first_dt: {first_dt}\n'
|
||||
f'trimmed_first_dt: {trimmed_first_dt}\n'
|
||||
)
|
||||
|
||||
# XXX, overwrite with start_dt-limited frame
|
||||
bars_array = trimmed_bars
|
||||
|
||||
return (
|
||||
bars_array,
|
||||
first_dt,
|
||||
last_dt,
|
||||
)
|
||||
return bars_array, first_dt, last_dt
|
||||
|
||||
# TODO: it seems like we can do async queries for ohlc
|
||||
# but getting the order right still isn't working and I'm not
|
||||
# quite sure why.. needs some tinkering and probably
|
||||
# a lookthrough of the `ib_insync` machinery, for eg. maybe
|
||||
# a lookthrough of the ``ib_insync`` machinery, for eg. maybe
|
||||
# we have to do the batch queries on the `asyncio` side?
|
||||
yield (
|
||||
get_hist,
|
||||
|
|
@ -328,9 +285,8 @@ _pacing: str = (
|
|||
|
||||
async def wait_on_data_reset(
|
||||
proxy: MethodProxy,
|
||||
|
||||
reset_type: str = 'data',
|
||||
timeout: float = 16,
|
||||
timeout: float = 16, # float('inf'),
|
||||
|
||||
task_status: TaskStatus[
|
||||
tuple[
|
||||
|
|
@ -339,47 +295,29 @@ async def wait_on_data_reset(
|
|||
]
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
) -> bool:
|
||||
'''
|
||||
Wait on a (global-ish) "data-farm" event to be emitted
|
||||
by the IB api server.
|
||||
|
||||
Allows syncing to reconnect event-messages emitted on the API
|
||||
console, such as:
|
||||
|
||||
- 'HMDS data farm connection is OK:ushmds'
|
||||
- 'Market data farm is connecting:usfuture'
|
||||
- 'Market data farm connection is OK:usfuture'
|
||||
|
||||
Deliver a `(cs, done: Event)` pair to the caller to support it
|
||||
waiting or cancelling the associated "data-reset-request";
|
||||
normally a manual data-reset-req is expected to be the cause and
|
||||
thus trigger such events (such as our click-hack-magic from
|
||||
`.ib._util`).
|
||||
|
||||
'''
|
||||
# ?TODO, do we need a task-lock around this method?
|
||||
#
|
||||
# register for an API "status event" wrapped for `trio`-sync.
|
||||
hist_ev: trio.Event = proxy.status_event(
|
||||
# TODO: we might have to put a task lock around this
|
||||
# method..
|
||||
hist_ev = proxy.status_event(
|
||||
'HMDS data farm connection is OK:ushmds'
|
||||
)
|
||||
#
|
||||
# ^TODO: other event-messages we might want to support waiting-for
|
||||
# but i wasn't able to get reliable..
|
||||
#
|
||||
|
||||
# TODO: other event messages we might want to try and
|
||||
# wait for but i wasn't able to get any of this
|
||||
# reliable..
|
||||
# reconnect_start = proxy.status_event(
|
||||
# 'Market data farm is connecting:usfuture'
|
||||
# )
|
||||
# live_ev = proxy.status_event(
|
||||
# 'Market data farm connection is OK:usfuture'
|
||||
# )
|
||||
|
||||
# try to wait on the reset event(s) to arrive, a timeout
|
||||
# will trigger a retry up to 6 times (for now).
|
||||
client: Client = proxy._aio_ns
|
||||
|
||||
done = trio.Event()
|
||||
with trio.move_on_after(timeout) as cs:
|
||||
|
||||
task_status.started((cs, done))
|
||||
|
||||
log.warning(
|
||||
|
|
@ -427,13 +365,14 @@ _failed_resets: int = 0
|
|||
|
||||
|
||||
async def get_bars(
|
||||
|
||||
proxy: MethodProxy,
|
||||
fqme: str,
|
||||
timeframe: int,
|
||||
|
||||
# blank to start which tells ib to look up the latest datum
|
||||
end_dt: datetime|None = None,
|
||||
start_dt: datetime|None = None,
|
||||
end_dt: str = '',
|
||||
start_dt: str | None = '',
|
||||
|
||||
# TODO: make this more dynamic based on measured frame rx latency?
|
||||
# how long before we trigger a feed reset (seconds)
|
||||
|
|
@ -457,9 +396,8 @@ async def get_bars(
|
|||
bool, # timed out hint
|
||||
]:
|
||||
'''
|
||||
Request-n-retrieve historical data frames from a `trio.Task`
|
||||
using a `MethoProxy` to query the `asyncio`-side's
|
||||
`.ib.api.Client` methods.
|
||||
Retrieve historical data from a ``trio``-side task using
|
||||
a ``MethoProxy``.
|
||||
|
||||
'''
|
||||
global _data_resetter_task, _failed_resets
|
||||
|
|
@ -487,9 +425,6 @@ async def get_bars(
|
|||
dt_duration,
|
||||
) = await proxy.bars(
|
||||
fqme=fqme,
|
||||
# XXX TODO! LOL we're not using this and IB dun
|
||||
# support it anyway..
|
||||
# start_dt=start_dt,
|
||||
end_dt=end_dt,
|
||||
sample_period_s=timeframe,
|
||||
|
||||
|
|
@ -652,7 +587,7 @@ async def get_bars(
|
|||
data_cs.cancel()
|
||||
|
||||
# spawn new data reset task
|
||||
data_cs, reset_done = await tn.start(
|
||||
data_cs, reset_done = await nurse.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
|
|
@ -672,14 +607,11 @@ async def get_bars(
|
|||
# such that simultaneous symbol queries don't try data resettingn
|
||||
# too fast..
|
||||
unset_resetter: bool = False
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
async with trio.open_nursery() as nurse:
|
||||
|
||||
# start history request that we allow
|
||||
# to run indefinitely until a result is acquired
|
||||
tn.start_soon(query)
|
||||
nurse.start_soon(query)
|
||||
|
||||
# start history reset loop which waits up to the timeout
|
||||
# for a result before triggering a data feed reset.
|
||||
|
|
@ -699,7 +631,7 @@ async def get_bars(
|
|||
unset_resetter: bool = True
|
||||
|
||||
# spawn new data reset task
|
||||
data_cs, reset_done = await tn.start(
|
||||
data_cs, reset_done = await nurse.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
|
|
@ -721,12 +653,14 @@ async def get_bars(
|
|||
)
|
||||
|
||||
|
||||
# per-actor cache of inter-eventloop-chans
|
||||
_quote_streams: dict[str, trio.abc.ReceiveStream] = {}
|
||||
|
||||
|
||||
async def _setup_quote_stream(
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
|
||||
symbol: str,
|
||||
opts: tuple[int] = (
|
||||
'375', # RT trade volume (excludes utrades)
|
||||
|
|
@ -740,17 +674,14 @@ async def _setup_quote_stream(
|
|||
# '294', # Trade rate / minute
|
||||
# '295', # Vlm rate / minute
|
||||
),
|
||||
contract: Contract|None = None,
|
||||
contract: Contract | None = None,
|
||||
|
||||
) -> trio.abc.ReceiveChannel:
|
||||
'''
|
||||
Stream L1 quotes via the `Ticker.updateEvent.connect(push)`
|
||||
callback API by registering a `push` callback which simply
|
||||
`chan.send_nowait()`s quote msgs back to the calling
|
||||
parent-`trio.Task`-side.
|
||||
Stream a ticker using the std L1 api.
|
||||
|
||||
NOTE, that this task-fn is run on the `asyncio.Task`-side ONLY
|
||||
and is thus run via `tractor.to_asyncio.open_channel_from()`.
|
||||
This task is ``asyncio``-side and must be called from
|
||||
``tractor.to_asyncio.open_channel_from()``.
|
||||
|
||||
'''
|
||||
global _quote_streams
|
||||
|
|
@ -758,84 +689,37 @@ async def _setup_quote_stream(
|
|||
async with load_aio_clients(
|
||||
disconnect_on_exit=False,
|
||||
) as accts2clients:
|
||||
caccount_name, client = get_preferred_data_client(accts2clients)
|
||||
contract = contract or (await client.find_contract(symbol))
|
||||
to_trio.send_nowait(contract) # cuz why not
|
||||
ticker: Ticker = client.ib.reqMktData(contract, ','.join(opts))
|
||||
|
||||
# XXX since this is an `asyncio.Task`, we must use
|
||||
# tractor.pause_from_sync()
|
||||
|
||||
(
|
||||
_account_name,
|
||||
client,
|
||||
) = get_preferred_data_client(
|
||||
accts2clients,
|
||||
)
|
||||
contract = (
|
||||
contract
|
||||
or
|
||||
(await client.find_contract(symbol))
|
||||
)
|
||||
chan.started_nowait(contract) # cuz why not
|
||||
ticker: Ticker = client.ib.reqMktData(
|
||||
contract,
|
||||
','.join(opts),
|
||||
)
|
||||
maybe_exc: BaseException|None = None
|
||||
handler_tries: int = 0
|
||||
aio_task: asyncio.Task = asyncio.current_task()
|
||||
|
||||
# ?TODO? this API is batch-wise and quite slow-af but,
|
||||
# - seems to be 5s updates?
|
||||
# - maybe we could use it for backchecking?
|
||||
#
|
||||
# NOTE: it's batch-wise and slow af but I guess could
|
||||
# be good for backchecking? Seems to be every 5s maybe?
|
||||
# ticker: Ticker = client.ib.reqTickByTickData(
|
||||
# contract, 'Last',
|
||||
# )
|
||||
|
||||
# define a very naive queue-pushing callback that relays
|
||||
# quote-packets directly the calling (parent) `trio.Task`.
|
||||
# Ensure on teardown we cancel the feed via their cancel API.
|
||||
#
|
||||
# # define a simple queue push routine that streams quote packets
|
||||
# # to trio over the ``to_trio`` memory channel.
|
||||
# to_trio, from_aio = trio.open_memory_channel(2**8) # type: ignore
|
||||
def teardown():
|
||||
'''
|
||||
Disconnect our `push`-er callback and cancel the data-feed
|
||||
for `contract`.
|
||||
|
||||
'''
|
||||
nonlocal maybe_exc
|
||||
ticker.updateEvent.disconnect(push)
|
||||
report: str = f'Disconnected mkt-data for {symbol!r} due to '
|
||||
if maybe_exc is not None:
|
||||
report += (
|
||||
'error,\n'
|
||||
f'{maybe_exc!r}\n'
|
||||
)
|
||||
log.error(report)
|
||||
else:
|
||||
report += (
|
||||
'cancellation.\n'
|
||||
)
|
||||
log.cancel(report)
|
||||
|
||||
log.error(f"Disconnected stream for `{symbol}`")
|
||||
client.ib.cancelMktData(contract)
|
||||
|
||||
# decouple broadcast mem chan
|
||||
_quote_streams.pop(symbol, None)
|
||||
|
||||
def push(
|
||||
t: Ticker,
|
||||
tries_before_raise: int = 6,
|
||||
) -> None:
|
||||
'''
|
||||
Push quotes verbatim to parent-side `trio.Task`.
|
||||
def push(t: Ticker) -> None:
|
||||
"""
|
||||
Push quotes to trio task.
|
||||
|
||||
'''
|
||||
nonlocal maybe_exc, handler_tries
|
||||
# log.debug(f'new IB quote: {t}\n')
|
||||
"""
|
||||
# log.debug(t)
|
||||
try:
|
||||
chan.send_nowait(t)
|
||||
to_trio.send_nowait(t)
|
||||
|
||||
# XXX TODO XXX replicate in `tractor` tests
|
||||
# as per `CancelledError`-handler notes below!
|
||||
# assert 0
|
||||
except (
|
||||
trio.BrokenResourceError,
|
||||
|
||||
|
|
@ -850,107 +734,35 @@ async def _setup_quote_stream(
|
|||
# resulting in tracebacks spammed to console..
|
||||
# Manually do the dereg ourselves.
|
||||
teardown()
|
||||
|
||||
# for slow debugging purposes to avoid clobbering prompt
|
||||
# with log msgs
|
||||
except trio.WouldBlock:
|
||||
log.exception(
|
||||
f'Asyncio->Trio `chan.send_nowait()` blocked !?\n'
|
||||
f'\n'
|
||||
f'{chan._to_trio.statistics()}\n'
|
||||
)
|
||||
|
||||
# ?TODO, handle re-connection attempts?
|
||||
except BaseException as _berr:
|
||||
berr = _berr
|
||||
if handler_tries >= tries_before_raise:
|
||||
# breakpoint()
|
||||
maybe_exc = _berr
|
||||
# task.set_exception(berr)
|
||||
aio_task.cancel(msg=berr.args)
|
||||
raise berr
|
||||
else:
|
||||
handler_tries += 1
|
||||
|
||||
log.exception(
|
||||
f'Failed to push ticker quote !?\n'
|
||||
f'handler_tries={handler_tries!r}\n'
|
||||
f'ticker: {t!r}\n'
|
||||
f'\n'
|
||||
f'{chan._to_trio.statistics()}\n'
|
||||
f'\n'
|
||||
f'CAUSE: {berr}\n'
|
||||
)
|
||||
# log.warning(
|
||||
# f'channel is blocking symbol feed for {symbol}?'
|
||||
# f'\n{to_trio.statistics}'
|
||||
# )
|
||||
pass
|
||||
|
||||
# except trio.WouldBlock:
|
||||
# # for slow debugging purposes to avoid clobbering prompt
|
||||
# # with log msgs
|
||||
# pass
|
||||
|
||||
ticker.updateEvent.connect(push)
|
||||
try:
|
||||
await asyncio.sleep(float('inf'))
|
||||
|
||||
# XXX, for debug.. TODO? can we rm again?
|
||||
#
|
||||
# tractor.pause_from_sync()
|
||||
# while True:
|
||||
# await asyncio.sleep(1.6)
|
||||
# if ticker.ticks:
|
||||
# log.debug(
|
||||
# f'ticker.ticks = \n'
|
||||
# f'{ticker.ticks}\n'
|
||||
# )
|
||||
# else:
|
||||
# log.warning(
|
||||
# 'UHH no ticker.ticks ??'
|
||||
# )
|
||||
|
||||
# XXX TODO XXX !?!?
|
||||
# apparently **without this handler** and the subsequent
|
||||
# re-raising of `maybe_exc from _taskc` cancelling the
|
||||
# `aio_task` from the `push()`-callback will cause a very
|
||||
# strange chain of exc raising that breaks alll sorts of
|
||||
# downstream callers, tasks and remote-actor tasks!?
|
||||
#
|
||||
# -[ ] we need some lowlevel reproducting tests to replicate
|
||||
# those worst-case scenarios in `tractor` core!!
|
||||
# -[ ] likely we should factor-out the `tractor.to_asyncio`
|
||||
# attempts at workarounds in `.translate_aio_errors()`
|
||||
# for failed `asyncio.Task.set_exception()` to either
|
||||
# call `aio_task.cancel()` and/or
|
||||
# `aio_task._fut_waiter.set_exception()` to a re-useable
|
||||
# toolset in something like a `.to_asyncio._utils`??
|
||||
#
|
||||
except asyncio.CancelledError as _taskc:
|
||||
if maybe_exc is not None:
|
||||
raise maybe_exc from _taskc
|
||||
|
||||
raise _taskc
|
||||
|
||||
except BaseException as _berr:
|
||||
# stash any crash cause for reporting in `teardown()`
|
||||
maybe_exc = _berr
|
||||
raise _berr
|
||||
|
||||
finally:
|
||||
# always disconnect our `push()` and cancel the
|
||||
# ib-"mkt-data-feed".
|
||||
teardown()
|
||||
|
||||
# return from_aio
|
||||
|
||||
|
||||
@acm
|
||||
async def open_aio_quote_stream(
|
||||
|
||||
symbol: str,
|
||||
contract: Contract|None = None,
|
||||
contract: Contract | None = None,
|
||||
|
||||
) -> (
|
||||
trio.abc.Channel| # iface
|
||||
tractor.to_asyncio.LinkedTaskChannel # actually
|
||||
):
|
||||
'''
|
||||
Open a real-time `Ticker` quote stream from an `asyncio.Task`
|
||||
spawned via `tractor.to_asyncio.open_channel_from()`, deliver the
|
||||
inter-event-loop channel to the `trio.Task` caller and cache it
|
||||
globally for re-use.
|
||||
) -> trio.abc.ReceiveStream:
|
||||
|
||||
'''
|
||||
from tractor.trionics import broadcast_receiver
|
||||
global _quote_streams
|
||||
|
||||
|
|
@ -966,7 +778,6 @@ async def open_aio_quote_stream(
|
|||
yield from_aio
|
||||
return
|
||||
|
||||
from_aio: tractor.to_asyncio.LinkedTaskChannel
|
||||
async with tractor.to_asyncio.open_channel_from(
|
||||
_setup_quote_stream,
|
||||
symbol=symbol,
|
||||
|
|
@ -976,10 +787,6 @@ async def open_aio_quote_stream(
|
|||
|
||||
assert contract
|
||||
|
||||
# TODO? de-reg on teardown of last consumer task?
|
||||
# -> why aren't we using `.trionics.maybe_open_context()`
|
||||
# here again?? (we are in `open_client_proxies()` tho?)
|
||||
#
|
||||
# cache feed for later consumers
|
||||
_quote_streams[symbol] = from_aio
|
||||
|
||||
|
|
@ -994,12 +801,7 @@ def normalize(
|
|||
calc_price: bool = False
|
||||
|
||||
) -> dict:
|
||||
'''
|
||||
Translate `ib_async`'s `Ticker.ticks` values to a `piker`
|
||||
normalized `dict` form for transmit to downstream `.data` layer
|
||||
consumers.
|
||||
|
||||
'''
|
||||
# check for special contract types
|
||||
con = ticker.contract
|
||||
fqme, calc_price = con2fqme(con)
|
||||
|
|
@ -1018,7 +820,7 @@ def normalize(
|
|||
|
||||
tbt = ticker.tickByTicks
|
||||
if tbt:
|
||||
log.info(f'tickbyticks:\n {ticker.tickByTicks}')
|
||||
print(f'tickbyticks:\n {ticker.tickByTicks}')
|
||||
|
||||
ticker.ticks = new_ticks
|
||||
|
||||
|
|
@ -1054,39 +856,27 @@ def normalize(
|
|||
return data
|
||||
|
||||
|
||||
# ?TODO? feels like this task-fn could be factored to reduce some
|
||||
# indentation levels?
|
||||
# -[ ] the reconnect while loop on ib-gw "data farm connection.."s
|
||||
# -[ ] everything embedded under the `async with aclosing(stream):`
|
||||
# as the "meat" of the quote delivery once the connection is
|
||||
# stable.
|
||||
#
|
||||
async def stream_quotes(
|
||||
|
||||
send_chan: trio.abc.SendChannel,
|
||||
symbols: list[str],
|
||||
feed_is_live: trio.Event,
|
||||
|
||||
# TODO? we need to hook into the `ib_async` logger like
|
||||
# we can with i3ipc from modden!
|
||||
# loglevel: str|None = None,
|
||||
loglevel: str = None,
|
||||
|
||||
# startup sync
|
||||
task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Stream `symbols[0]` quotes back via `send_chan`.
|
||||
Stream symbol quotes.
|
||||
|
||||
The `feed_is_live: Event` is set to signal the caller that it can
|
||||
begin processing msgs from the mem-chan.
|
||||
This is a ``trio`` callable routine meant to be invoked
|
||||
once the brokerd is up.
|
||||
|
||||
'''
|
||||
# TODO: support multiple subscriptions
|
||||
sym: str = symbols[0]
|
||||
log.info(
|
||||
f'request for real-time quotes\n'
|
||||
f'sym: {sym!r}\n'
|
||||
)
|
||||
sym = symbols[0]
|
||||
log.info(f'request for real-time quotes: {sym}')
|
||||
|
||||
init_msgs: list[FeedInit] = []
|
||||
|
||||
|
|
@ -1095,48 +885,34 @@ async def stream_quotes(
|
|||
details: ibis.ContractDetails
|
||||
async with (
|
||||
open_data_client() as proxy,
|
||||
# trio.open_nursery() as tn,
|
||||
):
|
||||
mkt, details = await get_mkt_info(
|
||||
sym,
|
||||
proxy=proxy, # passed to avoid implicit client load
|
||||
)
|
||||
|
||||
# is venue active rn?
|
||||
venue_is_open: bool = is_venue_open(
|
||||
con_deats=details,
|
||||
)
|
||||
init_msg = FeedInit(mkt_info=mkt)
|
||||
|
||||
# NOTE, tell sampler (via config) to skip vlm summing for dst
|
||||
# assets which provide no vlm data..
|
||||
if mkt.dst.atype in {
|
||||
'fiat',
|
||||
'index',
|
||||
'commodity',
|
||||
}:
|
||||
# tell sampler config that it shouldn't do vlm summing.
|
||||
init_msg.shm_write_opts['sum_tick_vlm'] = False
|
||||
init_msg.shm_write_opts['has_vlm'] = False
|
||||
|
||||
init_msgs.append(init_msg)
|
||||
|
||||
con: Contract = details.contract
|
||||
first_ticker: Ticker|None = None
|
||||
first_quote: dict[str, Any] = {}
|
||||
|
||||
timeout: float = 1.6
|
||||
with trio.move_on_after(timeout) as quote_cs:
|
||||
first_ticker: Ticker | None = None
|
||||
with trio.move_on_after(1):
|
||||
first_ticker: Ticker = await proxy.get_quote(
|
||||
contract=con,
|
||||
raise_on_timeout=False,
|
||||
)
|
||||
|
||||
# XXX should never happen with this ep right?
|
||||
# but if so then, more then likely mkt is closed?
|
||||
if quote_cs.cancelled_caught:
|
||||
log.warning(
|
||||
f'First quote req timed out after {timeout!r}s'
|
||||
)
|
||||
|
||||
if first_ticker:
|
||||
first_quote: dict = normalize(first_ticker)
|
||||
|
||||
|
|
@ -1148,42 +924,42 @@ async def stream_quotes(
|
|||
f'{pformat(first_quote)}\n'
|
||||
)
|
||||
|
||||
# XXX NOTE: whenever we're "outside regular trading hours"
|
||||
# (only relevant for assets coming from the "legacy markets"
|
||||
# space) so we basically (from an API/runtime-operational
|
||||
# perspective) "pretend the feed is live" even if it's
|
||||
# actually closed.
|
||||
#
|
||||
# IOW, we signal to the effective caller (task) that the live
|
||||
# feed is "already up" but really we're just indicating that
|
||||
# the OHLCV history can start being loaded immediately by the
|
||||
# `piker.data`/`.tsp` layers.
|
||||
#
|
||||
# XXX, deats: the "pretend we're live" is just done by
|
||||
# a `feed_is_live.set()` even though nothing is actually live
|
||||
# Bp
|
||||
if not venue_is_open:
|
||||
log.warning(
|
||||
f'Venue is closed, unable to establish real-time feed.\n'
|
||||
f'mkt: {mkt!r}\n'
|
||||
f'\n'
|
||||
f'first_ticker: {first_ticker}\n'
|
||||
)
|
||||
# NOTE: it might be outside regular trading hours for
|
||||
# assets with "standard venue operating hours" so we
|
||||
# only "pretend the feed is live" when the dst asset
|
||||
# type is NOT within the NON-NORMAL-venue set: aka not
|
||||
# commodities, forex or crypto currencies which CAN
|
||||
# always return a NaN on a snap quote request during
|
||||
# normal venue hours. In the case of a closed venue
|
||||
# (equitiies, futes, bonds etc.) we at least try to
|
||||
# grab the OHLC history.
|
||||
if (
|
||||
first_ticker
|
||||
and
|
||||
isnan(first_ticker.last)
|
||||
# SO, if the last quote price value is NaN we ONLY
|
||||
# "pretend to do" `feed_is_live.set()` if it's a known
|
||||
# dst asset venue with a lot of closed operating hours.
|
||||
and mkt.dst.atype not in {
|
||||
'commodity',
|
||||
'fiat',
|
||||
'crypto',
|
||||
}
|
||||
):
|
||||
task_status.started((
|
||||
init_msgs,
|
||||
first_quote,
|
||||
))
|
||||
|
||||
# block and let data history backfill code run.
|
||||
# XXX obvi given the venue is closed, we never expect feed
|
||||
# to come up; a taskc should be the only way to
|
||||
# terminate this task.
|
||||
await trio.sleep_forever()
|
||||
#
|
||||
# ^^XXX^^TODO! INSTEAD impl a `trio.sleep()` for the
|
||||
# duration until the venue opens!!
|
||||
# it's not really live but this will unblock
|
||||
# the brokerd feed task to tell the ui to update?
|
||||
feed_is_live.set()
|
||||
|
||||
# ?TODO, we could instead spawn a task that waits on a feed
|
||||
# block and let data history backfill code run.
|
||||
await trio.sleep_forever()
|
||||
return # we never expect feed to come up?
|
||||
|
||||
# TODO: we should instead spawn a task that waits on a feed
|
||||
# to start and let it wait indefinitely..instead of this
|
||||
# hard coded stuff.
|
||||
# async def wait_for_first_quote():
|
||||
|
|
@ -1205,30 +981,23 @@ async def stream_quotes(
|
|||
'Rxed init quote:\n'
|
||||
f'{pformat(first_quote)}'
|
||||
)
|
||||
# signal `.data.feed` layer that mkt quotes are LIVE
|
||||
feed_is_live.set()
|
||||
|
||||
cs: trio.CancelScope|None = None
|
||||
cs: trio.CancelScope | None = None
|
||||
startup: bool = True
|
||||
iter_quotes: trio.abc.Channel
|
||||
while (
|
||||
startup
|
||||
or
|
||||
cs.cancel_called
|
||||
or cs.cancel_called
|
||||
):
|
||||
with trio.CancelScope() as cs:
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
trio.open_nursery() as nurse,
|
||||
open_aio_quote_stream(
|
||||
symbol=sym,
|
||||
contract=con,
|
||||
) as iter_quotes,
|
||||
) as stream,
|
||||
):
|
||||
# ?TODO? can we rm this - particularly for `ib_async`?
|
||||
# ugh, clear ticks since we've consumed them
|
||||
# (ahem, ib_insync is stateful trash)
|
||||
# first_ticker.ticks = []
|
||||
first_ticker.ticks = []
|
||||
|
||||
# only on first entry at feed boot up
|
||||
if startup:
|
||||
|
|
@ -1242,8 +1011,8 @@ async def stream_quotes(
|
|||
# data feed event.
|
||||
async def reset_on_feed():
|
||||
|
||||
# ??TODO? this seems to be surpressed from the
|
||||
# traceback in `tractor`?
|
||||
# TODO: this seems to be surpressed from the
|
||||
# traceback in ``tractor``?
|
||||
# assert 0
|
||||
|
||||
rt_ev = proxy.status_event(
|
||||
|
|
@ -1252,28 +1021,60 @@ async def stream_quotes(
|
|||
await rt_ev.wait()
|
||||
cs.cancel() # cancel called should now be set
|
||||
|
||||
tn.start_soon(reset_on_feed)
|
||||
nurse.start_soon(reset_on_feed)
|
||||
|
||||
async with aclosing(stream):
|
||||
# if syminfo.get('no_vlm', False):
|
||||
if not init_msg.shm_write_opts['has_vlm']:
|
||||
|
||||
# generally speaking these feeds don't
|
||||
# include vlm data.
|
||||
atype: str = mkt.dst.atype
|
||||
log.info(
|
||||
f'No-vlm {mkt.fqme}@{atype}, skipping quote poll'
|
||||
)
|
||||
|
||||
else:
|
||||
# wait for real volume on feed (trading might be
|
||||
# closed)
|
||||
while True:
|
||||
ticker = await stream.receive()
|
||||
|
||||
# for a real volume contract we rait for
|
||||
# the first "real" trade to take place
|
||||
if (
|
||||
# not calc_price
|
||||
# and not ticker.rtTime
|
||||
not ticker.rtTime
|
||||
):
|
||||
# spin consuming tickers until we
|
||||
# get a real market datum
|
||||
log.debug(f"New unsent ticker: {ticker}")
|
||||
continue
|
||||
else:
|
||||
log.debug("Received first volume tick")
|
||||
# ugh, clear ticks since we've
|
||||
# consumed them (ahem, ib_insync is
|
||||
# truly stateful trash)
|
||||
ticker.ticks = []
|
||||
|
||||
# XXX: this works because we don't use
|
||||
# ``aclosing()`` above?
|
||||
break
|
||||
|
||||
quote = normalize(ticker)
|
||||
log.debug(f"First ticker received {quote}")
|
||||
|
||||
async with aclosing(iter_quotes):
|
||||
# tell data-layer spawner-caller that live
|
||||
# quotes are now active desptie not having
|
||||
# necessarily received a first vlm/clearing
|
||||
# tick.
|
||||
ticker = await iter_quotes.receive()
|
||||
quote = normalize(ticker)
|
||||
fqme: str = quote['fqme']
|
||||
await send_chan.send({fqme: quote})
|
||||
# quotes are now streaming.
|
||||
feed_is_live.set()
|
||||
|
||||
# last = time.time()
|
||||
async for ticker in iter_quotes:
|
||||
async for ticker in stream:
|
||||
quote = normalize(ticker)
|
||||
fqme: str = quote['fqme']
|
||||
log.debug(
|
||||
f'Sending quote\n'
|
||||
f'{quote}'
|
||||
)
|
||||
fqme = quote['fqme']
|
||||
await send_chan.send({fqme: quote})
|
||||
|
||||
# ugh, clear ticks since we've consumed them
|
||||
# ticker.ticks = []
|
||||
ticker.ticks = []
|
||||
# last = time.time()
|
||||
|
|
|
|||
|
|
@ -44,7 +44,6 @@ from ib_insync import (
|
|||
CommissionReport,
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
from piker.types import Struct
|
||||
from piker.data import (
|
||||
SymbologyCache,
|
||||
|
|
@ -58,6 +57,7 @@ from piker.accounting import (
|
|||
iter_by_dt,
|
||||
)
|
||||
from ._flex_reports import parse_flex_dt
|
||||
from ._util import log
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .api import (
|
||||
|
|
@ -65,9 +65,6 @@ if TYPE_CHECKING:
|
|||
MethodProxy,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
tx_sort: Callable = partial(
|
||||
iter_by_dt,
|
||||
|
|
|
|||
|
|
@ -42,7 +42,10 @@ from piker.accounting import (
|
|||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
)
|
||||
from piker.log import get_logger
|
||||
|
||||
from ._util import (
|
||||
log,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .api import (
|
||||
|
|
@ -50,10 +53,6 @@ if TYPE_CHECKING:
|
|||
Client,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
_futes_venues = (
|
||||
'GLOBEX',
|
||||
'NYMEX',
|
||||
|
|
@ -135,7 +134,7 @@ _adhoc_fiat_set = set((
|
|||
|
||||
# manually discovered tick discrepancies,
|
||||
# onl god knows how or why they'd cuck these up..
|
||||
_adhoc_mkt_infos: dict[int|str, dict] = {
|
||||
_adhoc_mkt_infos: dict[int | str, dict] = {
|
||||
'vtgn.nasdaq': {'price_tick': Decimal('0.01')},
|
||||
}
|
||||
|
||||
|
|
@ -489,7 +488,8 @@ def con2fqme(
|
|||
@async_lifo_cache()
|
||||
async def get_mkt_info(
|
||||
fqme: str,
|
||||
proxy: MethodProxy|None = None,
|
||||
|
||||
proxy: MethodProxy | None = None,
|
||||
|
||||
) -> tuple[MktPair, ibis.ContractDetails]:
|
||||
|
||||
|
|
@ -550,7 +550,7 @@ async def get_mkt_info(
|
|||
size_tick: Decimal = Decimal(
|
||||
str(details.minSize).rstrip('0')
|
||||
)
|
||||
# ?TODO, there is also the Contract.sizeIncrement, bt wtf is it?
|
||||
# |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
|
||||
|
||||
# NOTE: this is duplicate from the .broker.norm_trade_records()
|
||||
# routine, we should factor all this parsing somewhere..
|
||||
|
|
|
|||
|
|
@ -1,312 +0,0 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
(Multi-)venue mgmt helpers.
|
||||
|
||||
IB generally supports all "legacy" trading venues, those mostly owned
|
||||
by ICE and friends.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
from datetime import ( # noqa
|
||||
datetime,
|
||||
date,
|
||||
tzinfo as TzInfo,
|
||||
)
|
||||
from typing import (
|
||||
Iterator,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import exchange_calendars as xcals
|
||||
from pendulum import (
|
||||
now,
|
||||
Duration,
|
||||
Interval,
|
||||
Time,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ib_insync import (
|
||||
TradingSession,
|
||||
ContractDetails,
|
||||
)
|
||||
from exchange_calendars.exchange_calendars import (
|
||||
ExchangeCalendar,
|
||||
)
|
||||
from pandas import (
|
||||
# DatetimeIndex,
|
||||
TimeDelta,
|
||||
Timestamp,
|
||||
)
|
||||
|
||||
|
||||
def has_weekend(
|
||||
period: Interval,
|
||||
) -> bool:
|
||||
'''
|
||||
Predicate to for a period being within
|
||||
days 6->0 (sat->sun).
|
||||
|
||||
'''
|
||||
has_weekend: bool = False
|
||||
for dt in period:
|
||||
if dt.day_of_week in [0, 6]: # 0=Sunday, 6=Saturday
|
||||
has_weekend = True
|
||||
break
|
||||
|
||||
return has_weekend
|
||||
|
||||
|
||||
def has_holiday(
|
||||
con_deats: ContractDetails,
|
||||
period: Interval,
|
||||
) -> bool:
|
||||
'''
|
||||
Using the `exchange_calendars` lib detect if a time-gap `period`
|
||||
is contained in a known "cash hours" closure.
|
||||
|
||||
'''
|
||||
tz: str = con_deats.timeZoneId
|
||||
exch: str = con_deats.contract.primaryExchange
|
||||
cal: ExchangeCalendar = xcals.get_calendar(exch)
|
||||
end: datetime = period.end
|
||||
# _start: datetime = period.start
|
||||
# ?TODO, can rm ya?
|
||||
# => not that useful?
|
||||
# dti: DatetimeIndex = cal.sessions_in_range(
|
||||
# _start.date(),
|
||||
# end.date(),
|
||||
# )
|
||||
prev_close: Timestamp = cal.previous_close(
|
||||
end.date()
|
||||
).tz_convert(tz)
|
||||
prev_open: Timestamp = cal.previous_open(
|
||||
end.date()
|
||||
).tz_convert(tz)
|
||||
# now do relative from prev_ values ^
|
||||
# to get the next open which should match
|
||||
# "contain" the end of the gap.
|
||||
next_open: Timestamp = cal.next_open(
|
||||
prev_open,
|
||||
).tz_convert(tz)
|
||||
next_open: Timestamp = cal.next_open(
|
||||
prev_open,
|
||||
).tz_convert(tz)
|
||||
_next_close: Timestamp = cal.next_close(
|
||||
prev_close
|
||||
).tz_convert(tz)
|
||||
cash_gap: TimeDelta = next_open - prev_close
|
||||
is_holiday_gap = (
|
||||
cash_gap
|
||||
>
|
||||
period
|
||||
)
|
||||
# XXX, debug
|
||||
# breakpoint()
|
||||
return is_holiday_gap
|
||||
|
||||
|
||||
def is_current_time_in_range(
|
||||
sesh: Interval,
|
||||
when: datetime|None = None,
|
||||
) -> bool:
|
||||
'''
|
||||
Check if current time is within the datetime range.
|
||||
|
||||
Use any/the-same timezone as provided by `start_dt.tzinfo` value
|
||||
in the range.
|
||||
|
||||
'''
|
||||
when: datetime = when or now()
|
||||
return when in sesh
|
||||
|
||||
|
||||
def iter_sessions(
|
||||
con_deats: ContractDetails,
|
||||
) -> Iterator[Interval]:
|
||||
'''
|
||||
Yield `pendulum.Interval`s for all
|
||||
`ibas.ContractDetails.tradingSessions() -> TradingSession`s.
|
||||
|
||||
'''
|
||||
sesh: TradingSession
|
||||
for sesh in con_deats.tradingSessions():
|
||||
yield Interval(*sesh)
|
||||
|
||||
|
||||
def sesh_times(
|
||||
con_deats: ContractDetails,
|
||||
) -> tuple[Time, Time]:
|
||||
'''
|
||||
Based on the earliest trading session provided by the IB API,
|
||||
get the (day-agnostic) times for the start/end.
|
||||
|
||||
'''
|
||||
earliest_sesh: Interval = next(iter_sessions(con_deats))
|
||||
return (
|
||||
earliest_sesh.start.time(),
|
||||
earliest_sesh.end.time(),
|
||||
)
|
||||
# ^?TODO, use `.diff()` to get point-in-time-agnostic period?
|
||||
# https://pendulum.eustace.io/docs/#difference
|
||||
|
||||
|
||||
def is_venue_open(
|
||||
con_deats: ContractDetails,
|
||||
when: datetime|Duration|None = None,
|
||||
) -> bool:
|
||||
'''
|
||||
Check if market-venue is open during `when`, which defaults to
|
||||
"now".
|
||||
|
||||
'''
|
||||
sesh: Interval
|
||||
for sesh in iter_sessions(con_deats):
|
||||
if is_current_time_in_range(
|
||||
sesh=sesh,
|
||||
when=when,
|
||||
):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def is_venue_closure(
|
||||
gap: Interval,
|
||||
con_deats: ContractDetails,
|
||||
time_step_s: int,
|
||||
) -> bool:
|
||||
'''
|
||||
Check if a provided time-`gap` is just an (expected) trading
|
||||
venue closure period.
|
||||
|
||||
'''
|
||||
open: Time
|
||||
close: Time
|
||||
open, close = sesh_times(con_deats)
|
||||
|
||||
# ensure times are in mkt-native timezone
|
||||
tz: str = con_deats.timeZoneId
|
||||
start = gap.start.in_tz(tz)
|
||||
start_t = start.time()
|
||||
end = gap.end.in_tz(tz)
|
||||
end_t = end.time()
|
||||
if (
|
||||
(
|
||||
start_t in (
|
||||
close,
|
||||
close.subtract(seconds=time_step_s)
|
||||
)
|
||||
and
|
||||
end_t in (
|
||||
open,
|
||||
open.add(seconds=time_step_s),
|
||||
)
|
||||
)
|
||||
or
|
||||
has_weekend(gap)
|
||||
or
|
||||
has_holiday(
|
||||
con_deats=con_deats,
|
||||
period=gap,
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
# breakpoint()
|
||||
return False
|
||||
|
||||
|
||||
# TODO, put this into `._util` and call it from here!
|
||||
#
|
||||
# NOTE, this was generated by @guille from a gpt5 prompt
|
||||
# and was originally thot to be needed before learning about
|
||||
# `ib_insync.contract.ContractDetails._parseSessions()` and
|
||||
# it's downstream meths..
|
||||
#
|
||||
# This is still likely useful to keep for now to parse the
|
||||
# `.tradingHours: str` value manually if we ever decide
|
||||
# to move off `ib_async` and implement our own `trio`/`anyio`
|
||||
# based version Bp
|
||||
#
|
||||
# >attempt to parse the retarted ib "time stampy thing" they
|
||||
# >do for "venue hours" with this.. written by
|
||||
# >gpt5-"thinking",
|
||||
#
|
||||
|
||||
|
||||
def parse_trading_hours(
|
||||
spec: str,
|
||||
tz: TzInfo|None = None
|
||||
) -> dict[
|
||||
date,
|
||||
tuple[datetime, datetime]
|
||||
]|None:
|
||||
'''
|
||||
Parse venue hours like:
|
||||
'YYYYMMDD:HHMM-YYYYMMDD:HHMM;YYYYMMDD:CLOSED;...'
|
||||
|
||||
Returns `dict[date] = (open_dt, close_dt)` or `None` if
|
||||
closed.
|
||||
|
||||
'''
|
||||
if (
|
||||
not isinstance(spec, str)
|
||||
or
|
||||
not spec
|
||||
):
|
||||
raise ValueError('spec must be a non-empty string')
|
||||
|
||||
out: dict[
|
||||
date,
|
||||
tuple[datetime, datetime]
|
||||
]|None = {}
|
||||
|
||||
for part in (p.strip() for p in spec.split(';') if p.strip()):
|
||||
if part.endswith(':CLOSED'):
|
||||
day_s, _ = part.split(':', 1)
|
||||
d = datetime.strptime(day_s, '%Y%m%d').date()
|
||||
out[d] = None
|
||||
continue
|
||||
|
||||
try:
|
||||
start_s, end_s = part.split('-', 1)
|
||||
start_dt = datetime.strptime(start_s, '%Y%m%d:%H%M')
|
||||
end_dt = datetime.strptime(end_s, '%Y%m%d:%H%M')
|
||||
except ValueError as exc:
|
||||
raise ValueError(f'invalid segment: {part}') from exc
|
||||
|
||||
if tz is not None:
|
||||
start_dt = start_dt.replace(tzinfo=tz)
|
||||
end_dt = end_dt.replace(tzinfo=tz)
|
||||
|
||||
out[start_dt.date()] = (start_dt, end_dt)
|
||||
|
||||
return out
|
||||
|
||||
|
||||
# ORIG desired usage,
|
||||
#
|
||||
# TODO, for non-drunk tomorrow,
|
||||
# - call above fn and check that `output[today] is not None`
|
||||
# trading_hrs: dict = parse_trading_hours(
|
||||
# details.tradingHours
|
||||
# )
|
||||
# liq_hrs: dict = parse_trading_hours(
|
||||
# details.liquidHours
|
||||
# )
|
||||
|
|
@ -34,7 +34,6 @@ import urllib.parse
|
|||
import hashlib
|
||||
import hmac
|
||||
import base64
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
from piker import config
|
||||
|
|
@ -373,7 +372,8 @@ class Client:
|
|||
# 1658347714, 'status': 'Success'}]}
|
||||
|
||||
if xfers:
|
||||
await tractor.pause()
|
||||
import tractor
|
||||
await tractor.pp()
|
||||
|
||||
trans: dict[str, Transaction] = {}
|
||||
for entry in xfers:
|
||||
|
|
@ -501,8 +501,7 @@ class Client:
|
|||
for xkey, data in resp['result'].items():
|
||||
|
||||
# NOTE: always cache in pairs tables for faster lookup
|
||||
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
||||
pair = Pair(xname=xkey, **data)
|
||||
pair = Pair(xname=xkey, **data)
|
||||
|
||||
# register the above `Pair` structs for all
|
||||
# key-sets/monikers: a set of 4 (frickin) tables
|
||||
|
|
|
|||
|
|
@ -62,12 +62,9 @@ from piker.clearing._messages import (
|
|||
from piker.brokers import (
|
||||
open_cached_client,
|
||||
)
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from piker.data import open_symcache
|
||||
from .api import (
|
||||
log,
|
||||
Client,
|
||||
BrokerError,
|
||||
)
|
||||
|
|
@ -81,8 +78,6 @@ from .ledger import (
|
|||
verify_balances,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
MsgUnion = Union[
|
||||
BrokerdCancel,
|
||||
BrokerdError,
|
||||
|
|
@ -180,8 +175,9 @@ async def handle_order_requests(
|
|||
|
||||
case {
|
||||
'account': 'kraken.spot' as account,
|
||||
'action': 'buy'|'sell',
|
||||
}:
|
||||
'action': action,
|
||||
} if action in {'buy', 'sell'}:
|
||||
|
||||
# validate
|
||||
order = BrokerdOrder(**msg)
|
||||
|
||||
|
|
@ -266,12 +262,6 @@ async def handle_order_requests(
|
|||
} | extra
|
||||
|
||||
log.info(f'Submitting WS order request:\n{pformat(req)}')
|
||||
|
||||
# NOTE HOWTO, debug order requests
|
||||
#
|
||||
# if 'XRP' in pair:
|
||||
# await tractor.pause()
|
||||
|
||||
await ws.send_msg(req)
|
||||
|
||||
# placehold for sanity checking in relay loop
|
||||
|
|
@ -436,15 +426,9 @@ def trades2pps(
|
|||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
async with (
|
||||
# TODO: maybe bind these together and deliver
|
||||
# a tuple from `.open_cached_client()`?
|
||||
|
|
@ -560,7 +544,7 @@ async def open_trade_dialog(
|
|||
# to be reloaded.
|
||||
balances: dict[str, float] = await client.get_balances()
|
||||
|
||||
await verify_balances(
|
||||
verify_balances(
|
||||
acnt,
|
||||
src_fiat,
|
||||
balances,
|
||||
|
|
@ -1101,8 +1085,6 @@ async def handle_order_updates(
|
|||
f'Failed to {action} order {reqid}:\n'
|
||||
f'{errmsg}'
|
||||
)
|
||||
# if tractor._state.debug_mode():
|
||||
# await tractor.pause()
|
||||
|
||||
symbol: str = 'N/A'
|
||||
if chain := apiflows.get(reqid):
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ Symbology defs and search.
|
|||
from decimal import Decimal
|
||||
|
||||
import tractor
|
||||
from rapidfuzz import process as fuzzy
|
||||
|
||||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
|
|
@ -40,13 +41,8 @@ from piker.accounting._mktinfo import (
|
|||
)
|
||||
|
||||
|
||||
# https://www.kraken.com/features/api#get-tradable-pairs
|
||||
class Pair(Struct):
|
||||
'''
|
||||
A tradable asset pair as schema-defined by,
|
||||
|
||||
https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs
|
||||
|
||||
'''
|
||||
xname: str # idiotic bs_mktid equiv i guess?
|
||||
altname: str # alternate pair name
|
||||
wsname: str # WebSocket pair name (if available)
|
||||
|
|
@ -57,6 +53,7 @@ class Pair(Struct):
|
|||
lot: str # volume lot size
|
||||
|
||||
cost_decimals: int
|
||||
costmin: float
|
||||
pair_decimals: int # scaling decimal places for pair
|
||||
lot_decimals: int # scaling decimal places for volume
|
||||
|
||||
|
|
@ -82,7 +79,6 @@ class Pair(Struct):
|
|||
tick_size: float # min price step size
|
||||
status: str
|
||||
|
||||
costmin: str|None = None # XXX, only some mktpairs?
|
||||
short_position_limit: float = 0
|
||||
long_position_limit: float = float('inf')
|
||||
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ from piker._cacheables import (
|
|||
)
|
||||
from piker.log import get_logger
|
||||
from piker.data.validate import FeedInit
|
||||
from piker.types import Struct # NOTE, this is already a `tractor.msg.Struct`
|
||||
from piker.types import Struct
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
match_from_pairs,
|
||||
|
|
@ -98,18 +98,9 @@ class KucoinMktPair(Struct, frozen=True):
|
|||
def size_tick(self) -> Decimal:
|
||||
return Decimal(str(self.quoteMinSize))
|
||||
|
||||
callauctionFirstStageStartTime: None|float
|
||||
callauctionIsEnabled: bool
|
||||
callauctionPriceCeiling: float|None
|
||||
callauctionPriceFloor: float|None
|
||||
callauctionSecondStageStartTime: float|None
|
||||
callauctionThirdStageStartTime: float|None
|
||||
|
||||
enableTrading: bool
|
||||
feeCategory: int
|
||||
feeCurrency: str
|
||||
isMarginEnabled: bool
|
||||
makerFeeCoefficient: float
|
||||
market: str
|
||||
minFunds: float
|
||||
name: str
|
||||
|
|
@ -119,10 +110,7 @@ class KucoinMktPair(Struct, frozen=True):
|
|||
quoteIncrement: float
|
||||
quoteMaxSize: float
|
||||
quoteMinSize: float
|
||||
st: bool
|
||||
symbol: str # our bs_mktid, kucoin's internal id
|
||||
takerFeeCoefficient: float
|
||||
tradingStartTime: float|None
|
||||
|
||||
|
||||
class AccountTrade(Struct, frozen=True):
|
||||
|
|
@ -404,13 +392,7 @@ class Client:
|
|||
pairs: dict[str, KucoinMktPair] = {}
|
||||
fqmes2mktids: bidict[str, str] = bidict()
|
||||
for item in entries:
|
||||
try:
|
||||
pair = pairs[item['name']] = KucoinMktPair(**item)
|
||||
except TypeError as te:
|
||||
raise TypeError(
|
||||
'`KucoinMktPair` and reponse fields do not match ??\n'
|
||||
f'{KucoinMktPair.fields_diff(item)}\n'
|
||||
) from te
|
||||
pair = pairs[item['name']] = KucoinMktPair(**item)
|
||||
fqmes2mktids[
|
||||
item['name'].lower().replace('-', '')
|
||||
] = pair.name
|
||||
|
|
@ -611,7 +593,7 @@ async def get_client() -> AsyncGenerator[Client, None]:
|
|||
'''
|
||||
async with (
|
||||
httpx.AsyncClient(
|
||||
base_url='https://api.kucoin.com/api',
|
||||
base_url=f'https://api.kucoin.com/api',
|
||||
) as trio_client,
|
||||
):
|
||||
client = Client(httpx_client=trio_client)
|
||||
|
|
@ -655,7 +637,7 @@ async def open_ping_task(
|
|||
await trio.sleep((ping_interval - 1000) / 1000)
|
||||
await ws.send_msg({'id': connect_id, 'type': 'ping'})
|
||||
|
||||
log.warning('Starting ping task for kucoin ws connection')
|
||||
log.info('Starting ping task for kucoin ws connection')
|
||||
n.start_soon(ping_server)
|
||||
|
||||
yield
|
||||
|
|
@ -667,14 +649,9 @@ async def open_ping_task(
|
|||
async def get_mkt_info(
|
||||
fqme: str,
|
||||
|
||||
) -> tuple[
|
||||
MktPair,
|
||||
KucoinMktPair,
|
||||
]:
|
||||
) -> tuple[MktPair, KucoinMktPair]:
|
||||
'''
|
||||
Query for and return both a `piker.accounting.MktPair` and
|
||||
`KucoinMktPair` from provided `fqme: str`
|
||||
(fully-qualified-market-endpoint).
|
||||
Query for and return a `MktPair` and `KucoinMktPair`.
|
||||
|
||||
'''
|
||||
async with open_cached_client('kucoin') as client:
|
||||
|
|
@ -749,8 +726,6 @@ async def stream_quotes(
|
|||
|
||||
log.info(f'Starting up quote stream(s) for {symbols}')
|
||||
for sym_str in symbols:
|
||||
mkt: MktPair
|
||||
pair: KucoinMktPair
|
||||
mkt, pair = await get_mkt_info(sym_str)
|
||||
init_msgs.append(
|
||||
FeedInit(mkt_info=mkt)
|
||||
|
|
@ -758,11 +733,7 @@ async def stream_quotes(
|
|||
|
||||
ws: NoBsWs
|
||||
token, ping_interval = await client._get_ws_token()
|
||||
log.info('API reported ping_interval: {ping_interval}\n')
|
||||
|
||||
connect_id: str = str(uuid4())
|
||||
typ: str
|
||||
quote: dict
|
||||
connect_id = str(uuid4())
|
||||
async with (
|
||||
open_autorecon_ws(
|
||||
(
|
||||
|
|
@ -776,37 +747,20 @@ async def stream_quotes(
|
|||
),
|
||||
) as ws,
|
||||
open_ping_task(ws, ping_interval, connect_id),
|
||||
aclosing(
|
||||
iter_normed_quotes(
|
||||
ws, sym_str
|
||||
)
|
||||
) as iter_quotes,
|
||||
aclosing(stream_messages(ws, sym_str)) as msg_gen,
|
||||
):
|
||||
typ, quote = await anext(iter_quotes)
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
# take care to not unblock here until we get a real
|
||||
# trade quote?
|
||||
# ^TODO, remove this right?
|
||||
# -[ ] what often blocks chart boot/new-feed switching
|
||||
# since we'ere waiting for a live quote instead of just
|
||||
# loading history afap..
|
||||
# |_ XXX, not sure if we require a bit of rework to core
|
||||
# feed init logic or if backends justg gotta be
|
||||
# changed up.. feel like there was some causality
|
||||
# dilema prolly only seen with IB too..
|
||||
# while typ != 'trade':
|
||||
# typ, quote = await anext(iter_quotes)
|
||||
while typ != 'trade':
|
||||
# take care to not unblock here until we get a real
|
||||
# trade quote
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
task_status.started((init_msgs, quote))
|
||||
feed_is_live.set()
|
||||
|
||||
# XXX NOTE, DO NOT include the `.<backend>` suffix!
|
||||
# OW the sampling loop will not broadcast correctly..
|
||||
# since `bus._subscribers.setdefault(bs_fqme, set())`
|
||||
# is used inside `.data.open_feed_bus()` !!!
|
||||
topic: str = mkt.bs_fqme
|
||||
async for typ, quote in iter_quotes:
|
||||
await send_chan.send({topic: quote})
|
||||
async for typ, msg in msg_gen:
|
||||
await send_chan.send({sym_str: msg})
|
||||
|
||||
|
||||
@acm
|
||||
|
|
@ -861,7 +815,7 @@ async def subscribe(
|
|||
)
|
||||
|
||||
|
||||
async def iter_normed_quotes(
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
sym: str,
|
||||
|
||||
|
|
@ -892,9 +846,6 @@ async def iter_normed_quotes(
|
|||
|
||||
yield 'trade', {
|
||||
'symbol': sym,
|
||||
# TODO, is 'last' even used elsewhere/a-good
|
||||
# semantic? can't we just read the ticks with our
|
||||
# .data.ticktools.frame_ticks()`/
|
||||
'last': trade_data.price,
|
||||
'brokerd_ts': last_trade_ts,
|
||||
'ticks': [
|
||||
|
|
@ -987,7 +938,7 @@ async def open_history_client(
|
|||
if end_dt is None:
|
||||
inow = round(time.time())
|
||||
|
||||
log.debug(
|
||||
print(
|
||||
f'difference in time between load and processing'
|
||||
f'{inow - times[-1]}'
|
||||
)
|
||||
|
|
|
|||
|
|
@ -37,12 +37,6 @@ import tractor
|
|||
from async_generator import asynccontextmanager
|
||||
import numpy as np
|
||||
import wrapt
|
||||
|
||||
# TODO, port to `httpx`/`trio-websocket` whenver i get back to
|
||||
# writing a proper ws-api streamer for this backend (since the data
|
||||
# feeds are free now) as per GH feat-req:
|
||||
# https://github.com/pikers/piker/issues/509
|
||||
#
|
||||
import asks
|
||||
|
||||
from ..calc import humanize, percent_change
|
||||
|
|
@ -50,19 +44,13 @@ from . import open_cached_client
|
|||
from piker._cacheables import async_lifo_cache
|
||||
from .. import config
|
||||
from ._util import resproc, BrokerError, SymbolNotFound
|
||||
from piker.log import (
|
||||
from ..log import (
|
||||
colorize_json,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
)
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
|
||||
_use_practice_account = False
|
||||
_refresh_token_ep = 'https://{}login.questrade.com/oauth2/'
|
||||
|
|
@ -1211,10 +1199,7 @@ async def stream_quotes(
|
|||
# feed_type: str = 'stock',
|
||||
) -> AsyncGenerator[str, Dict[str, Any]]:
|
||||
# XXX: required to propagate ``tractor`` loglevel to piker logging
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(loglevel)
|
||||
|
||||
async with open_cached_client('questrade') as client:
|
||||
if feed_type == 'stock':
|
||||
|
|
|
|||
|
|
@ -30,16 +30,9 @@ import asks
|
|||
from ._util import (
|
||||
resproc,
|
||||
BrokerError,
|
||||
log,
|
||||
)
|
||||
from piker.calc import percent_change
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
from ..calc import percent_change
|
||||
|
||||
_service_ep = 'https://api.robinhood.com'
|
||||
|
||||
|
|
|
|||
|
|
@ -1,49 +0,0 @@
|
|||
piker.clearing
|
||||
______________
|
||||
trade execution-n-control subsys for both live and paper trading as
|
||||
well as algo-trading manual override/interaction across any backend
|
||||
broker and data provider.
|
||||
|
||||
avail UIs
|
||||
*********
|
||||
|
||||
order ctl
|
||||
---------
|
||||
the `piker.clearing` subsys is exposed mainly though
|
||||
the `piker chart` GUI as a "chart trader" style UX and
|
||||
is automatically enabled whenever a chart is opened.
|
||||
|
||||
.. ^TODO, more prose here!
|
||||
|
||||
the "manual" order control features are exposed via the
|
||||
`piker.ui.order_mode` API and can pretty much always be
|
||||
used (at least) in simulated-trading mode, aka "paper"-mode, and
|
||||
the micro-manual is as follows:
|
||||
|
||||
``order_mode`` (
|
||||
edge triggered activation by any of the following keys,
|
||||
``mouse-click`` on y-level to submit at that price
|
||||
):
|
||||
|
||||
- ``f``/ ``ctl-f`` to stage buy
|
||||
- ``d``/ ``ctl-d`` to stage sell
|
||||
- ``a`` to stage alert
|
||||
|
||||
|
||||
``search_mode`` (
|
||||
``ctl-l`` or ``ctl-space`` to open,
|
||||
``ctl-c`` or ``ctl-space`` to close
|
||||
) :
|
||||
|
||||
- begin typing to have symbol search automatically lookup
|
||||
symbols from all loaded backend (broker) providers
|
||||
- arrow keys and mouse click to navigate selection
|
||||
- vi-like ``ctl-[hjkl]`` for navigation
|
||||
|
||||
|
||||
position (pp) mgmt
|
||||
------------------
|
||||
you can also configure your position allocation limits from the
|
||||
sidepane.
|
||||
|
||||
.. ^TODO, explain and provide tut once more refined!
|
||||
|
|
@ -25,10 +25,7 @@ from typing import TYPE_CHECKING
|
|||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor.trionics import (
|
||||
broadcast_receiver,
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor.trionics import broadcast_receiver
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
|
|
@ -171,6 +168,7 @@ class OrderClient(Struct):
|
|||
|
||||
|
||||
async def relay_orders_from_sync_code(
|
||||
|
||||
client: OrderClient,
|
||||
symbol_key: str,
|
||||
to_ems_stream: tractor.MsgStream,
|
||||
|
|
@ -215,7 +213,7 @@ async def relay_orders_from_sync_code(
|
|||
async def open_ems(
|
||||
fqme: str,
|
||||
mode: str = 'live',
|
||||
loglevel: str = 'warning',
|
||||
loglevel: str = 'error',
|
||||
|
||||
) -> tuple[
|
||||
OrderClient, # client
|
||||
|
|
@ -244,11 +242,6 @@ async def open_ems(
|
|||
|
||||
async with maybe_open_emsd(
|
||||
broker,
|
||||
# XXX NOTE, LOL so this determines the daemon `emsd` loglevel
|
||||
# then FYI.. that's kinda wrong no?
|
||||
# -[ ] shouldn't it be set by `pikerd -l` or no?
|
||||
# -[ ] would make a lot more sense to have a subsys ctl for
|
||||
# levels.. like `-l emsd.info` or something?
|
||||
loglevel=loglevel,
|
||||
) as portal:
|
||||
|
||||
|
|
@ -288,11 +281,8 @@ async def open_ems(
|
|||
client._ems_stream = trades_stream
|
||||
|
||||
# start sync code order msg delivery task
|
||||
async with (
|
||||
collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
tn.start_soon(
|
||||
async with trio.open_nursery() as n:
|
||||
n.start_soon(
|
||||
relay_orders_from_sync_code,
|
||||
client,
|
||||
fqme,
|
||||
|
|
@ -308,4 +298,4 @@ async def open_ems(
|
|||
)
|
||||
|
||||
# stop the sync-msg-relay task on exit.
|
||||
tn.cancel_scope.cancel()
|
||||
n.cancel_scope.cancel()
|
||||
|
|
|
|||
|
|
@ -42,12 +42,10 @@ from bidict import bidict
|
|||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from tractor import trionics
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
subsys,
|
||||
)
|
||||
from ..accounting._mktinfo import (
|
||||
unpack_fqme,
|
||||
|
|
@ -78,6 +76,7 @@ if TYPE_CHECKING:
|
|||
|
||||
# TODO: numba all of this
|
||||
def mk_check(
|
||||
|
||||
trigger_price: float,
|
||||
known_last: float,
|
||||
action: str,
|
||||
|
|
@ -137,7 +136,7 @@ class DarkBook(Struct):
|
|||
tuple[
|
||||
Callable[[float], bool], # predicate
|
||||
tuple[str, ...], # tickfilter
|
||||
dict|Order, # cmd / msg type
|
||||
dict | Order, # cmd / msg type
|
||||
|
||||
# live submission constraint parameters
|
||||
float, # percent_away max price diff
|
||||
|
|
@ -163,7 +162,7 @@ async def clear_dark_triggers(
|
|||
|
||||
router: Router,
|
||||
brokerd_orders_stream: tractor.MsgStream,
|
||||
quote_stream: tractor.MsgStream,
|
||||
quote_stream: tractor.ReceiveMsgStream, # noqa
|
||||
broker: str,
|
||||
fqme: str,
|
||||
|
||||
|
|
@ -179,7 +178,6 @@ async def clear_dark_triggers(
|
|||
'''
|
||||
# XXX: optimize this for speed!
|
||||
# TODO:
|
||||
# - port to the new ringbuf stuff in `tractor.ipc`!
|
||||
# - numba all this!
|
||||
# - this stream may eventually contain multiple symbols
|
||||
quote_stream._raise_on_lag = False
|
||||
|
|
@ -279,7 +277,7 @@ async def clear_dark_triggers(
|
|||
|
||||
# remove exec-condition from set
|
||||
log.info(f'Removing trigger for {oid}')
|
||||
trigger: tuple|None = execs.pop(oid, None)
|
||||
trigger: tuple | None = execs.pop(oid, None)
|
||||
if not trigger:
|
||||
log.warning(
|
||||
f'trigger for {oid} was already removed!?'
|
||||
|
|
@ -337,8 +335,8 @@ async def open_brokerd_dialog(
|
|||
brokermod: ModuleType,
|
||||
portal: tractor.Portal,
|
||||
exec_mode: str,
|
||||
fqme: str|None = None,
|
||||
loglevel: str|None = None,
|
||||
fqme: str | None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> tuple[
|
||||
tractor.MsgStream,
|
||||
|
|
@ -352,21 +350,9 @@ async def open_brokerd_dialog(
|
|||
broker backend, configuration, or client code usage.
|
||||
|
||||
'''
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name='clearing',
|
||||
)
|
||||
# enable `.accounting` console since normally used by
|
||||
# each `brokerd`.
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name='piker.accounting',
|
||||
)
|
||||
broker: str = brokermod.name
|
||||
|
||||
def mk_paper_ep(
|
||||
loglevel: str,
|
||||
):
|
||||
def mk_paper_ep():
|
||||
from . import _paper_engine as paper_mod
|
||||
|
||||
nonlocal brokermod, exec_mode
|
||||
|
|
@ -401,7 +387,6 @@ async def open_brokerd_dialog(
|
|||
for ep_name in [
|
||||
'open_trade_dialog', # probably final name?
|
||||
'trades_dialogue', # legacy
|
||||
# ^!TODO, rm this since all backends ported no ?!?
|
||||
]:
|
||||
trades_endpoint = getattr(
|
||||
brokermod,
|
||||
|
|
@ -418,21 +403,17 @@ async def open_brokerd_dialog(
|
|||
|
||||
if (
|
||||
trades_endpoint is not None
|
||||
or
|
||||
exec_mode != 'paper'
|
||||
or exec_mode != 'paper'
|
||||
):
|
||||
# open live brokerd trades endpoint
|
||||
open_trades_endpoint = portal.open_context(
|
||||
trades_endpoint,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
|
||||
@acm
|
||||
async def maybe_open_paper_ep():
|
||||
if exec_mode == 'paper':
|
||||
async with mk_paper_ep(
|
||||
loglevel=loglevel,
|
||||
) as msg:
|
||||
async with mk_paper_ep() as msg:
|
||||
yield msg
|
||||
return
|
||||
|
||||
|
|
@ -443,9 +424,7 @@ async def open_brokerd_dialog(
|
|||
# runtime indication that the backend can't support live
|
||||
# order ctrl yet, so boot the paperboi B0
|
||||
if first == 'paper':
|
||||
async with mk_paper_ep(
|
||||
loglevel=loglevel,
|
||||
) as msg:
|
||||
async with mk_paper_ep() as msg:
|
||||
yield msg
|
||||
return
|
||||
else:
|
||||
|
|
@ -521,7 +500,7 @@ class Router(Struct):
|
|||
|
||||
'''
|
||||
# setup at actor spawn time
|
||||
_tn: trio.Nursery
|
||||
nursery: trio.Nursery
|
||||
|
||||
# broker to book map
|
||||
books: dict[str, DarkBook] = {}
|
||||
|
|
@ -674,11 +653,7 @@ class Router(Struct):
|
|||
flume = feed.flumes[fqme]
|
||||
first_quote: dict = flume.first_quote
|
||||
book: DarkBook = self.get_dark_book(broker)
|
||||
|
||||
if not (last := first_quote.get('last')):
|
||||
last: float = flume.rt_shm.array[-1]['close']
|
||||
|
||||
book.lasts[fqme]: float = float(last)
|
||||
book.lasts[fqme]: float = float(first_quote['last'])
|
||||
|
||||
async with self.maybe_open_brokerd_dialog(
|
||||
brokermod=brokermod,
|
||||
|
|
@ -691,7 +666,7 @@ class Router(Struct):
|
|||
# dark book clearing loop, also lives with parent
|
||||
# daemon to allow dark order clearing while no
|
||||
# client is connected.
|
||||
self._tn.start_soon(
|
||||
self.nursery.start_soon(
|
||||
clear_dark_triggers,
|
||||
self,
|
||||
relay.brokerd_stream,
|
||||
|
|
@ -714,7 +689,7 @@ class Router(Struct):
|
|||
|
||||
# spawn a ``brokerd`` order control dialog stream
|
||||
# that syncs lifetime with the parent `emsd` daemon.
|
||||
self._tn.start_soon(
|
||||
self.nursery.start_soon(
|
||||
translate_and_relay_brokerd_events,
|
||||
broker,
|
||||
relay.brokerd_stream,
|
||||
|
|
@ -741,14 +716,13 @@ class Router(Struct):
|
|||
subs = self.subscribers[sub_key]
|
||||
|
||||
sent_some: bool = False
|
||||
for client_stream in subs.copy():
|
||||
for client_stream in subs:
|
||||
try:
|
||||
await client_stream.send(msg)
|
||||
sent_some = True
|
||||
except (
|
||||
trio.ClosedResourceError,
|
||||
trio.BrokenResourceError,
|
||||
tractor.TransportClosed,
|
||||
):
|
||||
to_remove.add(client_stream)
|
||||
log.warning(
|
||||
|
|
@ -780,25 +754,19 @@ _router: Router = None
|
|||
@tractor.context
|
||||
async def _setup_persistent_emsd(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> None:
|
||||
|
||||
if loglevel:
|
||||
_log = get_console_log(
|
||||
level=loglevel,
|
||||
name=subsys,
|
||||
)
|
||||
assert _log.name == 'piker.clearing'
|
||||
get_console_log(loglevel)
|
||||
|
||||
global _router
|
||||
|
||||
# open a root "service task-nursery" for the `emsd`-actor
|
||||
async with (
|
||||
trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
_router = Router(_tn=tn)
|
||||
# open a root "service nursery" for the ``emsd`` actor
|
||||
async with trio.open_nursery() as service_nursery:
|
||||
|
||||
_router = Router(nursery=service_nursery)
|
||||
|
||||
# TODO: send back the full set of persistent
|
||||
# orders/execs?
|
||||
|
|
@ -845,7 +813,7 @@ async def translate_and_relay_brokerd_events(
|
|||
f'Rx brokerd trade msg:\n'
|
||||
f'{fmsg}'
|
||||
)
|
||||
status_msg: Status|None = None
|
||||
status_msg: Status | None = None
|
||||
|
||||
match brokerd_msg:
|
||||
# BrokerdPosition
|
||||
|
|
@ -1042,28 +1010,14 @@ async def translate_and_relay_brokerd_events(
|
|||
status_msg.brokerd_msg = msg
|
||||
status_msg.src = msg.broker_details['name']
|
||||
|
||||
if not status_msg.req:
|
||||
# likely some order change state?
|
||||
await tractor.pause()
|
||||
else:
|
||||
await router.client_broadcast(
|
||||
status_msg.req.symbol,
|
||||
status_msg,
|
||||
)
|
||||
await router.client_broadcast(
|
||||
status_msg.req.symbol,
|
||||
status_msg,
|
||||
)
|
||||
|
||||
if status == 'closed':
|
||||
log.info(
|
||||
f'Execution is complete!\n'
|
||||
f'oid: {oid!r}\n'
|
||||
)
|
||||
status_msg = book._active.pop(oid, None)
|
||||
if status_msg is None:
|
||||
log.warning(
|
||||
f'Order was already cleared from book ??\n'
|
||||
f'oid: {oid!r}\n'
|
||||
f'\n'
|
||||
f'Maybe the order cancelled before submitted ??\n'
|
||||
)
|
||||
log.info(f'Execution for {oid} is complete!')
|
||||
status_msg = book._active.pop(oid)
|
||||
|
||||
elif status == 'canceled':
|
||||
log.cancel(f'Cancellation for {oid} is complete!')
|
||||
|
|
@ -1228,16 +1182,12 @@ async def process_client_order_cmds(
|
|||
submitting live orders immediately if requested by the client.
|
||||
|
||||
'''
|
||||
# TODO, only allow `msgspec.Struct` form!
|
||||
cmd: dict
|
||||
# cmd: dict
|
||||
async for cmd in client_order_stream:
|
||||
log.info(
|
||||
f'Received order cmd:\n'
|
||||
f'{pformat(cmd)}\n'
|
||||
)
|
||||
log.info(f'Received order cmd:\n{pformat(cmd)}')
|
||||
|
||||
# CAWT DAMN we need struct support!
|
||||
oid: str = str(cmd['oid'])
|
||||
oid = str(cmd['oid'])
|
||||
|
||||
# register this stream as an active order dialog (msg flow) for
|
||||
# this order id such that translated message from the brokerd
|
||||
|
|
@ -1306,7 +1256,7 @@ async def process_client_order_cmds(
|
|||
and status.resp == 'dark_open'
|
||||
):
|
||||
# remove from dark book clearing
|
||||
entry: tuple|None = dark_book.triggers[fqme].pop(oid, None)
|
||||
entry: tuple | None = dark_book.triggers[fqme].pop(oid, None)
|
||||
if entry:
|
||||
(
|
||||
pred,
|
||||
|
|
@ -1343,7 +1293,7 @@ async def process_client_order_cmds(
|
|||
case {
|
||||
'oid': oid,
|
||||
'symbol': fqme,
|
||||
'price': price,
|
||||
'price': trigger_price,
|
||||
'size': size,
|
||||
'action': ('buy' | 'sell') as action,
|
||||
'exec_mode': ('live' | 'paper'),
|
||||
|
|
@ -1375,7 +1325,7 @@ async def process_client_order_cmds(
|
|||
|
||||
symbol=sym,
|
||||
action=action,
|
||||
price=price,
|
||||
price=trigger_price,
|
||||
size=size,
|
||||
account=req.account,
|
||||
)
|
||||
|
|
@ -1397,11 +1347,7 @@ async def process_client_order_cmds(
|
|||
# (``translate_and_relay_brokerd_events()`` above) will
|
||||
# handle relaying the ems side responses back to
|
||||
# the client/cmd sender from this request
|
||||
log.info(
|
||||
f'Sending live order to {broker}:\n'
|
||||
f'{pformat(msg)}'
|
||||
)
|
||||
|
||||
log.info(f'Sending live order to {broker}:\n{pformat(msg)}')
|
||||
await brokerd_order_stream.send(msg)
|
||||
|
||||
# an immediate response should be ``BrokerdOrderAck``
|
||||
|
|
@ -1417,7 +1363,7 @@ async def process_client_order_cmds(
|
|||
case {
|
||||
'oid': oid,
|
||||
'symbol': fqme,
|
||||
'price': price,
|
||||
'price': trigger_price,
|
||||
'size': size,
|
||||
'exec_mode': exec_mode,
|
||||
'action': action,
|
||||
|
|
@ -1445,12 +1391,7 @@ async def process_client_order_cmds(
|
|||
if isnan(last):
|
||||
last = flume.rt_shm.array[-1]['close']
|
||||
|
||||
trigger_price: float = float(price)
|
||||
pred = mk_check(
|
||||
trigger_price,
|
||||
last,
|
||||
action,
|
||||
)
|
||||
pred = mk_check(trigger_price, last, action)
|
||||
|
||||
# NOTE: for dark orders currently we submit
|
||||
# the triggered live order at a price 5 ticks
|
||||
|
|
@ -1557,7 +1498,7 @@ async def maybe_open_trade_relays(
|
|||
loglevel: str = 'info',
|
||||
):
|
||||
|
||||
fqme, relay, feed, client_ready = await _router._tn.start(
|
||||
fqme, relay, feed, client_ready = await _router.nursery.start(
|
||||
_router.open_trade_relays,
|
||||
fqme,
|
||||
exec_mode,
|
||||
|
|
@ -1587,18 +1528,19 @@ async def maybe_open_trade_relays(
|
|||
|
||||
@tractor.context
|
||||
async def _emsd_main(
|
||||
ctx: tractor.Context, # becomes `ems_ctx` below
|
||||
ctx: tractor.Context,
|
||||
fqme: str,
|
||||
exec_mode: str, # ('paper', 'live')
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> tuple[ # `ctx.started()` value!
|
||||
dict[ # positions
|
||||
tuple[str, str], # brokername, acctid
|
||||
) -> tuple[
|
||||
dict[
|
||||
# brokername, acctid
|
||||
tuple[str, str],
|
||||
list[BrokerdPosition],
|
||||
],
|
||||
list[str], # accounts
|
||||
dict[str, Status], # dialogs
|
||||
list[str],
|
||||
dict[str, Status],
|
||||
]:
|
||||
'''
|
||||
EMS (sub)actor entrypoint providing the execution management
|
||||
|
|
@ -1723,5 +1665,5 @@ async def _emsd_main(
|
|||
if not client_streams:
|
||||
log.warning(
|
||||
f'Order dialog is not being monitored:\n'
|
||||
f'{oid!r} <-> {client_stream.chan.aid.reprol()}\n'
|
||||
f'{oid} ->\n{client_stream._ctx.chan.uid}'
|
||||
)
|
||||
|
|
|
|||
|
|
@ -19,7 +19,6 @@ Clearing sub-system message and protocols.
|
|||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
from decimal import Decimal
|
||||
from typing import (
|
||||
Literal,
|
||||
)
|
||||
|
|
@ -72,15 +71,7 @@ class Order(Struct):
|
|||
symbol: str # | MktPair
|
||||
account: str # should we set a default as '' ?
|
||||
|
||||
# https://docs.python.org/3/library/decimal.html#decimal-objects
|
||||
#
|
||||
# ?TODO? decimal usage throughout?
|
||||
# -[ ] possibly leverage the `Encoder(decimal_format='number')`
|
||||
# bit?
|
||||
# |_https://jcristharif.com/msgspec/supported-types.html#decimal
|
||||
# -[ ] should we also use it for .size?
|
||||
#
|
||||
price: Decimal
|
||||
price: float
|
||||
size: float # -ve is "sell", +ve is "buy"
|
||||
|
||||
brokers: list[str] = []
|
||||
|
|
@ -187,7 +178,7 @@ class BrokerdOrder(Struct):
|
|||
time_ns: int
|
||||
|
||||
symbol: str # fqme
|
||||
price: Decimal
|
||||
price: float
|
||||
size: float
|
||||
|
||||
# TODO: if we instead rely on a +ve/-ve size to determine
|
||||
|
|
@ -301,9 +292,6 @@ class BrokerdError(Struct):
|
|||
|
||||
# TODO: yeah, so we REALLY need to completely deprecate
|
||||
# this and use the `.accounting.Position` msg-type instead..
|
||||
# -[ ] an alternative might be to add a `Position.summary() ->
|
||||
# `PositionSummary`-msg that we generate since `Position` has a lot
|
||||
# of fields by default we likely don't want to send over the wire?
|
||||
class BrokerdPosition(Struct):
|
||||
'''
|
||||
Position update event from brokerd.
|
||||
|
|
@ -316,4 +304,3 @@ class BrokerdPosition(Struct):
|
|||
avg_price: float
|
||||
currency: str = ''
|
||||
name: str = 'position'
|
||||
bs_mktid: str|int|None = None
|
||||
|
|
|
|||
|
|
@ -59,9 +59,9 @@ from piker.data import (
|
|||
open_symcache,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.log import (
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ._messages import (
|
||||
BrokerdCancel,
|
||||
|
|
@ -73,8 +73,6 @@ from ._messages import (
|
|||
BrokerdError,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
class PaperBoi(Struct):
|
||||
'''
|
||||
|
|
@ -299,8 +297,6 @@ class PaperBoi(Struct):
|
|||
|
||||
# transmit pp msg to ems
|
||||
pp: Position = self.acnt.pps[bs_mktid]
|
||||
# TODO, this will break if `require_only=True` was passed to
|
||||
# `.update_from_ledger()`
|
||||
|
||||
pp_msg = BrokerdPosition(
|
||||
broker=self.broker,
|
||||
|
|
@ -512,7 +508,7 @@ async def handle_order_requests(
|
|||
reqid = await client.submit_limit(
|
||||
oid=order.oid,
|
||||
symbol=f'{order.symbol}.{client.broker}',
|
||||
price=float(order.price),
|
||||
price=order.price,
|
||||
action=order.action,
|
||||
size=order.size,
|
||||
# XXX: by default 0 tells ``ib_insync`` methods that
|
||||
|
|
@ -552,18 +548,16 @@ _sells: defaultdict[
|
|||
|
||||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
|
||||
ctx: tractor.Context,
|
||||
broker: str,
|
||||
fqme: str|None = None, # if empty, we only boot broker mode
|
||||
fqme: str | None = None, # if empty, we only boot broker mode
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> None:
|
||||
|
||||
# enable piker.clearing console log for *this* `brokerd` subactor
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
# enable piker.clearing console log for *this* subactor
|
||||
get_console_log(loglevel)
|
||||
|
||||
symcache: SymbologyCache
|
||||
async with open_symcache(get_brokermod(broker)) as symcache:
|
||||
|
|
@ -659,7 +653,6 @@ async def open_trade_dialog(
|
|||
# in) use manually constructed table from calling
|
||||
# the `.get_mkt_info()` provider EP above.
|
||||
_mktmap_table=mkt_by_fqme,
|
||||
only_require=list(mkt_by_fqme),
|
||||
)
|
||||
|
||||
pp_msgs: list[BrokerdPosition] = []
|
||||
|
|
|
|||
|
|
@ -28,14 +28,11 @@ from ..log import (
|
|||
from piker.types import Struct
|
||||
subsys: str = 'piker.clearing'
|
||||
|
||||
log = get_logger(
|
||||
name='piker.clearing',
|
||||
)
|
||||
log = get_logger(subsys)
|
||||
|
||||
# TODO, oof doesn't this ignore the `loglevel` then???
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name='clearing',
|
||||
name=subsys,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -61,8 +61,7 @@ def load_trans_eps(
|
|||
|
||||
if (
|
||||
network
|
||||
and
|
||||
not maddrs
|
||||
and not maddrs
|
||||
):
|
||||
# load network section and (attempt to) connect all endpoints
|
||||
# which are reachable B)
|
||||
|
|
@ -113,83 +112,108 @@ def load_trans_eps(
|
|||
default=None,
|
||||
help='Multiaddrs to bind or contact',
|
||||
)
|
||||
# @click.option(
|
||||
# '--tsdb',
|
||||
# is_flag=True,
|
||||
# help='Enable local ``marketstore`` instance'
|
||||
# )
|
||||
# @click.option(
|
||||
# '--es',
|
||||
# is_flag=True,
|
||||
# help='Enable local ``elasticsearch`` instance'
|
||||
# )
|
||||
def pikerd(
|
||||
maddr: list[str] | None,
|
||||
loglevel: str,
|
||||
tl: bool,
|
||||
pdb: bool,
|
||||
# tsdb: bool,
|
||||
# es: bool,
|
||||
):
|
||||
'''
|
||||
Start the "root service actor", `pikerd`, run it until
|
||||
cancellation.
|
||||
|
||||
This "root daemon" operates as the top most service-mngr and
|
||||
subsys-as-subactor supervisor, think of it as the "init proc" of
|
||||
any of any `piker` application or daemon-process tree.
|
||||
Spawn the piker broker-daemon.
|
||||
|
||||
'''
|
||||
# from tractor.devx import maybe_open_crash_handler
|
||||
# with maybe_open_crash_handler(pdb=False):
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
with_tractor_log=tl,
|
||||
)
|
||||
from tractor.devx import maybe_open_crash_handler
|
||||
with maybe_open_crash_handler(pdb=pdb):
|
||||
log = get_console_log(loglevel, name='cli')
|
||||
|
||||
if pdb:
|
||||
log.warning((
|
||||
"\n"
|
||||
"!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
|
||||
"When a `piker` daemon crashes it will block the "
|
||||
"task-thread until resumed from console!\n"
|
||||
"\n"
|
||||
))
|
||||
|
||||
# service-actor registry endpoint socket-address set
|
||||
regaddrs: list[tuple[str, int]] = []
|
||||
|
||||
conf, _ = config.load(
|
||||
conf_name='conf',
|
||||
)
|
||||
network: dict = conf.get('network')
|
||||
if (
|
||||
network is None
|
||||
and not maddr
|
||||
):
|
||||
regaddrs = [(
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)]
|
||||
|
||||
else:
|
||||
eps: dict = load_trans_eps(
|
||||
network,
|
||||
maddr,
|
||||
)
|
||||
for layers in eps['pikerd']:
|
||||
regaddrs.append((
|
||||
layers['ipv4']['addr'],
|
||||
layers['tcp']['port'],
|
||||
if pdb:
|
||||
log.warning((
|
||||
"\n"
|
||||
"!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
|
||||
"When a `piker` daemon crashes it will block the "
|
||||
"task-thread until resumed from console!\n"
|
||||
"\n"
|
||||
))
|
||||
|
||||
from .. import service
|
||||
# service-actor registry endpoint socket-address set
|
||||
regaddrs: list[tuple[str, int]] = []
|
||||
|
||||
async def main():
|
||||
service_mngr: service.Services
|
||||
async with (
|
||||
service.open_pikerd(
|
||||
registry_addrs=regaddrs,
|
||||
loglevel=loglevel,
|
||||
debug_mode=pdb,
|
||||
# enable_transports=['uds'],
|
||||
enable_transports=['tcp'],
|
||||
) as service_mngr,
|
||||
conf, _ = config.load(
|
||||
conf_name='conf',
|
||||
)
|
||||
network: dict = conf.get('network')
|
||||
if (
|
||||
network is None
|
||||
and not maddr
|
||||
):
|
||||
assert service_mngr
|
||||
# ?TODO? spawn all other sub-actor daemons according to
|
||||
# multiaddress endpoint spec defined by user config
|
||||
await trio.sleep_forever()
|
||||
regaddrs = [(
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)]
|
||||
|
||||
trio.run(main)
|
||||
else:
|
||||
eps: dict = load_trans_eps(
|
||||
network,
|
||||
maddr,
|
||||
)
|
||||
for layers in eps['pikerd']:
|
||||
regaddrs.append((
|
||||
layers['ipv4']['addr'],
|
||||
layers['tcp']['port'],
|
||||
))
|
||||
|
||||
from .. import service
|
||||
|
||||
async def main():
|
||||
service_mngr: service.Services
|
||||
|
||||
async with (
|
||||
service.open_pikerd(
|
||||
registry_addrs=regaddrs,
|
||||
loglevel=loglevel,
|
||||
debug_mode=pdb,
|
||||
|
||||
) as service_mngr, # normally delivers a ``Services`` handle
|
||||
|
||||
# AsyncExitStack() as stack,
|
||||
):
|
||||
# TODO: spawn all other sub-actor daemons according to
|
||||
# multiaddress endpoint spec defined by user config
|
||||
assert service_mngr
|
||||
|
||||
# if tsdb:
|
||||
# dname, conf = await stack.enter_async_context(
|
||||
# service.marketstore.start_ahab_daemon(
|
||||
# service_mngr,
|
||||
# loglevel=loglevel,
|
||||
# )
|
||||
# )
|
||||
# log.info(f'TSDB `{dname}` up with conf:\n{conf}')
|
||||
|
||||
# if es:
|
||||
# dname, conf = await stack.enter_async_context(
|
||||
# service.elastic.start_ahab_daemon(
|
||||
# service_mngr,
|
||||
# loglevel=loglevel,
|
||||
# )
|
||||
# )
|
||||
# log.info(f'DB `{dname}` up with conf:\n{conf}')
|
||||
|
||||
await trio.sleep_forever()
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
@click.group(context_settings=config._context_defaults)
|
||||
|
|
@ -234,14 +258,6 @@ def cli(
|
|||
regaddr: str,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
The "root" `piker`-cmd CLI endpoint.
|
||||
|
||||
NOTE, this def generally relies on and requires a sub-cmd to be
|
||||
provided by the user, OW only a `--help` msg (listing said
|
||||
subcmds) will be dumped to console.
|
||||
|
||||
'''
|
||||
if configdir is not None:
|
||||
assert os.path.isdir(configdir), f"`{configdir}` is not a valid path"
|
||||
config._override_config_dir(configdir)
|
||||
|
|
@ -300,93 +316,42 @@ def cli(
|
|||
@click.option('--tl', is_flag=True, help='Enable tractor logging')
|
||||
@click.argument('ports', nargs=-1, required=False)
|
||||
@click.pass_obj
|
||||
def services(
|
||||
config,
|
||||
tl: bool,
|
||||
ports: list[int],
|
||||
):
|
||||
'''
|
||||
List all `piker` "service deamons" to the console in
|
||||
a `json`-table which maps each actor's UID in the form,
|
||||
def services(config, tl, ports):
|
||||
|
||||
`{service_name}.{subservice_name}.{UUID}`
|
||||
|
||||
to its (primary) IPC server address.
|
||||
|
||||
(^TODO, should be its multiaddr form once we support it)
|
||||
|
||||
Note that by convention actors which operate as "headless"
|
||||
processes (those without GUIs/graphics, and which generally
|
||||
parent some noteworthy subsystem) are normally suffixed by
|
||||
a "d" such as,
|
||||
|
||||
- pikerd: the root runtime supervisor
|
||||
- brokerd: a broker-backend order ctl daemon
|
||||
- emsd: the internal dark-clearing and order routing daemon
|
||||
- datad: a data-provider-backend data feed daemon
|
||||
- samplerd: the real-time data sampling and clock-syncing daemon
|
||||
|
||||
"Headed units" are normally just given an obvious app-like name
|
||||
with subactors indexed by `.` such as,
|
||||
- chart: the primary modal charting iface, a Qt app
|
||||
- chart.fsp_0: a financial-sig-proc cascade instance which
|
||||
delivers graphics to a parent `chart` app.
|
||||
- polars_boi: some (presumably) `polars` using console app.
|
||||
|
||||
'''
|
||||
from piker.service import (
|
||||
from ..service import (
|
||||
open_piker_runtime,
|
||||
_default_registry_port,
|
||||
_default_registry_host,
|
||||
)
|
||||
|
||||
# !TODO, mk this to work with UDS!
|
||||
host: str = _default_registry_host
|
||||
host = _default_registry_host
|
||||
if not ports:
|
||||
ports: list[int] = [_default_registry_port]
|
||||
|
||||
addr = tractor._addr.wrap_address(
|
||||
addr=(host, ports[0])
|
||||
)
|
||||
ports = [_default_registry_port]
|
||||
|
||||
async def list_services():
|
||||
nonlocal host
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
name='service_query',
|
||||
loglevel=(
|
||||
config['loglevel']
|
||||
if tl
|
||||
else None
|
||||
),
|
||||
loglevel=config['loglevel'] if tl else None,
|
||||
),
|
||||
tractor.get_registry(
|
||||
addr=addr,
|
||||
tractor.get_arbiter(
|
||||
host=host,
|
||||
port=ports[0]
|
||||
) as portal
|
||||
):
|
||||
registry = await portal.run_from_ns(
|
||||
'self',
|
||||
'get_registry',
|
||||
)
|
||||
registry = await portal.run_from_ns('self', 'get_registry')
|
||||
json_d = {}
|
||||
for key, socket in registry.items():
|
||||
json_d[key] = f'{socket}'
|
||||
|
||||
host, port = socket
|
||||
json_d[key] = f'{host}:{port}'
|
||||
click.echo(f"{colorize_json(json_d)}")
|
||||
|
||||
trio.run(list_services)
|
||||
|
||||
|
||||
def _load_clis() -> None:
|
||||
'''
|
||||
Dynamically load and register all subsys CLI endpoints (at call
|
||||
time).
|
||||
|
||||
NOTE, obviously this is normally expected to be called at
|
||||
`import` time and implicitly relies on our use of various
|
||||
`click`/`typer` decorator APIs.
|
||||
|
||||
'''
|
||||
# from ..service import elastic # noqa
|
||||
from ..brokers import cli # noqa
|
||||
from ..ui import cli # noqa
|
||||
from ..watchlists import cli # noqa
|
||||
|
|
@ -396,5 +361,5 @@ def _load_clis() -> None:
|
|||
from ..accounting import cli # noqa
|
||||
|
||||
|
||||
# load all subsytem cli eps
|
||||
# load downstream cli modules
|
||||
_load_clis()
|
||||
|
|
|
|||
|
|
@ -41,13 +41,10 @@ from .log import get_logger
|
|||
log = get_logger('broker-config')
|
||||
|
||||
|
||||
# XXX NOTE: taken from `click`
|
||||
# |_https://github.com/pallets/click/blob/main/src/click/utils.py#L449
|
||||
#
|
||||
# (since apparently they have some super weirdness with SIGINT and
|
||||
# sudo.. no clue we're probably going to slowly just modify it to our
|
||||
# own version over time..)
|
||||
#
|
||||
# XXX NOTE: taken from ``click`` since apparently they have some
|
||||
# super weirdness with sigint and sudo..no clue
|
||||
# we're probably going to slowly just modify it to our own version over
|
||||
# time..
|
||||
def get_app_dir(
|
||||
app_name: str,
|
||||
roaming: bool = True,
|
||||
|
|
@ -107,15 +104,14 @@ def get_app_dir(
|
|||
# `tractor`) with the testing dir and check for it whenever we
|
||||
# detect `pytest` is being used (which it isn't under normal
|
||||
# operation).
|
||||
# if "pytest" in sys.modules:
|
||||
# import tractor
|
||||
# actor = tractor.current_actor(err_on_no_runtime=False)
|
||||
# if actor: # runtime is up
|
||||
# rvs = tractor._state._runtime_vars
|
||||
# import pdbp; pdbp.set_trace()
|
||||
# testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
|
||||
# assert testdirpath.exists(), 'piker test harness might be borked!?'
|
||||
# app_name = str(testdirpath)
|
||||
if "pytest" in sys.modules:
|
||||
import tractor
|
||||
actor = tractor.current_actor(err_on_no_runtime=False)
|
||||
if actor: # runtime is up
|
||||
rvs = tractor._state._runtime_vars
|
||||
testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
|
||||
assert testdirpath.exists(), 'piker test harness might be borked!?'
|
||||
app_name = str(testdirpath)
|
||||
|
||||
if platform.system() == 'Windows':
|
||||
key = "APPDATA" if roaming else "LOCALAPPDATA"
|
||||
|
|
@ -264,7 +260,7 @@ def load(
|
|||
MutableMapping,
|
||||
] = tomllib.loads,
|
||||
|
||||
touch_if_dne: bool = True,
|
||||
touch_if_dne: bool = False,
|
||||
|
||||
**tomlkws,
|
||||
|
||||
|
|
@ -273,7 +269,7 @@ def load(
|
|||
Load config file by name.
|
||||
|
||||
If desired config is not in the top level piker-user config path then
|
||||
pass the `path: Path` explicitly.
|
||||
pass the ``path: Path`` explicitly.
|
||||
|
||||
'''
|
||||
# create the $HOME/.config/piker dir if dne
|
||||
|
|
@ -288,8 +284,7 @@ def load(
|
|||
|
||||
if (
|
||||
not path.is_file()
|
||||
and
|
||||
touch_if_dne
|
||||
and touch_if_dne
|
||||
):
|
||||
# only do a template if no path provided,
|
||||
# just touch an empty file with same name.
|
||||
|
|
|
|||
|
|
@ -80,27 +80,20 @@ class Sampler:
|
|||
This non-instantiated type is meant to be a singleton within
|
||||
a `samplerd` actor-service spawned once by the user wishing to
|
||||
time-step-sample (real-time) quote feeds, see
|
||||
`.service.maybe_open_samplerd()` and the below
|
||||
`register_with_sampler()`.
|
||||
``.service.maybe_open_samplerd()`` and the below
|
||||
``register_with_sampler()``.
|
||||
|
||||
'''
|
||||
service_nursery: None|trio.Nursery = None
|
||||
service_nursery: None | trio.Nursery = None
|
||||
|
||||
# TODO: we could stick these in a composed type to avoid angering
|
||||
# the "i hate module scoped variables crowd" (yawn).
|
||||
# TODO: we could stick these in a composed type to avoid
|
||||
# angering the "i hate module scoped variables crowd" (yawn).
|
||||
ohlcv_shms: dict[float, list[ShmArray]] = {}
|
||||
|
||||
# holds one-task-per-sample-period tasks which are spawned as-needed by
|
||||
# data feed requests with a given detected time step usually from
|
||||
# history loading.
|
||||
incr_task_cs: trio.CancelScope|None = None
|
||||
|
||||
bcast_errors: tuple[Exception] = (
|
||||
trio.BrokenResourceError,
|
||||
trio.ClosedResourceError,
|
||||
trio.EndOfChannel,
|
||||
tractor.TransportClosed,
|
||||
)
|
||||
incr_task_cs: trio.CancelScope | None = None
|
||||
|
||||
# holds all the ``tractor.Context`` remote subscriptions for
|
||||
# a particular sample period increment event: all subscribers are
|
||||
|
|
@ -249,8 +242,8 @@ class Sampler:
|
|||
async def broadcast(
|
||||
self,
|
||||
period_s: float,
|
||||
time_stamp: float|None = None,
|
||||
info: dict|None = None,
|
||||
time_stamp: float | None = None,
|
||||
info: dict | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -265,15 +258,14 @@ class Sampler:
|
|||
subs: set
|
||||
last_ts, subs = pair
|
||||
|
||||
# NOTE, for debugging pub-sub issues
|
||||
# task = trio.lowlevel.current_task()
|
||||
# log.debug(
|
||||
# f'AlL-SUBS@{period_s!r}: {self.subscribers}\n'
|
||||
# f'PAIR: {pair}\n'
|
||||
# f'TASK: {task}: {id(task)}\n'
|
||||
# f'broadcasting {period_s} -> {last_ts}\n'
|
||||
# f'consumers: {subs}'
|
||||
# )
|
||||
task = trio.lowlevel.current_task()
|
||||
log.debug(
|
||||
f'SUBS {self.subscribers}\n'
|
||||
f'PAIR {pair}\n'
|
||||
f'TASK: {task}: {id(task)}\n'
|
||||
f'broadcasting {period_s} -> {last_ts}\n'
|
||||
# f'consumers: {subs}'
|
||||
)
|
||||
borked: set[MsgStream] = set()
|
||||
sent: set[MsgStream] = set()
|
||||
while True:
|
||||
|
|
@ -290,12 +282,12 @@ class Sampler:
|
|||
await stream.send(msg)
|
||||
sent.add(stream)
|
||||
|
||||
except self.bcast_errors as err:
|
||||
except (
|
||||
trio.BrokenResourceError,
|
||||
trio.ClosedResourceError
|
||||
):
|
||||
log.error(
|
||||
f'Connection dropped for IPC ctx due to,\n'
|
||||
f'{type(err)!r}\n'
|
||||
f'\n'
|
||||
f'{stream._ctx}'
|
||||
f'{stream._ctx.chan.uid} dropped connection'
|
||||
)
|
||||
borked.add(stream)
|
||||
else:
|
||||
|
|
@ -315,7 +307,7 @@ class Sampler:
|
|||
@classmethod
|
||||
async def broadcast_all(
|
||||
self,
|
||||
info: dict|None = None,
|
||||
info: dict | None = None,
|
||||
) -> None:
|
||||
|
||||
# NOTE: take a copy of subs since removals can happen
|
||||
|
|
@ -332,22 +324,14 @@ class Sampler:
|
|||
async def register_with_sampler(
|
||||
ctx: Context,
|
||||
period_s: float,
|
||||
shms_by_period: dict[float, dict]|None = None,
|
||||
shms_by_period: dict[float, dict] | None = None,
|
||||
|
||||
open_index_stream: bool = True, # open a 2way stream for sample step msgs?
|
||||
sub_for_broadcasts: bool = True, # sampler side to send step updates?
|
||||
loglevel: str|None = None,
|
||||
|
||||
) -> set[int]:
|
||||
) -> None:
|
||||
|
||||
get_console_log(
|
||||
level=(
|
||||
loglevel
|
||||
or
|
||||
tractor.current_actor().loglevel
|
||||
),
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(tractor.current_actor().loglevel)
|
||||
incr_was_started: bool = False
|
||||
|
||||
try:
|
||||
|
|
@ -372,12 +356,7 @@ async def register_with_sampler(
|
|||
|
||||
# insert the base 1s period (for OHLC style sampling) into
|
||||
# the increment buffer set to update and shift every second.
|
||||
if (
|
||||
shms_by_period is not None
|
||||
# and
|
||||
# feed_is_live.is_set()
|
||||
# ^TODO? pass it in instead?
|
||||
):
|
||||
if shms_by_period is not None:
|
||||
from ._sharedmem import (
|
||||
attach_shm_array,
|
||||
_Token,
|
||||
|
|
@ -391,17 +370,12 @@ async def register_with_sampler(
|
|||
readonly=False,
|
||||
)
|
||||
shms_by_period[period] = shm
|
||||
Sampler.ohlcv_shms.setdefault(
|
||||
period,
|
||||
[],
|
||||
).append(shm)
|
||||
Sampler.ohlcv_shms.setdefault(period, []).append(shm)
|
||||
|
||||
assert Sampler.ohlcv_shms
|
||||
|
||||
# unblock caller
|
||||
await ctx.started(
|
||||
set(Sampler.ohlcv_shms.keys())
|
||||
)
|
||||
await ctx.started(set(Sampler.ohlcv_shms.keys()))
|
||||
|
||||
if open_index_stream:
|
||||
try:
|
||||
|
|
@ -420,8 +394,7 @@ async def register_with_sampler(
|
|||
finally:
|
||||
if (
|
||||
sub_for_broadcasts
|
||||
and
|
||||
subs
|
||||
and subs
|
||||
):
|
||||
try:
|
||||
subs.remove(stream)
|
||||
|
|
@ -447,7 +420,7 @@ async def register_with_sampler(
|
|||
|
||||
async def spawn_samplerd(
|
||||
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
**extra_tractor_kwargs
|
||||
|
||||
) -> bool:
|
||||
|
|
@ -484,7 +457,6 @@ async def spawn_samplerd(
|
|||
register_with_sampler,
|
||||
period_s=1,
|
||||
sub_for_broadcasts=False,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
return True
|
||||
|
||||
|
|
@ -493,7 +465,8 @@ async def spawn_samplerd(
|
|||
|
||||
@acm
|
||||
async def maybe_open_samplerd(
|
||||
loglevel: str|None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) -> tractor.Portal: # noqa
|
||||
|
|
@ -518,13 +491,13 @@ async def maybe_open_samplerd(
|
|||
@acm
|
||||
async def open_sample_stream(
|
||||
period_s: float,
|
||||
shms_by_period: dict[float, dict]|None = None,
|
||||
shms_by_period: dict[float, dict] | None = None,
|
||||
open_index_stream: bool = True,
|
||||
sub_for_broadcasts: bool = True,
|
||||
loglevel: str|None = None,
|
||||
|
||||
# cache_key: str|None = None,
|
||||
# allow_new_sampler: bool = True,
|
||||
cache_key: str | None = None,
|
||||
allow_new_sampler: bool = True,
|
||||
|
||||
ensure_is_active: bool = False,
|
||||
|
||||
) -> AsyncIterator[dict[str, float]]:
|
||||
|
|
@ -553,15 +526,11 @@ async def open_sample_stream(
|
|||
# yield bistream
|
||||
# else:
|
||||
|
||||
ctx: tractor.Context
|
||||
shm_periods: set[int] # in `int`-seconds
|
||||
async with (
|
||||
# XXX: this should be singleton on a host,
|
||||
# a lone broker-daemon per provider should be
|
||||
# created for all practical purposes
|
||||
maybe_open_samplerd(
|
||||
loglevel=loglevel,
|
||||
) as portal,
|
||||
maybe_open_samplerd() as portal,
|
||||
|
||||
portal.open_context(
|
||||
register_with_sampler,
|
||||
|
|
@ -570,12 +539,11 @@ async def open_sample_stream(
|
|||
'shms_by_period': shms_by_period,
|
||||
'open_index_stream': open_index_stream,
|
||||
'sub_for_broadcasts': sub_for_broadcasts,
|
||||
'loglevel': loglevel,
|
||||
},
|
||||
) as (ctx, shm_periods)
|
||||
) as (ctx, first)
|
||||
):
|
||||
if ensure_is_active:
|
||||
assert len(shm_periods) > 1
|
||||
assert len(first) > 1
|
||||
|
||||
async with (
|
||||
ctx.open_stream(
|
||||
|
|
@ -593,7 +561,8 @@ async def open_sample_stream(
|
|||
|
||||
|
||||
async def sample_and_broadcast(
|
||||
bus: _FeedsBus,
|
||||
|
||||
bus: _FeedsBus, # noqa
|
||||
rt_shm: ShmArray,
|
||||
hist_shm: ShmArray,
|
||||
quote_stream: trio.abc.ReceiveChannel,
|
||||
|
|
@ -613,33 +582,11 @@ async def sample_and_broadcast(
|
|||
|
||||
overruns = Counter()
|
||||
|
||||
# NOTE, only used for debugging live-data-feed issues, though
|
||||
# this should be resolved more correctly in the future using the
|
||||
# new typed-msgspec feats of `tractor`!
|
||||
#
|
||||
# XXX, a multiline nested `dict` formatter (since rn quote-msgs
|
||||
# are just that).
|
||||
# pfmt: Callable[[str], str] = mk_repr()
|
||||
|
||||
# iterate stream delivered by broker
|
||||
async for quotes in quote_stream:
|
||||
# print(quotes)
|
||||
|
||||
# XXX WARNING XXX only enable for debugging bc ow can cost
|
||||
# ALOT of perf with HF-feedz!!!
|
||||
#
|
||||
# log.info(
|
||||
# 'Rx live quotes:\n'
|
||||
# f'{pfmt(quotes)}'
|
||||
# )
|
||||
|
||||
# TODO,
|
||||
# -[ ] `numba` or `cython`-nize this loop possibly?
|
||||
# |_alternatively could we do it in rust somehow by upacking
|
||||
# arrow msgs instead of using `msgspec`?
|
||||
# -[ ] use `msgspec.Struct` support in new typed-msging from
|
||||
# `tractor` to ensure only allowed msgs are transmitted?
|
||||
#
|
||||
# TODO: ``numba`` this!
|
||||
for broker_symbol, quote in quotes.items():
|
||||
# TODO: in theory you can send the IPC msg *before* writing
|
||||
# to the sharedmem array to decrease latency, however, that
|
||||
|
|
@ -712,21 +659,6 @@ async def sample_and_broadcast(
|
|||
sub_key: str = broker_symbol.lower()
|
||||
subs: set[Sub] = bus.get_subs(sub_key)
|
||||
|
||||
# TODO, figure out how to make this useful whilst
|
||||
# incoporating feed "pausing" ..
|
||||
#
|
||||
# if not subs:
|
||||
# all_bs_fqmes: list[str] = list(
|
||||
# bus._subscribers.keys()
|
||||
# )
|
||||
# log.warning(
|
||||
# f'No subscribers for {brokername!r} live-quote ??\n'
|
||||
# f'broker_symbol: {broker_symbol}\n\n'
|
||||
|
||||
# f'Maybe the backend-sys symbol does not match one of,\n'
|
||||
# f'{pfmt(all_bs_fqmes)}\n'
|
||||
# )
|
||||
|
||||
# NOTE: by default the broker backend doesn't append
|
||||
# it's own "name" into the fqme schema (but maybe it
|
||||
# should?) so we have to manually generate the correct
|
||||
|
|
@ -765,8 +697,8 @@ async def sample_and_broadcast(
|
|||
|
||||
log.warning(
|
||||
f'Feed OVERRUN {sub_key}'
|
||||
f'@{bus.brokername} -> \n'
|
||||
f'feed @ {chan.aid.reprol()}\n'
|
||||
'@{bus.brokername} -> \n'
|
||||
f'feed @ {chan.uid}\n'
|
||||
f'throttle = {throttle} Hz'
|
||||
)
|
||||
|
||||
|
|
@ -796,14 +728,18 @@ async def sample_and_broadcast(
|
|||
if lags > 10:
|
||||
await tractor.pause()
|
||||
|
||||
except Sampler.bcast_errors as ipc_err:
|
||||
except (
|
||||
trio.BrokenResourceError,
|
||||
trio.ClosedResourceError,
|
||||
trio.EndOfChannel,
|
||||
):
|
||||
ctx: Context = ipc._ctx
|
||||
chan: Channel = ctx.chan
|
||||
if ctx:
|
||||
log.warning(
|
||||
f'Dropped `brokerd`-feed for {broker_symbol!r} due to,\n'
|
||||
f'x>) {ctx.cid}@{chan.uid}'
|
||||
f'|_{ipc_err!r}\n\n'
|
||||
'Dropped `brokerd`-quotes-feed connection:\n'
|
||||
f'{broker_symbol}:'
|
||||
f'{ctx.cid}@{chan.uid}'
|
||||
)
|
||||
if sub.throttle_rate:
|
||||
assert ipc._closed
|
||||
|
|
@ -820,11 +756,12 @@ async def sample_and_broadcast(
|
|||
|
||||
|
||||
async def uniform_rate_send(
|
||||
|
||||
rate: float,
|
||||
quote_stream: trio.abc.ReceiveChannel,
|
||||
stream: MsgStream,
|
||||
|
||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -842,16 +779,13 @@ async def uniform_rate_send(
|
|||
https://gist.github.com/njsmith/7ea44ec07e901cb78ebe1dd8dd846cb9
|
||||
|
||||
'''
|
||||
# ?TODO? dynamically compute the **actual** approx overhead latency per cycle
|
||||
# instead of this magic # bidinezz?
|
||||
throttle_period: float = 1/rate - 0.000616
|
||||
left_to_sleep: float = throttle_period
|
||||
# TODO: compute the approx overhead latency per cycle
|
||||
left_to_sleep = throttle_period = 1/rate - 0.000616
|
||||
|
||||
# send cycle state
|
||||
first_quote: dict|None
|
||||
first_quote = last_quote = None
|
||||
last_send: float = time.time()
|
||||
diff: float = 0
|
||||
last_send = time.time()
|
||||
diff = 0
|
||||
|
||||
task_status.started()
|
||||
ticks_by_type: dict[
|
||||
|
|
@ -862,28 +796,22 @@ async def uniform_rate_send(
|
|||
clear_types = _tick_groups['clears']
|
||||
|
||||
while True:
|
||||
|
||||
# compute the remaining time to sleep for this throttled cycle
|
||||
left_to_sleep: float = throttle_period - diff
|
||||
left_to_sleep = throttle_period - diff
|
||||
|
||||
if left_to_sleep > 0:
|
||||
cs: trio.CancelScope
|
||||
with trio.move_on_after(left_to_sleep) as cs:
|
||||
sym: str
|
||||
last_quote: dict
|
||||
try:
|
||||
sym, last_quote = await quote_stream.receive()
|
||||
except trio.EndOfChannel:
|
||||
log.exception(
|
||||
f'Live stream for feed for ended?\n'
|
||||
f'<=c\n'
|
||||
f' |_[{stream!r}\n'
|
||||
)
|
||||
log.exception(f"feed for {stream} ended?")
|
||||
break
|
||||
|
||||
diff: float = time.time() - last_send
|
||||
diff = time.time() - last_send
|
||||
|
||||
if not first_quote:
|
||||
first_quote: float = last_quote
|
||||
first_quote = last_quote
|
||||
# first_quote['tbt'] = ticks_by_type
|
||||
|
||||
if (throttle_period - diff) > 0:
|
||||
|
|
@ -944,9 +872,7 @@ async def uniform_rate_send(
|
|||
# TODO: now if only we could sync this to the display
|
||||
# rate timing exactly lul
|
||||
try:
|
||||
await stream.send({
|
||||
sym: first_quote
|
||||
})
|
||||
await stream.send({sym: first_quote})
|
||||
except tractor.RemoteActorError as rme:
|
||||
if rme.type is not tractor._exceptions.StreamOverrun:
|
||||
raise
|
||||
|
|
@ -957,28 +883,19 @@ async def uniform_rate_send(
|
|||
f'{sym}:{ctx.cid}@{chan.uid}'
|
||||
)
|
||||
|
||||
# NOTE: any of these can be raised by `tractor`'s IPC
|
||||
# transport-layer and we want to be highly resilient
|
||||
# to consumers which crash or lose network connection.
|
||||
# I.e. we **DO NOT** want to crash and propagate up to
|
||||
# ``pikerd`` these kinds of errors!
|
||||
except (
|
||||
# NOTE: any of these can be raised by ``tractor``'s IPC
|
||||
# transport-layer and we want to be highly resilient
|
||||
# to consumers which crash or lose network connection.
|
||||
# I.e. we **DO NOT** want to crash and propagate up to
|
||||
# ``pikerd`` these kinds of errors!
|
||||
trio.ClosedResourceError,
|
||||
trio.BrokenResourceError,
|
||||
ConnectionResetError,
|
||||
) + Sampler.bcast_errors as ipc_err:
|
||||
match ipc_err:
|
||||
case trio.EndOfChannel():
|
||||
log.info(
|
||||
f'{stream} terminated by peer,\n'
|
||||
f'{ipc_err!r}'
|
||||
)
|
||||
case _:
|
||||
# if the feed consumer goes down then drop
|
||||
# out of this rate limiter
|
||||
log.warning(
|
||||
f'{stream} closed due to,\n'
|
||||
f'{ipc_err!r}'
|
||||
)
|
||||
|
||||
):
|
||||
# if the feed consumer goes down then drop
|
||||
# out of this rate limiter
|
||||
log.warning(f'{stream} closed')
|
||||
await stream.aclose()
|
||||
return
|
||||
|
||||
|
|
|
|||
|
|
@ -520,12 +520,9 @@ def open_shm_array(
|
|||
|
||||
# "unlink" created shm on process teardown by
|
||||
# pushing teardown calls onto actor context stack
|
||||
stack = tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
).lifetime_stack
|
||||
if stack:
|
||||
stack.callback(shmarr.close)
|
||||
stack.callback(shmarr.destroy)
|
||||
stack = tractor.current_actor().lifetime_stack
|
||||
stack.callback(shmarr.close)
|
||||
stack.callback(shmarr.destroy)
|
||||
|
||||
return shmarr
|
||||
|
||||
|
|
@ -610,10 +607,7 @@ def attach_shm_array(
|
|||
_known_tokens[key] = token
|
||||
|
||||
# "close" attached shm on actor teardown
|
||||
if (actor := tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
)):
|
||||
actor.lifetime_stack.callback(sha.close)
|
||||
tractor.current_actor().lifetime_stack.callback(sha.close)
|
||||
|
||||
return sha
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,6 @@ from pathlib import Path
|
|||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
Sequence,
|
||||
Hashable,
|
||||
TYPE_CHECKING,
|
||||
|
|
@ -57,7 +56,7 @@ from piker.brokers import (
|
|||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.accounting import (
|
||||
from ..accounting import (
|
||||
Asset,
|
||||
MktPair,
|
||||
)
|
||||
|
|
@ -91,18 +90,6 @@ class SymbologyCache(Struct):
|
|||
# provided by the backend pkg.
|
||||
mktmaps: dict[str, MktPair] = field(default_factory=dict)
|
||||
|
||||
def pformat(self) -> str:
|
||||
return (
|
||||
f'<{type(self).__name__}(\n'
|
||||
f' .mod: {self.mod!r}\n'
|
||||
f' .assets: {len(self.assets)!r}\n'
|
||||
f' .pairs: {len(self.pairs)!r}\n'
|
||||
f' .mktmaps: {len(self.mktmaps)!r}\n'
|
||||
f')>'
|
||||
)
|
||||
|
||||
__repr__ = pformat
|
||||
|
||||
def write_config(self) -> None:
|
||||
|
||||
# put the backend's pair-struct type ref at the top
|
||||
|
|
@ -162,68 +149,57 @@ class SymbologyCache(Struct):
|
|||
'Implement `Client.get_assets()`!'
|
||||
)
|
||||
|
||||
get_mkt_pairs: Callable|None = getattr(
|
||||
client,
|
||||
'get_mkt_pairs',
|
||||
None,
|
||||
)
|
||||
if not get_mkt_pairs:
|
||||
if get_mkt_pairs := getattr(client, 'get_mkt_pairs', None):
|
||||
|
||||
pairs: dict[str, Struct] = await get_mkt_pairs()
|
||||
for bs_fqme, pair in pairs.items():
|
||||
|
||||
# NOTE: every backend defined pair should
|
||||
# declare it's ns path for roundtrip
|
||||
# serialization lookup.
|
||||
if not getattr(pair, 'ns_path', None):
|
||||
raise TypeError(
|
||||
f'Pair-struct for {self.mod.name} MUST define a '
|
||||
'`.ns_path: str`!\n'
|
||||
f'{pair}'
|
||||
)
|
||||
|
||||
entry = await self.mod.get_mkt_info(pair.bs_fqme)
|
||||
if not entry:
|
||||
continue
|
||||
|
||||
mkt: MktPair
|
||||
pair: Struct
|
||||
mkt, _pair = entry
|
||||
assert _pair is pair, (
|
||||
f'`{self.mod.name}` backend probably has a '
|
||||
'keying-symmetry problem between the pair-`Struct` '
|
||||
'returned from `Client.get_mkt_pairs()`and the '
|
||||
'module level endpoint: `.get_mkt_info()`\n\n'
|
||||
"Here's the struct diff:\n"
|
||||
f'{_pair - pair}'
|
||||
)
|
||||
# NOTE XXX: this means backends MUST implement
|
||||
# a `Struct.bs_mktid: str` field to provide
|
||||
# a native-keyed map to their own symbol
|
||||
# set(s).
|
||||
self.pairs[pair.bs_mktid] = pair
|
||||
|
||||
# NOTE: `MktPair`s are keyed here using piker's
|
||||
# internal FQME schema so that search,
|
||||
# accounting and feed init can be accomplished
|
||||
# a sane, uniform, normalized basis.
|
||||
self.mktmaps[mkt.fqme] = mkt
|
||||
|
||||
self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
|
||||
pair,
|
||||
)
|
||||
|
||||
else:
|
||||
log.warning(
|
||||
'No symbology cache `Pair` support for `{provider}`..\n'
|
||||
'Implement `Client.get_mkt_pairs()`!'
|
||||
)
|
||||
return self
|
||||
|
||||
pairs: dict[str, Struct] = await get_mkt_pairs()
|
||||
if not pairs:
|
||||
log.warning(
|
||||
'No pairs from intial {provider!r} sym-cache request?\n\n'
|
||||
'`Client.get_mkt_pairs()` -> {pairs!r} ?'
|
||||
)
|
||||
return self
|
||||
|
||||
for bs_fqme, pair in pairs.items():
|
||||
if not getattr(pair, 'ns_path', None):
|
||||
# XXX: every backend defined pair must declare
|
||||
# a `.ns_path: tractor.NamespacePath` to enable
|
||||
# roundtrip serialization lookup from a local
|
||||
# cache file.
|
||||
raise TypeError(
|
||||
f'Pair-struct for {self.mod.name} MUST define a '
|
||||
'`.ns_path: str`!\n\n'
|
||||
f'{pair!r}'
|
||||
)
|
||||
|
||||
entry = await self.mod.get_mkt_info(pair.bs_fqme)
|
||||
if not entry:
|
||||
continue
|
||||
|
||||
mkt: MktPair
|
||||
pair: Struct
|
||||
mkt, _pair = entry
|
||||
assert _pair is pair, (
|
||||
f'`{self.mod.name}` backend probably has a '
|
||||
'keying-symmetry problem between the pair-`Struct` '
|
||||
'returned from `Client.get_mkt_pairs()`and the '
|
||||
'module level endpoint: `.get_mkt_info()`\n\n'
|
||||
"Here's the struct diff:\n"
|
||||
f'{_pair - pair}'
|
||||
)
|
||||
# NOTE XXX: this means backends MUST implement
|
||||
# a `Struct.bs_mktid: str` field to provide
|
||||
# a native-keyed map to their own symbol
|
||||
# set(s).
|
||||
self.pairs[pair.bs_mktid] = pair
|
||||
|
||||
# NOTE: `MktPair`s are keyed here using piker's
|
||||
# internal FQME schema so that search,
|
||||
# accounting and feed init can be accomplished
|
||||
# a sane, uniform, normalized basis.
|
||||
self.mktmaps[mkt.fqme] = mkt
|
||||
|
||||
self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
|
||||
pair,
|
||||
)
|
||||
|
||||
return self
|
||||
|
||||
|
|
|
|||
|
|
@ -26,9 +26,7 @@ from ..log import (
|
|||
)
|
||||
subsys: str = 'piker.data'
|
||||
|
||||
log = get_logger(
|
||||
name=subsys,
|
||||
)
|
||||
log = get_logger(subsys)
|
||||
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
|
|
|
|||
|
|
@ -27,15 +27,14 @@ from functools import partial
|
|||
from types import ModuleType
|
||||
from typing import (
|
||||
Any,
|
||||
Optional,
|
||||
Callable,
|
||||
AsyncContextManager,
|
||||
AsyncGenerator,
|
||||
Iterable,
|
||||
Type,
|
||||
)
|
||||
import json
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
from trio_websocket import (
|
||||
|
|
@ -68,7 +67,7 @@ class NoBsWs:
|
|||
|
||||
'''
|
||||
# apparently we can QoS for all sorts of reasons..so catch em.
|
||||
recon_errors: tuple[Type[Exception]] = (
|
||||
recon_errors = (
|
||||
ConnectionClosed,
|
||||
DisconnectionTimeout,
|
||||
ConnectionRejected,
|
||||
|
|
@ -106,10 +105,7 @@ class NoBsWs:
|
|||
def connected(self) -> bool:
|
||||
return self._connected.is_set()
|
||||
|
||||
async def reset(
|
||||
self,
|
||||
timeout: float,
|
||||
) -> bool:
|
||||
async def reset(self) -> None:
|
||||
'''
|
||||
Reset the underlying ws connection by cancelling
|
||||
the bg relay task and waiting for it to signal
|
||||
|
|
@ -118,31 +114,18 @@ class NoBsWs:
|
|||
'''
|
||||
self._connected = trio.Event()
|
||||
self._cs.cancel()
|
||||
with trio.move_on_after(timeout) as cs:
|
||||
await self._connected.wait()
|
||||
return True
|
||||
|
||||
assert cs.cancelled_caught
|
||||
return False
|
||||
await self._connected.wait()
|
||||
|
||||
async def send_msg(
|
||||
self,
|
||||
data: Any,
|
||||
timeout: float = 3,
|
||||
) -> None:
|
||||
while True:
|
||||
try:
|
||||
msg: Any = self._dumps(data)
|
||||
return await self._ws.send_message(msg)
|
||||
except self.recon_errors:
|
||||
with trio.CancelScope(shield=True):
|
||||
reconnected: bool = await self.reset(
|
||||
timeout=timeout,
|
||||
)
|
||||
if not reconnected:
|
||||
log.warning(
|
||||
'Failed to reconnect after {timeout!r}s ??'
|
||||
)
|
||||
await self.reset()
|
||||
|
||||
async def recv_msg(self) -> Any:
|
||||
msg: Any = await self._rx.receive()
|
||||
|
|
@ -184,7 +167,7 @@ async def _reconnect_forever(
|
|||
|
||||
async def proxy_msgs(
|
||||
ws: WebSocketConnection,
|
||||
rent_cs: trio.CancelScope, # parent cancel scope
|
||||
pcs: trio.CancelScope, # parent cancel scope
|
||||
):
|
||||
'''
|
||||
Receive (under `timeout` deadline) all msgs from from underlying
|
||||
|
|
@ -208,10 +191,8 @@ async def _reconnect_forever(
|
|||
f'{src_mod}\n'
|
||||
f'{url} connection bail with:'
|
||||
)
|
||||
with trio.CancelScope(shield=True):
|
||||
await trio.sleep(0.5)
|
||||
|
||||
rent_cs.cancel()
|
||||
await trio.sleep(0.5)
|
||||
pcs.cancel()
|
||||
|
||||
# go back to reonnect loop in parent task
|
||||
return
|
||||
|
|
@ -223,7 +204,7 @@ async def _reconnect_forever(
|
|||
f'{src_mod}\n'
|
||||
'WS feed seems down and slow af.. reconnecting\n'
|
||||
)
|
||||
rent_cs.cancel()
|
||||
pcs.cancel()
|
||||
|
||||
# go back to reonnect loop in parent task
|
||||
return
|
||||
|
|
@ -247,12 +228,7 @@ async def _reconnect_forever(
|
|||
nobsws._connected = trio.Event()
|
||||
task_status.started()
|
||||
|
||||
mc_state: trio._channel.MemoryChannelState = snd._state
|
||||
while (
|
||||
mc_state.open_receive_channels > 0
|
||||
and
|
||||
mc_state.open_send_channels > 0
|
||||
):
|
||||
while not snd._closed:
|
||||
log.info(
|
||||
f'{src_mod}\n'
|
||||
f'{url} trying (RE)CONNECT'
|
||||
|
|
@ -261,11 +237,10 @@ async def _reconnect_forever(
|
|||
ws: WebSocketConnection
|
||||
try:
|
||||
async with (
|
||||
trio.open_nursery() as n,
|
||||
open_websocket_url(url) as ws,
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
cs = nobsws._cs = tn.cancel_scope
|
||||
cs = nobsws._cs = n.cancel_scope
|
||||
nobsws._ws = ws
|
||||
log.info(
|
||||
f'{src_mod}\n'
|
||||
|
|
@ -273,7 +248,7 @@ async def _reconnect_forever(
|
|||
)
|
||||
|
||||
# begin relay loop to forward msgs
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
proxy_msgs,
|
||||
ws,
|
||||
cs,
|
||||
|
|
@ -287,7 +262,7 @@ async def _reconnect_forever(
|
|||
|
||||
# TODO: should we return an explicit sub-cs
|
||||
# from this fixture task?
|
||||
await tn.start(
|
||||
await n.start(
|
||||
open_fixture,
|
||||
fixture,
|
||||
nobsws,
|
||||
|
|
@ -297,24 +272,11 @@ async def _reconnect_forever(
|
|||
# to let tasks run **inside** the ws open block above.
|
||||
nobsws._connected.set()
|
||||
await trio.sleep_forever()
|
||||
except HandshakeError:
|
||||
log.exception(f'Retrying connection')
|
||||
|
||||
except (
|
||||
HandshakeError,
|
||||
ConnectionRejected,
|
||||
):
|
||||
log.exception('Retrying connection')
|
||||
await trio.sleep(0.5) # throttle
|
||||
# ws & nursery block ends
|
||||
|
||||
except BaseException as _berr:
|
||||
berr = _berr
|
||||
log.exception(
|
||||
'Reconnect-attempt failed ??\n'
|
||||
)
|
||||
with trio.CancelScope(shield=True):
|
||||
await trio.sleep(0.2) # throttle
|
||||
raise berr
|
||||
|
||||
#|_ws & nursery block ends
|
||||
nobsws._connected = trio.Event()
|
||||
if cs.cancelled_caught:
|
||||
log.cancel(
|
||||
|
|
@ -362,53 +324,43 @@ async def open_autorecon_ws(
|
|||
connetivity errors, or some user defined recv timeout.
|
||||
|
||||
You can provide a ``fixture`` async-context-manager which will be
|
||||
entered/exitted around each connection reset; eg. for
|
||||
(re)requesting subscriptions without requiring streaming setup
|
||||
code to rerun.
|
||||
entered/exitted around each connection reset; eg. for (re)requesting
|
||||
subscriptions without requiring streaming setup code to rerun.
|
||||
|
||||
'''
|
||||
snd: trio.MemorySendChannel
|
||||
rcv: trio.MemoryReceiveChannel
|
||||
snd, rcv = trio.open_memory_channel(616)
|
||||
|
||||
try:
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
nobsws = NoBsWs(
|
||||
url,
|
||||
rcv,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
)
|
||||
await tn.start(
|
||||
partial(
|
||||
_reconnect_forever,
|
||||
url,
|
||||
snd,
|
||||
nobsws,
|
||||
fixture=fixture,
|
||||
reset_after=reset_after,
|
||||
)
|
||||
)
|
||||
await nobsws._connected.wait()
|
||||
assert nobsws._cs
|
||||
assert nobsws.connected()
|
||||
try:
|
||||
yield nobsws
|
||||
finally:
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
except NoBsWs.recon_errors as con_err:
|
||||
log.warning(
|
||||
f'Entire ws-channel disconnect due to,\n'
|
||||
f'con_err: {con_err!r}\n'
|
||||
async with trio.open_nursery() as n:
|
||||
nobsws = NoBsWs(
|
||||
url,
|
||||
rcv,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
)
|
||||
await n.start(
|
||||
partial(
|
||||
_reconnect_forever,
|
||||
url,
|
||||
snd,
|
||||
nobsws,
|
||||
fixture=fixture,
|
||||
reset_after=reset_after,
|
||||
)
|
||||
)
|
||||
await nobsws._connected.wait()
|
||||
assert nobsws._cs
|
||||
assert nobsws.connected()
|
||||
|
||||
try:
|
||||
yield nobsws
|
||||
finally:
|
||||
n.cancel_scope.cancel()
|
||||
|
||||
|
||||
'''
|
||||
JSONRPC response-request style machinery for transparent multiplexing
|
||||
of msgs over a `NoBsWs`.
|
||||
JSONRPC response-request style machinery for transparent multiplexing of msgs
|
||||
over a NoBsWs.
|
||||
|
||||
'''
|
||||
|
||||
|
|
@ -416,8 +368,8 @@ of msgs over a `NoBsWs`.
|
|||
class JSONRPCResult(Struct):
|
||||
id: int
|
||||
jsonrpc: str = '2.0'
|
||||
result: dict|None = None
|
||||
error: dict|None = None
|
||||
result: Optional[dict] = None
|
||||
error: Optional[dict] = None
|
||||
|
||||
|
||||
@acm
|
||||
|
|
@ -425,82 +377,43 @@ async def open_jsonrpc_session(
|
|||
url: str,
|
||||
start_id: int = 0,
|
||||
response_type: type = JSONRPCResult,
|
||||
msg_recv_timeout: float = float('inf'),
|
||||
# ^NOTE, since only `deribit` is using this jsonrpc stuff atm
|
||||
# and options mkts are generally "slow moving"..
|
||||
#
|
||||
# FURTHER if we break the underlying ws connection then since we
|
||||
# don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
|
||||
# `_reconnect_forever()`, the jsonrpc "transport pipe" get's
|
||||
# broken and never restored with wtv init sequence is required to
|
||||
# re-establish a working req-resp session.
|
||||
|
||||
request_type: Optional[type] = None,
|
||||
request_hook: Optional[Callable] = None,
|
||||
error_hook: Optional[Callable] = None,
|
||||
) -> Callable[[str, dict], dict]:
|
||||
'''
|
||||
Init a json-RPC-over-websocket connection to the provided `url`.
|
||||
|
||||
A `json_rpc: Callable[[str, dict], dict` is delivered to the
|
||||
caller for sending requests and a bg-`trio.Task` handles
|
||||
processing of response msgs including error reporting/raising in
|
||||
the parent/caller task.
|
||||
|
||||
'''
|
||||
# NOTE, store all request msgs so we can raise errors on the
|
||||
# caller side!
|
||||
req_msgs: dict[int, dict] = {}
|
||||
|
||||
async with (
|
||||
trio.open_nursery() as tn,
|
||||
open_autorecon_ws(
|
||||
url=url,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
) as ws
|
||||
trio.open_nursery() as n,
|
||||
open_autorecon_ws(url) as ws
|
||||
):
|
||||
rpc_id: Iterable[int] = count(start_id)
|
||||
rpc_id: Iterable = count(start_id)
|
||||
rpc_results: dict[int, dict] = {}
|
||||
|
||||
async def json_rpc(
|
||||
method: str,
|
||||
params: dict,
|
||||
) -> dict:
|
||||
async def json_rpc(method: str, params: dict) -> dict:
|
||||
'''
|
||||
perform a json rpc call and wait for the result, raise exception in
|
||||
case of error field present on response
|
||||
'''
|
||||
nonlocal req_msgs
|
||||
|
||||
req_id: int = next(rpc_id)
|
||||
msg = {
|
||||
'jsonrpc': '2.0',
|
||||
'id': req_id,
|
||||
'id': next(rpc_id),
|
||||
'method': method,
|
||||
'params': params
|
||||
}
|
||||
_id = msg['id']
|
||||
|
||||
result = rpc_results[_id] = {
|
||||
rpc_results[_id] = {
|
||||
'result': None,
|
||||
'error': None,
|
||||
'event': trio.Event(), # signal caller resp arrived
|
||||
'event': trio.Event()
|
||||
}
|
||||
req_msgs[_id] = msg
|
||||
|
||||
await ws.send_msg(msg)
|
||||
|
||||
# wait for reponse before unblocking requester code
|
||||
await rpc_results[_id]['event'].wait()
|
||||
|
||||
if (maybe_result := result['result']):
|
||||
ret = maybe_result
|
||||
del rpc_results[_id]
|
||||
ret = rpc_results[_id]['result']
|
||||
|
||||
else:
|
||||
err = result['error']
|
||||
raise Exception(
|
||||
f'JSONRPC request failed\n'
|
||||
f'req: {msg}\n'
|
||||
f'resp: {err}\n'
|
||||
)
|
||||
del rpc_results[_id]
|
||||
|
||||
if ret.error is not None:
|
||||
raise Exception(json.dumps(ret.error, indent=4))
|
||||
|
|
@ -515,7 +428,6 @@ async def open_jsonrpc_session(
|
|||
the server side.
|
||||
|
||||
'''
|
||||
nonlocal req_msgs
|
||||
async for msg in ws:
|
||||
match msg:
|
||||
case {
|
||||
|
|
@ -539,28 +451,19 @@ async def open_jsonrpc_session(
|
|||
'params': _,
|
||||
}:
|
||||
log.debug(f'Recieved\n{msg}')
|
||||
if request_hook:
|
||||
await request_hook(request_type(**msg))
|
||||
|
||||
case {
|
||||
'error': error
|
||||
}:
|
||||
# retreive orig request msg, set error
|
||||
# response in original "result" msg,
|
||||
# THEN FINALLY set the event to signal caller
|
||||
# to raise the error in the parent task.
|
||||
req_id: int = error['id']
|
||||
req_msg: dict = req_msgs[req_id]
|
||||
result: dict = rpc_results[req_id]
|
||||
result['error'] = error
|
||||
result['event'].set()
|
||||
log.error(
|
||||
f'JSONRPC request failed\n'
|
||||
f'req: {req_msg}\n'
|
||||
f'resp: {error}\n'
|
||||
)
|
||||
log.warning(f'Recieved\n{error}')
|
||||
if error_hook:
|
||||
await error_hook(response_type(**msg))
|
||||
|
||||
case _:
|
||||
log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
|
||||
|
||||
tn.start_soon(recv_task)
|
||||
n.start_soon(recv_task)
|
||||
yield json_rpc
|
||||
tn.cancel_scope.cancel()
|
||||
n.cancel_scope.cancel()
|
||||
|
|
|
|||
|
|
@ -39,7 +39,6 @@ from typing import (
|
|||
AsyncContextManager,
|
||||
Awaitable,
|
||||
Sequence,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import trio
|
||||
|
|
@ -62,6 +61,7 @@ from ._util import (
|
|||
log,
|
||||
get_console_log,
|
||||
)
|
||||
from .flows import Flume
|
||||
from .validate import (
|
||||
FeedInit,
|
||||
validate_backend,
|
||||
|
|
@ -75,11 +75,6 @@ from ._sampling import (
|
|||
uniform_rate_send,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .flows import Flume
|
||||
from tractor._addr import Address
|
||||
from tractor.msg.types import Aid
|
||||
|
||||
|
||||
class Sub(Struct, frozen=True):
|
||||
'''
|
||||
|
|
@ -239,6 +234,7 @@ async def allocate_persistent_feed(
|
|||
|
||||
brokername: str,
|
||||
symstr: str,
|
||||
|
||||
loglevel: str,
|
||||
start_stream: bool = True,
|
||||
init_timeout: float = 616,
|
||||
|
|
@ -277,7 +273,7 @@ async def allocate_persistent_feed(
|
|||
# ``stream_quotes()``, a required broker backend endpoint.
|
||||
init_msgs: (
|
||||
list[FeedInit] # new
|
||||
|dict[str, dict[str, str]] # legacy / deprecated
|
||||
| dict[str, dict[str, str]] # legacy / deprecated
|
||||
)
|
||||
|
||||
# TODO: probably make a struct msg type for this as well
|
||||
|
|
@ -347,25 +343,18 @@ async def allocate_persistent_feed(
|
|||
izero_rt,
|
||||
rt_shm,
|
||||
) = await bus.nursery.start(
|
||||
partial(
|
||||
manage_history,
|
||||
mod=mod,
|
||||
mkt=mkt,
|
||||
some_data_ready=some_data_ready,
|
||||
feed_is_live=feed_is_live,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
manage_history,
|
||||
mod,
|
||||
mkt,
|
||||
some_data_ready,
|
||||
feed_is_live,
|
||||
)
|
||||
|
||||
# yield back control to starting nursery once we receive either
|
||||
# some history or a real-time quote.
|
||||
log.info(
|
||||
f'loading OHLCV history: {fqme!r}\n'
|
||||
)
|
||||
log.info(f'loading OHLCV history: {fqme}')
|
||||
await some_data_ready.wait()
|
||||
|
||||
# XXX, avoid cycle; it imports this mod.
|
||||
from .flows import Flume
|
||||
flume = Flume(
|
||||
|
||||
# TODO: we have to use this for now since currently the
|
||||
|
|
@ -462,6 +451,7 @@ async def allocate_persistent_feed(
|
|||
|
||||
@tractor.context
|
||||
async def open_feed_bus(
|
||||
|
||||
ctx: tractor.Context,
|
||||
brokername: str,
|
||||
symbols: list[str], # normally expected to the broker-specific fqme
|
||||
|
|
@ -482,16 +472,13 @@ async def open_feed_bus(
|
|||
|
||||
'''
|
||||
if loglevel is None:
|
||||
loglevel: str = tractor.current_actor().loglevel
|
||||
loglevel = tractor.current_actor().loglevel
|
||||
|
||||
# XXX: required to propagate ``tractor`` loglevel to piker
|
||||
# logging
|
||||
get_console_log(
|
||||
level=(loglevel
|
||||
or
|
||||
tractor.current_actor().loglevel
|
||||
),
|
||||
name=__name__,
|
||||
loglevel
|
||||
or tractor.current_actor().loglevel
|
||||
)
|
||||
|
||||
# local state sanity checks
|
||||
|
|
@ -506,6 +493,7 @@ async def open_feed_bus(
|
|||
sub_registered = trio.Event()
|
||||
|
||||
flumes: dict[str, Flume] = {}
|
||||
|
||||
for symbol in symbols:
|
||||
|
||||
# if no cached feed for this symbol has been created for this
|
||||
|
|
@ -689,7 +677,6 @@ class Feed(Struct):
|
|||
'''
|
||||
mods: dict[str, ModuleType] = {}
|
||||
portals: dict[ModuleType, tractor.Portal] = {}
|
||||
|
||||
flumes: dict[
|
||||
str, # FQME
|
||||
Flume,
|
||||
|
|
@ -736,10 +723,7 @@ class Feed(Struct):
|
|||
async for msg in stream:
|
||||
await tx.send(msg)
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as nurse
|
||||
):
|
||||
async with trio.open_nursery() as nurse:
|
||||
# spawn a relay task for each stream so that they all
|
||||
# multiplex to a common channel.
|
||||
for brokername in mods:
|
||||
|
|
@ -802,8 +786,9 @@ async def install_brokerd_search(
|
|||
|
||||
@acm
|
||||
async def maybe_open_feed(
|
||||
|
||||
fqmes: list[str],
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
**kwargs,
|
||||
|
||||
|
|
@ -855,12 +840,13 @@ async def maybe_open_feed(
|
|||
|
||||
@acm
|
||||
async def open_feed(
|
||||
|
||||
fqmes: list[str],
|
||||
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
allow_overruns: bool = True,
|
||||
start_stream: bool = True,
|
||||
tick_throttle: float|None = None, # Hz
|
||||
tick_throttle: float | None = None, # Hz
|
||||
|
||||
allow_remote_ctl_ui: bool = False,
|
||||
|
||||
|
|
@ -887,6 +873,7 @@ async def open_feed(
|
|||
|
||||
# one actor per brokerd for now
|
||||
brokerd_ctxs = []
|
||||
|
||||
for brokermod, bfqmes in providers.items():
|
||||
|
||||
# if no `brokerd` for this backend exists yet we spawn
|
||||
|
|
@ -912,19 +899,19 @@ async def open_feed(
|
|||
feed.portals[brokermod] = portal
|
||||
|
||||
# fill out "status info" that the UI can show
|
||||
chan: tractor.Channel = portal.chan
|
||||
raddr: Address = chan.raddr
|
||||
aid: Aid = chan.aid
|
||||
# TAG_feed_status_update
|
||||
host, port = portal.channel.raddr
|
||||
if host == '127.0.0.1':
|
||||
host = 'localhost'
|
||||
|
||||
feed.status.update({
|
||||
'actor_id': aid,
|
||||
'actor_short_id': f'{aid.name}@{aid.pid}',
|
||||
'ipc': chan.raddr.proto_key,
|
||||
'ipc_addr': raddr,
|
||||
'actor_name': portal.channel.uid[0],
|
||||
'host': host,
|
||||
'port': port,
|
||||
'hist_shm': 'NA',
|
||||
'rt_shm': 'NA',
|
||||
'throttle_hz': tick_throttle,
|
||||
'throttle_rate': tick_throttle,
|
||||
})
|
||||
# feed.status.update(init_msg.pop('status', {}))
|
||||
|
||||
# (allocate and) connect to any feed bus for this broker
|
||||
bus_ctxs.append(
|
||||
|
|
@ -956,8 +943,6 @@ async def open_feed(
|
|||
|
||||
assert len(feed.mods) == len(feed.portals)
|
||||
|
||||
# XXX, avoid cycle; it imports this mod.
|
||||
from .flows import Flume
|
||||
async with (
|
||||
trionics.gather_contexts(bus_ctxs) as ctxs,
|
||||
):
|
||||
|
|
|
|||
|
|
@ -36,10 +36,10 @@ from ._sharedmem import (
|
|||
ShmArray,
|
||||
_Token,
|
||||
)
|
||||
from piker.accounting import MktPair
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.data.feed import Feed
|
||||
from ..accounting import MktPair
|
||||
from .feed import Feed
|
||||
|
||||
|
||||
class Flume(Struct):
|
||||
|
|
@ -82,7 +82,7 @@ class Flume(Struct):
|
|||
|
||||
# TODO: do we need this really if we can pull the `Portal` from
|
||||
# ``tractor``'s internals?
|
||||
feed: Feed|None = None
|
||||
feed: Feed | None = None
|
||||
|
||||
@property
|
||||
def rt_shm(self) -> ShmArray:
|
||||
|
|
|
|||
|
|
@ -113,9 +113,9 @@ def validate_backend(
|
|||
)
|
||||
if ep is None:
|
||||
log.warning(
|
||||
f'Provider backend {mod.name!r} is missing '
|
||||
f'{daemon_name!r} support?\n'
|
||||
f'|_module endpoint-func missing: {name!r}\n'
|
||||
f'Provider backend {mod.name} is missing '
|
||||
f'{daemon_name} support :(\n'
|
||||
f'The following endpoint is missing: {name}'
|
||||
)
|
||||
|
||||
inits: list[
|
||||
|
|
|
|||
|
|
@ -24,7 +24,6 @@ from functools import partial
|
|||
from typing import (
|
||||
AsyncIterator,
|
||||
Callable,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import numpy as np
|
||||
|
|
@ -34,12 +33,12 @@ import tractor
|
|||
from tractor.msg import NamespacePath
|
||||
|
||||
from piker.types import Struct
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from ..log import get_logger, get_console_log
|
||||
from .. import data
|
||||
from ..data.flows import Flume
|
||||
from ..data.feed import (
|
||||
Flume,
|
||||
Feed,
|
||||
)
|
||||
from ..data._sharedmem import ShmArray
|
||||
from ..data._sampling import (
|
||||
_default_delay_s,
|
||||
|
|
@ -53,9 +52,6 @@ from ._api import (
|
|||
)
|
||||
from ..toolz import Profiler
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..data.feed import Feed
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
|
|
@ -173,10 +169,8 @@ class Cascade(Struct):
|
|||
if not synced:
|
||||
fsp: Fsp = self.fsp
|
||||
log.warning(
|
||||
f'***DESYNCED fsp***\n'
|
||||
f'------------------\n'
|
||||
f'ns-path: {fsp.ns_path!r}\n'
|
||||
f'shm-token: {src_shm.token}\n'
|
||||
'***DESYNCED FSP***\n'
|
||||
f'{fsp.ns_path}@{src_shm.token}\n'
|
||||
f'step_diff: {step_diff}\n'
|
||||
f'len_diff: {len_diff}\n'
|
||||
)
|
||||
|
|
@ -404,6 +398,7 @@ async def connect_streams(
|
|||
|
||||
@tractor.context
|
||||
async def cascade(
|
||||
|
||||
ctx: tractor.Context,
|
||||
|
||||
# data feed key
|
||||
|
|
@ -417,7 +412,7 @@ async def cascade(
|
|||
shm_registry: dict[str, _Token],
|
||||
|
||||
zero_on_step: bool = False,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -431,17 +426,7 @@ async def cascade(
|
|||
)
|
||||
|
||||
if loglevel:
|
||||
log = get_console_log(
|
||||
loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
# XXX TODO!
|
||||
# figure out why this writes a dict to,
|
||||
# `tractor._state._runtime_vars['_root_mailbox']`
|
||||
# XD .. wtf
|
||||
# TODO, solve this as reported in,
|
||||
# https://www.pikers.dev/pikers/piker/issues/70
|
||||
# await tractor.pause()
|
||||
get_console_log(loglevel)
|
||||
|
||||
src: Flume = Flume.from_msg(src_flume_addr)
|
||||
dst: Flume = Flume.from_msg(
|
||||
|
|
@ -484,8 +469,7 @@ async def cascade(
|
|||
# open a data feed stream with requested broker
|
||||
feed: Feed
|
||||
async with data.feed.maybe_open_feed(
|
||||
fqmes=[fqme],
|
||||
loglevel=loglevel,
|
||||
[fqme],
|
||||
|
||||
# TODO throttle tick outputs from *this* daemon since
|
||||
# it'll emit tons of ticks due to the throttle only
|
||||
|
|
@ -514,7 +498,6 @@ async def cascade(
|
|||
|
||||
func_name: str = func.__name__
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(), # avoid multi-taskc tb in console
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
# TODO: might be better to just make a "restart" method where
|
||||
|
|
@ -583,8 +566,7 @@ async def cascade(
|
|||
# on every step msg received from the global `samplerd`
|
||||
# service.
|
||||
async with open_sample_stream(
|
||||
period_s=float(delay_s),
|
||||
loglevel=loglevel,
|
||||
float(delay_s)
|
||||
) as istream:
|
||||
|
||||
profiler(f'{func_name}: sample stream up')
|
||||
|
|
|
|||
101
piker/log.py
101
piker/log.py
|
|
@ -19,10 +19,6 @@ Log like a forester!
|
|||
"""
|
||||
import logging
|
||||
import json
|
||||
import reprlib
|
||||
from typing import (
|
||||
Callable,
|
||||
)
|
||||
|
||||
import tractor
|
||||
from pygments import (
|
||||
|
|
@ -37,84 +33,35 @@ _proj_name: str = 'piker'
|
|||
|
||||
|
||||
def get_logger(
|
||||
name: str|None = None,
|
||||
**tractor_log_kwargs,
|
||||
name: str = None,
|
||||
|
||||
) -> logging.Logger:
|
||||
'''
|
||||
Return the package log or a sub-logger if a `name=` is provided,
|
||||
which defaults to the calling module's pkg-namespace path.
|
||||
|
||||
See `tractor.log.get_logger()` for details.
|
||||
Return the package log or a sub-log for `name` if provided.
|
||||
|
||||
'''
|
||||
pkg_name: str = _proj_name
|
||||
if (
|
||||
name
|
||||
and
|
||||
pkg_name in name
|
||||
):
|
||||
name: str = name.lstrip(f'{_proj_name}.')
|
||||
|
||||
return tractor.log.get_logger(
|
||||
name=name,
|
||||
pkg_name=pkg_name,
|
||||
**tractor_log_kwargs,
|
||||
_root_name=_proj_name,
|
||||
)
|
||||
|
||||
|
||||
def get_console_log(
|
||||
level: str|None = None,
|
||||
name: str|None = None,
|
||||
pkg_name: str|None = None,
|
||||
with_tractor_log: bool = False,
|
||||
# ?TODO, support a "log-spec" style `str|dict[str, str]` which
|
||||
# dictates both the sublogger-key and a level?
|
||||
# -> see similar idea in `modden`'s usage.
|
||||
**tractor_log_kwargs,
|
||||
level: str | None = None,
|
||||
name: str | None = None,
|
||||
|
||||
) -> logging.Logger:
|
||||
'''
|
||||
Get the package logger and enable a handler which writes to
|
||||
stderr.
|
||||
Get the package logger and enable a handler which writes to stderr.
|
||||
|
||||
Yeah yeah, i know we can use `DictConfig`.
|
||||
You do it.. Bp
|
||||
Yeah yeah, i know we can use ``DictConfig``. You do it...
|
||||
|
||||
'''
|
||||
pkg_name: str = _proj_name
|
||||
if (
|
||||
name
|
||||
and
|
||||
pkg_name in name
|
||||
):
|
||||
name: str = name.lstrip(f'{_proj_name}.')
|
||||
|
||||
tll: str|None = None
|
||||
if (
|
||||
with_tractor_log is not False
|
||||
):
|
||||
tll = level
|
||||
|
||||
elif maybe_actor := tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
):
|
||||
tll = maybe_actor.loglevel
|
||||
|
||||
if tll:
|
||||
t_log = tractor.log.get_console_log(
|
||||
level=tll,
|
||||
name='tractor', # <- XXX, force root tractor log!
|
||||
**tractor_log_kwargs,
|
||||
)
|
||||
# TODO/ allow only enabling certain tractor sub-logs?
|
||||
assert t_log.name == 'tractor'
|
||||
|
||||
return tractor.log.get_console_log(
|
||||
level=level,
|
||||
level,
|
||||
name=name,
|
||||
pkg_name=pkg_name,
|
||||
**tractor_log_kwargs,
|
||||
)
|
||||
_root_name=_proj_name,
|
||||
) # our root logger
|
||||
|
||||
|
||||
def colorize_json(
|
||||
|
|
@ -137,29 +84,3 @@ def colorize_json(
|
|||
# likeable styles: algol_nu, tango, monokai
|
||||
formatters.TerminalTrueColorFormatter(style=style)
|
||||
)
|
||||
|
||||
|
||||
# TODO, eventually defer to the version in `modden` once
|
||||
# it becomes a dep!
|
||||
def mk_repr(
|
||||
**repr_kws,
|
||||
) -> Callable[[str], str]:
|
||||
'''
|
||||
Allocate and deliver a `repr.Repr` instance with provided input
|
||||
settings using the std-lib's `reprlib` mod,
|
||||
* https://docs.python.org/3/library/reprlib.html
|
||||
|
||||
------ Ex. ------
|
||||
An up to 6-layer-nested `dict` as multi-line:
|
||||
- https://stackoverflow.com/a/79102479
|
||||
- https://docs.python.org/3/library/reprlib.html#reprlib.Repr.maxlevel
|
||||
|
||||
'''
|
||||
def_kws: dict[str, int] = dict(
|
||||
indent=2,
|
||||
maxlevel=6, # recursion levels
|
||||
maxstring=66, # match editor line-len limit
|
||||
)
|
||||
def_kws |= repr_kws
|
||||
reprr = reprlib.Repr(**def_kws)
|
||||
return reprr.repr
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@
|
|||
from __future__ import annotations
|
||||
import os
|
||||
from typing import (
|
||||
Optional,
|
||||
Any,
|
||||
ClassVar,
|
||||
)
|
||||
|
|
@ -31,11 +32,8 @@ from contextlib import (
|
|||
import tractor
|
||||
import trio
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
)
|
||||
from ._util import (
|
||||
subsys,
|
||||
get_console_log,
|
||||
)
|
||||
from ._mngr import (
|
||||
Services,
|
||||
|
|
@ -61,7 +59,7 @@ async def open_piker_runtime(
|
|||
registry_addrs: list[tuple[str, int]] = [],
|
||||
|
||||
enable_modules: list[str] = [],
|
||||
loglevel: str|None = None,
|
||||
loglevel: Optional[str] = None,
|
||||
|
||||
# XXX NOTE XXX: you should pretty much never want debug mode
|
||||
# for data daemons when running in production.
|
||||
|
|
@ -71,7 +69,7 @@ async def open_piker_runtime(
|
|||
# and spawn the service tree distributed per that.
|
||||
start_method: str = 'trio',
|
||||
|
||||
tractor_runtime_overrides: dict|None = None,
|
||||
tractor_runtime_overrides: dict | None = None,
|
||||
**tractor_kwargs,
|
||||
|
||||
) -> tuple[
|
||||
|
|
@ -99,8 +97,7 @@ async def open_piker_runtime(
|
|||
# setting it as the root actor on localhost.
|
||||
registry_addrs = (
|
||||
registry_addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
or [_default_reg_addr]
|
||||
)
|
||||
|
||||
if ems := tractor_kwargs.pop('enable_modules', None):
|
||||
|
|
@ -110,22 +107,17 @@ async def open_piker_runtime(
|
|||
async with (
|
||||
tractor.open_root_actor(
|
||||
|
||||
# passed through to `open_root_actor`
|
||||
# passed through to ``open_root_actor``
|
||||
registry_addrs=registry_addrs,
|
||||
name=name,
|
||||
start_method=start_method,
|
||||
loglevel=loglevel,
|
||||
debug_mode=debug_mode,
|
||||
|
||||
# XXX NOTE MEMBER DAT der's a perf hit yo!!
|
||||
# https://greenback.readthedocs.io/en/latest/principle.html#performance
|
||||
maybe_enable_greenback=True,
|
||||
start_method=start_method,
|
||||
|
||||
# TODO: eventually we should be able to avoid
|
||||
# having the root have more then permissions to
|
||||
# spawn other specialized daemons I think?
|
||||
enable_modules=enable_modules,
|
||||
hide_tb=False,
|
||||
|
||||
**tractor_kwargs,
|
||||
) as actor,
|
||||
|
|
@ -166,7 +158,8 @@ _root_modules: list[str] = [
|
|||
@acm
|
||||
async def open_pikerd(
|
||||
registry_addrs: list[tuple[str, int]],
|
||||
loglevel: str|None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
|
||||
# XXX: you should pretty much never want debug mode
|
||||
# for data daemons when running in production.
|
||||
|
|
@ -194,6 +187,7 @@ async def open_pikerd(
|
|||
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
|
||||
name=_root_dname,
|
||||
loglevel=loglevel,
|
||||
debug_mode=debug_mode,
|
||||
|
|
@ -206,8 +200,7 @@ async def open_pikerd(
|
|||
reg_addrs,
|
||||
),
|
||||
tractor.open_nursery() as actor_nursery,
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as service_tn,
|
||||
trio.open_nursery() as service_nursery,
|
||||
):
|
||||
for addr in reg_addrs:
|
||||
if addr not in root_actor.accept_addrs:
|
||||
|
|
@ -218,7 +211,7 @@ async def open_pikerd(
|
|||
|
||||
# assign globally for future daemon/task creation
|
||||
Services.actor_n = actor_nursery
|
||||
Services.service_n = service_tn
|
||||
Services.service_n = service_nursery
|
||||
Services.debug_mode = debug_mode
|
||||
|
||||
try:
|
||||
|
|
@ -228,7 +221,7 @@ async def open_pikerd(
|
|||
# TODO: is this more clever/efficient?
|
||||
# if 'samplerd' in Services.service_tasks:
|
||||
# await Services.cancel_service('samplerd')
|
||||
service_tn.cancel_scope.cancel()
|
||||
service_nursery.cancel_scope.cancel()
|
||||
|
||||
|
||||
# TODO: do we even need this?
|
||||
|
|
@ -263,10 +256,7 @@ async def maybe_open_pikerd(
|
|||
loglevel: str | None = None,
|
||||
**kwargs,
|
||||
|
||||
) -> (
|
||||
tractor._portal.Portal
|
||||
|ClassVar[Services]
|
||||
):
|
||||
) -> tractor._portal.Portal | ClassVar[Services]:
|
||||
'''
|
||||
If no ``pikerd`` daemon-root-actor can be found start it and
|
||||
yield up (we should probably figure out returning a portal to self
|
||||
|
|
@ -274,10 +264,7 @@ async def maybe_open_pikerd(
|
|||
|
||||
'''
|
||||
if loglevel:
|
||||
get_console_log(
|
||||
name=subsys,
|
||||
level=loglevel
|
||||
)
|
||||
get_console_log(loglevel)
|
||||
|
||||
# subtle, we must have the runtime up here or portal lookup will fail
|
||||
query_name = kwargs.pop(
|
||||
|
|
@ -294,11 +281,10 @@ async def maybe_open_pikerd(
|
|||
|
||||
registry_addrs: list[tuple[str, int]] = (
|
||||
registry_addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
or [_default_reg_addr]
|
||||
)
|
||||
|
||||
pikerd_portal: tractor.Portal|None
|
||||
pikerd_portal: tractor.Portal | None
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
name=query_name,
|
||||
|
|
|
|||
|
|
@ -49,15 +49,13 @@ from requests.exceptions import (
|
|||
ReadTimeout,
|
||||
)
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ._mngr import Services
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
)
|
||||
from .. import config
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
class DockerNotStarted(Exception):
|
||||
'Prolly you dint start da daemon bruh'
|
||||
|
|
@ -338,16 +336,13 @@ class Container:
|
|||
async def open_ahabd(
|
||||
ctx: tractor.Context,
|
||||
endpoint: str, # ns-pointer str-msg-type
|
||||
loglevel: str = 'cancel',
|
||||
loglevel: str | None = None,
|
||||
|
||||
**ep_kwargs,
|
||||
|
||||
) -> None:
|
||||
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
name='piker.service',
|
||||
)
|
||||
log = get_console_log(loglevel or 'cancel')
|
||||
|
||||
async with open_docker() as client:
|
||||
|
||||
|
|
|
|||
|
|
@ -28,11 +28,9 @@ from contextlib import (
|
|||
)
|
||||
|
||||
import tractor
|
||||
from trio.lowlevel import current_task
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
)
|
||||
from ._mngr import (
|
||||
Services,
|
||||
|
|
@ -40,17 +38,16 @@ from ._mngr import (
|
|||
from ._actor_runtime import maybe_open_pikerd
|
||||
from ._registry import find_service
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
@acm
|
||||
async def maybe_spawn_daemon(
|
||||
|
||||
service_name: str,
|
||||
service_task_target: Callable,
|
||||
|
||||
spawn_args: dict[str, Any],
|
||||
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
singleton: bool = False,
|
||||
|
||||
**pikerd_kwargs,
|
||||
|
|
@ -68,99 +65,79 @@ async def maybe_spawn_daemon(
|
|||
clients.
|
||||
|
||||
'''
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
assert log.name == 'piker.service'
|
||||
|
||||
# serialize access to this section to avoid
|
||||
# 2 or more tasks racing to create a daemon
|
||||
lock = Services.locks[service_name]
|
||||
await lock.acquire()
|
||||
|
||||
try:
|
||||
async with find_service(
|
||||
service_name,
|
||||
registry_addrs=[('127.0.0.1', 6116)],
|
||||
) as portal:
|
||||
if portal is not None:
|
||||
lock.release()
|
||||
yield portal
|
||||
return
|
||||
|
||||
log.warning(
|
||||
f"Couldn't find any existing {service_name}\n"
|
||||
'Attempting to spawn new daemon-service..'
|
||||
)
|
||||
|
||||
# ask root ``pikerd`` daemon to spawn the daemon we need if
|
||||
# pikerd is not live we now become the root of the
|
||||
# process tree
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) as pikerd_portal:
|
||||
|
||||
# we are the root and thus are `pikerd`
|
||||
# so spawn the target service directly by calling
|
||||
# the provided target routine.
|
||||
# XXX: this assumes that the target is well formed and will
|
||||
# do the right things to setup both a sub-actor **and** call
|
||||
# the ``_Services`` api from above to start the top level
|
||||
# service task for that actor.
|
||||
started: bool
|
||||
if pikerd_portal is None:
|
||||
started = await service_task_target(
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
else:
|
||||
# request a remote `pikerd` (service manager) to start the
|
||||
# target daemon-task, the target can't return
|
||||
# a non-serializable value since it is expected that service
|
||||
# starting is non-blocking and the target task will persist
|
||||
# running "under" or "within" the `pikerd` actor tree after
|
||||
# the questing client disconnects. in other words this
|
||||
# spawns a persistent daemon actor that continues to live
|
||||
# for the lifespan of whatever the service manager inside
|
||||
# `pikerd` says it should.
|
||||
started = await pikerd_portal.run(
|
||||
service_task_target,
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
if started:
|
||||
log.info(f'Service {service_name} started!')
|
||||
|
||||
# block until we can discover (by IPC connection) to the newly
|
||||
# spawned daemon-actor and then deliver the portal to the
|
||||
# caller.
|
||||
async with tractor.wait_for_actor(service_name) as portal:
|
||||
lock.release()
|
||||
yield portal
|
||||
await portal.cancel_actor()
|
||||
|
||||
except BaseException as _err:
|
||||
err = _err
|
||||
if (
|
||||
lock.locked()
|
||||
and
|
||||
lock.statistics().owner is current_task()
|
||||
):
|
||||
log.exception(
|
||||
f'Releasing stale lock after crash..?'
|
||||
f'{err!r}\n'
|
||||
)
|
||||
async with find_service(
|
||||
service_name,
|
||||
registry_addrs=[('127.0.0.1', 6116)],
|
||||
) as portal:
|
||||
if portal is not None:
|
||||
lock.release()
|
||||
raise err
|
||||
yield portal
|
||||
return
|
||||
|
||||
log.warning(
|
||||
f"Couldn't find any existing {service_name}\n"
|
||||
'Attempting to spawn new daemon-service..'
|
||||
)
|
||||
|
||||
# ask root ``pikerd`` daemon to spawn the daemon we need if
|
||||
# pikerd is not live we now become the root of the
|
||||
# process tree
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) as pikerd_portal:
|
||||
|
||||
# we are the root and thus are `pikerd`
|
||||
# so spawn the target service directly by calling
|
||||
# the provided target routine.
|
||||
# XXX: this assumes that the target is well formed and will
|
||||
# do the right things to setup both a sub-actor **and** call
|
||||
# the ``_Services`` api from above to start the top level
|
||||
# service task for that actor.
|
||||
started: bool
|
||||
if pikerd_portal is None:
|
||||
started = await service_task_target(
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
else:
|
||||
# request a remote `pikerd` (service manager) to start the
|
||||
# target daemon-task, the target can't return
|
||||
# a non-serializable value since it is expected that service
|
||||
# starting is non-blocking and the target task will persist
|
||||
# running "under" or "within" the `pikerd` actor tree after
|
||||
# the questing client disconnects. in other words this
|
||||
# spawns a persistent daemon actor that continues to live
|
||||
# for the lifespan of whatever the service manager inside
|
||||
# `pikerd` says it should.
|
||||
started = await pikerd_portal.run(
|
||||
service_task_target,
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
if started:
|
||||
log.info(f'Service {service_name} started!')
|
||||
|
||||
# block until we can discover (by IPC connection) to the newly
|
||||
# spawned daemon-actor and then deliver the portal to the
|
||||
# caller.
|
||||
async with tractor.wait_for_actor(service_name) as portal:
|
||||
lock.release()
|
||||
yield portal
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
||||
async def spawn_emsd(
|
||||
loglevel: str|None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
**extra_tractor_kwargs
|
||||
|
||||
) -> bool:
|
||||
|
|
@ -197,8 +174,9 @@ async def spawn_emsd(
|
|||
|
||||
@acm
|
||||
async def maybe_open_emsd(
|
||||
|
||||
brokername: str,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
**pikerd_kwargs,
|
||||
|
||||
|
|
|
|||
|
|
@ -34,9 +34,9 @@ from tractor import (
|
|||
Portal,
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
)
|
||||
|
||||
|
||||
# TODO: we need remote wrapping and a general soln:
|
||||
|
|
@ -109,7 +109,7 @@ class Services:
|
|||
# wait on any context's return value
|
||||
# and any final portal result from the
|
||||
# sub-actor.
|
||||
ctx_res: Any = await ctx.wait_for_result()
|
||||
ctx_res: Any = await ctx.result()
|
||||
|
||||
# NOTE: blocks indefinitely until cancelled
|
||||
# either by error from the target context
|
||||
|
|
|
|||
|
|
@ -27,29 +27,15 @@ from typing import (
|
|||
)
|
||||
|
||||
import tractor
|
||||
from tractor import (
|
||||
msg,
|
||||
Actor,
|
||||
Portal,
|
||||
from tractor import Portal
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
# TODO? default path-space for UDS registry?
|
||||
# [ ] needs to be Xplatform tho!
|
||||
# _default_registry_path: Path = (
|
||||
# Path(os.environ['XDG_RUNTIME_DIR'])
|
||||
# /'piker'
|
||||
# )
|
||||
|
||||
_default_registry_host: str = '127.0.0.1'
|
||||
_default_registry_port: int = 6116
|
||||
_default_reg_addr: tuple[
|
||||
str,
|
||||
int, # |str TODO, once we support UDS, see above.
|
||||
] = (
|
||||
_default_reg_addr: tuple[str, int] = (
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)
|
||||
|
|
@ -89,22 +75,16 @@ async def open_registry(
|
|||
|
||||
'''
|
||||
global _tractor_kwargs
|
||||
actor: Actor = tractor.current_actor()
|
||||
aid: msg.Aid = actor.aid
|
||||
uid: tuple[str, str] = aid.uid
|
||||
preset_reg_addrs: list[
|
||||
tuple[str, int]
|
||||
] = Registry.addrs
|
||||
actor = tractor.current_actor()
|
||||
uid = actor.uid
|
||||
preset_reg_addrs: list[tuple[str, int]] = Registry.addrs
|
||||
if (
|
||||
preset_reg_addrs
|
||||
and
|
||||
addrs
|
||||
and addrs
|
||||
):
|
||||
if preset_reg_addrs != addrs:
|
||||
# if any(addr in preset_reg_addrs for addr in addrs):
|
||||
diff: set[
|
||||
tuple[str, int]
|
||||
] = set(preset_reg_addrs) - set(addrs)
|
||||
diff: set[tuple[str, int]] = set(preset_reg_addrs) - set(addrs)
|
||||
if diff:
|
||||
log.warning(
|
||||
f'`{uid}` requested only subset of registrars: {addrs}\n'
|
||||
|
|
@ -118,39 +98,31 @@ async def open_registry(
|
|||
)
|
||||
|
||||
was_set: bool = False
|
||||
|
||||
if (
|
||||
not tractor.is_root_process()
|
||||
and
|
||||
not Registry.addrs
|
||||
and not Registry.addrs
|
||||
):
|
||||
Registry.addrs.extend(actor.reg_addrs)
|
||||
|
||||
if (
|
||||
ensure_exists
|
||||
and
|
||||
not Registry.addrs
|
||||
and not Registry.addrs
|
||||
):
|
||||
raise RuntimeError(
|
||||
f"`{uid}` registry should already exist but doesn't?"
|
||||
)
|
||||
|
||||
if not Registry.addrs:
|
||||
if (
|
||||
not Registry.addrs
|
||||
):
|
||||
was_set = True
|
||||
Registry.addrs = (
|
||||
addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
)
|
||||
Registry.addrs = addrs or [_default_reg_addr]
|
||||
|
||||
# NOTE: only spot this seems currently used is inside
|
||||
# `.ui._exec` which is the (eventual qtloops) bootstrapping
|
||||
# with guest mode.
|
||||
reg_addrs: list[tuple[str, str|int]] = Registry.addrs
|
||||
# !TODO, a struct-API to stringently allow this only in special
|
||||
# cases?
|
||||
# -> better would be to have some way to (atomically) rewrite
|
||||
# and entire `RuntimeVars`?? ideas welcome obvi..
|
||||
_tractor_kwargs['registry_addrs'] = reg_addrs
|
||||
_tractor_kwargs['registry_addrs'] = Registry.addrs
|
||||
|
||||
try:
|
||||
yield Registry.addrs
|
||||
|
|
@ -174,8 +146,8 @@ async def find_service(
|
|||
| list[Portal]
|
||||
| None
|
||||
):
|
||||
# try:
|
||||
reg_addrs: list[tuple[str, int|str]]
|
||||
|
||||
reg_addrs: list[tuple[str, int]]
|
||||
async with open_registry(
|
||||
addrs=(
|
||||
registry_addrs
|
||||
|
|
@ -185,41 +157,27 @@ async def find_service(
|
|||
or Registry.addrs
|
||||
),
|
||||
) as reg_addrs:
|
||||
log.info(f'Scanning for service `{service_name}`')
|
||||
|
||||
log.info(
|
||||
f'Scanning for service {service_name!r}'
|
||||
)
|
||||
maybe_portals: list[Portal] | Portal | None
|
||||
|
||||
# attach to existing daemon by name if possible
|
||||
maybe_portals: list[Portal]|Portal|None
|
||||
async with tractor.find_actor(
|
||||
service_name,
|
||||
registry_addrs=reg_addrs,
|
||||
only_first=first_only, # if set only returns single ref
|
||||
) as maybe_portals:
|
||||
if not maybe_portals:
|
||||
log.info(
|
||||
f'Could NOT find service {service_name!r} -> {maybe_portals!r}'
|
||||
)
|
||||
yield None
|
||||
return
|
||||
|
||||
log.info(
|
||||
f'Found service {service_name!r} -> {maybe_portals}'
|
||||
)
|
||||
yield maybe_portals
|
||||
|
||||
# except BaseException as _berr:
|
||||
# berr = _berr
|
||||
# log.exception(
|
||||
# 'tractor.find_actor() failed with,\n'
|
||||
# )
|
||||
# raise berr
|
||||
|
||||
|
||||
async def check_for_service(
|
||||
service_name: str,
|
||||
) -> None|tuple[str, int]:
|
||||
|
||||
) -> None | tuple[str, int]:
|
||||
'''
|
||||
Service daemon "liveness" predicate.
|
||||
|
||||
|
|
|
|||
|
|
@ -14,12 +14,20 @@
|
|||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
"""
|
||||
Sub-sys module commons (if any ?? Bp).
|
||||
Sub-sys module commons.
|
||||
|
||||
"""
|
||||
from functools import partial
|
||||
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
subsys: str = 'piker.service'
|
||||
|
||||
# ?TODO, if we were going to keep a `get_console_log()` in here to be
|
||||
# invoked at `import`-time, how do we dynamically hand in the
|
||||
# `level=` value? seems too early in the runtime to be injected
|
||||
# right?
|
||||
log = get_logger(subsys)
|
||||
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name=subsys,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -16,7 +16,6 @@
|
|||
|
||||
from __future__ import annotations
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
TYPE_CHECKING,
|
||||
|
|
@ -27,17 +26,12 @@ import asks
|
|||
if TYPE_CHECKING:
|
||||
import docker
|
||||
from ._ahab import DockerContainer
|
||||
from . import (
|
||||
Services,
|
||||
)
|
||||
|
||||
from piker.log import (
|
||||
from ._util import log # sub-sys logger
|
||||
from ._util import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
# container level config
|
||||
_config = {
|
||||
|
|
@ -73,10 +67,7 @@ def start_elasticsearch(
|
|||
elastic
|
||||
|
||||
'''
|
||||
get_console_log(
|
||||
level='info',
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log('info', name=__name__)
|
||||
|
||||
dcntr: DockerContainer = client.containers.run(
|
||||
'piker:elastic',
|
||||
|
|
|
|||
|
|
@ -52,18 +52,17 @@ import pendulum
|
|||
# TODO: import this for specific error set expected by mkts client
|
||||
# import purerpc
|
||||
|
||||
from piker.data.feed import maybe_open_feed
|
||||
from ..data.feed import maybe_open_feed
|
||||
from . import Services
|
||||
from piker.log import (
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import docker
|
||||
from ._ahab import DockerContainer
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
# ahabd-supervisor and container level config
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ from typing import (
|
|||
|
||||
import numpy as np
|
||||
|
||||
|
||||
from .. import config
|
||||
from ..service import (
|
||||
check_for_service,
|
||||
|
|
@ -151,10 +152,7 @@ class StorageConnectionError(ConnectionError):
|
|||
|
||||
'''
|
||||
|
||||
def get_storagemod(
|
||||
name: str,
|
||||
|
||||
) -> ModuleType:
|
||||
def get_storagemod(name: str) -> ModuleType:
|
||||
mod: ModuleType = import_module(
|
||||
'.' + name,
|
||||
'piker.storage',
|
||||
|
|
@ -167,12 +165,9 @@ def get_storagemod(
|
|||
|
||||
@acm
|
||||
async def open_storage_client(
|
||||
backend: str|None = None,
|
||||
backend: str | None = None,
|
||||
|
||||
) -> tuple[
|
||||
ModuleType,
|
||||
StorageClient,
|
||||
]:
|
||||
) -> tuple[ModuleType, StorageClient]:
|
||||
'''
|
||||
Load the ``StorageClient`` for named backend.
|
||||
|
||||
|
|
@ -272,10 +267,7 @@ async def open_tsdb_client(
|
|||
from ..data.feed import maybe_open_feed
|
||||
|
||||
async with (
|
||||
open_storage_client() as (
|
||||
_,
|
||||
storage,
|
||||
),
|
||||
open_storage_client() as (_, storage),
|
||||
|
||||
maybe_open_feed(
|
||||
[fqme],
|
||||
|
|
@ -283,7 +275,7 @@ async def open_tsdb_client(
|
|||
|
||||
) as feed,
|
||||
):
|
||||
profiler(f'opened feed for {fqme!r}')
|
||||
profiler(f'opened feed for {fqme}')
|
||||
|
||||
# to_append = feed.hist_shm.array
|
||||
# to_prepend = None
|
||||
|
|
|
|||
|
|
@ -19,10 +19,16 @@ Storage middle-ware CLIs.
|
|||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
# from datetime import datetime
|
||||
# from contextlib import (
|
||||
# AsyncExitStack,
|
||||
# )
|
||||
from pathlib import Path
|
||||
from math import copysign
|
||||
import time
|
||||
from types import ModuleType
|
||||
from typing import (
|
||||
Any,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
|
|
@ -41,6 +47,7 @@ from piker.data import (
|
|||
ShmArray,
|
||||
)
|
||||
from piker import tsp
|
||||
from piker.data._formatters import BGM
|
||||
from . import log
|
||||
from . import (
|
||||
__tsdbs__,
|
||||
|
|
@ -235,12 +242,122 @@ def anal(
|
|||
trio.run(main)
|
||||
|
||||
|
||||
async def markup_gaps(
|
||||
fqme: str,
|
||||
timeframe: float,
|
||||
actl: AnnotCtl,
|
||||
wdts: pl.DataFrame,
|
||||
gaps: pl.DataFrame,
|
||||
|
||||
) -> dict[int, dict]:
|
||||
'''
|
||||
Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
|
||||
with rectangles.
|
||||
|
||||
'''
|
||||
aids: dict[int] = {}
|
||||
for i in range(gaps.height):
|
||||
|
||||
row: pl.DataFrame = gaps[i]
|
||||
|
||||
# the gap's RIGHT-most bar's OPEN value
|
||||
# at that time (sample) step.
|
||||
iend: int = row['index'][0]
|
||||
# dt: datetime = row['dt'][0]
|
||||
# dt_prev: datetime = row['dt_prev'][0]
|
||||
# dt_end_t: float = dt.timestamp()
|
||||
|
||||
|
||||
# TODO: can we eventually remove this
|
||||
# once we figure out why the epoch cols
|
||||
# don't match?
|
||||
# TODO: FIX HOW/WHY these aren't matching
|
||||
# and are instead off by 4hours (EST
|
||||
# vs. UTC?!?!)
|
||||
# end_t: float = row['time']
|
||||
# assert (
|
||||
# dt.timestamp()
|
||||
# ==
|
||||
# end_t
|
||||
# )
|
||||
|
||||
# the gap's LEFT-most bar's CLOSE value
|
||||
# at that time (sample) step.
|
||||
prev_r: pl.DataFrame = wdts.filter(
|
||||
pl.col('index') == iend - 1
|
||||
)
|
||||
# XXX: probably a gap in the (newly sorted or de-duplicated)
|
||||
# dt-df, so we might need to re-index first..
|
||||
if prev_r.is_empty():
|
||||
await tractor.pause()
|
||||
|
||||
istart: int = prev_r['index'][0]
|
||||
# dt_start_t: float = dt_prev.timestamp()
|
||||
|
||||
# start_t: float = prev_r['time']
|
||||
# assert (
|
||||
# dt_start_t
|
||||
# ==
|
||||
# start_t
|
||||
# )
|
||||
|
||||
# TODO: implement px-col width measure
|
||||
# and ensure at least as many px-cols
|
||||
# shown per rect as configured by user.
|
||||
# gap_w: float = abs((iend - istart))
|
||||
# if gap_w < 6:
|
||||
# margin: float = 6
|
||||
# iend += margin
|
||||
# istart -= margin
|
||||
|
||||
rect_gap: float = BGM*3/8
|
||||
opn: float = row['open'][0]
|
||||
ro: tuple[float, float] = (
|
||||
# dt_end_t,
|
||||
iend + rect_gap + 1,
|
||||
opn,
|
||||
)
|
||||
cls: float = prev_r['close'][0]
|
||||
lc: tuple[float, float] = (
|
||||
# dt_start_t,
|
||||
istart - rect_gap, # + 1 ,
|
||||
cls,
|
||||
)
|
||||
|
||||
color: str = 'dad_blue'
|
||||
diff: float = cls - opn
|
||||
sgn: float = copysign(1, diff)
|
||||
color: str = {
|
||||
-1: 'buy_green',
|
||||
1: 'sell_red',
|
||||
}[sgn]
|
||||
|
||||
rect_kwargs: dict[str, Any] = dict(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
start_pos=lc,
|
||||
end_pos=ro,
|
||||
color=color,
|
||||
)
|
||||
|
||||
aid: int = await actl.add_rect(**rect_kwargs)
|
||||
assert aid
|
||||
aids[aid] = rect_kwargs
|
||||
|
||||
# tell chart to redraw all its
|
||||
# graphics view layers Bo
|
||||
await actl.redraw(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
return aids
|
||||
|
||||
|
||||
@store.command()
|
||||
def ldshm(
|
||||
fqme: str,
|
||||
write_parquet: bool = True,
|
||||
reload_parquet_to_shm: bool = True,
|
||||
pdb: bool = False, # --pdb passed?
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -260,7 +377,7 @@ def ldshm(
|
|||
open_piker_runtime(
|
||||
'polars_boi',
|
||||
enable_modules=['piker.data._sharedmem'],
|
||||
debug_mode=pdb,
|
||||
debug_mode=True,
|
||||
),
|
||||
open_storage_client() as (
|
||||
mod,
|
||||
|
|
@ -269,8 +386,6 @@ def ldshm(
|
|||
open_annot_ctl() as actl,
|
||||
):
|
||||
shm_df: pl.DataFrame | None = None
|
||||
tf2aids: dict[float, dict] = {}
|
||||
|
||||
for (
|
||||
shmfile,
|
||||
shm,
|
||||
|
|
@ -280,19 +395,17 @@ def ldshm(
|
|||
|
||||
times: np.ndarray = shm.array['time']
|
||||
d1: float = float(times[-1] - times[-2])
|
||||
d2: float = 0
|
||||
# XXX, take a median sample rate if sufficient data
|
||||
if times.size > 2:
|
||||
d2: float = float(times[-2] - times[-3])
|
||||
med: float = np.median(np.diff(times))
|
||||
if (
|
||||
d1 < 1.
|
||||
and d2 < 1.
|
||||
and med < 1.
|
||||
):
|
||||
raise ValueError(
|
||||
f'Something is wrong with time period for {shm}:\n{times}'
|
||||
)
|
||||
d2: float = float(times[-2] - times[-3])
|
||||
med: float = np.median(np.diff(times))
|
||||
if (
|
||||
d1 < 1.
|
||||
and d2 < 1.
|
||||
and med < 1.
|
||||
):
|
||||
raise ValueError(
|
||||
f'Something is wrong with time period for {shm}:\n{times}'
|
||||
)
|
||||
|
||||
period_s: float = float(max(d1, d2, med))
|
||||
|
||||
null_segs: tuple = tsp.get_null_segs(
|
||||
|
|
@ -302,9 +415,7 @@ def ldshm(
|
|||
|
||||
# TODO: call null-seg fixer somehow?
|
||||
if null_segs:
|
||||
|
||||
if tractor._state.is_debug_mode():
|
||||
await tractor.pause()
|
||||
await tractor.pause()
|
||||
# async with (
|
||||
# trio.open_nursery() as tn,
|
||||
# mod.open_history_client(
|
||||
|
|
@ -328,37 +439,11 @@ def ldshm(
|
|||
wdts,
|
||||
deduped,
|
||||
diff,
|
||||
valid_races,
|
||||
dq_issues,
|
||||
) = tsp.dedupe_ohlcv_smart(
|
||||
) = tsp.dedupe(
|
||||
shm_df,
|
||||
period=period_s,
|
||||
)
|
||||
|
||||
# Report duplicate analysis
|
||||
if diff > 0:
|
||||
log.info(
|
||||
f'Removed {diff} duplicate timestamp(s)\n'
|
||||
)
|
||||
if valid_races is not None:
|
||||
identical: int = (
|
||||
valid_races
|
||||
.filter(pl.col('identical_bars'))
|
||||
.height
|
||||
)
|
||||
monotonic: int = valid_races.height - identical
|
||||
log.info(
|
||||
f'Valid race conditions: {valid_races.height}\n'
|
||||
f' - Identical bars: {identical}\n'
|
||||
f' - Volume monotonic: {monotonic}\n'
|
||||
)
|
||||
|
||||
if dq_issues is not None:
|
||||
log.warning(
|
||||
f'DATA QUALITY ISSUES from provider: '
|
||||
f'{dq_issues.height} timestamp(s)\n'
|
||||
f'{dq_issues}\n'
|
||||
)
|
||||
|
||||
# detect gaps from in expected (uniform OHLC) sample period
|
||||
step_gaps: pl.DataFrame = tsp.detect_time_gaps(
|
||||
deduped,
|
||||
|
|
@ -373,8 +458,7 @@ def ldshm(
|
|||
|
||||
# TODO: actually pull the exact duration
|
||||
# expected for each venue operational period?
|
||||
# gap_dt_unit='day',
|
||||
gap_dt_unit='day',
|
||||
gap_dt_unit='days',
|
||||
gap_thresh=1,
|
||||
)
|
||||
|
||||
|
|
@ -385,11 +469,8 @@ def ldshm(
|
|||
if (
|
||||
not venue_gaps.is_empty()
|
||||
or (
|
||||
not step_gaps.is_empty()
|
||||
# XXX, i presume i put this bc i was guarding
|
||||
# for ib venue gaps?
|
||||
# and
|
||||
# period_s < 60
|
||||
period_s < 60
|
||||
and not step_gaps.is_empty()
|
||||
)
|
||||
):
|
||||
# write repaired ts to parquet-file?
|
||||
|
|
@ -438,35 +519,23 @@ def ldshm(
|
|||
do_markup_gaps: bool = True
|
||||
if do_markup_gaps:
|
||||
new_df: pl.DataFrame = tsp.np2pl(new)
|
||||
aids: dict = await tsp._annotate.markup_gaps(
|
||||
aids: dict = await markup_gaps(
|
||||
fqme,
|
||||
period_s,
|
||||
actl,
|
||||
new_df,
|
||||
step_gaps,
|
||||
)
|
||||
|
||||
# last chance manual overwrites in REPL
|
||||
# await tractor.pause()
|
||||
if not aids:
|
||||
log.warning(
|
||||
f'No gaps were found !?\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
f'timeframe: {period_s!r}\n'
|
||||
f"WELL THAT'S GOOD NOOZ!\n"
|
||||
)
|
||||
tf2aids[period_s] = aids
|
||||
await tractor.pause()
|
||||
assert aids
|
||||
|
||||
else:
|
||||
# No significant gaps to handle, but may have had
|
||||
# duplicates removed (valid race conditions are ok)
|
||||
if diff > 0 and dq_issues is not None:
|
||||
log.warning(
|
||||
'Found duplicates with data quality issues '
|
||||
'but no significant time gaps!\n'
|
||||
)
|
||||
# allow interaction even when no ts problems.
|
||||
await tractor.pause()
|
||||
# assert not diff
|
||||
|
||||
await tractor.pause()
|
||||
log.info('Exiting TSP shm anal-izer!')
|
||||
|
||||
if shm_df is None:
|
||||
log.error(
|
||||
|
|
|
|||
|
|
@ -161,13 +161,7 @@ class NativeStorageClient:
|
|||
|
||||
def index_files(self):
|
||||
for path in self._datadir.iterdir():
|
||||
if (
|
||||
path.is_dir()
|
||||
or
|
||||
'.parquet' not in str(path)
|
||||
# or
|
||||
# path.name in {'borked', 'expired',}
|
||||
):
|
||||
if path.name in {'borked', 'expired',}:
|
||||
continue
|
||||
|
||||
key: str = path.name.rstrip('.parquet')
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -54,10 +54,10 @@ from ..log import (
|
|||
# for "time series processing"
|
||||
subsys: str = 'piker.tsp'
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
log = get_logger(subsys)
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name=subsys, # activate for subsys-pkg "downward"
|
||||
name=subsys,
|
||||
)
|
||||
|
||||
# NOTE: union type-defs to handle generic `numpy` and `polars` types
|
||||
|
|
@ -275,18 +275,6 @@ def get_null_segs(
|
|||
# diff of abs index steps between each zeroed row
|
||||
absi_zdiff: np.ndarray = np.diff(absi_zeros)
|
||||
|
||||
if zero_t.size < 2:
|
||||
try:
|
||||
breakpoint()
|
||||
except RuntimeError:
|
||||
# XXX, if greenback not active from
|
||||
# piker store ldshm cmd..
|
||||
log.exception(
|
||||
"Can't debug single-sample null!\n"
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
# scan for all frame-indices where the
|
||||
# zeroed-row-abs-index-step-diff is greater then the
|
||||
# expected increment of 1.
|
||||
|
|
@ -446,8 +434,8 @@ def get_null_segs(
|
|||
|
||||
def iter_null_segs(
|
||||
timeframe: float,
|
||||
frame: Frame|None = None,
|
||||
null_segs: tuple|None = None,
|
||||
frame: Frame | None = None,
|
||||
null_segs: tuple | None = None,
|
||||
|
||||
) -> Generator[
|
||||
tuple[
|
||||
|
|
@ -499,8 +487,7 @@ def iter_null_segs(
|
|||
start_dt = None
|
||||
if (
|
||||
absi_start is not None
|
||||
and
|
||||
start_t != 0
|
||||
and start_t != 0
|
||||
):
|
||||
fi_start: int = absi_start - absi_first
|
||||
start_row: Seq = frame[fi_start]
|
||||
|
|
@ -514,8 +501,8 @@ def iter_null_segs(
|
|||
yield (
|
||||
absi_start, absi_end, # abs indices
|
||||
fi_start, fi_end, # relative "frame" indices
|
||||
start_t, end_t, # epoch times
|
||||
start_dt, end_dt, # dts
|
||||
start_t, end_t,
|
||||
start_dt, end_dt,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -530,7 +517,7 @@ def with_dts(
|
|||
|
||||
'''
|
||||
return df.with_columns([
|
||||
pl.col(time_col).shift(1).name.suffix('_prev'),
|
||||
pl.col(time_col).shift(1).suffix('_prev'),
|
||||
pl.col(time_col).diff().alias('s_diff'),
|
||||
pl.from_epoch(pl.col(time_col)).alias('dt'),
|
||||
]).with_columns([
|
||||
|
|
@ -591,22 +578,11 @@ def detect_time_gaps(
|
|||
# NOTE: this flag is to indicate that on this (sampling) time
|
||||
# scale we expect to only be filtering against larger venue
|
||||
# closures-scale time gaps.
|
||||
#
|
||||
# Map to total_ method since `dt_diff` is a duration type,
|
||||
# not datetime - modern polars requires `total_*` methods
|
||||
# for duration types (e.g. `total_days()` not `day()`)
|
||||
# Ensure plural form for polars API (e.g. 'day' -> 'days')
|
||||
unit_plural: str = (
|
||||
gap_dt_unit
|
||||
if gap_dt_unit.endswith('s')
|
||||
else f'{gap_dt_unit}s'
|
||||
)
|
||||
duration_method: str = f'total_{unit_plural}'
|
||||
return step_gaps.filter(
|
||||
# Second by an arbitrary dt-unit step size
|
||||
getattr(
|
||||
pl.col('dt_diff').dt,
|
||||
duration_method,
|
||||
gap_dt_unit,
|
||||
)().abs() > gap_thresh
|
||||
)
|
||||
|
||||
|
|
@ -640,18 +616,6 @@ def detect_price_gaps(
|
|||
# ])
|
||||
...
|
||||
|
||||
# TODO: probably just use the null_segs impl above?
|
||||
def detect_vlm_gaps(
|
||||
df: pl.DataFrame,
|
||||
col: str = 'volume',
|
||||
|
||||
) -> pl.DataFrame:
|
||||
|
||||
vnull: pl.DataFrame = df.filter(
|
||||
pl.col(col) == 0
|
||||
)
|
||||
return vnull
|
||||
|
||||
|
||||
def dedupe(
|
||||
src_df: pl.DataFrame,
|
||||
|
|
@ -662,6 +626,7 @@ def dedupe(
|
|||
|
||||
) -> tuple[
|
||||
pl.DataFrame, # with dts
|
||||
pl.DataFrame, # gaps
|
||||
pl.DataFrame, # with deduplicated dts (aka gap/repeat removal)
|
||||
int, # len diff between input and deduped
|
||||
]:
|
||||
|
|
@ -674,22 +639,19 @@ def dedupe(
|
|||
'''
|
||||
wdts: pl.DataFrame = with_dts(src_df)
|
||||
|
||||
deduped = wdts
|
||||
|
||||
# remove duplicated datetime samples/sections
|
||||
deduped: pl.DataFrame = wdts.unique(
|
||||
# subset=['dt'],
|
||||
subset=['time'],
|
||||
maintain_order=True,
|
||||
)
|
||||
|
||||
# maybe sort on any time field
|
||||
if sort:
|
||||
deduped = deduped.sort(by='time')
|
||||
wdts = wdts.sort(by='time')
|
||||
# TODO: detect out-of-order segments which were corrected!
|
||||
# -[ ] report in log msg
|
||||
# -[ ] possibly return segment sections which were moved?
|
||||
|
||||
# remove duplicated datetime samples/sections
|
||||
deduped: pl.DataFrame = wdts.unique(
|
||||
subset=['dt'],
|
||||
maintain_order=True,
|
||||
)
|
||||
|
||||
diff: int = (
|
||||
wdts.height
|
||||
-
|
||||
|
|
|
|||
|
|
@ -1,306 +0,0 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
"""
|
||||
Time-series (remote) annotation APIs.
|
||||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
from math import copysign
|
||||
from typing import (
|
||||
Any,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import polars as pl
|
||||
import tractor
|
||||
|
||||
from piker.data._formatters import BGM
|
||||
from piker.storage import log
|
||||
from piker.ui._style import get_fonts
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.ui._remote_ctl import AnnotCtl
|
||||
|
||||
|
||||
def humanize_duration(
|
||||
seconds: float,
|
||||
) -> str:
|
||||
'''
|
||||
Convert duration in seconds to short human-readable form.
|
||||
|
||||
Uses smallest appropriate time unit:
|
||||
- d: days
|
||||
- h: hours
|
||||
- m: minutes
|
||||
- s: seconds
|
||||
|
||||
Examples:
|
||||
- 86400 -> "1d"
|
||||
- 28800 -> "8h"
|
||||
- 180 -> "3m"
|
||||
- 45 -> "45s"
|
||||
|
||||
'''
|
||||
abs_secs: float = abs(seconds)
|
||||
|
||||
if abs_secs >= 86400:
|
||||
days: float = abs_secs / 86400
|
||||
if days >= 10 or days == int(days):
|
||||
return f'{int(days)}d'
|
||||
return f'{days:.1f}d'
|
||||
|
||||
elif abs_secs >= 3600:
|
||||
hours: float = abs_secs / 3600
|
||||
if hours >= 10 or hours == int(hours):
|
||||
return f'{int(hours)}h'
|
||||
return f'{hours:.1f}h'
|
||||
|
||||
elif abs_secs >= 60:
|
||||
mins: float = abs_secs / 60
|
||||
if mins >= 10 or mins == int(mins):
|
||||
return f'{int(mins)}m'
|
||||
return f'{mins:.1f}m'
|
||||
|
||||
else:
|
||||
if abs_secs >= 10 or abs_secs == int(abs_secs):
|
||||
return f'{int(abs_secs)}s'
|
||||
return f'{abs_secs:.1f}s'
|
||||
|
||||
|
||||
async def markup_gaps(
|
||||
fqme: str,
|
||||
timeframe: float,
|
||||
actl: AnnotCtl,
|
||||
wdts: pl.DataFrame,
|
||||
gaps: pl.DataFrame,
|
||||
|
||||
# XXX, switch on to see txt showing a "humanized" label of each
|
||||
# gap's duration.
|
||||
show_txt: bool = False,
|
||||
|
||||
) -> dict[int, dict]:
|
||||
'''
|
||||
Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
|
||||
with rectangles.
|
||||
|
||||
'''
|
||||
# XXX: force chart redraw FIRST to ensure PlotItem coordinate
|
||||
# system is properly initialized before we position annotations!
|
||||
# Without this, annotations may be misaligned on first creation
|
||||
# due to Qt/pyqtgraph initialization race conditions.
|
||||
await actl.redraw(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
|
||||
aids: dict[int] = {}
|
||||
for i in range(gaps.height):
|
||||
row: pl.DataFrame = gaps[i]
|
||||
|
||||
# the gap's RIGHT-most bar's OPEN value
|
||||
# at that time (sample) step.
|
||||
iend: int = row['index'][0]
|
||||
|
||||
# dt: datetime = row['dt'][0]
|
||||
# dt_prev: datetime = row['dt_prev'][0]
|
||||
# dt_end_t: float = dt.timestamp()
|
||||
|
||||
|
||||
# TODO: can we eventually remove this
|
||||
# once we figure out why the epoch cols
|
||||
# don't match?
|
||||
# TODO: FIX HOW/WHY these aren't matching
|
||||
# and are instead off by 4hours (EST
|
||||
# vs. UTC?!?!)
|
||||
# end_t: float = row['time']
|
||||
# assert (
|
||||
# dt.timestamp()
|
||||
# ==
|
||||
# end_t
|
||||
# )
|
||||
|
||||
# the gap's LEFT-most bar's CLOSE value
|
||||
# at that time (sample) step.
|
||||
prev_r: pl.DataFrame = wdts.filter(
|
||||
pl.col('index') == iend - 1
|
||||
)
|
||||
# XXX: probably a gap in the (newly sorted or de-duplicated)
|
||||
# dt-df, so we might need to re-index first..
|
||||
dt: pl.Series = row['dt']
|
||||
dt_prev: pl.Series = row['dt_prev']
|
||||
if prev_r.is_empty():
|
||||
|
||||
# XXX, filter out any special ignore cases,
|
||||
# - UNIX-epoch stamped datums
|
||||
# - first row
|
||||
if (
|
||||
dt_prev.dt.epoch()[0] == 0
|
||||
or
|
||||
dt.dt.epoch()[0] == 0
|
||||
):
|
||||
log.warning('Skipping row with UNIX epoch timestamp ??')
|
||||
continue
|
||||
|
||||
if wdts[0]['index'][0] == iend: # first row
|
||||
log.warning('Skipping first-row (has no previous obvi) !!')
|
||||
continue
|
||||
|
||||
# XXX, if the previous-row by shm-index is missing,
|
||||
# meaning there is a missing sample (set), get the prior
|
||||
# row by df index and attempt to use it?
|
||||
i_wdts: pl.DataFrame = wdts.with_row_index(name='i')
|
||||
i_row: int = i_wdts.filter(pl.col('index') == iend)['i'][0]
|
||||
prev_row_by_i = wdts[i_row]
|
||||
prev_r: pl.DataFrame = prev_row_by_i
|
||||
|
||||
# debug any missing pre-row
|
||||
if tractor._state.is_debug_mode():
|
||||
await tractor.pause()
|
||||
|
||||
istart: int = prev_r['index'][0]
|
||||
# TODO: implement px-col width measure
|
||||
# and ensure at least as many px-cols
|
||||
# shown per rect as configured by user.
|
||||
# gap_w: float = abs((iend - istart))
|
||||
# if gap_w < 6:
|
||||
# margin: float = 6
|
||||
# iend += margin
|
||||
# istart -= margin
|
||||
|
||||
opn: float = row['open'][0]
|
||||
cls: float = prev_r['close'][0]
|
||||
|
||||
# get gap duration for humanized label
|
||||
gap_dur_s: float = row['s_diff'][0]
|
||||
gap_label: str = humanize_duration(gap_dur_s)
|
||||
|
||||
# XXX: get timestamps for server-side index lookup
|
||||
start_time: float = prev_r['time'][0]
|
||||
end_time: float = row['time'][0]
|
||||
|
||||
# BGM=0.16 is the normal diff from overlap between bars, SO
|
||||
# just go slightly "in" from that "between them".
|
||||
from_idx: int = BGM - .06 # = .10
|
||||
lc: tuple[float, float] = (
|
||||
istart + 1 - from_idx,
|
||||
cls,
|
||||
)
|
||||
ro: tuple[float, float] = (
|
||||
iend + from_idx,
|
||||
opn,
|
||||
)
|
||||
|
||||
diff: float = cls - opn
|
||||
sgn: float = copysign(1, diff)
|
||||
up_gap: bool = sgn == -1
|
||||
down_gap: bool = sgn == 1
|
||||
flat: bool = sgn == 0
|
||||
|
||||
color: str = 'dad_blue'
|
||||
# TODO? mks more sense to have up/down coloring?
|
||||
# color: str = {
|
||||
# -1: 'lilypad_green', # up-gap
|
||||
# 1: 'wine', # down-gap
|
||||
# }[sgn]
|
||||
|
||||
rect_kwargs: dict[str, Any] = dict(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
start_pos=lc,
|
||||
end_pos=ro,
|
||||
color=color,
|
||||
start_time=start_time,
|
||||
end_time=end_time,
|
||||
)
|
||||
|
||||
# add up/down rects
|
||||
aid: int|None = await actl.add_rect(**rect_kwargs)
|
||||
if aid is None:
|
||||
log.error(
|
||||
f'Failed to add rect for,\n'
|
||||
f'{rect_kwargs!r}\n'
|
||||
f'\n'
|
||||
f'Skipping to next gap!\n'
|
||||
)
|
||||
continue
|
||||
|
||||
assert aid
|
||||
aids[aid] = rect_kwargs
|
||||
direction: str = (
|
||||
'down' if down_gap
|
||||
else 'up'
|
||||
)
|
||||
# TODO! mk this a `msgspec.Struct` which we deserialize
|
||||
# on the server side!
|
||||
# XXX: send timestamp for server-side index lookup
|
||||
# to ensure alignment with current shm state
|
||||
gap_time: float = row['time'][0]
|
||||
arrow_kwargs: dict[str, Any] = dict(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
x=iend, # fallback if timestamp lookup fails
|
||||
y=cls,
|
||||
time=gap_time, # for server-side index lookup
|
||||
color=color,
|
||||
alpha=169,
|
||||
pointing=direction,
|
||||
# TODO: expose these as params to markup_gaps()?
|
||||
headLen=10,
|
||||
headWidth=2.222,
|
||||
pxMode=True,
|
||||
)
|
||||
|
||||
aid: int = await actl.add_arrow(
|
||||
**arrow_kwargs
|
||||
)
|
||||
|
||||
# add duration label to RHS of arrow
|
||||
if up_gap:
|
||||
anchor = (0, 0)
|
||||
# ^XXX? i dun get dese dims.. XD
|
||||
elif down_gap:
|
||||
anchor = (0, 1) # XXX y, x?
|
||||
else: # no-gap?
|
||||
assert flat
|
||||
anchor = (0, 0) # up from bottom
|
||||
|
||||
# use a slightly smaller font for gap label txt.
|
||||
font, small_font = get_fonts()
|
||||
font_size: int = small_font.px_size - 1
|
||||
assert isinstance(font_size, int)
|
||||
|
||||
if show_txt:
|
||||
text_aid: int = await actl.add_text(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
text=gap_label,
|
||||
x=iend + 1, # fallback if timestamp lookup fails
|
||||
y=cls,
|
||||
time=gap_time, # server-side index lookup
|
||||
color=color,
|
||||
anchor=anchor,
|
||||
font_size=font_size,
|
||||
)
|
||||
aids[text_aid] = {'text': gap_label}
|
||||
|
||||
# tell chart to redraw all its
|
||||
# graphics view layers Bo
|
||||
await actl.redraw(
|
||||
fqme=fqme,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
return aids
|
||||
|
|
@ -1,206 +0,0 @@
|
|||
'''
|
||||
Smart OHLCV deduplication with data quality validation.
|
||||
|
||||
Handles concurrent write conflicts by keeping the most complete bar
|
||||
(highest volume) while detecting data quality anomalies.
|
||||
|
||||
'''
|
||||
import polars as pl
|
||||
|
||||
from ._anal import with_dts
|
||||
|
||||
|
||||
def dedupe_ohlcv_smart(
|
||||
src_df: pl.DataFrame,
|
||||
time_col: str = 'time',
|
||||
volume_col: str = 'volume',
|
||||
sort: bool = True,
|
||||
|
||||
) -> tuple[
|
||||
pl.DataFrame, # with dts
|
||||
pl.DataFrame, # deduped (keeping higher volume bars)
|
||||
int, # count of dupes removed
|
||||
pl.DataFrame|None, # valid race conditions
|
||||
pl.DataFrame|None, # data quality violations
|
||||
]:
|
||||
'''
|
||||
Smart OHLCV deduplication keeping most complete bars.
|
||||
|
||||
For duplicate timestamps, keeps bar with highest volume under
|
||||
the assumption that higher volume indicates more complete/final
|
||||
data from backfill vs partial live updates.
|
||||
|
||||
Returns
|
||||
-------
|
||||
Tuple of:
|
||||
- wdts: original dataframe with datetime columns added
|
||||
- deduped: deduplicated frame keeping highest-volume bars
|
||||
- diff: number of duplicate rows removed
|
||||
- valid_races: duplicates meeting expected race condition pattern
|
||||
(volume monotonic, OHLC ranges valid)
|
||||
- data_quality_issues: duplicates violating expected relationships
|
||||
indicating provider data problems
|
||||
|
||||
'''
|
||||
wdts: pl.DataFrame = with_dts(src_df)
|
||||
|
||||
# Find duplicate timestamps
|
||||
dupes: pl.DataFrame = wdts.filter(
|
||||
pl.col(time_col).is_duplicated()
|
||||
)
|
||||
|
||||
if dupes.is_empty():
|
||||
# No duplicates, return as-is
|
||||
return (wdts, wdts, 0, None, None)
|
||||
|
||||
# Analyze duplicate groups for validation
|
||||
dupe_analysis: pl.DataFrame = (
|
||||
dupes
|
||||
.sort([time_col, 'index'])
|
||||
.group_by(time_col, maintain_order=True)
|
||||
.agg([
|
||||
pl.col('index').alias('indices'),
|
||||
pl.col('volume').alias('volumes'),
|
||||
pl.col('high').alias('highs'),
|
||||
pl.col('low').alias('lows'),
|
||||
pl.col('open').alias('opens'),
|
||||
pl.col('close').alias('closes'),
|
||||
pl.col('dt').first().alias('dt'),
|
||||
pl.len().alias('count'),
|
||||
])
|
||||
)
|
||||
|
||||
# Validate OHLCV monotonicity for each duplicate group
|
||||
def check_ohlcv_validity(row) -> dict[str, bool]:
|
||||
'''
|
||||
Check if duplicate bars follow expected race condition pattern.
|
||||
|
||||
For a valid live-update → backfill race:
|
||||
- volume should be monotonically increasing
|
||||
- high should be monotonically non-decreasing
|
||||
- low should be monotonically non-increasing
|
||||
- open should be identical (fixed at bar start)
|
||||
|
||||
Returns dict of violation flags.
|
||||
|
||||
'''
|
||||
vols: list = row['volumes']
|
||||
highs: list = row['highs']
|
||||
lows: list = row['lows']
|
||||
opens: list = row['opens']
|
||||
|
||||
violations: dict[str, bool] = {
|
||||
'volume_non_monotonic': False,
|
||||
'high_decreased': False,
|
||||
'low_increased': False,
|
||||
'open_mismatch': False,
|
||||
'identical_bars': False,
|
||||
}
|
||||
|
||||
# Check if all bars are identical (pure duplicate)
|
||||
if (
|
||||
len(set(vols)) == 1
|
||||
and len(set(highs)) == 1
|
||||
and len(set(lows)) == 1
|
||||
and len(set(opens)) == 1
|
||||
):
|
||||
violations['identical_bars'] = True
|
||||
return violations
|
||||
|
||||
# Check volume monotonicity
|
||||
for i in range(1, len(vols)):
|
||||
if vols[i] < vols[i-1]:
|
||||
violations['volume_non_monotonic'] = True
|
||||
break
|
||||
|
||||
# Check high monotonicity (can only increase or stay same)
|
||||
for i in range(1, len(highs)):
|
||||
if highs[i] < highs[i-1]:
|
||||
violations['high_decreased'] = True
|
||||
break
|
||||
|
||||
# Check low monotonicity (can only decrease or stay same)
|
||||
for i in range(1, len(lows)):
|
||||
if lows[i] > lows[i-1]:
|
||||
violations['low_increased'] = True
|
||||
break
|
||||
|
||||
# Check open consistency (should be fixed)
|
||||
if len(set(opens)) > 1:
|
||||
violations['open_mismatch'] = True
|
||||
|
||||
return violations
|
||||
|
||||
# Apply validation
|
||||
dupe_analysis = dupe_analysis.with_columns([
|
||||
pl.struct(['volumes', 'highs', 'lows', 'opens'])
|
||||
.map_elements(
|
||||
check_ohlcv_validity,
|
||||
return_dtype=pl.Struct([
|
||||
pl.Field('volume_non_monotonic', pl.Boolean),
|
||||
pl.Field('high_decreased', pl.Boolean),
|
||||
pl.Field('low_increased', pl.Boolean),
|
||||
pl.Field('open_mismatch', pl.Boolean),
|
||||
pl.Field('identical_bars', pl.Boolean),
|
||||
])
|
||||
)
|
||||
.alias('validity')
|
||||
])
|
||||
|
||||
# Unnest validity struct
|
||||
dupe_analysis = dupe_analysis.unnest('validity')
|
||||
|
||||
# Separate valid races from data quality issues
|
||||
valid_races: pl.DataFrame|None = (
|
||||
dupe_analysis
|
||||
.filter(
|
||||
# Valid if no violations OR just identical bars
|
||||
~pl.col('volume_non_monotonic')
|
||||
& ~pl.col('high_decreased')
|
||||
& ~pl.col('low_increased')
|
||||
& ~pl.col('open_mismatch')
|
||||
)
|
||||
)
|
||||
if valid_races.is_empty():
|
||||
valid_races = None
|
||||
|
||||
data_quality_issues: pl.DataFrame|None = (
|
||||
dupe_analysis
|
||||
.filter(
|
||||
# Issues if any non-identical violation exists
|
||||
(
|
||||
pl.col('volume_non_monotonic')
|
||||
| pl.col('high_decreased')
|
||||
| pl.col('low_increased')
|
||||
| pl.col('open_mismatch')
|
||||
)
|
||||
& ~pl.col('identical_bars')
|
||||
)
|
||||
)
|
||||
if data_quality_issues.is_empty():
|
||||
data_quality_issues = None
|
||||
|
||||
# Deduplicate: keep highest volume bar for each timestamp
|
||||
deduped: pl.DataFrame = (
|
||||
wdts
|
||||
.sort([time_col, volume_col])
|
||||
.unique(
|
||||
subset=[time_col],
|
||||
keep='last',
|
||||
maintain_order=False,
|
||||
)
|
||||
)
|
||||
|
||||
# Re-sort by time or index
|
||||
if sort:
|
||||
deduped = deduped.sort(by=time_col)
|
||||
|
||||
diff: int = wdts.height - deduped.height
|
||||
|
||||
return (
|
||||
wdts,
|
||||
deduped,
|
||||
diff,
|
||||
valid_races,
|
||||
data_quality_issues,
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
228
piker/types.py
228
piker/types.py
|
|
@ -21,6 +21,230 @@ Extensions to built-in or (heavily used but 3rd party) friend-lib
|
|||
types.
|
||||
|
||||
'''
|
||||
from tractor.msg.pretty_struct import (
|
||||
Struct as Struct,
|
||||
from __future__ import annotations
|
||||
from collections import UserList
|
||||
from pprint import (
|
||||
saferepr,
|
||||
)
|
||||
from typing import Any
|
||||
|
||||
from msgspec import (
|
||||
msgpack,
|
||||
Struct as _Struct,
|
||||
structs,
|
||||
)
|
||||
|
||||
|
||||
class DiffDump(UserList):
|
||||
'''
|
||||
Very simple list delegator that repr() dumps (presumed) tuple
|
||||
elements of the form `tuple[str, Any, Any]` in a nice
|
||||
multi-line readable form for analyzing `Struct` diffs.
|
||||
|
||||
'''
|
||||
def __repr__(self) -> str:
|
||||
if not len(self):
|
||||
return super().__repr__()
|
||||
|
||||
# format by displaying item pair's ``repr()`` on multiple,
|
||||
# indented lines such that they are more easily visually
|
||||
# comparable when printed to console when printed to
|
||||
# console.
|
||||
repstr: str = '[\n'
|
||||
for k, left, right in self:
|
||||
repstr += (
|
||||
f'({k},\n'
|
||||
f'\t{repr(left)},\n'
|
||||
f'\t{repr(right)},\n'
|
||||
')\n'
|
||||
)
|
||||
repstr += ']\n'
|
||||
return repstr
|
||||
|
||||
|
||||
class Struct(
|
||||
_Struct,
|
||||
|
||||
# https://jcristharif.com/msgspec/structs.html#tagged-unions
|
||||
# tag='pikerstruct',
|
||||
# tag=True,
|
||||
):
|
||||
'''
|
||||
A "human friendlier" (aka repl buddy) struct subtype.
|
||||
|
||||
'''
|
||||
def _sin_props(self) -> Iterator[
|
||||
tuple[
|
||||
structs.FieldIinfo,
|
||||
str,
|
||||
Any,
|
||||
]
|
||||
]:
|
||||
'''
|
||||
Iterate over all non-@property fields of this struct.
|
||||
|
||||
'''
|
||||
fi: structs.FieldInfo
|
||||
for fi in structs.fields(self):
|
||||
key: str = fi.name
|
||||
val: Any = getattr(self, key)
|
||||
yield fi, key, val
|
||||
|
||||
def to_dict(
|
||||
self,
|
||||
include_non_members: bool = True,
|
||||
|
||||
) -> dict:
|
||||
'''
|
||||
Like it sounds.. direct delegation to:
|
||||
https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
|
||||
|
||||
BUT, by default we pop all non-member (aka not defined as
|
||||
struct fields) fields by default.
|
||||
|
||||
'''
|
||||
asdict: dict = structs.asdict(self)
|
||||
if include_non_members:
|
||||
return asdict
|
||||
|
||||
# only return a dict of the struct members
|
||||
# which were provided as input, NOT anything
|
||||
# added as type-defined `@property` methods!
|
||||
sin_props: dict = {}
|
||||
fi: structs.FieldInfo
|
||||
for fi, k, v in self._sin_props():
|
||||
sin_props[k] = asdict[k]
|
||||
|
||||
return sin_props
|
||||
|
||||
def pformat(
|
||||
self,
|
||||
field_indent: int = 2,
|
||||
indent: int = 0,
|
||||
|
||||
) -> str:
|
||||
'''
|
||||
Recursion-safe `pprint.pformat()` style formatting of
|
||||
a `msgspec.Struct` for sane reading by a human using a REPL.
|
||||
|
||||
'''
|
||||
# global whitespace indent
|
||||
ws: str = ' '*indent
|
||||
|
||||
# field whitespace indent
|
||||
field_ws: str = ' '*(field_indent + indent)
|
||||
|
||||
# qtn: str = ws + self.__class__.__qualname__
|
||||
qtn: str = self.__class__.__qualname__
|
||||
|
||||
obj_str: str = '' # accumulator
|
||||
fi: structs.FieldInfo
|
||||
k: str
|
||||
v: Any
|
||||
for fi, k, v in self._sin_props():
|
||||
|
||||
# TODO: how can we prefer `Literal['option1', 'option2,
|
||||
# ..]` over .__name__ == `Literal` but still get only the
|
||||
# latter for simple types like `str | int | None` etc..?
|
||||
ft: type = fi.type
|
||||
typ_name: str = getattr(ft, '__name__', str(ft))
|
||||
|
||||
# recurse to get sub-struct's `.pformat()` output Bo
|
||||
if isinstance(v, Struct):
|
||||
val_str: str = v.pformat(
|
||||
indent=field_indent + indent,
|
||||
field_indent=indent + field_indent,
|
||||
)
|
||||
|
||||
else: # the `pprint` recursion-safe format:
|
||||
# https://docs.python.org/3.11/library/pprint.html#pprint.saferepr
|
||||
val_str: str = saferepr(v)
|
||||
|
||||
obj_str += (field_ws + f'{k}: {typ_name} = {val_str},\n')
|
||||
|
||||
return (
|
||||
f'{qtn}(\n'
|
||||
f'{obj_str}'
|
||||
f'{ws})'
|
||||
)
|
||||
|
||||
# TODO: use a pprint.PrettyPrinter instance around ONLY rendering
|
||||
# inside a known tty?
|
||||
# def __repr__(self) -> str:
|
||||
# ...
|
||||
|
||||
# __str__ = __repr__ = pformat
|
||||
__repr__ = pformat
|
||||
|
||||
def copy(
|
||||
self,
|
||||
update: dict | None = None,
|
||||
|
||||
) -> Struct:
|
||||
'''
|
||||
Validate-typecast all self defined fields, return a copy of
|
||||
us with all such fields.
|
||||
|
||||
NOTE: This is kinda like the default behaviour in
|
||||
`pydantic.BaseModel` except a copy of the object is
|
||||
returned making it compat with `frozen=True`.
|
||||
|
||||
'''
|
||||
if update:
|
||||
for k, v in update.items():
|
||||
setattr(self, k, v)
|
||||
|
||||
# NOTE: roundtrip serialize to validate
|
||||
# - enode to msgpack binary format,
|
||||
# - decode that back to a struct.
|
||||
return msgpack.Decoder(type=type(self)).decode(
|
||||
msgpack.Encoder().encode(self)
|
||||
)
|
||||
|
||||
def typecast(
|
||||
self,
|
||||
|
||||
# TODO: allow only casting a named subset?
|
||||
# fields: set[str] | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Cast all fields using their declared type annotations
|
||||
(kinda like what `pydantic` does by default).
|
||||
|
||||
NOTE: this of course won't work on frozen types, use
|
||||
``.copy()`` above in such cases.
|
||||
|
||||
'''
|
||||
# https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
|
||||
fi: structs.FieldInfo
|
||||
for fi in structs.fields(self):
|
||||
setattr(
|
||||
self,
|
||||
fi.name,
|
||||
fi.type(getattr(self, fi.name)),
|
||||
)
|
||||
|
||||
def __sub__(
|
||||
self,
|
||||
other: Struct,
|
||||
|
||||
) -> DiffDump[tuple[str, Any, Any]]:
|
||||
'''
|
||||
Compare fields/items key-wise and return a ``DiffDump``
|
||||
for easy visual REPL comparison B)
|
||||
|
||||
'''
|
||||
diffs: DiffDump[tuple[str, Any, Any]] = DiffDump()
|
||||
for fi in structs.fields(self):
|
||||
attr_name: str = fi.name
|
||||
ours: Any = getattr(self, attr_name)
|
||||
theirs: Any = getattr(other, attr_name)
|
||||
if ours != theirs:
|
||||
diffs.append((
|
||||
attr_name,
|
||||
ours,
|
||||
theirs,
|
||||
))
|
||||
|
||||
return diffs
|
||||
|
|
|
|||
|
|
@ -21,24 +21,20 @@ Main app startup and run.
|
|||
from functools import partial
|
||||
from types import ModuleType
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
from piker.ui.qt import (
|
||||
QEvent,
|
||||
)
|
||||
from . import _chart
|
||||
from . import _event
|
||||
from . import _search
|
||||
from ..accounting import unpack_fqme
|
||||
from ..data._symcache import open_symcache
|
||||
from ..data.feed import install_brokerd_search
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from ..service import maybe_spawn_brokerd
|
||||
from . import _event
|
||||
from ._exec import run_qtractor
|
||||
from ..data.feed import install_brokerd_search
|
||||
from ..data._symcache import open_symcache
|
||||
from ..accounting import unpack_fqme
|
||||
from . import _search
|
||||
from ._chart import GodWidget
|
||||
from ..log import get_logger
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
|
@ -76,8 +72,8 @@ async def load_provider_search(
|
|||
|
||||
async def _async_main(
|
||||
|
||||
# implicit required argument provided by `qtractor_run()`
|
||||
main_widget: _chart.GodWidget,
|
||||
# implicit required argument provided by ``qtractor_run()``
|
||||
main_widget: GodWidget,
|
||||
|
||||
syms: list[str],
|
||||
brokers: dict[str, ModuleType],
|
||||
|
|
@ -90,16 +86,6 @@ async def _async_main(
|
|||
Provision the "main" widget with initial symbol data and root nursery.
|
||||
|
||||
"""
|
||||
# enable chart's console logging
|
||||
if loglevel:
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# set as singleton
|
||||
_chart._godw = main_widget
|
||||
|
||||
from . import _display
|
||||
from ._pg_overrides import _do_overrides
|
||||
_do_overrides()
|
||||
|
|
@ -130,7 +116,6 @@ async def _async_main(
|
|||
needed_brokermods[brokername] = brokers[brokername]
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as root_n,
|
||||
):
|
||||
# set root nursery and task stack for spawning other charts/feeds
|
||||
|
|
@ -214,6 +199,6 @@ def _main(
|
|||
brokermods,
|
||||
piker_loglevel,
|
||||
),
|
||||
main_widget_type=_chart.GodWidget,
|
||||
main_widget_type=GodWidget,
|
||||
tractor_kwargs=tractor_kwargs,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -29,9 +29,11 @@ from typing import (
|
|||
)
|
||||
|
||||
import pyqtgraph as pg
|
||||
import trio
|
||||
|
||||
from piker.ui.qt import (
|
||||
QtCore,
|
||||
QtWidgets,
|
||||
Qt,
|
||||
QLineF,
|
||||
QFrame,
|
||||
|
|
@ -40,7 +42,6 @@ from piker.ui.qt import (
|
|||
QVBoxLayout,
|
||||
QSplitter,
|
||||
)
|
||||
from ._widget import GodWidget
|
||||
from ._axes import (
|
||||
DynamicDateAxis,
|
||||
PriceAxis,
|
||||
|
|
@ -61,6 +62,10 @@ from ._style import (
|
|||
_xaxis_at,
|
||||
# _min_points_to_show,
|
||||
)
|
||||
from ..data.feed import (
|
||||
Feed,
|
||||
Flume,
|
||||
)
|
||||
from ..accounting import (
|
||||
MktPair,
|
||||
)
|
||||
|
|
@ -74,12 +79,286 @@ from . import _pg_overrides as pgo
|
|||
|
||||
if TYPE_CHECKING:
|
||||
from ._display import DisplayState
|
||||
from ..data.flows import Flume
|
||||
from ..data.feed import Feed
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
class GodWidget(QWidget):
|
||||
'''
|
||||
"Our lord and savior, the holy child of window-shua, there is no
|
||||
widget above thee." - 6|6
|
||||
|
||||
The highest level composed widget which contains layouts for
|
||||
organizing charts as well as other sub-widgets used to control or
|
||||
modify them.
|
||||
|
||||
'''
|
||||
search: SearchWidget
|
||||
mode_name: str = 'god'
|
||||
|
||||
def __init__(
|
||||
|
||||
self,
|
||||
parent=None,
|
||||
|
||||
) -> None:
|
||||
|
||||
super().__init__(parent)
|
||||
|
||||
self.search: SearchWidget | None = None
|
||||
|
||||
self.hbox = QHBoxLayout(self)
|
||||
self.hbox.setContentsMargins(0, 0, 0, 0)
|
||||
self.hbox.setSpacing(6)
|
||||
self.hbox.setAlignment(Qt.AlignTop)
|
||||
|
||||
self.vbox = QVBoxLayout()
|
||||
self.vbox.setContentsMargins(0, 0, 0, 0)
|
||||
self.vbox.setSpacing(2)
|
||||
self.vbox.setAlignment(Qt.AlignTop)
|
||||
|
||||
self.hbox.addLayout(self.vbox)
|
||||
|
||||
self._chart_cache: dict[
|
||||
str,
|
||||
tuple[LinkedSplits, LinkedSplits],
|
||||
] = {}
|
||||
|
||||
self.hist_linked: LinkedSplits | None = None
|
||||
self.rt_linked: LinkedSplits | None = None
|
||||
self._active_cursor: Cursor | None = None
|
||||
|
||||
# assigned in the startup func `_async_main()`
|
||||
self._root_n: trio.Nursery = None
|
||||
|
||||
self._widgets: dict[str, QWidget] = {}
|
||||
self._resizing: bool = False
|
||||
|
||||
# TODO: do we need this, when would god get resized
|
||||
# and the window does not? Never right?!
|
||||
# self.reg_for_resize(self)
|
||||
|
||||
# TODO: strat loader/saver that we don't need yet.
|
||||
# def init_strategy_ui(self):
|
||||
# self.toolbar_layout = QHBoxLayout()
|
||||
# self.toolbar_layout.setContentsMargins(0, 0, 0, 0)
|
||||
# self.vbox.addLayout(self.toolbar_layout)
|
||||
# self.strategy_box = StrategyBoxWidget(self)
|
||||
# self.toolbar_layout.addWidget(self.strategy_box)
|
||||
|
||||
@property
|
||||
def linkedsplits(self) -> LinkedSplits:
|
||||
return self.rt_linked
|
||||
|
||||
def set_chart_symbols(
|
||||
self,
|
||||
group_key: tuple[str], # of form <fqme>.<providername>
|
||||
all_linked: tuple[LinkedSplits, LinkedSplits], # type: ignore
|
||||
|
||||
) -> None:
|
||||
# re-sort org cache symbol list in LIFO order
|
||||
cache = self._chart_cache
|
||||
cache.pop(group_key, None)
|
||||
cache[group_key] = all_linked
|
||||
|
||||
def get_chart_symbols(
|
||||
self,
|
||||
symbol_key: str,
|
||||
|
||||
) -> tuple[LinkedSplits, LinkedSplits]: # type: ignore
|
||||
return self._chart_cache.get(symbol_key)
|
||||
|
||||
async def load_symbols(
|
||||
self,
|
||||
fqmes: list[str],
|
||||
loglevel: str,
|
||||
reset: bool = False,
|
||||
|
||||
) -> trio.Event:
|
||||
'''
|
||||
Load a new contract into the charting app.
|
||||
|
||||
Expects a ``numpy`` structured array containing all the ohlcv fields.
|
||||
|
||||
'''
|
||||
# NOTE: for now we use the first symbol in the set as the "key"
|
||||
# for the overlay of feeds on the chart.
|
||||
group_key: tuple[str] = tuple(fqmes)
|
||||
|
||||
all_linked = self.get_chart_symbols(group_key)
|
||||
order_mode_started = trio.Event()
|
||||
|
||||
if not self.vbox.isEmpty():
|
||||
|
||||
# XXX: seems to make switching slower?
|
||||
# qframe = self.hist_linked.chart.qframe
|
||||
# if qframe.sidepane is self.search:
|
||||
# qframe.hbox.removeWidget(self.search)
|
||||
|
||||
for linked in [self.rt_linked, self.hist_linked]:
|
||||
# XXX: this is CRITICAL especially with pixel buffer caching
|
||||
linked.hide()
|
||||
linked.unfocus()
|
||||
|
||||
# XXX: pretty sure we don't need this
|
||||
# remove any existing plots?
|
||||
# XXX: ahh we might want to support cache unloading..
|
||||
# self.vbox.removeWidget(linked)
|
||||
|
||||
# switching to a new viewable chart
|
||||
if all_linked is None or reset:
|
||||
from ._display import display_symbol_data
|
||||
|
||||
# we must load a fresh linked charts set
|
||||
self.rt_linked = rt_charts = LinkedSplits(self)
|
||||
self.hist_linked = hist_charts = LinkedSplits(self)
|
||||
|
||||
# spawn new task to start up and update new sub-chart instances
|
||||
self._root_n.start_soon(
|
||||
display_symbol_data,
|
||||
self,
|
||||
fqmes,
|
||||
loglevel,
|
||||
order_mode_started,
|
||||
)
|
||||
|
||||
# self.vbox.addWidget(hist_charts)
|
||||
self.vbox.addWidget(rt_charts)
|
||||
self.set_chart_symbols(
|
||||
group_key,
|
||||
(hist_charts, rt_charts),
|
||||
)
|
||||
|
||||
for linked in [hist_charts, rt_charts]:
|
||||
linked.show()
|
||||
linked.focus()
|
||||
|
||||
await trio.sleep(0)
|
||||
|
||||
else:
|
||||
# symbol is already loaded and ems ready
|
||||
order_mode_started.set()
|
||||
|
||||
self.hist_linked, self.rt_linked = all_linked
|
||||
|
||||
for linked in all_linked:
|
||||
# TODO:
|
||||
# - we'll probably want per-instrument/provider state here?
|
||||
# change the order config form over to the new chart
|
||||
|
||||
# chart is already in memory so just focus it
|
||||
linked.show()
|
||||
linked.focus()
|
||||
linked.graphics_cycle()
|
||||
await trio.sleep(0)
|
||||
|
||||
# resume feeds *after* rendering chart view asap
|
||||
chart = linked.chart
|
||||
if chart:
|
||||
chart.resume_all_feeds()
|
||||
|
||||
# TODO: we need a check to see if the chart
|
||||
# last had the xlast in view, if so then shift so it's
|
||||
# still in view, if the user was viewing history then
|
||||
# do nothing yah?
|
||||
self.rt_linked.chart.main_viz.default_view(
|
||||
do_min_bars=True,
|
||||
)
|
||||
|
||||
# if a history chart instance is already up then
|
||||
# set the search widget as its sidepane.
|
||||
hist_chart = self.hist_linked.chart
|
||||
if hist_chart:
|
||||
hist_chart.qframe.set_sidepane(self.search)
|
||||
|
||||
# NOTE: this is really stupid/hard to follow.
|
||||
# we have to reposition the active position nav
|
||||
# **AFTER** applying the search bar as a sidepane
|
||||
# to the newly switched to symbol.
|
||||
await trio.sleep(0)
|
||||
|
||||
# TODO: probably stick this in some kinda `LooknFeel` API?
|
||||
for tracker in self.rt_linked.mode.trackers.values():
|
||||
pp_nav = tracker.nav
|
||||
if tracker.live_pp.cumsize:
|
||||
pp_nav.show()
|
||||
pp_nav.hide_info()
|
||||
else:
|
||||
pp_nav.hide()
|
||||
|
||||
# set window titlebar info
|
||||
symbol = self.rt_linked.mkt
|
||||
if symbol is not None:
|
||||
self.window.setWindowTitle(
|
||||
f'{symbol.fqme} '
|
||||
f'tick:{symbol.size_tick}'
|
||||
)
|
||||
|
||||
return order_mode_started
|
||||
|
||||
def focus(self) -> None:
|
||||
'''
|
||||
Focus the top level widget which in turn focusses the chart
|
||||
ala "view mode".
|
||||
|
||||
'''
|
||||
# go back to view-mode focus (aka chart focus)
|
||||
self.clearFocus()
|
||||
chart = self.rt_linked.chart
|
||||
if chart:
|
||||
chart.setFocus()
|
||||
|
||||
def reg_for_resize(
|
||||
self,
|
||||
widget: QWidget,
|
||||
) -> None:
|
||||
getattr(widget, 'on_resize')
|
||||
self._widgets[widget.mode_name] = widget
|
||||
|
||||
def on_win_resize(self, event: QtCore.QEvent) -> None:
|
||||
'''
|
||||
Top level god widget handler from window (the real yaweh) resize
|
||||
events such that any registered widgets which wish to be
|
||||
notified are invoked using our pythonic `.on_resize()` method
|
||||
api.
|
||||
|
||||
Where we do UX magic to make things not suck B)
|
||||
|
||||
'''
|
||||
if self._resizing:
|
||||
return
|
||||
|
||||
self._resizing = True
|
||||
|
||||
log.info('God widget resize')
|
||||
for name, widget in self._widgets.items():
|
||||
widget.on_resize()
|
||||
|
||||
self._resizing = False
|
||||
|
||||
# on_resize = on_win_resize
|
||||
|
||||
def get_cursor(self) -> Cursor:
|
||||
return self._active_cursor
|
||||
|
||||
def iter_linked(self) -> Iterator[LinkedSplits]:
|
||||
for linked in [self.hist_linked, self.rt_linked]:
|
||||
yield linked
|
||||
|
||||
def resize_all(self) -> None:
|
||||
'''
|
||||
Dynamic resize sequence: adjusts all sub-widgets/charts to
|
||||
sensible default ratios of what space is detected as available
|
||||
on the display / window.
|
||||
|
||||
'''
|
||||
rt_linked = self.rt_linked
|
||||
rt_linked.set_split_sizes()
|
||||
self.rt_linked.resize_sidepanes()
|
||||
self.hist_linked.resize_sidepanes(from_linked=rt_linked)
|
||||
self.search.on_resize()
|
||||
|
||||
|
||||
class ChartnPane(QFrame):
|
||||
'''
|
||||
One-off ``QFrame`` composite which pairs a chart
|
||||
|
|
@ -91,9 +370,9 @@ class ChartnPane(QFrame):
|
|||
https://doc.qt.io/qt-5/qwidget.html#composite-widgets
|
||||
|
||||
'''
|
||||
sidepane: FieldsForm|SearchWidget
|
||||
sidepane: FieldsForm | SearchWidget
|
||||
hbox: QHBoxLayout
|
||||
chart: ChartPlotWidget|None = None
|
||||
chart: ChartPlotWidget | None = None
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
|
@ -109,13 +388,13 @@ class ChartnPane(QFrame):
|
|||
self.chart = None
|
||||
|
||||
hbox = self.hbox = QHBoxLayout(self)
|
||||
hbox.setAlignment(Qt.AlignTop|Qt.AlignLeft)
|
||||
hbox.setAlignment(Qt.AlignTop | Qt.AlignLeft)
|
||||
hbox.setContentsMargins(0, 0, 0, 0)
|
||||
hbox.setSpacing(3)
|
||||
|
||||
def set_sidepane(
|
||||
self,
|
||||
sidepane: FieldsForm|SearchWidget,
|
||||
sidepane: FieldsForm | SearchWidget,
|
||||
) -> None:
|
||||
|
||||
# add sidepane **after** chart; place it on axis side
|
||||
|
|
@ -126,7 +405,7 @@ class ChartnPane(QFrame):
|
|||
self._sidepane = sidepane
|
||||
|
||||
@property
|
||||
def sidepane(self) -> FieldsForm|SearchWidget:
|
||||
def sidepane(self) -> FieldsForm | SearchWidget:
|
||||
return self._sidepane
|
||||
|
||||
|
||||
|
|
@ -141,6 +420,7 @@ class LinkedSplits(QWidget):
|
|||
|
||||
'''
|
||||
def __init__(
|
||||
|
||||
self,
|
||||
godwidget: GodWidget,
|
||||
|
||||
|
|
@ -171,7 +451,7 @@ class LinkedSplits(QWidget):
|
|||
# chart-local graphics state that can be passed to
|
||||
# a ``graphic_update_cycle()`` call by any task wishing to
|
||||
# update the UI for a given "chart instance".
|
||||
self.display_state: DisplayState|None = None
|
||||
self.display_state: DisplayState | None = None
|
||||
|
||||
self._mkt: MktPair = None
|
||||
|
||||
|
|
@ -207,7 +487,7 @@ class LinkedSplits(QWidget):
|
|||
|
||||
def set_split_sizes(
|
||||
self,
|
||||
prop: float|None = None,
|
||||
prop: float | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -288,8 +568,8 @@ class LinkedSplits(QWidget):
|
|||
|
||||
# style?
|
||||
self.chart.setFrameStyle(
|
||||
QFrame.Shape.StyledPanel
|
||||
|QFrame.Shadow.Plain
|
||||
QFrame.Shape.StyledPanel |
|
||||
QFrame.Shadow.Plain
|
||||
)
|
||||
|
||||
return self.chart
|
||||
|
|
@ -301,11 +581,11 @@ class LinkedSplits(QWidget):
|
|||
shm: ShmArray,
|
||||
flume: Flume,
|
||||
|
||||
array_key: str|None = None,
|
||||
array_key: str | None = None,
|
||||
style: str = 'line',
|
||||
_is_main: bool = False,
|
||||
|
||||
sidepane: QWidget|None = None,
|
||||
sidepane: QWidget | None = None,
|
||||
draw_kwargs: dict = {},
|
||||
|
||||
**cpw_kwargs,
|
||||
|
|
@ -408,7 +688,7 @@ class LinkedSplits(QWidget):
|
|||
cpw.plotItem.vb.linked = self
|
||||
cpw.setFrameStyle(
|
||||
QFrame.Shape.StyledPanel
|
||||
# |QFrame.Shadow.Plain
|
||||
# | QFrame.Shadow.Plain
|
||||
)
|
||||
|
||||
# don't show the little "autoscale" A label.
|
||||
|
|
@ -521,7 +801,7 @@ class LinkedSplits(QWidget):
|
|||
|
||||
def resize_sidepanes(
|
||||
self,
|
||||
from_linked: LinkedSplits|None = None,
|
||||
from_linked: LinkedSplits | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -595,7 +875,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
# TODO: load from config
|
||||
use_open_gl: bool = False,
|
||||
|
||||
static_yrange: tuple[float, float]|None = None,
|
||||
static_yrange: tuple[float, float] | None = None,
|
||||
|
||||
parent=None,
|
||||
**kwargs,
|
||||
|
|
@ -610,7 +890,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
|
||||
# NOTE: must be set bfore calling ``.mk_vb()``
|
||||
self.linked = linkedsplits
|
||||
self.sidepane: FieldsForm|None = None
|
||||
self.sidepane: FieldsForm | None = None
|
||||
|
||||
# source of our custom interactions
|
||||
self.cv = self.mk_vb(name)
|
||||
|
|
@ -644,7 +924,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
self.useOpenGL(use_open_gl)
|
||||
self.name = name
|
||||
self.data_key = data_key or name
|
||||
self.qframe: ChartnPane|None = None
|
||||
self.qframe: ChartnPane | None = None
|
||||
|
||||
# scene-local placeholder for book graphics
|
||||
# sizing to avoid overlap with data contents
|
||||
|
|
@ -655,7 +935,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
# registry of overlay curve names
|
||||
self._vizs: dict[str, Viz] = {}
|
||||
|
||||
self.feed: Feed|None = None
|
||||
self.feed: Feed | None = None
|
||||
|
||||
self._labels = {} # registry of underlying graphics
|
||||
self._ysticks = {} # registry of underlying graphics
|
||||
|
|
@ -748,11 +1028,11 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
def increment_view(
|
||||
self,
|
||||
datums: int = 1,
|
||||
vb: ChartView|None = None,
|
||||
vb: ChartView | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Increment the data view `datums`` steps toward y-axis thus
|
||||
Increment the data view ``datums``` steps toward y-axis thus
|
||||
"following" the current time slot/step/bar.
|
||||
|
||||
'''
|
||||
|
|
@ -762,7 +1042,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
x_shift = viz.index_step() * datums
|
||||
|
||||
if datums >= 300:
|
||||
log.warning('FUCKING FIX THE GLOBAL STEP BULLSHIT')
|
||||
print("FUCKING FIX THE GLOBAL STEP BULLSHIT")
|
||||
# breakpoint()
|
||||
return
|
||||
|
||||
|
|
@ -779,8 +1059,8 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
def overlay_plotitem(
|
||||
self,
|
||||
name: str,
|
||||
index: int|None = None,
|
||||
axis_title: str|None = None,
|
||||
index: int | None = None,
|
||||
axis_title: str | None = None,
|
||||
axis_side: str = 'right',
|
||||
axis_kwargs: dict = {},
|
||||
|
||||
|
|
@ -868,14 +1148,14 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
shm: ShmArray,
|
||||
flume: Flume,
|
||||
|
||||
array_key: str|None = None,
|
||||
array_key: str | None = None,
|
||||
overlay: bool = False,
|
||||
color: str|None = None,
|
||||
color: str | None = None,
|
||||
add_label: bool = True,
|
||||
pi: pg.PlotItem|None = None,
|
||||
pi: pg.PlotItem | None = None,
|
||||
step_mode: bool = False,
|
||||
is_ohlc: bool = False,
|
||||
add_sticky: None|str = 'right',
|
||||
add_sticky: None | str = 'right',
|
||||
|
||||
**graphics_kwargs,
|
||||
|
||||
|
|
@ -973,7 +1253,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
# use the tick size precision for display
|
||||
name = name or pi.name
|
||||
mkt: MktPair = self.linked.mkt
|
||||
digits: int|None = None
|
||||
digits: int | None = None
|
||||
if name in mkt.fqme:
|
||||
digits = mkt.price_tick_digits
|
||||
|
||||
|
|
@ -1007,7 +1287,7 @@ class ChartPlotWidget(pg.PlotWidget):
|
|||
shm: ShmArray,
|
||||
flume: Flume,
|
||||
|
||||
array_key: str|None = None,
|
||||
array_key: str | None = None,
|
||||
**draw_curve_kwargs,
|
||||
|
||||
) -> Viz:
|
||||
|
|
|
|||
|
|
@ -413,18 +413,9 @@ class Cursor(pg.GraphicsObject):
|
|||
self,
|
||||
item: pg.GraphicsObject,
|
||||
) -> None:
|
||||
assert getattr(
|
||||
item,
|
||||
'delete',
|
||||
), f"{item} must define a ``.delete()``"
|
||||
assert getattr(item, 'delete'), f"{item} must define a ``.delete()``"
|
||||
self._hovered.add(item)
|
||||
|
||||
def is_hovered(
|
||||
self,
|
||||
item: pg.GraphicsObject,
|
||||
) -> bool:
|
||||
return item in self._hovered
|
||||
|
||||
def add_plot(
|
||||
self,
|
||||
plot: ChartPlotWidget, # noqa
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ import pyqtgraph as pg
|
|||
|
||||
from piker.ui.qt import (
|
||||
QtWidgets,
|
||||
QGraphicsItem,
|
||||
Qt,
|
||||
QLineF,
|
||||
QRectF,
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ from piker.ui.qt import QLineF
|
|||
from ..data._sharedmem import (
|
||||
ShmArray,
|
||||
)
|
||||
from ..data.flows import Flume
|
||||
from ..data.feed import Flume
|
||||
from ..data._formatters import (
|
||||
IncrementalFormatter,
|
||||
OHLCBarsFmtr, # Plain OHLC renderer
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ this module ties together quote and computational (fsp) streams with
|
|||
graphics update methods via our custom ``pyqtgraph`` charting api.
|
||||
|
||||
'''
|
||||
from functools import partial
|
||||
import itertools
|
||||
from math import floor
|
||||
import time
|
||||
|
|
@ -209,7 +208,6 @@ class DisplayState(Struct):
|
|||
async def increment_history_view(
|
||||
# min_istream: tractor.MsgStream,
|
||||
ds: DisplayState,
|
||||
loglevel: str = 'warning',
|
||||
):
|
||||
hist_chart: ChartPlotWidget = ds.hist_chart
|
||||
hist_viz: Viz = ds.hist_viz
|
||||
|
|
@ -231,10 +229,7 @@ async def increment_history_view(
|
|||
hist_viz.reset_graphics()
|
||||
# hist_viz.update_graphics(force_redraw=True)
|
||||
|
||||
async with open_sample_stream(
|
||||
period_s=1.,
|
||||
loglevel=loglevel,
|
||||
) as min_istream:
|
||||
async with open_sample_stream(1.) as min_istream:
|
||||
async for msg in min_istream:
|
||||
|
||||
profiler = Profiler(
|
||||
|
|
@ -315,6 +310,7 @@ async def increment_history_view(
|
|||
|
||||
|
||||
async def graphics_update_loop(
|
||||
|
||||
dss: dict[str, DisplayState],
|
||||
nurse: trio.Nursery,
|
||||
godwidget: GodWidget,
|
||||
|
|
@ -323,7 +319,6 @@ async def graphics_update_loop(
|
|||
|
||||
pis: dict[str, list[pgo.PlotItem, pgo.PlotItem]] = {},
|
||||
vlm_charts: dict[str, ChartPlotWidget] = {},
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -467,12 +462,9 @@ async def graphics_update_loop(
|
|||
# })
|
||||
|
||||
nurse.start_soon(
|
||||
partial(
|
||||
increment_history_view,
|
||||
# min_istream,
|
||||
ds=ds,
|
||||
loglevel=loglevel,
|
||||
),
|
||||
increment_history_view,
|
||||
# min_istream,
|
||||
ds,
|
||||
)
|
||||
await trio.sleep(0)
|
||||
|
||||
|
|
@ -519,19 +511,14 @@ async def graphics_update_loop(
|
|||
fast_chart.linked.isHidden()
|
||||
or not rt_pi.isVisible()
|
||||
):
|
||||
log.debug(
|
||||
f'{fqme} skipping update for HIDDEN CHART'
|
||||
)
|
||||
print(f'{fqme} skipping update for HIDDEN CHART')
|
||||
fast_chart.pause_all_feeds()
|
||||
continue
|
||||
|
||||
ic = fast_chart.view._in_interact
|
||||
if ic:
|
||||
fast_chart.pause_all_feeds()
|
||||
log.debug(
|
||||
f'Pausing chart updaates during interaction\n'
|
||||
f'fqme: {fqme!r}'
|
||||
)
|
||||
print(f'{fqme} PAUSING DURING INTERACTION')
|
||||
await ic.wait()
|
||||
fast_chart.resume_all_feeds()
|
||||
|
||||
|
|
@ -1458,10 +1445,7 @@ async def display_symbol_data(
|
|||
# for pause/resume on mouse interaction
|
||||
rt_chart.feed = feed
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as ln,
|
||||
):
|
||||
async with trio.open_nursery() as ln:
|
||||
# if available load volume related built-in display(s)
|
||||
vlm_charts: dict[
|
||||
str,
|
||||
|
|
@ -1604,18 +1588,15 @@ async def display_symbol_data(
|
|||
# start update loop task
|
||||
dss: dict[str, DisplayState] = {}
|
||||
ln.start_soon(
|
||||
partial(
|
||||
graphics_update_loop,
|
||||
dss=dss,
|
||||
nurse=ln,
|
||||
godwidget=godwidget,
|
||||
feed=feed,
|
||||
# min_istream,
|
||||
graphics_update_loop,
|
||||
dss,
|
||||
ln,
|
||||
godwidget,
|
||||
feed,
|
||||
# min_istream,
|
||||
|
||||
pis=pis,
|
||||
vlm_charts=vlm_charts,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
pis,
|
||||
vlm_charts,
|
||||
)
|
||||
|
||||
# boot order-mode
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ Higher level annotation editors.
|
|||
from __future__ import annotations
|
||||
from collections import defaultdict
|
||||
from typing import (
|
||||
Literal,
|
||||
Sequence,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
|
@ -55,11 +54,6 @@ from ._style import (
|
|||
from ._lines import LevelLine
|
||||
from ..log import get_logger
|
||||
|
||||
# TODO, rm the cycle here!
|
||||
from ._widget import (
|
||||
GodWidget,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._chart import (
|
||||
GodWidget,
|
||||
|
|
@ -72,18 +66,9 @@ log = get_logger(__name__)
|
|||
|
||||
|
||||
class ArrowEditor(Struct):
|
||||
'''
|
||||
Annotate a chart-view with arrows most often used for indicating,
|
||||
- order txns/clears,
|
||||
- positions directions,
|
||||
- general points-of-interest like nooz events.
|
||||
|
||||
'''
|
||||
godw: GodWidget = None # type: ignore # noqa
|
||||
_arrows: dict[
|
||||
str,
|
||||
list[pg.ArrowItem]
|
||||
] = {}
|
||||
_arrows: dict[str, list[pg.ArrowItem]] = {}
|
||||
|
||||
def add(
|
||||
self,
|
||||
|
|
@ -91,19 +76,8 @@ class ArrowEditor(Struct):
|
|||
uid: str,
|
||||
x: float,
|
||||
y: float,
|
||||
color: str|None = None,
|
||||
pointing: Literal[
|
||||
'up',
|
||||
'down',
|
||||
None,
|
||||
] = None,
|
||||
alpha: int = 255,
|
||||
zval: float = 1e9,
|
||||
headLen: float|None = None,
|
||||
headWidth: float|None = None,
|
||||
tailLen: float|None = None,
|
||||
tailWidth: float|None = None,
|
||||
pxMode: bool = True,
|
||||
color: str = 'default',
|
||||
pointing: str | None = None,
|
||||
|
||||
) -> pg.ArrowItem:
|
||||
'''
|
||||
|
|
@ -119,83 +93,29 @@ class ArrowEditor(Struct):
|
|||
# scale arrow sizing to dpi-aware font
|
||||
size = _font.font.pixelSize() * 0.8
|
||||
|
||||
# allow caller override of head dimensions
|
||||
if headLen is None:
|
||||
headLen = size
|
||||
if headWidth is None:
|
||||
headWidth = size/2
|
||||
# tail params default to None (no tail)
|
||||
if tailWidth is None:
|
||||
tailWidth = 3
|
||||
|
||||
color = color or 'default'
|
||||
color = QColor(hcolor(color))
|
||||
color.setAlpha(alpha)
|
||||
pen = fn.mkPen(color, width=1)
|
||||
brush = fn.mkBrush(color)
|
||||
arrow = pg.ArrowItem(
|
||||
angle=angle,
|
||||
baseAngle=0,
|
||||
headLen=headLen,
|
||||
headWidth=headWidth,
|
||||
tailLen=tailLen,
|
||||
tailWidth=tailWidth,
|
||||
pxMode=pxMode,
|
||||
# coloring
|
||||
pen=pen,
|
||||
brush=brush,
|
||||
)
|
||||
arrow.setZValue(zval)
|
||||
arrow.setPos(x, y)
|
||||
plot.addItem(arrow) # render to view
|
||||
headLen=size,
|
||||
headWidth=size/2,
|
||||
tailLen=None,
|
||||
pxMode=True,
|
||||
|
||||
# register for removal
|
||||
arrow._uid = uid
|
||||
self._arrows.setdefault(
|
||||
uid, []
|
||||
).append(arrow)
|
||||
# coloring
|
||||
pen=pg.mkPen(hcolor('papas_special')),
|
||||
brush=pg.mkBrush(hcolor(color)),
|
||||
)
|
||||
arrow.setPos(x, y)
|
||||
self._arrows.setdefault(uid, []).append(arrow)
|
||||
|
||||
# render to view
|
||||
plot.addItem(arrow)
|
||||
|
||||
return arrow
|
||||
|
||||
def remove(
|
||||
self,
|
||||
arrow: pg.ArrowItem,
|
||||
) -> None:
|
||||
'''
|
||||
Remove a *single arrow* from all chart views to which it was
|
||||
added.
|
||||
|
||||
'''
|
||||
uid: str = arrow._uid
|
||||
arrows: list[pg.ArrowItem] = self._arrows[uid]
|
||||
log.info(
|
||||
f'Removing arrow from views\n'
|
||||
f'uid: {uid!r}\n'
|
||||
f'{arrow!r}\n'
|
||||
)
|
||||
def remove(self, arrow) -> bool:
|
||||
for linked in self.godw.iter_linked():
|
||||
if not (chart := linked.chart):
|
||||
continue
|
||||
|
||||
chart.plotItem.removeItem(arrow)
|
||||
try:
|
||||
arrows.remove(arrow)
|
||||
except ValueError:
|
||||
log.warning(
|
||||
f'Arrow was already removed?\n'
|
||||
f'uid: {uid!r}\n'
|
||||
f'{arrow!r}\n'
|
||||
)
|
||||
|
||||
def remove_all(self) -> set[pg.ArrowItem]:
|
||||
'''
|
||||
Remove all arrows added by this editor from all
|
||||
chart-views.
|
||||
|
||||
'''
|
||||
for uid, arrows in self._arrows.items():
|
||||
for arrow in arrows:
|
||||
self.remove(arrow)
|
||||
linked.chart.plotItem.removeItem(arrow)
|
||||
|
||||
|
||||
class LineEditor(Struct):
|
||||
|
|
@ -341,9 +261,6 @@ class LineEditor(Struct):
|
|||
|
||||
return lines
|
||||
|
||||
# compat with ArrowEditor
|
||||
remove = remove_line
|
||||
|
||||
|
||||
def as_point(
|
||||
pair: Sequence[float, float] | QPointF,
|
||||
|
|
@ -376,7 +293,7 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
|
|||
def __init__(
|
||||
self,
|
||||
viewbox: ViewBox,
|
||||
color: str|None = None,
|
||||
color: str | None = None,
|
||||
) -> None:
|
||||
super().__init__(0, 0, 1, 1)
|
||||
|
||||
|
|
@ -692,6 +609,3 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
|
|||
|
||||
):
|
||||
scen.removeItem(self._label_proxy)
|
||||
|
||||
# compat with ArrowEditor
|
||||
remove = delete
|
||||
|
|
|
|||
|
|
@ -22,10 +22,7 @@ from contextlib import asynccontextmanager as acm
|
|||
from typing import Callable
|
||||
|
||||
import trio
|
||||
from tractor.trionics import (
|
||||
gather_contexts,
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor.trionics import gather_contexts
|
||||
|
||||
from piker.ui.qt import (
|
||||
QtCore,
|
||||
|
|
@ -210,10 +207,7 @@ async def open_signal_handler(
|
|||
async for args in recv:
|
||||
await async_handler(*args)
|
||||
|
||||
async with (
|
||||
collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(proxy_to_handler)
|
||||
async with send:
|
||||
yield
|
||||
|
|
@ -248,7 +242,6 @@ async def open_handlers(
|
|||
widget: QWidget
|
||||
streams: list[trio.abc.ReceiveChannel]
|
||||
async with (
|
||||
collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
gather_contexts([
|
||||
open_event_stream(
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@ from . import _style
|
|||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._widget import GodWidget
|
||||
from ._chart import GodWidget
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
|
@ -91,10 +91,6 @@ def run_qtractor(
|
|||
window_type: QMainWindow = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Run the Qt event loop and embed `trio` via guest mode on it.
|
||||
|
||||
'''
|
||||
# avoids annoying message when entering debugger from qt loop
|
||||
pyqtRemoveInputHook()
|
||||
|
||||
|
|
@ -174,7 +170,7 @@ def run_qtractor(
|
|||
# hook into app focus change events
|
||||
app.focusChanged.connect(window.on_focus_change)
|
||||
|
||||
instance: GodWidget = main_widget_type()
|
||||
instance = main_widget_type()
|
||||
instance.window = window
|
||||
|
||||
# override tractor's defaults
|
||||
|
|
|
|||
|
|
@ -18,11 +18,10 @@
|
|||
Feed status and controls widget(s) for embedding in a UI-pane.
|
||||
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
from typing import (
|
||||
Any,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
from textwrap import dedent
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
# from PyQt5.QtCore import Qt
|
||||
|
||||
|
|
@ -50,55 +49,35 @@ def mk_feed_label(
|
|||
a feed control protocol.
|
||||
|
||||
'''
|
||||
status: dict[str, Any] = feed.status
|
||||
status = feed.status
|
||||
assert status
|
||||
|
||||
# SO tips on ws/nls,
|
||||
# https://stackoverflow.com/a/15721400
|
||||
ws: str = ' '
|
||||
# nl: str = '<br>' # dun work?
|
||||
actor_info_repr: str = (
|
||||
f')> **{status["actor_short_id"]}**\n'
|
||||
'\n' # bc md?
|
||||
)
|
||||
msg = dedent("""
|
||||
actor: **{actor_name}**\n
|
||||
|_ @**{host}:{port}**\n
|
||||
""")
|
||||
|
||||
# fields to select *IN* for display
|
||||
# (see `.data.feed.open_feed()` status
|
||||
# update -> TAG_feed_status_update)
|
||||
for key in [
|
||||
'ipc',
|
||||
'hist_shm',
|
||||
'rt_shm',
|
||||
'throttle_hz',
|
||||
]:
|
||||
# NOTE, the 2nd key is filled via `.format()` updates.
|
||||
actor_info_repr += (
|
||||
f'\n' # bc md?
|
||||
f'{ws}|_{key}: **{{{key}}}**\n'
|
||||
)
|
||||
# ^TODO? formatting and content..
|
||||
# -[ ] showing which fqme is "forward" on the
|
||||
# chart/fsp/order-mode?
|
||||
# '|_ flows: **{symbols}**\n'
|
||||
#
|
||||
# -[x] why isn't the indent working?
|
||||
# => markdown, now solved..
|
||||
for key, val in status.items():
|
||||
if key in ('host', 'port', 'actor_name'):
|
||||
continue
|
||||
msg += f'\n|_ {key}: **{{{key}}}**\n'
|
||||
|
||||
feed_label = FormatLabel(
|
||||
fmt_str=actor_info_repr,
|
||||
fmt_str=msg,
|
||||
# |_ streams: **{symbols}**\n
|
||||
font=_font.font,
|
||||
font_size=_font_small.px_size,
|
||||
font_color='default_lightest',
|
||||
)
|
||||
|
||||
# ?TODO, remove this?
|
||||
# form.vbox.setAlignment(feed_label, Qt.AlignBottom)
|
||||
# form.vbox.setAlignment(Qt.AlignBottom)
|
||||
# _ = chart.height() - (
|
||||
# form.height() +
|
||||
# form.fill_bar.height()
|
||||
# # feed_label.height()
|
||||
# )
|
||||
_ = chart.height() - (
|
||||
form.height() +
|
||||
form.fill_bar.height()
|
||||
# feed_label.height()
|
||||
)
|
||||
|
||||
feed_label.format(**feed.status)
|
||||
|
||||
return feed_label
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ log = get_logger(__name__)
|
|||
def update_fsp_chart(
|
||||
viz,
|
||||
graphics_name: str,
|
||||
array_key: str|None,
|
||||
array_key: str | None,
|
||||
**kwargs,
|
||||
|
||||
) -> None:
|
||||
|
|
@ -87,11 +87,7 @@ def update_fsp_chart(
|
|||
|
||||
# guard against unreadable case
|
||||
if not last_row:
|
||||
log.warning(
|
||||
f'Read-race on shm array,\n'
|
||||
f'graphics_name: {graphics_name!r}\n'
|
||||
f'shm.token: {shm.token}\n'
|
||||
)
|
||||
log.warning(f'Read-race on shm array: {graphics_name}@{shm.token}')
|
||||
return
|
||||
|
||||
# update graphics
|
||||
|
|
@ -183,17 +179,13 @@ async def open_fsp_sidepane(
|
|||
|
||||
@acm
|
||||
async def open_fsp_actor_cluster(
|
||||
names: list[str] = [
|
||||
'fsp_0',
|
||||
'fsp_1',
|
||||
],
|
||||
names: list[str] = ['fsp_0', 'fsp_1'],
|
||||
|
||||
) -> AsyncGenerator[
|
||||
int,
|
||||
dict[str, tractor.Portal]
|
||||
]:
|
||||
|
||||
# TODO! change to .experimental!
|
||||
from tractor._clustering import open_actor_cluster
|
||||
|
||||
# profiler = Profiler(
|
||||
|
|
@ -201,7 +193,7 @@ async def open_fsp_actor_cluster(
|
|||
# disabled=False
|
||||
# )
|
||||
async with open_actor_cluster(
|
||||
count=len(names),
|
||||
count=2,
|
||||
names=names,
|
||||
modules=['piker.fsp._engine'],
|
||||
|
||||
|
|
@ -211,6 +203,7 @@ async def open_fsp_actor_cluster(
|
|||
|
||||
|
||||
async def run_fsp_ui(
|
||||
|
||||
linkedsplits: LinkedSplits,
|
||||
flume: Flume,
|
||||
started: trio.Event,
|
||||
|
|
@ -478,7 +471,7 @@ class FspAdmin:
|
|||
target: Fsp,
|
||||
conf: dict[str, dict[str, Any]],
|
||||
|
||||
worker_name: str|None = None,
|
||||
worker_name: str | None = None,
|
||||
loglevel: str = 'info',
|
||||
|
||||
) -> (Flume, trio.Event):
|
||||
|
|
@ -501,8 +494,7 @@ class FspAdmin:
|
|||
|
||||
portal: tractor.Portal = (
|
||||
self.cluster.get(worker_name)
|
||||
or
|
||||
self.rr_next_portal()
|
||||
or self.rr_next_portal()
|
||||
)
|
||||
|
||||
# TODO: this should probably be turned into a
|
||||
|
|
@ -608,7 +600,6 @@ async def open_fsp_admin(
|
|||
kwargs=kwargs,
|
||||
) as (cache_hit, cluster_map),
|
||||
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
if cache_hit:
|
||||
|
|
@ -622,8 +613,6 @@ async def open_fsp_admin(
|
|||
)
|
||||
try:
|
||||
yield admin
|
||||
|
||||
# ??TODO, does this *need* to be inside a finally?
|
||||
finally:
|
||||
# terminate all tasks via signals
|
||||
for key, entry in admin._registry.items():
|
||||
|
|
@ -631,10 +620,8 @@ async def open_fsp_admin(
|
|||
event.set()
|
||||
|
||||
|
||||
# TODO, passing in `pikerd` related settings here!
|
||||
# [ ] read in the `tractor` setting for `enable_transports: list`
|
||||
# from the root `conf.toml`!
|
||||
async def open_vlm_displays(
|
||||
|
||||
linked: LinkedSplits,
|
||||
flume: Flume,
|
||||
dvlm: bool = True,
|
||||
|
|
@ -644,12 +631,12 @@ async def open_vlm_displays(
|
|||
|
||||
) -> None:
|
||||
'''
|
||||
Vlm (volume) subchart displays.
|
||||
Volume subchart displays.
|
||||
|
||||
Since "volume" is often included directly alongside OHLCV price
|
||||
data, we don't really need a separate FSP-actor + shm array for
|
||||
it since it's likely already directly adjacent to OHLC samples
|
||||
from the data provider.
|
||||
data, we don't really need a separate FSP-actor + shm array for it
|
||||
since it's likely already directly adjacent to OHLC samples from the
|
||||
data provider.
|
||||
|
||||
Further only if volume data is detected (it sometimes isn't provided
|
||||
eg. forex, certain commodities markets) will volume dependent FSPs
|
||||
|
|
|
|||
|
|
@ -43,7 +43,6 @@ from pyqtgraph import (
|
|||
functions as fn,
|
||||
)
|
||||
import numpy as np
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
from piker.ui.qt import (
|
||||
|
|
@ -73,10 +72,7 @@ if TYPE_CHECKING:
|
|||
GodWidget,
|
||||
)
|
||||
from ._dataviz import Viz
|
||||
from .order_mode import (
|
||||
OrderMode,
|
||||
Dialog,
|
||||
)
|
||||
from .order_mode import OrderMode
|
||||
from ._display import DisplayState
|
||||
|
||||
|
||||
|
|
@ -134,12 +130,7 @@ async def handle_viewmode_kb_inputs(
|
|||
|
||||
async for kbmsg in recv_chan:
|
||||
event, etype, key, mods, text = kbmsg.to_tuple()
|
||||
log.debug(
|
||||
f'View-mode kb-msg received,\n'
|
||||
f'mods: {mods!r}\n'
|
||||
f'key: {key!r}\n'
|
||||
f'text: {text!r}\n'
|
||||
)
|
||||
log.debug(f'key: {key}, mods: {mods}, text: {text}')
|
||||
now = time.time()
|
||||
period = now - last
|
||||
|
||||
|
|
@ -167,12 +158,8 @@ async def handle_viewmode_kb_inputs(
|
|||
# have no previous keys or we do and the min_tap period is
|
||||
# met
|
||||
if (
|
||||
not fast_key_seq
|
||||
or (
|
||||
period <= min_tap
|
||||
and
|
||||
fast_key_seq
|
||||
)
|
||||
not fast_key_seq or
|
||||
period <= min_tap and fast_key_seq
|
||||
):
|
||||
fast_key_seq.append(text)
|
||||
log.debug(f'fast keys seqs {fast_key_seq}')
|
||||
|
|
@ -187,8 +174,7 @@ async def handle_viewmode_kb_inputs(
|
|||
# UI REPL-shell, with ctrl-p (for "pause")
|
||||
if (
|
||||
ctrl
|
||||
and
|
||||
key in {
|
||||
and key in {
|
||||
Qt.Key_P,
|
||||
}
|
||||
):
|
||||
|
|
@ -198,6 +184,7 @@ async def handle_viewmode_kb_inputs(
|
|||
vlm_chart = chart.linked.subplots['volume'] # noqa
|
||||
vlm_viz = vlm_chart.main_viz # noqa
|
||||
dvlm_pi = vlm_chart._vizs['dolla_vlm'].plot # noqa
|
||||
import tractor
|
||||
await tractor.pause()
|
||||
view.interact_graphics_cycle()
|
||||
|
||||
|
|
@ -205,8 +192,7 @@ async def handle_viewmode_kb_inputs(
|
|||
# shown data `Viz`s for the current chart app.
|
||||
if (
|
||||
ctrl
|
||||
and
|
||||
key in {
|
||||
and key in {
|
||||
Qt.Key_R,
|
||||
}
|
||||
):
|
||||
|
|
@ -245,8 +231,7 @@ async def handle_viewmode_kb_inputs(
|
|||
key == Qt.Key_Escape
|
||||
or (
|
||||
ctrl
|
||||
and
|
||||
key == Qt.Key_C
|
||||
and key == Qt.Key_C
|
||||
)
|
||||
):
|
||||
# ctrl-c as cancel
|
||||
|
|
@ -257,35 +242,17 @@ async def handle_viewmode_kb_inputs(
|
|||
# cancel order or clear graphics
|
||||
if (
|
||||
key == Qt.Key_C
|
||||
or
|
||||
key == Qt.Key_Delete
|
||||
or key == Qt.Key_Delete
|
||||
):
|
||||
# log.info('Handling <c> hotkey!')
|
||||
try:
|
||||
dialogs: list[Dialog] = order_mode.cancel_orders_under_cursor()
|
||||
except BaseException:
|
||||
log.exception('Failed to cancel orders !?\n')
|
||||
await tractor.pause()
|
||||
|
||||
if not dialogs:
|
||||
log.warning(
|
||||
'No orders were cancelled?\n'
|
||||
'Is there an order-line under the cursor?\n'
|
||||
'If you think there IS your DE might be "hiding the mouse" before '
|
||||
'we rx the keyboard input via Qt..\n'
|
||||
'=> Check your DE and/or TWM settings to be sure! <=\n'
|
||||
)
|
||||
# ^TODO?, some way to detect if there's lines and
|
||||
# the DE is cuckin with things?
|
||||
# await tractor.pause()
|
||||
order_mode.cancel_orders_under_cursor()
|
||||
|
||||
# View modes
|
||||
if (
|
||||
ctrl
|
||||
and (
|
||||
key == Qt.Key_Equal
|
||||
or
|
||||
key == Qt.Key_I
|
||||
or key == Qt.Key_I
|
||||
)
|
||||
):
|
||||
view.wheelEvent(
|
||||
|
|
@ -297,8 +264,7 @@ async def handle_viewmode_kb_inputs(
|
|||
ctrl
|
||||
and (
|
||||
key == Qt.Key_Minus
|
||||
or
|
||||
key == Qt.Key_O
|
||||
or key == Qt.Key_O
|
||||
)
|
||||
):
|
||||
view.wheelEvent(
|
||||
|
|
@ -309,8 +275,7 @@ async def handle_viewmode_kb_inputs(
|
|||
|
||||
elif (
|
||||
not ctrl
|
||||
and
|
||||
key == Qt.Key_R
|
||||
and key == Qt.Key_R
|
||||
):
|
||||
# NOTE: seems that if we don't yield a Qt render
|
||||
# cycle then the m4 downsampled curves will show here
|
||||
|
|
@ -512,8 +477,7 @@ async def handle_viewmode_mouse(
|
|||
# view.raiseContextMenu(event)
|
||||
|
||||
if (
|
||||
view.order_mode.active
|
||||
and
|
||||
view.order_mode.active and
|
||||
button == QtCore.Qt.LeftButton
|
||||
):
|
||||
# when in order mode, submit execution
|
||||
|
|
@ -817,8 +781,7 @@ class ChartView(ViewBox):
|
|||
|
||||
# Scale or translate based on mouse button
|
||||
if btn & (
|
||||
QtCore.Qt.LeftButton
|
||||
| QtCore.Qt.MidButton
|
||||
QtCore.Qt.LeftButton | QtCore.Qt.MidButton
|
||||
):
|
||||
# zoom y-axis ONLY when click-n-drag on it
|
||||
# if axis == 1:
|
||||
|
|
|
|||
|
|
@ -237,8 +237,8 @@ class LevelLabel(YAxisLabel):
|
|||
class L1Label(LevelLabel):
|
||||
|
||||
text_flags = (
|
||||
QtCore.Qt.TextFlag.TextDontClip
|
||||
| QtCore.Qt.AlignmentFlag.AlignLeft
|
||||
QtCore.Qt.TextDontClip
|
||||
| QtCore.Qt.AlignLeft
|
||||
)
|
||||
|
||||
def set_label_str(
|
||||
|
|
|
|||
|
|
@ -285,20 +285,18 @@ class FormatLabel(QLabel):
|
|||
font_size: int,
|
||||
font_color: str,
|
||||
|
||||
use_md: bool = True,
|
||||
|
||||
parent=None,
|
||||
|
||||
) -> None:
|
||||
|
||||
super().__init__(parent)
|
||||
|
||||
# by default set the format string verbatim and expect user
|
||||
# to call ``.format()`` later (presumably they'll notice the
|
||||
# by default set the format string verbatim and expect user to
|
||||
# call ``.format()`` later (presumably they'll notice the
|
||||
# unformatted content if ``fmt_str`` isn't meant to be
|
||||
# unformatted).
|
||||
self.fmt_str = fmt_str
|
||||
# self.setText(fmt_str) # ?TODO, why here?
|
||||
self.setText(fmt_str)
|
||||
|
||||
self.setStyleSheet(
|
||||
f"""QLabel {{
|
||||
|
|
@ -308,10 +306,9 @@ class FormatLabel(QLabel):
|
|||
"""
|
||||
)
|
||||
self.setFont(_font.font)
|
||||
if use_md:
|
||||
self.setTextFormat(
|
||||
Qt.TextFormat.MarkdownText
|
||||
)
|
||||
self.setTextFormat(
|
||||
Qt.TextFormat.MarkdownText
|
||||
)
|
||||
self.setMargin(0)
|
||||
|
||||
self.setSizePolicy(
|
||||
|
|
@ -319,10 +316,7 @@ class FormatLabel(QLabel):
|
|||
size_policy.Expanding,
|
||||
)
|
||||
self.setAlignment(
|
||||
Qt.AlignLeft
|
||||
|
|
||||
Qt.AlignBottom
|
||||
# Qt.AlignVCenter
|
||||
Qt.AlignVCenter | Qt.AlignLeft
|
||||
)
|
||||
self.setText(self.fmt_str)
|
||||
|
||||
|
|
|
|||
|
|
@ -52,13 +52,10 @@ from ._anchors import (
|
|||
from ..calc import humanize
|
||||
from ._label import Label
|
||||
from ._style import hcolor, _font
|
||||
from ..log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._cursor import Cursor
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
# TODO: probably worth investigating if we can
|
||||
# make .boundingRect() faster:
|
||||
|
|
@ -350,7 +347,7 @@ class LevelLine(pg.InfiniteLine):
|
|||
|
||||
) -> None:
|
||||
# TODO: enter labels edit mode
|
||||
log.debug(f'double click {ev}')
|
||||
print(f'double click {ev}')
|
||||
|
||||
def paint(
|
||||
self,
|
||||
|
|
@ -464,19 +461,10 @@ class LevelLine(pg.InfiniteLine):
|
|||
# hovered
|
||||
if (
|
||||
not ev.isExit()
|
||||
and
|
||||
ev.acceptDrags(QtCore.Qt.LeftButton)
|
||||
and ev.acceptDrags(QtCore.Qt.LeftButton)
|
||||
):
|
||||
# if already hovered we don't need to run again
|
||||
if (
|
||||
self.mouseHovering is True
|
||||
and
|
||||
cur.is_hovered(self)
|
||||
):
|
||||
log.debug(
|
||||
f'Already hovering ??\n'
|
||||
f'cur._hovered: {cur._hovered!r}\n'
|
||||
)
|
||||
if self.mouseHovering is True:
|
||||
return
|
||||
|
||||
if self.only_show_markers_on_hover:
|
||||
|
|
@ -493,7 +481,6 @@ class LevelLine(pg.InfiniteLine):
|
|||
cur._y_label_update = False
|
||||
|
||||
# add us to cursor state
|
||||
log.debug(f'Adding line {self!r}\n')
|
||||
cur.add_hovered(self)
|
||||
|
||||
if self._hide_xhair_on_hover:
|
||||
|
|
@ -521,7 +508,6 @@ class LevelLine(pg.InfiniteLine):
|
|||
|
||||
self.currentPen = self.pen
|
||||
|
||||
log.debug(f'Removing line {self!r}\n')
|
||||
cur._hovered.remove(self)
|
||||
|
||||
if self.only_show_markers_on_hover:
|
||||
|
|
|
|||
|
|
@ -15,8 +15,8 @@
|
|||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Remote control tasks for sending annotations (and maybe more cmds) to
|
||||
a chart from some other actor.
|
||||
Remote control tasks for sending annotations (and maybe more cmds)
|
||||
to a chart from some other actor.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
|
@ -27,14 +27,11 @@ from contextlib import (
|
|||
from functools import partial
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
# Any,
|
||||
AsyncContextManager,
|
||||
Literal,
|
||||
)
|
||||
from uuid import uuid4
|
||||
|
||||
import pyqtgraph as pg
|
||||
import tractor
|
||||
import trio
|
||||
from tractor import trionics
|
||||
from tractor import (
|
||||
Portal,
|
||||
|
|
@ -49,16 +46,12 @@ from piker.brokers import SymbolNotFound
|
|||
from piker.ui.qt import (
|
||||
QGraphicsItem,
|
||||
)
|
||||
from PyQt6.QtGui import QFont
|
||||
from ._display import DisplayState
|
||||
from ._interaction import ChartView
|
||||
from ._editors import (
|
||||
SelectRect,
|
||||
ArrowEditor,
|
||||
)
|
||||
from ._editors import SelectRect
|
||||
from ._chart import ChartPlotWidget
|
||||
from ._dataviz import Viz
|
||||
from ._style import hcolor
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
|
@ -89,40 +82,8 @@ _ctxs: IpcCtxTable = {}
|
|||
# the "annotations server" which actually renders to a Qt canvas).
|
||||
# type AnnotsTable = dict[int, QGraphicsItem]
|
||||
AnnotsTable = dict[int, QGraphicsItem]
|
||||
EditorsTable = dict[int, ArrowEditor]
|
||||
|
||||
_annots: AnnotsTable = {}
|
||||
_editors: EditorsTable = {}
|
||||
|
||||
def rm_annot(
|
||||
annot: ArrowEditor|SelectRect|pg.TextItem
|
||||
) -> bool:
|
||||
global _editors
|
||||
match annot:
|
||||
case pg.ArrowItem():
|
||||
editor = _editors[annot._uid]
|
||||
editor.remove(annot)
|
||||
# ^TODO? only remove each arrow or all?
|
||||
# if editor._arrows:
|
||||
# editor.remove_all()
|
||||
# else:
|
||||
# log.warning(
|
||||
# f'Annot already removed!\n'
|
||||
# f'{annot!r}\n'
|
||||
# )
|
||||
return True
|
||||
|
||||
case SelectRect():
|
||||
annot.delete()
|
||||
return True
|
||||
|
||||
case pg.TextItem():
|
||||
scene = annot.scene()
|
||||
if scene:
|
||||
scene.removeItem(annot)
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
async def serve_rc_annots(
|
||||
|
|
@ -133,12 +94,6 @@ async def serve_rc_annots(
|
|||
annots: AnnotsTable,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
A small viz(ualization) server for remote ctl of chart
|
||||
annotations.
|
||||
|
||||
'''
|
||||
global _editors
|
||||
async for msg in annot_req_stream:
|
||||
match msg:
|
||||
case {
|
||||
|
|
@ -148,77 +103,14 @@ async def serve_rc_annots(
|
|||
'meth': str(meth),
|
||||
'kwargs': dict(kwargs),
|
||||
}:
|
||||
|
||||
ds: DisplayState = _dss[fqme]
|
||||
try:
|
||||
chart: ChartPlotWidget = {
|
||||
60: ds.hist_chart,
|
||||
1: ds.chart,
|
||||
}[timeframe]
|
||||
except KeyError:
|
||||
msg: str = (
|
||||
f'No chart for timeframe={timeframe}s, '
|
||||
f'skipping rect annotation'
|
||||
)
|
||||
log.exeception(msg)
|
||||
await annot_req_stream.send({'error': msg})
|
||||
continue
|
||||
|
||||
chart: ChartPlotWidget = {
|
||||
60: ds.hist_chart,
|
||||
1: ds.chart,
|
||||
}[timeframe]
|
||||
cv: ChartView = chart.cv
|
||||
|
||||
# NEW: if timestamps provided, lookup current indices
|
||||
# from shm to ensure alignment with current buffer
|
||||
# state
|
||||
start_time = kwargs.pop('start_time', None)
|
||||
end_time = kwargs.pop('end_time', None)
|
||||
if (
|
||||
start_time is not None
|
||||
and end_time is not None
|
||||
):
|
||||
viz: Viz = chart.get_viz(fqme)
|
||||
shm = viz.shm
|
||||
arr = shm.array
|
||||
|
||||
# lookup start index
|
||||
start_matches = arr[arr['time'] == start_time]
|
||||
if len(start_matches) == 0:
|
||||
msg: str = (
|
||||
f'No shm entry for start_time={start_time}, '
|
||||
f'skipping rect'
|
||||
)
|
||||
log.error(msg)
|
||||
await annot_req_stream.send({'error': msg})
|
||||
continue
|
||||
|
||||
# lookup end index
|
||||
end_matches = arr[arr['time'] == end_time]
|
||||
if len(end_matches) == 0:
|
||||
msg: str = (
|
||||
f'No shm entry for end_time={end_time}, '
|
||||
f'skipping rect'
|
||||
)
|
||||
log.error(msg)
|
||||
await annot_req_stream.send({'error': msg})
|
||||
continue
|
||||
|
||||
# get close price from start bar, open from end
|
||||
# bar
|
||||
start_idx = float(start_matches[0]['index'])
|
||||
end_idx = float(end_matches[0]['index'])
|
||||
start_close = float(start_matches[0]['close'])
|
||||
end_open = float(end_matches[0]['open'])
|
||||
|
||||
# reconstruct start_pos and end_pos with
|
||||
# looked-up indices
|
||||
from_idx: float = 0.16 - 0.06 # BGM offset
|
||||
kwargs['start_pos'] = (
|
||||
start_idx + 1 - from_idx,
|
||||
start_close,
|
||||
)
|
||||
kwargs['end_pos'] = (
|
||||
end_idx + from_idx,
|
||||
end_open,
|
||||
)
|
||||
|
||||
# annot type lookup from cmd
|
||||
rect = SelectRect(
|
||||
viewbox=cv,
|
||||
|
|
@ -237,207 +129,21 @@ async def serve_rc_annots(
|
|||
# delegate generically to the requested method
|
||||
getattr(rect, meth)(**kwargs)
|
||||
rect.show()
|
||||
|
||||
# XXX: store absolute coords for repositioning
|
||||
# during viz redraws (eg backfill updates)
|
||||
rect._meth = meth
|
||||
rect._kwargs = kwargs
|
||||
|
||||
aid: int = id(rect)
|
||||
annots[aid] = rect
|
||||
aids: set[int] = ctxs[ipc_key][1]
|
||||
aids.add(aid)
|
||||
await annot_req_stream.send(aid)
|
||||
|
||||
case {
|
||||
'cmd': 'ArrowEditor',
|
||||
'fqme': fqme,
|
||||
'timeframe': timeframe,
|
||||
'meth': 'add'|'remove' as meth,
|
||||
'kwargs': {
|
||||
'x': float(x),
|
||||
'y': float(y),
|
||||
'pointing': pointing,
|
||||
'color': color,
|
||||
'aid': str()|None as aid,
|
||||
'alpha': int(alpha),
|
||||
'headLen': int()|float()|None as headLen,
|
||||
'headWidth': int()|float()|None as headWidth,
|
||||
'tailLen': int()|float()|None as tailLen,
|
||||
'tailWidth': int()|float()|None as tailWidth,
|
||||
'pxMode': bool(pxMode),
|
||||
'time': int()|float()|None as timestamp,
|
||||
},
|
||||
# ?TODO? split based on method fn-sigs?
|
||||
# 'pointing',
|
||||
}:
|
||||
ds: DisplayState = _dss[fqme]
|
||||
try:
|
||||
chart: ChartPlotWidget = {
|
||||
60: ds.hist_chart,
|
||||
1: ds.chart,
|
||||
}[timeframe]
|
||||
except KeyError:
|
||||
log.warning(
|
||||
f'No chart for timeframe={timeframe}s, '
|
||||
f'skipping arrow annotation'
|
||||
)
|
||||
# return -1 to indicate failure
|
||||
await annot_req_stream.send(-1)
|
||||
continue
|
||||
cv: ChartView = chart.cv
|
||||
godw = chart.linked.godwidget
|
||||
|
||||
# NEW: if timestamp provided, lookup current index
|
||||
# from shm to ensure alignment with current buffer
|
||||
# state
|
||||
if timestamp is not None:
|
||||
viz: Viz = chart.get_viz(fqme)
|
||||
shm = viz.shm
|
||||
arr = shm.array
|
||||
# find index where time matches timestamp
|
||||
matches = arr[arr['time'] == timestamp]
|
||||
if len(matches) == 0:
|
||||
log.error(
|
||||
f'No shm entry for timestamp={timestamp}, '
|
||||
f'skipping arrow annotation'
|
||||
)
|
||||
await annot_req_stream.send(-1)
|
||||
continue
|
||||
# use the matched row's index as x
|
||||
x = float(matches[0]['index'])
|
||||
|
||||
arrows = ArrowEditor(godw=godw)
|
||||
# `.add/.remove()` API
|
||||
if meth != 'add':
|
||||
# await tractor.pause()
|
||||
raise ValueError(
|
||||
f'Invalid arrow-edit request ?\n'
|
||||
f'{msg!r}\n'
|
||||
)
|
||||
|
||||
aid: str = str(uuid4())
|
||||
arrow: pg.ArrowItem = arrows.add(
|
||||
plot=chart.plotItem,
|
||||
uid=aid,
|
||||
x=x,
|
||||
y=y,
|
||||
pointing=pointing,
|
||||
color=color,
|
||||
alpha=alpha,
|
||||
headLen=headLen,
|
||||
headWidth=headWidth,
|
||||
tailLen=tailLen,
|
||||
tailWidth=tailWidth,
|
||||
pxMode=pxMode,
|
||||
)
|
||||
# XXX: store absolute coords for repositioning
|
||||
# during viz redraws (eg backfill updates)
|
||||
arrow._abs_x = x
|
||||
arrow._abs_y = y
|
||||
|
||||
annots[aid] = arrow
|
||||
_editors[aid] = arrows
|
||||
aids: set[int] = ctxs[ipc_key][1]
|
||||
aids.add(aid)
|
||||
await annot_req_stream.send(aid)
|
||||
|
||||
case {
|
||||
'cmd': 'TextItem',
|
||||
'fqme': fqme,
|
||||
'timeframe': timeframe,
|
||||
'kwargs': {
|
||||
'text': str(text),
|
||||
'x': int()|float() as x,
|
||||
'y': int()|float() as y,
|
||||
'color': color,
|
||||
'anchor': list(anchor),
|
||||
'font_size': int()|None as font_size,
|
||||
'time': int()|float()|None as timestamp,
|
||||
},
|
||||
}:
|
||||
ds: DisplayState = _dss[fqme]
|
||||
try:
|
||||
chart: ChartPlotWidget = {
|
||||
60: ds.hist_chart,
|
||||
1: ds.chart,
|
||||
}[timeframe]
|
||||
except KeyError:
|
||||
log.warning(
|
||||
f'No chart for timeframe={timeframe}s, '
|
||||
f'skipping text annotation'
|
||||
)
|
||||
await annot_req_stream.send(-1)
|
||||
continue
|
||||
|
||||
# NEW: if timestamp provided, lookup current index
|
||||
# from shm to ensure alignment with current buffer
|
||||
# state
|
||||
if timestamp is not None:
|
||||
viz: Viz = chart.get_viz(fqme)
|
||||
shm = viz.shm
|
||||
arr = shm.array
|
||||
# find index where time matches timestamp
|
||||
matches = arr[arr['time'] == timestamp]
|
||||
if len(matches) == 0:
|
||||
log.error(
|
||||
f'No shm entry for timestamp={timestamp}, '
|
||||
f'skipping text annotation'
|
||||
)
|
||||
await annot_req_stream.send(-1)
|
||||
continue
|
||||
# use the matched row's index as x, +1 for text
|
||||
# offset
|
||||
x = float(matches[0]['index']) + 1
|
||||
|
||||
# convert named color to hex
|
||||
color_hex: str = hcolor(color)
|
||||
|
||||
# create text item
|
||||
text_item: pg.TextItem = pg.TextItem(
|
||||
text=text,
|
||||
color=color_hex,
|
||||
anchor=anchor,
|
||||
|
||||
# ?TODO, pin to github:main for this?
|
||||
# legacy, can have scaling ish?
|
||||
# ensureInBounds=True,
|
||||
)
|
||||
|
||||
# apply font size (default to DpiAwareFont if not
|
||||
# provided)
|
||||
if font_size is None:
|
||||
from ._style import get_fonts
|
||||
font, font_small = get_fonts()
|
||||
font_size = font_small.px_size - 1
|
||||
|
||||
qfont: QFont = text_item.textItem.font()
|
||||
qfont.setPixelSize(font_size)
|
||||
text_item.setFont(qfont)
|
||||
|
||||
text_item.setPos(x, y)
|
||||
chart.plotItem.addItem(text_item)
|
||||
|
||||
# XXX: store absolute coords for repositioning
|
||||
# during viz redraws (eg backfill updates)
|
||||
text_item._abs_x = x
|
||||
text_item._abs_y = y
|
||||
|
||||
aid: str = str(uuid4())
|
||||
annots[aid] = text_item
|
||||
aids: set[int] = ctxs[ipc_key][1]
|
||||
aids.add(aid)
|
||||
await annot_req_stream.send(aid)
|
||||
|
||||
case {
|
||||
'cmd': 'remove',
|
||||
'aid': int(aid)|str(aid),
|
||||
'aid': int(aid),
|
||||
}:
|
||||
# NOTE: this is normally entered on
|
||||
# a client's annotation de-alloc normally
|
||||
# prior to detach or modify.
|
||||
annot: QGraphicsItem = annots[aid]
|
||||
assert rm_annot(annot)
|
||||
annot.delete()
|
||||
|
||||
# respond to client indicating annot
|
||||
# was indeed deleted.
|
||||
|
|
@ -468,38 +174,6 @@ async def serve_rc_annots(
|
|||
)
|
||||
viz.reset_graphics()
|
||||
|
||||
# XXX: reposition all annotations to ensure they
|
||||
# stay aligned with viz data after reset (eg during
|
||||
# backfill when abs-index range changes)
|
||||
n_repositioned: int = 0
|
||||
for aid, annot in annots.items():
|
||||
# arrows and text items use abs x,y coords
|
||||
if (
|
||||
hasattr(annot, '_abs_x')
|
||||
and
|
||||
hasattr(annot, '_abs_y')
|
||||
):
|
||||
annot.setPos(
|
||||
annot._abs_x,
|
||||
annot._abs_y,
|
||||
)
|
||||
n_repositioned += 1
|
||||
|
||||
# rects use method + kwargs
|
||||
elif (
|
||||
hasattr(annot, '_meth')
|
||||
and
|
||||
hasattr(annot, '_kwargs')
|
||||
):
|
||||
getattr(annot, annot._meth)(**annot._kwargs)
|
||||
n_repositioned += 1
|
||||
|
||||
if n_repositioned:
|
||||
log.info(
|
||||
f'Repositioned {n_repositioned} annotation(s) '
|
||||
f'after viz redraw'
|
||||
)
|
||||
|
||||
case _:
|
||||
log.error(
|
||||
'Unknown remote annotation cmd:\n'
|
||||
|
|
@ -513,12 +187,6 @@ async def remote_annotate(
|
|||
) -> None:
|
||||
|
||||
global _dss, _ctxs
|
||||
if not _dss:
|
||||
raise RuntimeError(
|
||||
'Race condition on chart-init state ??\n'
|
||||
'Anoter actor is trying to annoate this chart '
|
||||
'before it has fully spawned.\n'
|
||||
)
|
||||
assert _dss
|
||||
|
||||
_ctxs[ctx.cid] = (ctx, set())
|
||||
|
|
@ -543,7 +211,7 @@ async def remote_annotate(
|
|||
assert _ctx is ctx
|
||||
for aid in aids:
|
||||
annot: QGraphicsItem = _annots[aid]
|
||||
assert rm_annot(annot)
|
||||
annot.delete()
|
||||
|
||||
|
||||
class AnnotCtl(Struct):
|
||||
|
|
@ -588,47 +256,36 @@ class AnnotCtl(Struct):
|
|||
|
||||
from_acm: bool = False,
|
||||
|
||||
# NEW: optional timestamps for server-side index lookup
|
||||
start_time: float|None = None,
|
||||
end_time: float|None = None,
|
||||
|
||||
) -> int|None:
|
||||
) -> int:
|
||||
'''
|
||||
Add a `SelectRect` annotation to the target view, return
|
||||
the instances `id(obj)` from the remote UI actor.
|
||||
|
||||
'''
|
||||
ipc: MsgStream = self._get_ipc(fqme)
|
||||
with trio.fail_after(3):
|
||||
await ipc.send({
|
||||
'fqme': fqme,
|
||||
'cmd': 'SelectRect',
|
||||
'timeframe': timeframe,
|
||||
# 'meth': str(meth),
|
||||
'meth': 'set_view_pos' if domain == 'view' else 'set_scene_pos',
|
||||
'kwargs': {
|
||||
'start_pos': tuple(start_pos),
|
||||
'end_pos': tuple(end_pos),
|
||||
'color': color,
|
||||
'update_label': False,
|
||||
'start_time': start_time,
|
||||
'end_time': end_time,
|
||||
},
|
||||
})
|
||||
aid: int|dict = await ipc.receive()
|
||||
match aid:
|
||||
case {'error': str(msg)}:
|
||||
log.error(msg)
|
||||
return None
|
||||
self._ipcs[aid] = ipc
|
||||
if not from_acm:
|
||||
self._annot_stack.push_async_callback(
|
||||
partial(
|
||||
self.remove,
|
||||
aid,
|
||||
)
|
||||
await ipc.send({
|
||||
'fqme': fqme,
|
||||
'cmd': 'SelectRect',
|
||||
'timeframe': timeframe,
|
||||
# 'meth': str(meth),
|
||||
'meth': 'set_view_pos' if domain == 'view' else 'set_scene_pos',
|
||||
'kwargs': {
|
||||
'start_pos': tuple(start_pos),
|
||||
'end_pos': tuple(end_pos),
|
||||
'color': color,
|
||||
'update_label': False,
|
||||
},
|
||||
})
|
||||
aid: int = await ipc.receive()
|
||||
self._ipcs[aid] = ipc
|
||||
if not from_acm:
|
||||
self._annot_stack.push_async_callback(
|
||||
partial(
|
||||
self.remove,
|
||||
aid,
|
||||
)
|
||||
return aid
|
||||
)
|
||||
return aid
|
||||
|
||||
async def remove(
|
||||
self,
|
||||
|
|
@ -659,9 +316,7 @@ class AnnotCtl(Struct):
|
|||
)
|
||||
yield aid
|
||||
finally:
|
||||
# async ipc send op
|
||||
with trio.CancelScope(shield=True):
|
||||
await self.remove(aid)
|
||||
await self.remove(aid)
|
||||
|
||||
async def redraw(
|
||||
self,
|
||||
|
|
@ -676,130 +331,20 @@ class AnnotCtl(Struct):
|
|||
'timeframe': timeframe,
|
||||
})
|
||||
|
||||
async def add_arrow(
|
||||
self,
|
||||
fqme: str,
|
||||
timeframe: float,
|
||||
x: float,
|
||||
y: float,
|
||||
pointing: Literal[
|
||||
'up',
|
||||
'down',
|
||||
],
|
||||
# TODO: a `Literal['view', 'scene']` for this?
|
||||
# domain: str = 'view', # or 'scene'
|
||||
color: str = 'dad_blue',
|
||||
alpha: int = 116,
|
||||
headLen: float|None = None,
|
||||
headWidth: float|None = None,
|
||||
tailLen: float|None = None,
|
||||
tailWidth: float|None = None,
|
||||
pxMode: bool = True,
|
||||
# TODO: do we even need this?
|
||||
# async def modify(
|
||||
# self,
|
||||
# aid: int, # annotation id
|
||||
# meth: str, # far end graphics object method to invoke
|
||||
# params: dict[str, Any], # far end `meth(**kwargs)`
|
||||
# ) -> bool:
|
||||
# '''
|
||||
# Modify an existing (remote) annotation's graphics
|
||||
# paramters, thus changing it's appearance / state in real
|
||||
# time.
|
||||
|
||||
from_acm: bool = False,
|
||||
|
||||
# NEW: optional timestamp for server-side index lookup
|
||||
time: float|None = None,
|
||||
|
||||
) -> int|None:
|
||||
'''
|
||||
Add a `SelectRect` annotation to the target view, return
|
||||
the instances `id(obj)` from the remote UI actor.
|
||||
|
||||
'''
|
||||
ipc: MsgStream = self._get_ipc(fqme)
|
||||
with trio.fail_after(3):
|
||||
await ipc.send({
|
||||
'fqme': fqme,
|
||||
'cmd': 'ArrowEditor',
|
||||
'timeframe': timeframe,
|
||||
# 'meth': str(meth),
|
||||
'meth': 'add',
|
||||
'kwargs': {
|
||||
'x': float(x),
|
||||
'y': float(y),
|
||||
'color': color,
|
||||
'pointing': pointing, # up|down
|
||||
'alpha': alpha,
|
||||
'aid': None,
|
||||
'headLen': headLen,
|
||||
'headWidth': headWidth,
|
||||
'tailLen': tailLen,
|
||||
'tailWidth': tailWidth,
|
||||
'pxMode': pxMode,
|
||||
'time': time, # for server-side index lookup
|
||||
},
|
||||
})
|
||||
aid: int|dict = await ipc.receive()
|
||||
match aid:
|
||||
case {'error': str(msg)}:
|
||||
log.error(msg)
|
||||
return None
|
||||
|
||||
self._ipcs[aid] = ipc
|
||||
if not from_acm:
|
||||
self._annot_stack.push_async_callback(
|
||||
partial(
|
||||
self.remove,
|
||||
aid,
|
||||
)
|
||||
)
|
||||
return aid
|
||||
|
||||
async def add_text(
|
||||
self,
|
||||
fqme: str,
|
||||
timeframe: float,
|
||||
text: str,
|
||||
x: float,
|
||||
y: float,
|
||||
color: str|tuple = 'dad_blue',
|
||||
anchor: tuple[float, float] = (0, 1),
|
||||
font_size: int|None = None,
|
||||
|
||||
from_acm: bool = False,
|
||||
|
||||
# NEW: optional timestamp for server-side index lookup
|
||||
time: float|None = None,
|
||||
|
||||
) -> int|None:
|
||||
'''
|
||||
Add a `pg.TextItem` annotation to the target view.
|
||||
|
||||
anchor: (x, y) where (0,0) is upper-left, (1,1) is lower-right
|
||||
font_size: pixel size for font, defaults to `_font.font.pixelSize()`
|
||||
|
||||
'''
|
||||
ipc: MsgStream = self._get_ipc(fqme)
|
||||
with trio.fail_after(3):
|
||||
await ipc.send({
|
||||
'fqme': fqme,
|
||||
'cmd': 'TextItem',
|
||||
'timeframe': timeframe,
|
||||
'kwargs': {
|
||||
'text': text,
|
||||
'x': float(x),
|
||||
'y': float(y),
|
||||
'color': color,
|
||||
'anchor': tuple(anchor),
|
||||
'font_size': font_size,
|
||||
'time': time, # for server-side index lookup
|
||||
},
|
||||
})
|
||||
aid: int|dict = await ipc.receive()
|
||||
match aid:
|
||||
case {'error': str(msg)}:
|
||||
log.error(msg)
|
||||
return None
|
||||
self._ipcs[aid] = ipc
|
||||
if not from_acm:
|
||||
self._annot_stack.push_async_callback(
|
||||
partial(
|
||||
self.remove,
|
||||
aid,
|
||||
)
|
||||
)
|
||||
return aid
|
||||
# '''
|
||||
# raise NotImplementedError
|
||||
|
||||
|
||||
@acm
|
||||
|
|
@ -826,9 +371,7 @@ async def open_annot_ctl(
|
|||
# TODO: print the current discoverable actor UID set
|
||||
# here as well?
|
||||
if not maybe_portals:
|
||||
raise RuntimeError(
|
||||
'No chart actors found in service domain?'
|
||||
)
|
||||
raise RuntimeError('No chart UI actors found in service domain?')
|
||||
|
||||
for portal in maybe_portals:
|
||||
ctx_mngrs.append(
|
||||
|
|
|
|||
|
|
@ -15,8 +15,7 @@
|
|||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
"""
|
||||
qompleterz: embeddable search and complete using trio, Qt and
|
||||
rapidfuzz.
|
||||
qompleterz: embeddable search and complete using trio, Qt and rapidfuzz.
|
||||
|
||||
"""
|
||||
|
||||
|
|
@ -47,7 +46,6 @@ import time
|
|||
from pprint import pformat
|
||||
|
||||
from rapidfuzz import process as fuzzy
|
||||
import tractor
|
||||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
|
||||
|
|
@ -55,7 +53,7 @@ from piker.ui.qt import (
|
|||
size_policy,
|
||||
align_flag,
|
||||
Qt,
|
||||
# QtCore,
|
||||
QtCore,
|
||||
QtWidgets,
|
||||
QModelIndex,
|
||||
QItemSelectionModel,
|
||||
|
|
@ -922,10 +920,7 @@ async def fill_results(
|
|||
|
||||
# issue multi-provider fan-out search request and place
|
||||
# "searching.." statuses on outstanding results providers
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
async with trio.open_nursery() as n:
|
||||
|
||||
for provider, (search, pause) in (
|
||||
_searcher_cache.copy().items()
|
||||
|
|
@ -949,7 +944,7 @@ async def fill_results(
|
|||
status_field='-> searchin..',
|
||||
)
|
||||
|
||||
await tn.start(
|
||||
await n.start(
|
||||
pack_matches,
|
||||
view,
|
||||
has_results,
|
||||
|
|
@ -1009,14 +1004,12 @@ async def handle_keyboard_input(
|
|||
view.set_font_size(searchbar.dpi_font.px_size)
|
||||
send, recv = trio.open_memory_channel(616)
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(), # needed?
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
async with trio.open_nursery() as n:
|
||||
|
||||
# start a background multi-searcher task which receives
|
||||
# patterns relayed from this keyboard input handler and
|
||||
# async updates the completer view's results.
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
partial(
|
||||
fill_results,
|
||||
searchw,
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@ class DpiAwareFont:
|
|||
) -> None:
|
||||
|
||||
self._font_size_calc_key: str = _font_size_key
|
||||
self._font_size: int|None = None
|
||||
self._font_size: int | None = None
|
||||
|
||||
# Read preferred font size from main config file if it exists
|
||||
conf, path = config.load('conf', touch_if_dne=True)
|
||||
|
|
@ -107,22 +107,7 @@ class DpiAwareFont:
|
|||
|
||||
@property
|
||||
def px_size(self) -> int:
|
||||
size: int = self._qfont.pixelSize()
|
||||
|
||||
# XXX, when no Qt app has been spawned this will always be
|
||||
# invalid..
|
||||
# SO, just return any conf.toml value.
|
||||
if size == -1:
|
||||
if (conf_size := self._font_size) is None:
|
||||
raise ValueError(
|
||||
f'No valid `{type(_font).__name__}.px_size` set?\n'
|
||||
f'\n'
|
||||
f'-> `ui.font_size` is NOT set in `conf.toml`\n'
|
||||
f'-> no Qt app is active ??\n'
|
||||
)
|
||||
return conf_size
|
||||
|
||||
return size
|
||||
return self._qfont.pixelSize()
|
||||
|
||||
def configure_to_dpi(self, screen: QtGui.QScreen | None = None):
|
||||
'''
|
||||
|
|
@ -236,20 +221,6 @@ def _config_fonts_to_screen() -> None:
|
|||
_font_small.configure_to_dpi()
|
||||
|
||||
|
||||
def get_fonts() -> tuple[
|
||||
DpiAwareFont,
|
||||
DpiAwareFont,
|
||||
]:
|
||||
'''
|
||||
Get the singleton font pair (of instances) from which all other
|
||||
UI/UX should be "scaled around".
|
||||
|
||||
See `DpiAwareFont` for (internal) deats.
|
||||
|
||||
'''
|
||||
return _font, _font_small
|
||||
|
||||
|
||||
# TODO: re-compute font size when main widget switches screens?
|
||||
# https://forum.qt.io/topic/54136/how-do-i-get-the-qscreen-my-widget-is-on-qapplication-desktop-screen-returns-a-qwidget-and-qobject_cast-qscreen-returns-null/3
|
||||
|
||||
|
|
@ -298,8 +269,6 @@ def hcolor(name: str) -> str:
|
|||
|
||||
# default ohlc-bars/curve gray
|
||||
'bracket': '#666666', # like the logo
|
||||
'pikers': '#616161', # a trader shade of..
|
||||
'beast': '#161616', # in the dark alone.
|
||||
|
||||
# bluish
|
||||
'charcoal': '#36454F',
|
||||
|
|
@ -337,7 +306,6 @@ def hcolor(name: str) -> str:
|
|||
'cool_green': '#33b864',
|
||||
'dull_green': '#74a662',
|
||||
'hedge_green': '#518360',
|
||||
'lilypad_green': '#839c84',
|
||||
|
||||
# orders and alerts
|
||||
'alert_yellow': '#e2d083',
|
||||
|
|
@ -365,7 +333,6 @@ def hcolor(name: str) -> str:
|
|||
'sell_red': '#b6003f',
|
||||
# 'sell_red': '#d00048',
|
||||
'sell_red_light': '#f85462',
|
||||
'wine': '#69212d',
|
||||
|
||||
# 'sell_red': '#f85462',
|
||||
# 'sell_red_light': '#ff4d5c',
|
||||
|
|
|
|||
|
|
@ -1,352 +0,0 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Root-most (what they call a "central widget") of every Qt-UI-app's
|
||||
window.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
from typing import (
|
||||
Iterator,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import trio
|
||||
|
||||
from piker.ui.qt import (
|
||||
QtCore,
|
||||
Qt,
|
||||
QWidget,
|
||||
QHBoxLayout,
|
||||
QVBoxLayout,
|
||||
)
|
||||
from ..log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._search import SearchWidget
|
||||
from ._chart import (
|
||||
LinkedSplits,
|
||||
)
|
||||
from ._cursor import (
|
||||
Cursor,
|
||||
)
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
_godw: GodWidget|None = None
|
||||
|
||||
def get_godw() -> GodWidget:
|
||||
'''
|
||||
Get the top level "god widget", the root/central-most Qt
|
||||
widget-object set as `QMainWindow.setCentralWidget(_godw)`.
|
||||
|
||||
See `piker.ui._exec` for the runtime init details and all the
|
||||
machinery for running `trio` on the Qt event loop in guest mode.
|
||||
|
||||
'''
|
||||
if _godw is None:
|
||||
raise RuntimeError(
|
||||
'No god-widget initialized ??\n'
|
||||
'Have you called `run_qtractor()` yet?\n'
|
||||
)
|
||||
return _godw
|
||||
|
||||
|
||||
class GodWidget(QWidget):
|
||||
'''
|
||||
"Our lord and savior, the holy child of window-shua, there is no
|
||||
widget above thee." - 6|6
|
||||
|
||||
The highest level composed widget which contains layouts for
|
||||
organizing charts as well as other sub-widgets used to control or
|
||||
modify them.
|
||||
|
||||
'''
|
||||
search: SearchWidget
|
||||
mode_name: str = 'god'
|
||||
|
||||
def __init__(
|
||||
|
||||
self,
|
||||
parent=None,
|
||||
|
||||
) -> None:
|
||||
|
||||
super().__init__(parent)
|
||||
|
||||
self.search: SearchWidget|None = None
|
||||
|
||||
self.hbox = QHBoxLayout(self)
|
||||
self.hbox.setContentsMargins(0, 0, 0, 0)
|
||||
self.hbox.setSpacing(6)
|
||||
self.hbox.setAlignment(Qt.AlignTop)
|
||||
|
||||
self.vbox = QVBoxLayout()
|
||||
self.vbox.setContentsMargins(0, 0, 0, 0)
|
||||
self.vbox.setSpacing(2)
|
||||
self.vbox.setAlignment(Qt.AlignTop)
|
||||
|
||||
self.hbox.addLayout(self.vbox)
|
||||
|
||||
self._chart_cache: dict[
|
||||
str,
|
||||
tuple[LinkedSplits, LinkedSplits],
|
||||
] = {}
|
||||
|
||||
self.hist_linked: LinkedSplits|None = None
|
||||
self.rt_linked: LinkedSplits|None = None
|
||||
self._active_cursor: Cursor|None = None
|
||||
|
||||
# assigned in the startup func `_async_main()`
|
||||
self._root_n: trio.Nursery = None
|
||||
|
||||
self._widgets: dict[str, QWidget] = {}
|
||||
self._resizing: bool = False
|
||||
|
||||
# TODO: do we need this, when would god get resized
|
||||
# and the window does not? Never right?!
|
||||
# self.reg_for_resize(self)
|
||||
|
||||
# TODO: strat loader/saver that we don't need yet.
|
||||
# def init_strategy_ui(self):
|
||||
# self.toolbar_layout = QHBoxLayout()
|
||||
# self.toolbar_layout.setContentsMargins(0, 0, 0, 0)
|
||||
# self.vbox.addLayout(self.toolbar_layout)
|
||||
# self.strategy_box = StrategyBoxWidget(self)
|
||||
# self.toolbar_layout.addWidget(self.strategy_box)
|
||||
|
||||
@property
|
||||
def linkedsplits(self) -> LinkedSplits:
|
||||
return self.rt_linked
|
||||
|
||||
def set_chart_symbols(
|
||||
self,
|
||||
group_key: tuple[str], # of form <fqme>.<providername>
|
||||
all_linked: tuple[LinkedSplits, LinkedSplits], # type: ignore
|
||||
|
||||
) -> None:
|
||||
# re-sort org cache symbol list in LIFO order
|
||||
cache = self._chart_cache
|
||||
cache.pop(group_key, None)
|
||||
cache[group_key] = all_linked
|
||||
|
||||
def get_chart_symbols(
|
||||
self,
|
||||
symbol_key: str,
|
||||
|
||||
) -> tuple[LinkedSplits, LinkedSplits]: # type: ignore
|
||||
return self._chart_cache.get(symbol_key)
|
||||
|
||||
async def load_symbols(
|
||||
self,
|
||||
fqmes: list[str],
|
||||
loglevel: str,
|
||||
reset: bool = False,
|
||||
|
||||
) -> trio.Event:
|
||||
'''
|
||||
Load a new contract into the charting app.
|
||||
|
||||
Expects a ``numpy`` structured array containing all the ohlcv fields.
|
||||
|
||||
'''
|
||||
# NOTE: for now we use the first symbol in the set as the "key"
|
||||
# for the overlay of feeds on the chart.
|
||||
group_key: tuple[str] = tuple(fqmes)
|
||||
|
||||
all_linked = self.get_chart_symbols(group_key)
|
||||
order_mode_started = trio.Event()
|
||||
|
||||
if not self.vbox.isEmpty():
|
||||
|
||||
# XXX: seems to make switching slower?
|
||||
# qframe = self.hist_linked.chart.qframe
|
||||
# if qframe.sidepane is self.search:
|
||||
# qframe.hbox.removeWidget(self.search)
|
||||
|
||||
for linked in [self.rt_linked, self.hist_linked]:
|
||||
# XXX: this is CRITICAL especially with pixel buffer caching
|
||||
linked.hide()
|
||||
linked.unfocus()
|
||||
|
||||
# XXX: pretty sure we don't need this
|
||||
# remove any existing plots?
|
||||
# XXX: ahh we might want to support cache unloading..
|
||||
# self.vbox.removeWidget(linked)
|
||||
|
||||
# switching to a new viewable chart
|
||||
if all_linked is None or reset:
|
||||
from ._display import display_symbol_data
|
||||
|
||||
# we must load a fresh linked charts set
|
||||
from ._chart import LinkedSplits
|
||||
self.rt_linked = rt_charts = LinkedSplits(self)
|
||||
self.hist_linked = hist_charts = LinkedSplits(self)
|
||||
|
||||
# spawn new task to start up and update new sub-chart instances
|
||||
self._root_n.start_soon(
|
||||
display_symbol_data,
|
||||
self,
|
||||
fqmes,
|
||||
loglevel,
|
||||
order_mode_started,
|
||||
)
|
||||
|
||||
# self.vbox.addWidget(hist_charts)
|
||||
self.vbox.addWidget(rt_charts)
|
||||
self.set_chart_symbols(
|
||||
group_key,
|
||||
(hist_charts, rt_charts),
|
||||
)
|
||||
|
||||
for linked in [hist_charts, rt_charts]:
|
||||
linked.show()
|
||||
linked.focus()
|
||||
|
||||
await trio.sleep(0)
|
||||
|
||||
else:
|
||||
# symbol is already loaded and ems ready
|
||||
order_mode_started.set()
|
||||
|
||||
self.hist_linked, self.rt_linked = all_linked
|
||||
|
||||
for linked in all_linked:
|
||||
# TODO:
|
||||
# - we'll probably want per-instrument/provider state here?
|
||||
# change the order config form over to the new chart
|
||||
|
||||
# chart is already in memory so just focus it
|
||||
linked.show()
|
||||
linked.focus()
|
||||
linked.graphics_cycle()
|
||||
await trio.sleep(0)
|
||||
|
||||
# resume feeds *after* rendering chart view asap
|
||||
chart = linked.chart
|
||||
if chart:
|
||||
chart.resume_all_feeds()
|
||||
|
||||
# TODO: we need a check to see if the chart
|
||||
# last had the xlast in view, if so then shift so it's
|
||||
# still in view, if the user was viewing history then
|
||||
# do nothing yah?
|
||||
self.rt_linked.chart.main_viz.default_view(
|
||||
do_min_bars=True,
|
||||
)
|
||||
|
||||
# if a history chart instance is already up then
|
||||
# set the search widget as its sidepane.
|
||||
hist_chart = self.hist_linked.chart
|
||||
if hist_chart:
|
||||
hist_chart.qframe.set_sidepane(self.search)
|
||||
|
||||
# NOTE: this is really stupid/hard to follow.
|
||||
# we have to reposition the active position nav
|
||||
# **AFTER** applying the search bar as a sidepane
|
||||
# to the newly switched to symbol.
|
||||
await trio.sleep(0)
|
||||
|
||||
# TODO: probably stick this in some kinda `LooknFeel` API?
|
||||
for tracker in self.rt_linked.mode.trackers.values():
|
||||
pp_nav = tracker.nav
|
||||
if tracker.live_pp.cumsize:
|
||||
pp_nav.show()
|
||||
pp_nav.hide_info()
|
||||
else:
|
||||
pp_nav.hide()
|
||||
|
||||
# set window titlebar info
|
||||
symbol = self.rt_linked.mkt
|
||||
if symbol is not None:
|
||||
self.window.setWindowTitle(
|
||||
f'{symbol.fqme} '
|
||||
f'tick:{symbol.size_tick}'
|
||||
)
|
||||
|
||||
return order_mode_started
|
||||
|
||||
def focus(self) -> None:
|
||||
'''
|
||||
Focus the top level widget which in turn focusses the chart
|
||||
ala "view mode".
|
||||
|
||||
'''
|
||||
# go back to view-mode focus (aka chart focus)
|
||||
self.clearFocus()
|
||||
chart = self.rt_linked.chart
|
||||
if chart:
|
||||
chart.setFocus()
|
||||
|
||||
def reg_for_resize(
|
||||
self,
|
||||
widget: QWidget,
|
||||
) -> None:
|
||||
getattr(widget, 'on_resize')
|
||||
self._widgets[widget.mode_name] = widget
|
||||
|
||||
def on_win_resize(
|
||||
self,
|
||||
event: QtCore.QEvent,
|
||||
) -> None:
|
||||
'''
|
||||
Top level god widget handler from window (the real yaweh) resize
|
||||
events such that any registered widgets which wish to be
|
||||
notified are invoked using our pythonic `.on_resize()` method
|
||||
api.
|
||||
|
||||
Where we do UX magic to make things not suck B)
|
||||
|
||||
'''
|
||||
if self._resizing:
|
||||
return
|
||||
|
||||
self._resizing = True
|
||||
|
||||
log.debug(
|
||||
f'God widget resize\n'
|
||||
f'{event}\n'
|
||||
)
|
||||
for name, widget in self._widgets.items():
|
||||
widget.on_resize()
|
||||
|
||||
self._resizing = False
|
||||
|
||||
# on_resize = on_win_resize
|
||||
|
||||
def get_cursor(self) -> Cursor:
|
||||
return self._active_cursor
|
||||
|
||||
def iter_linked(self) -> Iterator[LinkedSplits]:
|
||||
for linked in [self.hist_linked, self.rt_linked]:
|
||||
yield linked
|
||||
|
||||
def resize_all(self) -> None:
|
||||
'''
|
||||
Dynamic resize sequence: adjusts all sub-widgets/charts to
|
||||
sensible default ratios of what space is detected as available
|
||||
on the display / window.
|
||||
|
||||
'''
|
||||
rt_linked = self.rt_linked
|
||||
rt_linked.set_split_sizes()
|
||||
self.rt_linked.resize_sidepanes()
|
||||
self.hist_linked.resize_sidepanes(from_linked=rt_linked)
|
||||
self.search.on_resize()
|
||||
|
||||
|
||||
|
|
@ -40,7 +40,7 @@ from piker.ui.qt import (
|
|||
)
|
||||
from ..log import get_logger
|
||||
from ._style import _font_small, hcolor
|
||||
from ._widget import GodWidget
|
||||
from ._chart import GodWidget
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
|
@ -61,9 +61,9 @@ class MultiStatus:
|
|||
|
||||
self,
|
||||
msg: str,
|
||||
final_msg: str|None = None,
|
||||
final_msg: str | None = None,
|
||||
clear_on_next: bool = False,
|
||||
group_key: Union[bool, str]|None = False,
|
||||
group_key: Union[bool, str] | None = False,
|
||||
|
||||
) -> Union[Callable[..., None], str]:
|
||||
'''
|
||||
|
|
@ -175,11 +175,11 @@ class MainWindow(QMainWindow):
|
|||
self.setWindowTitle(self.title)
|
||||
|
||||
# set by runtime after `trio` is engaged.
|
||||
self.godwidget: GodWidget|None = None
|
||||
self.godwidget: GodWidget | None = None
|
||||
|
||||
self._status_bar: QStatusBar = None
|
||||
self._status_label: QLabel = None
|
||||
self._size: tuple[int, int]|None = None
|
||||
self._size: tuple[int, int] | None = None
|
||||
|
||||
@property
|
||||
def mode_label(self) -> QLabel:
|
||||
|
|
@ -202,7 +202,7 @@ class MainWindow(QMainWindow):
|
|||
label.setMargin(2)
|
||||
label.setAlignment(
|
||||
QtCore.Qt.AlignVCenter
|
||||
|QtCore.Qt.AlignRight
|
||||
| QtCore.Qt.AlignRight
|
||||
)
|
||||
self.statusBar().addPermanentWidget(label)
|
||||
label.show()
|
||||
|
|
@ -255,16 +255,8 @@ class MainWindow(QMainWindow):
|
|||
current: QWidget,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Focus handler.
|
||||
|
||||
For now updates the "current mode" name.
|
||||
|
||||
'''
|
||||
log.debug(
|
||||
f'widget focus changed from,\n'
|
||||
f'{last} -> {current}'
|
||||
)
|
||||
log.info(f'widget focus changed from {last} -> {current}')
|
||||
|
||||
if current is not None:
|
||||
# cursor left window?
|
||||
|
|
@ -296,7 +288,7 @@ class MainWindow(QMainWindow):
|
|||
|
||||
def configure_to_desktop(
|
||||
self,
|
||||
size: tuple[int, int]|None = None,
|
||||
size: tuple[int, int] | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
|
|||
|
|
@ -177,7 +177,7 @@ def chart(
|
|||
return
|
||||
|
||||
# global opts
|
||||
# brokernames: list[str] = config['brokers']
|
||||
brokernames = config['brokers']
|
||||
brokermods = config['brokermods']
|
||||
assert brokermods
|
||||
tractorloglevel = config['tractorloglevel']
|
||||
|
|
@ -216,7 +216,6 @@ def chart(
|
|||
layers['tcp']['port'],
|
||||
))
|
||||
|
||||
# breakpoint()
|
||||
from tractor.devx import maybe_open_crash_handler
|
||||
pdb: bool = config['pdb']
|
||||
with maybe_open_crash_handler(pdb=pdb):
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ Chart trading, the only way to scalp.
|
|||
from __future__ import annotations
|
||||
from contextlib import asynccontextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from decimal import Decimal
|
||||
from functools import partial
|
||||
from pprint import pformat
|
||||
import time
|
||||
|
|
@ -42,6 +41,7 @@ from piker.accounting import (
|
|||
Position,
|
||||
mk_allocator,
|
||||
MktPair,
|
||||
Symbol,
|
||||
)
|
||||
from piker.clearing import (
|
||||
open_ems,
|
||||
|
|
@ -59,14 +59,8 @@ from piker.data import (
|
|||
from piker.types import Struct
|
||||
from piker.log import get_logger
|
||||
from piker.ui.qt import Qt
|
||||
from ._editors import (
|
||||
LineEditor,
|
||||
ArrowEditor,
|
||||
)
|
||||
from ._lines import (
|
||||
order_line,
|
||||
LevelLine,
|
||||
)
|
||||
from ._editors import LineEditor, ArrowEditor
|
||||
from ._lines import order_line, LevelLine
|
||||
from ._position import (
|
||||
PositionTracker,
|
||||
SettingsPane,
|
||||
|
|
@ -77,6 +71,7 @@ from ._style import _font
|
|||
from ._forms import open_form_input_handling
|
||||
from ._notify import notify_from_ems_status_msg
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._chart import (
|
||||
ChartPlotWidget,
|
||||
|
|
@ -148,15 +143,6 @@ class OrderMode:
|
|||
}
|
||||
_staged_order: Order | None = None
|
||||
|
||||
@property
|
||||
def curr_mkt(self) -> MktPair:
|
||||
'''
|
||||
Deliver the currently selected `MktPair` according
|
||||
chart state.
|
||||
|
||||
'''
|
||||
return self.chart.linked.mkt
|
||||
|
||||
def on_level_change_update_next_order_info(
|
||||
self,
|
||||
level: float,
|
||||
|
|
@ -186,11 +172,7 @@ class OrderMode:
|
|||
line.update_labels(order_info)
|
||||
|
||||
# update bound-in staged order
|
||||
mkt: MktPair = self.curr_mkt
|
||||
order.price: Decimal = mkt.quantize(
|
||||
size=level,
|
||||
quantity_type='price',
|
||||
)
|
||||
order.price = level
|
||||
order.size = order_info['size']
|
||||
|
||||
# when an order is changed we flip the settings side-pane to
|
||||
|
|
@ -205,9 +187,7 @@ class OrderMode:
|
|||
|
||||
) -> LevelLine:
|
||||
|
||||
# TODO, if we instead just always decimalize at the ems layer
|
||||
# we can avoid this back-n-forth casting?
|
||||
level = float(order.price)
|
||||
level = order.price
|
||||
|
||||
line = order_line(
|
||||
chart or self.chart,
|
||||
|
|
@ -244,11 +224,7 @@ class OrderMode:
|
|||
# the order mode allocator but we still need to update the
|
||||
# "staged" order message we'll send to the ems
|
||||
def update_order_price(y: float) -> None:
|
||||
mkt: MktPair = self.curr_mkt
|
||||
order.price: Decimal = mkt.quantize(
|
||||
size=y,
|
||||
quantity_type='price',
|
||||
)
|
||||
order.price = y
|
||||
|
||||
line._on_level_change = update_order_price
|
||||
|
||||
|
|
@ -299,31 +275,34 @@ class OrderMode:
|
|||
chart = cursor.linked.chart
|
||||
if (
|
||||
not chart
|
||||
and
|
||||
cursor
|
||||
and
|
||||
cursor.active_plot
|
||||
and cursor
|
||||
and cursor.active_plot
|
||||
):
|
||||
return
|
||||
|
||||
chart = cursor.active_plot
|
||||
price: float = cursor._datum_xy[1]
|
||||
price = cursor._datum_xy[1]
|
||||
if not price:
|
||||
# zero prices are not supported by any means
|
||||
# since that's illogical / a no-op.
|
||||
return
|
||||
|
||||
mkt: MktPair = self.chart.linked.mkt
|
||||
|
||||
# NOTE : we could also use instead,
|
||||
# mkt.quantize(price, quantity_type='price')
|
||||
# but it returns a Decimal and it's probably gonna
|
||||
# be slower?
|
||||
# TODO: should we be enforcing this precision
|
||||
# at a different layer in the stack?
|
||||
# |_ might require `MktPair` tracking in the EMS?
|
||||
# |_ right now any precision error will be relayed
|
||||
# all the way back from the backend and vice-versa..
|
||||
#
|
||||
mkt: MktPair = self.curr_mkt
|
||||
price: Decimal = mkt.quantize(
|
||||
size=price,
|
||||
quantity_type='price',
|
||||
# at a different layer in the stack? right now
|
||||
# any precision error will literally be relayed
|
||||
# all the way back from the backend.
|
||||
|
||||
price = round(
|
||||
price,
|
||||
ndigits=mkt.price_tick_digits,
|
||||
)
|
||||
|
||||
order = self._staged_order = Order(
|
||||
action=action,
|
||||
price=price,
|
||||
|
|
@ -399,7 +378,7 @@ class OrderMode:
|
|||
'oid': oid,
|
||||
})
|
||||
|
||||
if float(order.price) <= 0:
|
||||
if order.price <= 0:
|
||||
log.error(
|
||||
'*!? Invalid `Order.price <= 0` ?!*\n'
|
||||
# TODO: make this present multi-line in object form
|
||||
|
|
@ -435,7 +414,7 @@ class OrderMode:
|
|||
lines=lines,
|
||||
last_status_close=self.multistatus.open_status(
|
||||
f'submitting {order.exec_mode}-{order.action}',
|
||||
# final_msg=f'submitted {order.exec_mode}-{order.action}',
|
||||
final_msg=f'submitted {order.exec_mode}-{order.action}',
|
||||
clear_on_next=True,
|
||||
)
|
||||
)
|
||||
|
|
@ -513,14 +492,13 @@ class OrderMode:
|
|||
def on_submit(
|
||||
self,
|
||||
uuid: str,
|
||||
order: Order|None = None,
|
||||
order: Order | None = None,
|
||||
|
||||
) -> Dialog|None:
|
||||
) -> Dialog | None:
|
||||
'''
|
||||
Order submitted status event handler.
|
||||
|
||||
Commit the order line and registered order uuid, store ack
|
||||
time stamp.
|
||||
Commit the order line and registered order uuid, store ack time stamp.
|
||||
|
||||
'''
|
||||
lines = self.lines.commit_line(uuid)
|
||||
|
|
@ -528,7 +506,7 @@ class OrderMode:
|
|||
# a submission is the start of a new order dialog
|
||||
dialog = self.dialogs[uuid]
|
||||
dialog.lines = lines
|
||||
cls: Callable|None = dialog.last_status_close
|
||||
cls: Callable | None = dialog.last_status_close
|
||||
if cls:
|
||||
cls()
|
||||
|
||||
|
|
@ -537,15 +515,14 @@ class OrderMode:
|
|||
# if an order msg is provided update the line
|
||||
# **from** that msg.
|
||||
if order:
|
||||
price: float = float(order.price)
|
||||
if price <= 0:
|
||||
if order.price <= 0:
|
||||
log.error(f'Order has 0 price, cancelling..\n{order}')
|
||||
self.cancel_orders([order.oid])
|
||||
return None
|
||||
|
||||
line.set_level(price)
|
||||
line.set_level(order.price)
|
||||
self.on_level_change_update_next_order_info(
|
||||
level=price,
|
||||
level=order.price,
|
||||
line=line,
|
||||
order=order,
|
||||
# use the corresponding position tracker for the
|
||||
|
|
@ -561,13 +538,14 @@ class OrderMode:
|
|||
|
||||
def on_fill(
|
||||
self,
|
||||
|
||||
uuid: str,
|
||||
price: float,
|
||||
time_s: float,
|
||||
|
||||
pointing: str | None = None,
|
||||
|
||||
) -> bool:
|
||||
) -> None:
|
||||
'''
|
||||
Fill msg handler.
|
||||
|
||||
|
|
@ -580,85 +558,62 @@ class OrderMode:
|
|||
- update fill bar size
|
||||
|
||||
'''
|
||||
# XXX WARNING XXX
|
||||
# if a `Status(resp='error')` arrives *before* this
|
||||
# fill-status, the `.dialogs` entry may have already been
|
||||
# popped and thus the below will skipped.
|
||||
#
|
||||
# NOTE, to avoid this confusing scenario ensure that any
|
||||
# errors delivered thru from the broker-backend are not just
|
||||
# "noisy reporting" (like is very common from IB..) and are
|
||||
# instead ONLY errors-causing-order-dialog-cancellation!
|
||||
if not (dialog := self.dialogs.get(uuid)):
|
||||
log.warning(
|
||||
f'Order was already cleared from `.dialogs` ??\n'
|
||||
f'uuid: {uuid!r}\n'
|
||||
)
|
||||
return False
|
||||
|
||||
dialog = self.dialogs[uuid]
|
||||
lines = dialog.lines
|
||||
chart = self.chart
|
||||
|
||||
if not lines:
|
||||
log.warn("No line(s) for order {uuid}!?")
|
||||
return False
|
||||
|
||||
# update line state(s)
|
||||
#
|
||||
# ?XXX this fails on certain types of races?
|
||||
# XXX: seems to fail on certain types of races?
|
||||
# assert len(lines) == 2
|
||||
flume: Flume = self.feed.flumes[chart.linked.mkt.fqme]
|
||||
_, _, ratio = flume.get_ds_info()
|
||||
if lines:
|
||||
flume: Flume = self.feed.flumes[chart.linked.mkt.fqme]
|
||||
_, _, ratio = flume.get_ds_info()
|
||||
|
||||
for chart, shm in [
|
||||
(self.chart, flume.rt_shm),
|
||||
(self.hist_chart, flume.hist_shm),
|
||||
]:
|
||||
viz = chart.get_viz(chart.name)
|
||||
index_field = viz.index_field
|
||||
arr = shm.array
|
||||
for chart, shm in [
|
||||
(self.chart, flume.rt_shm),
|
||||
(self.hist_chart, flume.hist_shm),
|
||||
]:
|
||||
viz = chart.get_viz(chart.name)
|
||||
index_field = viz.index_field
|
||||
arr = shm.array
|
||||
|
||||
# TODO: borked for int index based..
|
||||
index = flume.get_index(time_s, arr)
|
||||
# TODO: borked for int index based..
|
||||
index = flume.get_index(time_s, arr)
|
||||
|
||||
# get absolute index for arrow placement
|
||||
arrow_index = arr[index_field][index]
|
||||
# get absolute index for arrow placement
|
||||
arrow_index = arr[index_field][index]
|
||||
|
||||
self.arrows.add(
|
||||
chart.plotItem,
|
||||
uuid,
|
||||
arrow_index,
|
||||
price,
|
||||
pointing=pointing,
|
||||
color=lines[0].color
|
||||
)
|
||||
self.arrows.add(
|
||||
chart.plotItem,
|
||||
uuid,
|
||||
arrow_index,
|
||||
price,
|
||||
pointing=pointing,
|
||||
color=lines[0].color
|
||||
)
|
||||
else:
|
||||
log.warn("No line(s) for order {uuid}!?")
|
||||
|
||||
def on_cancel(
|
||||
self,
|
||||
uuid: str,
|
||||
uuid: str
|
||||
|
||||
) -> bool:
|
||||
) -> None:
|
||||
|
||||
msg: Order|None = self.client._sent_orders.pop(uuid, None)
|
||||
if msg is None:
|
||||
msg: Order = self.client._sent_orders.pop(uuid, None)
|
||||
|
||||
if msg is not None:
|
||||
self.lines.remove_line(uuid=uuid)
|
||||
self.chart.linked.cursor.show_xhair()
|
||||
|
||||
dialog = self.dialogs.pop(uuid, None)
|
||||
if dialog:
|
||||
dialog.last_status_close()
|
||||
else:
|
||||
log.warning(
|
||||
f'Received cancel for unsubmitted order {pformat(msg)}'
|
||||
)
|
||||
return False
|
||||
|
||||
# remove GUI line, show cursor.
|
||||
self.lines.remove_line(uuid=uuid)
|
||||
self.chart.linked.cursor.show_xhair()
|
||||
|
||||
# remove msg dialog (history)
|
||||
dialog: Dialog|None = self.dialogs.pop(uuid, None)
|
||||
if dialog:
|
||||
dialog.last_status_close()
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def cancel_orders_under_cursor(self) -> list[Dialog]:
|
||||
def cancel_orders_under_cursor(self) -> list[str]:
|
||||
return self.cancel_orders(
|
||||
self.oids_from_lines(
|
||||
self.lines.lines_under_cursor()
|
||||
|
|
@ -687,28 +642,24 @@ class OrderMode:
|
|||
self,
|
||||
oids: list[str],
|
||||
|
||||
) -> list[Dialog]:
|
||||
) -> None:
|
||||
'''
|
||||
Cancel all orders from a list of order ids: `oids`.
|
||||
|
||||
'''
|
||||
# key = self.multistatus.open_status(
|
||||
# f'cancelling {len(oids)} orders',
|
||||
# final_msg=f'cancelled orders:\n{oids}',
|
||||
# group_key=True
|
||||
# )
|
||||
dialogs: list[Dialog] = []
|
||||
key = self.multistatus.open_status(
|
||||
f'cancelling {len(oids)} orders',
|
||||
final_msg=f'cancelled orders:\n{oids}',
|
||||
group_key=True
|
||||
)
|
||||
for oid in oids:
|
||||
if dialog := self.dialogs.get(oid):
|
||||
self.client.cancel_nowait(uuid=oid)
|
||||
# cancel_status_close = self.multistatus.open_status(
|
||||
# f'cancelling order {oid}',
|
||||
# group_key=key,
|
||||
# )
|
||||
# dialog.last_status_close = cancel_status_close
|
||||
dialogs.append(dialog)
|
||||
|
||||
return dialogs
|
||||
cancel_status_close = self.multistatus.open_status(
|
||||
f'cancelling order {oid}',
|
||||
group_key=key,
|
||||
)
|
||||
dialog.last_status_close = cancel_status_close
|
||||
|
||||
def cancel_all_orders(self) -> None:
|
||||
'''
|
||||
|
|
@ -730,9 +681,9 @@ class OrderMode:
|
|||
) -> Dialog | None:
|
||||
# NOTE: the `.order` attr **must** be set with the
|
||||
# equivalent order msg in order to be loaded.
|
||||
order: Order = msg.req
|
||||
order = msg.req
|
||||
oid = str(msg.oid)
|
||||
symbol: str = order.symbol
|
||||
symbol = order.symbol
|
||||
|
||||
# TODO: MEGA UGGG ZONEEEE!
|
||||
src = msg.src
|
||||
|
|
@ -751,22 +702,13 @@ class OrderMode:
|
|||
order.oid = str(order.oid)
|
||||
order.brokers = [brokername]
|
||||
|
||||
# ?TODO? change this over to `MktPair`, but it's gonna be
|
||||
# tough since we don't have any such data really in our
|
||||
# clearing msg schema..
|
||||
# BUT WAIT! WHY do we even want/need this!?
|
||||
#
|
||||
# order.symbol = self.curr_mkt
|
||||
#
|
||||
# XXX, the old approach.. which i don't quire member why..
|
||||
# -[ ] verify we for sure don't require this any more!
|
||||
# |_https://github.com/pikers/piker/issues/517
|
||||
#
|
||||
# order.symbol = Symbol.from_fqme(
|
||||
# fqsn=fqme,
|
||||
# info={},
|
||||
# )
|
||||
|
||||
# TODO: change this over to `MktPair`, but it's
|
||||
# gonna be tough since we don't have any such data
|
||||
# really in our clearing msg schema..
|
||||
order.symbol = Symbol.from_fqme(
|
||||
fqsn=fqme,
|
||||
info={},
|
||||
)
|
||||
maybe_dialog: Dialog | None = self.submit_order(
|
||||
send_msg=False,
|
||||
order=order,
|
||||
|
|
@ -780,6 +722,7 @@ class OrderMode:
|
|||
|
||||
@asynccontextmanager
|
||||
async def open_order_mode(
|
||||
|
||||
feed: Feed,
|
||||
godw: GodWidget,
|
||||
fqme: str,
|
||||
|
|
@ -823,7 +766,6 @@ async def open_order_mode(
|
|||
brokerd_accounts,
|
||||
ems_dialog_msgs,
|
||||
),
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
|
||||
):
|
||||
|
|
@ -1088,23 +1030,13 @@ async def process_trade_msg(
|
|||
if name in (
|
||||
'position',
|
||||
):
|
||||
mkt: MktPair = mode.chart.linked.mkt
|
||||
sym: MktPair = mode.chart.linked.mkt
|
||||
pp_msg_symbol = msg['symbol'].lower()
|
||||
pp_msg_bsmktid = msg['bs_mktid']
|
||||
fqme = mkt.fqme
|
||||
broker = mkt.broker
|
||||
fqme = sym.fqme
|
||||
broker = sym.broker
|
||||
if (
|
||||
# match on any backed-specific(-unique)-ID first!
|
||||
(
|
||||
pp_msg_bsmktid
|
||||
and
|
||||
mkt.bs_mktid == pp_msg_bsmktid
|
||||
)
|
||||
or
|
||||
# OW try against what's provided as an FQME..
|
||||
pp_msg_symbol == fqme
|
||||
or
|
||||
pp_msg_symbol == fqme.removesuffix(f'.{broker}')
|
||||
or pp_msg_symbol == fqme.removesuffix(f'.{broker}')
|
||||
):
|
||||
log.info(
|
||||
f'Loading position for `{fqme}`:\n'
|
||||
|
|
@ -1127,7 +1059,7 @@ async def process_trade_msg(
|
|||
return
|
||||
|
||||
msg = Status(**msg)
|
||||
# resp: str = msg.resp
|
||||
resp = msg.resp
|
||||
oid = msg.oid
|
||||
dialog: Dialog = mode.dialogs.get(oid)
|
||||
|
||||
|
|
@ -1169,7 +1101,7 @@ async def process_trade_msg(
|
|||
)
|
||||
)
|
||||
):
|
||||
msg.req: Order = order
|
||||
msg.req = order
|
||||
dialog: (
|
||||
Dialog
|
||||
# NOTE: on an invalid order submission (eg.
|
||||
|
|
@ -1191,33 +1123,20 @@ async def process_trade_msg(
|
|||
mode.on_submit(oid)
|
||||
|
||||
case Status(resp='error'):
|
||||
# TODO: parse into broker-side msg, or should we
|
||||
# expect it to just be **that** msg verbatim (since
|
||||
# we'd presumably have only 1 `Error` msg-struct)
|
||||
broker_msg: dict = msg.brokerd_msg
|
||||
|
||||
# XXX NOTE, this presumes the rxed "error" is
|
||||
# order-dialog-cancel-causing, THUS backends much ONLY
|
||||
# relay errors of this "severity"!!
|
||||
log.error(
|
||||
f'Order errored ??\n'
|
||||
f'oid: {oid!r}\n'
|
||||
f'\n'
|
||||
f'{pformat(broker_msg)}\n'
|
||||
f'\n'
|
||||
f'=> CANCELLING ORDER DIALOG <=\n'
|
||||
|
||||
# from tractor.devx.pformat import ppfmt
|
||||
# !TODO LOL, wtf the msg is causing
|
||||
# a recursion bug!
|
||||
# -[ ] get this shit on msgspec stat!
|
||||
# f'{ppfmt(broker_msg)}'
|
||||
)
|
||||
# do all the things for a cancel:
|
||||
# - drop order-msg dialog from client table
|
||||
# - delete level line from view
|
||||
mode.on_cancel(oid)
|
||||
|
||||
# TODO: parse into broker-side msg, or should we
|
||||
# expect it to just be **that** msg verbatim (since
|
||||
# we'd presumably have only 1 `Error` msg-struct)
|
||||
broker_msg: dict = msg.brokerd_msg
|
||||
log.error(
|
||||
f'Order {oid}->{resp} with:\n{pformat(broker_msg)}'
|
||||
)
|
||||
|
||||
case Status(resp='canceled'):
|
||||
# delete level line from view
|
||||
mode.on_cancel(oid)
|
||||
|
|
@ -1232,10 +1151,10 @@ async def process_trade_msg(
|
|||
# TODO: UX for a "pending" clear/live order
|
||||
log.info(f'Dark order triggered for {fmtmsg}')
|
||||
|
||||
# TODO: do the struct-msg version, blah blah..
|
||||
# req=Order(exec_mode='live', action='alert') as req,
|
||||
case Status(
|
||||
resp='triggered',
|
||||
# TODO: do the struct-msg version, blah blah..
|
||||
# req=Order(exec_mode='live', action='alert') as req,
|
||||
req={
|
||||
'exec_mode': 'live',
|
||||
'action': 'alert',
|
||||
|
|
@ -1247,7 +1166,7 @@ async def process_trade_msg(
|
|||
tm = time.time()
|
||||
mode.on_fill(
|
||||
oid,
|
||||
price=float(req.price),
|
||||
price=req.price,
|
||||
time_s=tm,
|
||||
)
|
||||
mode.lines.remove_line(uuid=oid)
|
||||
|
|
@ -1302,7 +1221,7 @@ async def process_trade_msg(
|
|||
tm = details['broker_time']
|
||||
mode.on_fill(
|
||||
oid,
|
||||
price=float(details['price']),
|
||||
price=details['price'],
|
||||
time_s=tm,
|
||||
pointing='up' if action == 'buy' else 'down',
|
||||
)
|
||||
|
|
|
|||
323
pyproject.toml
323
pyproject.toml
|
|
@ -15,196 +15,141 @@
|
|||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
requires = ["poetry-core"]
|
||||
build-backend = "poetry.core.masonry.api"
|
||||
|
||||
# ------ - ------
|
||||
|
||||
[tool.ruff.lint]
|
||||
# https://docs.astral.sh/ruff/settings/#lint_ignore
|
||||
ignore = []
|
||||
|
||||
# https://docs.astral.sh/ruff/settings/#lint_per-file-ignores
|
||||
"piker/ui/qt.py" = [
|
||||
"E402",
|
||||
'F401', # unused imports (without __all__ or blah as blah)
|
||||
# "F841", # unused variable rules
|
||||
]
|
||||
# ignore-init-module-imports = false
|
||||
|
||||
# ------ - ------
|
||||
|
||||
[tool.poetry]
|
||||
name = "piker"
|
||||
version = "0.1.0.alpha0.dev0"
|
||||
description = "trading gear for hackers"
|
||||
authors = ["Tyler Goodlet <goodboy_foss@protonmail.com>"]
|
||||
license = "AGPLv3"
|
||||
readme = "README.rst"
|
||||
|
||||
# ------ - ------
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
async-generator = "^1.10"
|
||||
attrs = "^23.1.0"
|
||||
bidict = "^0.22.1"
|
||||
colorama = "^0.4.6"
|
||||
colorlog = "^6.7.0"
|
||||
cython = "^3.0.0"
|
||||
greenback = "^1.1.1"
|
||||
ib-insync = "^0.9.86"
|
||||
msgspec = "^0.18.0"
|
||||
numba = "^0.59.0"
|
||||
numpy = "^1.25"
|
||||
polars = "^0.18.13"
|
||||
pygments = "^2.16.1"
|
||||
python = ">=3.11, <3.13"
|
||||
rich = "^13.5.2"
|
||||
# setuptools = "^68.0.0"
|
||||
tomli = "^2.0.1"
|
||||
tomli-w = "^1.0.0"
|
||||
trio-util = "^0.7.0"
|
||||
trio-websocket = "^0.10.3"
|
||||
typer = "^0.9.0"
|
||||
rapidfuzz = "^3.5.2"
|
||||
pdbp = "^1.5.0"
|
||||
trio = "^0.24"
|
||||
pendulum = "^3.0.0"
|
||||
httpx = "^0.27.0"
|
||||
|
||||
[tool.poetry.dependencies.tractor]
|
||||
develop = true
|
||||
git = 'https://github.com/goodboy/tractor.git'
|
||||
branch = 'asyncio_debugger_support'
|
||||
# path = "../tractor"
|
||||
|
||||
[tool.poetry.dependencies.asyncvnc]
|
||||
git = 'https://github.com/pikers/asyncvnc.git'
|
||||
branch = 'main'
|
||||
|
||||
[tool.poetry.dependencies.tomlkit]
|
||||
develop = true
|
||||
git = 'https://github.com/pikers/tomlkit.git'
|
||||
branch = 'piker_pin'
|
||||
# path = "../tomlkit/"
|
||||
|
||||
[tool.poetry.group.uis]
|
||||
optional = true
|
||||
[tool.poetry.group.uis.dependencies]
|
||||
# https://python-poetry.org/docs/managing-dependencies/#dependency-groups
|
||||
# TODO: make sure the levenshtein shit compiles on nix..
|
||||
# rapidfuzz = {extras = ["speedup"], version = "^0.18.0"}
|
||||
rapidfuzz = "^3.2.0"
|
||||
qdarkstyle = ">=3.0.2"
|
||||
pyqtgraph = { git = 'https://github.com/pikers/pyqtgraph.git' }
|
||||
|
||||
# ------ - ------
|
||||
pyqt6 = "^6.7.0"
|
||||
|
||||
[tool.poetry.group.dev]
|
||||
optional = true
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
# testing / CI
|
||||
pytest = "^6.0.0"
|
||||
elasticsearch = "^8.9.0"
|
||||
xonsh = "^0.14.2"
|
||||
prompt-toolkit = "3.0.40"
|
||||
|
||||
# console ehancements and eventually remote debugging
|
||||
# extras/helpers.
|
||||
# TODO: add a toolset that makes debugging a `pikerd` service
|
||||
# (tree) easy to hack on directly using more or less the local env:
|
||||
# - xonsh + xxh
|
||||
# - rsyscall + pdbp
|
||||
# - actor runtime control console like BEAM/OTP
|
||||
|
||||
# ------ - ------
|
||||
|
||||
# TODO: add an `--only daemon` group for running non-ui / pikerd
|
||||
# service tree in distributed mode B)
|
||||
# https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies
|
||||
# [tool.poetry.group.daemon.dependencies]
|
||||
|
||||
[tool.poetry.scripts]
|
||||
piker = 'piker.cli:cli'
|
||||
pikerd = 'piker.cli:pikerd'
|
||||
ledger = 'piker.accounting.cli:ledger'
|
||||
|
||||
|
||||
[project]
|
||||
name = "piker"
|
||||
version = "0.1.0a0dev0"
|
||||
description = "trading gear for hackers"
|
||||
authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
|
||||
requires-python = ">=3.12"
|
||||
license = "AGPL-3.0-or-later"
|
||||
readme = "README.rst"
|
||||
keywords = [
|
||||
"async",
|
||||
"trading",
|
||||
"finance",
|
||||
"quant",
|
||||
"charting",
|
||||
name="piker"
|
||||
keywords=[
|
||||
"async",
|
||||
"trading",
|
||||
"finance",
|
||||
"quant",
|
||||
"charting",
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 3 - Alpha",
|
||||
"License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
|
||||
"Operating System :: POSIX :: Linux",
|
||||
"Programming Language :: Python :: Implementation :: CPython",
|
||||
"Programming Language :: Python :: 3 :: Only",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"Intended Audience :: Financial and Insurance Industry",
|
||||
"Intended Audience :: Science/Research",
|
||||
"Intended Audience :: Developers",
|
||||
"Intended Audience :: Education",
|
||||
classifiers=[
|
||||
'Development Status :: 3 - Alpha',
|
||||
"License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
|
||||
'Operating System :: POSIX :: Linux',
|
||||
"Programming Language :: Python :: Implementation :: CPython",
|
||||
"Programming Language :: Python :: 3 :: Only",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
'Intended Audience :: Financial and Insurance Industry',
|
||||
'Intended Audience :: Science/Research',
|
||||
'Intended Audience :: Developers',
|
||||
'Intended Audience :: Education',
|
||||
]
|
||||
dependencies = [
|
||||
"async-generator >=1.10, <2.0.0",
|
||||
"attrs >=23.1.0, <24.0.0",
|
||||
"bidict >=0.23.1",
|
||||
"colorama >=0.4.6, <0.5.0",
|
||||
"colorlog >=6.7.0, <7.0.0",
|
||||
"ib-insync >=0.9.86, <0.10.0",
|
||||
"numpy>=2.0",
|
||||
"polars >=0.20.6",
|
||||
"polars-fuzzy-match>=0.1.5",
|
||||
"pygments >=2.16.1, <3.0.0",
|
||||
"rich >=13.5.2, <14.0.0",
|
||||
"tomli >=2.0.1, <3.0.0",
|
||||
"tomli-w >=1.0.0, <2.0.0",
|
||||
"trio-util >=0.7.0, <0.8.0",
|
||||
"trio-websocket >=0.10.3, <0.11.0",
|
||||
"typer >=0.9.0, <1.0.0",
|
||||
"trio >=0.27",
|
||||
"pendulum",
|
||||
"httpx >=0.27.0, <0.28.0",
|
||||
"cryptofeed >=2.4.0, <3.0.0",
|
||||
"pyarrow>=18.0.0",
|
||||
"websockets ==12.0",
|
||||
"msgspec>=0.19.0,<0.20",
|
||||
"tractor",
|
||||
"tomlkit",
|
||||
"trio-typing>=0.10.0",
|
||||
"numba>=0.61.0",
|
||||
"pyvnc",
|
||||
"exchange-calendars>=4.13.1",
|
||||
]
|
||||
# ------ dependencies ------
|
||||
# NOTE, by default we ship only a "headless" deps set bc
|
||||
# the `uis` group is not listed in the optional set.
|
||||
|
||||
# [optional-dependencies]
|
||||
# uis = []
|
||||
# ?TODO? really we should be able to mv this `uis` group
|
||||
# to be under [optional-dependencies] and then include
|
||||
# it in the dev deps?
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#optional-dependencies
|
||||
# -> uis should be included in pubbed pkgs.
|
||||
# [ ] uv seems to have no way to do this though?
|
||||
|
||||
# TODO? move to a `uv.toml`?
|
||||
[tool.uv]
|
||||
# https://docs.astral.sh/uv/reference/settings/#python-preference
|
||||
python-preference = 'system'
|
||||
# https://docs.astral.sh/uv/reference/settings/#python-downloads
|
||||
python-downloads = 'manual'
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#default-groups
|
||||
default-groups = [
|
||||
'uis',
|
||||
'repl',
|
||||
]
|
||||
# ------ tool.uv ------
|
||||
|
||||
[dependency-groups]
|
||||
uis = [
|
||||
"pyqtgraph",
|
||||
"qdarkstyle >=3.0.2, <4.0.0",
|
||||
"pyqt6 >=6.7.0, <7.0.0",
|
||||
|
||||
# fuzzy search
|
||||
"rapidfuzz >=3.2.0, <4.0.0",
|
||||
]
|
||||
|
||||
# dev deps enabled by `uv --dev`
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
|
||||
dev = [
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
|
||||
"cython >=3.0.0, <4.0.0",
|
||||
# nested deps-groups
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#nesting-groups
|
||||
{include-group = 'uis'},
|
||||
{include-group = 'repl'},
|
||||
{include-group = 'testing'},
|
||||
{include-group = 'de'},
|
||||
]
|
||||
repl = [
|
||||
# `tractor`'s debugger
|
||||
"pdbp >=1.8.2, <2.0.0",
|
||||
"greenback >=1.1.1, <2.0.0",
|
||||
|
||||
# @goodboy's preferred console toolz
|
||||
"xonsh>=0.22.2",
|
||||
"prompt-toolkit ==3.0.40",
|
||||
"pyperclip>=1.9.0",
|
||||
|
||||
# for @claude's `snippets/claude_debug_helper.py` it uses to do
|
||||
# "offline" debug/crash REPL-in alongside a dev.
|
||||
"pexpect>=4.9.0",
|
||||
|
||||
# ?TODO, new stuff to consider..
|
||||
# "visidata" # console numerics
|
||||
# "xxh" # for remote `xonsh`-ing
|
||||
# "rsyscall" # (eventual) optional `tractor` backend
|
||||
# - an actor-runtime-ctl console like BEAM/OTP
|
||||
]
|
||||
testing = [
|
||||
"pytest",
|
||||
]
|
||||
de = [ # (linux) specific DEs
|
||||
"i3ipc>=2.2.1",
|
||||
]
|
||||
lint = [
|
||||
# XXX, with flake.nix needs to be from nixpkgs
|
||||
"ruff>=0.9.6"
|
||||
# ; os_name != 'nixos' and platform_system != 'NixOS'",
|
||||
# ?TODO? since ^ markers won't work, use a deps-flags to toggle for
|
||||
# now.
|
||||
]
|
||||
dbs = [
|
||||
"elasticsearch >=8.9.0, <9.0.0",
|
||||
]
|
||||
# ------ dependency-groups ------
|
||||
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
# https://docs.pytest.org/en/stable/reference/reference.html#configuration-options
|
||||
testpaths = [
|
||||
"tests",
|
||||
]
|
||||
# https://docs.pytest.org/en/stable/reference/reference.html#confval-console_output_style
|
||||
console_output_style = 'progress'
|
||||
|
||||
# https://docs.pytest.org/en/stable/how-to/plugins.html#disabling-plugins-from-autoloading
|
||||
# https://docs.pytest.org/en/stable/how-to/plugins.html#deactivating-unregistering-a-plugin-by-name
|
||||
addopts = '-p no:xonsh'
|
||||
# ------ tool.pytest ------
|
||||
|
||||
|
||||
[project.scripts]
|
||||
piker = "piker.cli:cli"
|
||||
pikerd = "piker.cli:pikerd"
|
||||
ledger = "piker.accounting.cli:ledger"
|
||||
# ------ project.scripts ------
|
||||
|
||||
[tool.hatch.build.targets.sdist]
|
||||
include = ["piker"]
|
||||
[tool.hatch.build.targets.wheel]
|
||||
include = ["piker"]
|
||||
# ------ tool.hatch ------
|
||||
|
||||
|
||||
[tool.uv.sources]
|
||||
pyqtgraph = { git = "https://github.com/pikers/pyqtgraph.git" }
|
||||
tomlkit = { git = "https://github.com/pikers/tomlkit.git", branch ="piker_pin" }
|
||||
pyvnc = { git = "https://github.com/regulad/pyvnc.git" }
|
||||
|
||||
# to get fancy next-cmd/suggestion feats prior to 0.22.2 B)
|
||||
# https://github.com/xonsh/xonsh/pull/6037
|
||||
# https://github.com/xonsh/xonsh/pull/6048
|
||||
# xonsh = { git = 'https://github.com/xonsh/xonsh.git', branch = 'main' }
|
||||
|
||||
# XXX since, we're like, always hacking new shite all-the-time. Bp
|
||||
tractor = { git = "https://github.com/goodboy/tractor.git", branch ="piker_pin" }
|
||||
# tractor = { git = "https://pikers.dev/goodboy/tractor", branch = "piker_pin" }
|
||||
# tractor = { git = "https://pikers.dev/goodboy/tractor", branch = "main" }
|
||||
# ------ goodboy ------
|
||||
# hackin dev-envs, usually there's something new he's hackin in..
|
||||
# tractor = { path = "../tractor", editable = true }
|
||||
|
|
|
|||
94
ruff.toml
94
ruff.toml
|
|
@ -1,94 +0,0 @@
|
|||
# from default `ruff.toml` @
|
||||
# https://docs.astral.sh/ruff/configuration/
|
||||
|
||||
# Exclude a variety of commonly ignored directories.
|
||||
exclude = [
|
||||
".bzr",
|
||||
".direnv",
|
||||
".eggs",
|
||||
".git",
|
||||
".git-rewrite",
|
||||
".hg",
|
||||
".ipynb_checkpoints",
|
||||
".mypy_cache",
|
||||
".nox",
|
||||
".pants.d",
|
||||
".pyenv",
|
||||
".pytest_cache",
|
||||
".pytype",
|
||||
".ruff_cache",
|
||||
".svn",
|
||||
".tox",
|
||||
".venv",
|
||||
".vscode",
|
||||
"__pypackages__",
|
||||
"_build",
|
||||
"buck-out",
|
||||
"build",
|
||||
"dist",
|
||||
"node_modules",
|
||||
"site-packages",
|
||||
"venv",
|
||||
]
|
||||
|
||||
# Same as Black.
|
||||
line-length = 88
|
||||
indent-width = 4
|
||||
|
||||
# Assume Python 3.9
|
||||
target-version = "py312"
|
||||
|
||||
# ------ - ------
|
||||
# TODO, stop warnings around `anext()` builtin use?
|
||||
# tool.ruff.target-version = "py310"
|
||||
|
||||
|
||||
[lint]
|
||||
# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default.
|
||||
# Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
|
||||
# McCabe complexity (`C901`) by default.
|
||||
select = ["E4", "E7", "E9", "F"]
|
||||
ignore = []
|
||||
ignore-init-module-imports = false
|
||||
|
||||
[lint.per-file-ignores]
|
||||
"piker/ui/qt.py" = [
|
||||
"E402",
|
||||
'F401', # unused imports (without __all__ or blah as blah)
|
||||
# "F841", # unused variable rules
|
||||
]
|
||||
|
||||
# Allow fix for all enabled rules (when `--fix`) is provided.
|
||||
fixable = ["ALL"]
|
||||
unfixable = []
|
||||
|
||||
# TODO? uhh why no work!?
|
||||
# Allow unused variables when underscore-prefixed.
|
||||
# dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
|
||||
|
||||
[format]
|
||||
# Use single quotes in `ruff format`.
|
||||
quote-style = "single"
|
||||
|
||||
# Like Black, indent with spaces, rather than tabs.
|
||||
indent-style = "space"
|
||||
|
||||
# Like Black, respect magic trailing commas.
|
||||
skip-magic-trailing-comma = false
|
||||
|
||||
# Like Black, automatically detect the appropriate line ending.
|
||||
line-ending = "auto"
|
||||
|
||||
# Enable auto-formatting of code examples in docstrings. Markdown,
|
||||
# reStructuredText code/literal blocks and doctests are all supported.
|
||||
#
|
||||
# This is currently disabled by default, but it is planned for this
|
||||
# to be opt-out in the future.
|
||||
docstring-code-format = false
|
||||
|
||||
# Set the line length limit used when formatting code snippets in
|
||||
# docstrings.
|
||||
#
|
||||
# This only has an effect when the `docstring-code-format` setting is
|
||||
# enabled.
|
||||
docstring-code-line-length = "dynamic"
|
||||
|
|
@ -1,256 +0,0 @@
|
|||
#!/usr/bin/env python
|
||||
'''
|
||||
Programmatic debugging helper for `pdbp` REPL human-like
|
||||
interaction but built to allow `claude` to interact with
|
||||
crashes and `tractor.pause()` breakpoints along side a human dev.
|
||||
|
||||
Originally written by `clauded` during a backfiller inspection
|
||||
session with @goodboy trying to resolve duplicate/gappy ohlcv ts
|
||||
issues discovered while testing the new `nativedb` tsdb.
|
||||
|
||||
Allows `claude` to run `pdb` commands and capture output in an "offline"
|
||||
manner but generating similar output as if it was iteracting with
|
||||
the debug REPL.
|
||||
|
||||
The use of `pexpect` is heavily based on tractor's REPL UX test
|
||||
suite(s), namely various `tests/devx/test_debugger.py` patterns.
|
||||
|
||||
'''
|
||||
import sys
|
||||
import os
|
||||
import time
|
||||
|
||||
import pexpect
|
||||
from pexpect.exceptions import (
|
||||
TIMEOUT,
|
||||
EOF,
|
||||
)
|
||||
|
||||
|
||||
PROMPT: str = r'\(Pdb\+\)'
|
||||
|
||||
|
||||
def expect(
|
||||
child: pexpect.spawn,
|
||||
patt: str,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
'''
|
||||
Expect wrapper that prints last console data before failing.
|
||||
|
||||
'''
|
||||
try:
|
||||
child.expect(
|
||||
patt,
|
||||
**kwargs,
|
||||
)
|
||||
except TIMEOUT:
|
||||
before: str = (
|
||||
str(child.before.decode())
|
||||
if isinstance(child.before, bytes)
|
||||
else str(child.before)
|
||||
)
|
||||
print(
|
||||
f'TIMEOUT waiting for pattern: {patt}\n'
|
||||
f'Last seen output:\n{before}'
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
def run_pdb_commands(
|
||||
commands: list[str],
|
||||
initial_cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance',
|
||||
timeout: int = 30,
|
||||
print_output: bool = True,
|
||||
) -> dict[str, str]:
|
||||
'''
|
||||
Spawn piker process, wait for pdb prompt, execute commands.
|
||||
|
||||
Returns dict mapping command -> output.
|
||||
|
||||
'''
|
||||
results: dict[str, str] = {}
|
||||
|
||||
# Disable colored output for easier parsing
|
||||
os.environ['PYTHON_COLORS'] = '0'
|
||||
|
||||
# Spawn the process
|
||||
if print_output:
|
||||
print(f'Spawning: {initial_cmd}')
|
||||
|
||||
child: pexpect.spawn = pexpect.spawn(
|
||||
initial_cmd,
|
||||
timeout=timeout,
|
||||
encoding='utf-8',
|
||||
echo=False,
|
||||
)
|
||||
|
||||
# Wait for pdb prompt
|
||||
try:
|
||||
expect(child, PROMPT, timeout=timeout)
|
||||
if print_output:
|
||||
print('Reached pdb prompt!')
|
||||
|
||||
# Execute each command
|
||||
for cmd in commands:
|
||||
if print_output:
|
||||
print(f'\n>>> {cmd}')
|
||||
|
||||
child.sendline(cmd)
|
||||
time.sleep(0.1)
|
||||
|
||||
# Wait for next prompt
|
||||
expect(child, PROMPT, timeout=timeout)
|
||||
|
||||
# Capture output (everything before the prompt)
|
||||
output: str = (
|
||||
str(child.before.decode())
|
||||
if isinstance(child.before, bytes)
|
||||
else str(child.before)
|
||||
)
|
||||
results[cmd] = output
|
||||
|
||||
if print_output:
|
||||
print(output)
|
||||
|
||||
# Quit debugger gracefully
|
||||
child.sendline('quit')
|
||||
try:
|
||||
child.expect(EOF, timeout=5)
|
||||
except (TIMEOUT, EOF):
|
||||
pass
|
||||
|
||||
except TIMEOUT as e:
|
||||
print(f'Timeout: {e}')
|
||||
if child.before:
|
||||
before: str = (
|
||||
str(child.before.decode())
|
||||
if isinstance(child.before, bytes)
|
||||
else str(child.before)
|
||||
)
|
||||
print(f'Buffer:\n{before}')
|
||||
results['_error'] = str(e)
|
||||
|
||||
finally:
|
||||
if child.isalive():
|
||||
child.close(force=True)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
class InteractivePdbSession:
|
||||
'''
|
||||
Interactive pdb session manager for incremental debugging.
|
||||
|
||||
'''
|
||||
def __init__(
|
||||
self,
|
||||
cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance',
|
||||
timeout: int = 30,
|
||||
):
|
||||
self.cmd: str = cmd
|
||||
self.timeout: int = timeout
|
||||
self.child: pexpect.spawn|None = None
|
||||
self.history: list[tuple[str, str]] = []
|
||||
|
||||
def start(self) -> None:
|
||||
'''
|
||||
Start the piker process and wait for first prompt.
|
||||
|
||||
'''
|
||||
os.environ['PYTHON_COLORS'] = '0'
|
||||
|
||||
print(f'Starting: {self.cmd}')
|
||||
self.child = pexpect.spawn(
|
||||
self.cmd,
|
||||
timeout=self.timeout,
|
||||
encoding='utf-8',
|
||||
echo=False,
|
||||
)
|
||||
|
||||
# Wait for initial prompt
|
||||
expect(self.child, PROMPT, timeout=self.timeout)
|
||||
print('Ready at pdb prompt!')
|
||||
|
||||
def run(
|
||||
self,
|
||||
cmd: str,
|
||||
print_output: bool = True,
|
||||
) -> str:
|
||||
'''
|
||||
Execute a single pdb command and return output.
|
||||
|
||||
'''
|
||||
if not self.child or not self.child.isalive():
|
||||
raise RuntimeError('Session not started or dead')
|
||||
|
||||
if print_output:
|
||||
print(f'\n>>> {cmd}')
|
||||
|
||||
self.child.sendline(cmd)
|
||||
time.sleep(0.1)
|
||||
|
||||
# Wait for next prompt
|
||||
expect(self.child, PROMPT, timeout=self.timeout)
|
||||
|
||||
output: str = (
|
||||
str(self.child.before.decode())
|
||||
if isinstance(self.child.before, bytes)
|
||||
else str(self.child.before)
|
||||
)
|
||||
self.history.append((cmd, output))
|
||||
|
||||
if print_output:
|
||||
print(output)
|
||||
|
||||
return output
|
||||
|
||||
def quit(self) -> None:
|
||||
'''
|
||||
Exit the debugger and cleanup.
|
||||
|
||||
'''
|
||||
if self.child and self.child.isalive():
|
||||
self.child.sendline('quit')
|
||||
try:
|
||||
self.child.expect(EOF, timeout=5)
|
||||
except (TIMEOUT, EOF):
|
||||
pass
|
||||
self.child.close(force=True)
|
||||
|
||||
def __enter__(self):
|
||||
self.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.quit()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Example inspection commands
|
||||
inspect_cmds: list[str] = [
|
||||
'locals().keys()',
|
||||
'type(deduped)',
|
||||
'deduped.shape',
|
||||
(
|
||||
'step_gaps.shape '
|
||||
'if "step_gaps" in locals() '
|
||||
'else "N/A"'
|
||||
),
|
||||
(
|
||||
'venue_gaps.shape '
|
||||
'if "venue_gaps" in locals() '
|
||||
'else "N/A"'
|
||||
),
|
||||
]
|
||||
|
||||
# Allow commands from CLI args
|
||||
if len(sys.argv) > 1:
|
||||
inspect_cmds = sys.argv[1:]
|
||||
|
||||
# Interactive session example
|
||||
with InteractivePdbSession() as session:
|
||||
for cmd in inspect_cmds:
|
||||
session.run(cmd)
|
||||
|
||||
print('\n=== Session Complete ===')
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue