Compare commits
8 Commits
gitea_feat
...
kucoin_and
| Author | SHA1 | Date |
|---|---|---|
Tyler Goodlet
|
2a9fdec253 | |
|
Tyler Goodlet
|
8f4697bc19 | |
|
Tyler Goodlet
|
32fe5c4942 | |
 Nelson Torres
|
6e9759a7f1 | |
|
Nelson Torres
|
07b13a99b9 | |
|
Tyler Goodlet
|
2377fec665 | |
|
Tyler Goodlet
|
4de11e717c | |
|
Tyler Goodlet
|
525b7df3ee |
234
README.rst
234
README.rst
|
|
@ -1,161 +1,162 @@
|
|||
piker
|
||||
-----
|
||||
trading gear for hackers
|
||||
trading gear for hackers.
|
||||
|
||||
|gh_actions|
|
||||
|
||||
.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
|
||||
:target: https://actions-badge.atrox.dev/piker/pikers/goto
|
||||
|
||||
``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
|
||||
real-time computational trading targeted at `hardcore Linux users
|
||||
<comp_trader>`_ .
|
||||
``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
|
||||
computational trading targeted at `hardcore Linux users <comp_trader>`_ .
|
||||
|
||||
we use much bleeding edge tech including (but not limited to):
|
||||
we use as much bleeding edge tech as possible including (but not limited to):
|
||||
|
||||
- latest python for glue_
|
||||
- uv_ for packaging and distribution
|
||||
- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
|
||||
- Qt_ for pristine low latency UIs
|
||||
- pyqtgraph_ (which we've extended) for real-time charting and graphics
|
||||
- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
|
||||
- `apache arrow and parquet`_ for time-series storage
|
||||
- trio_ & tractor_ for our distributed, multi-core, real-time streaming
|
||||
`structured concurrency`_ runtime B)
|
||||
- Qt_ for pristine high performance UIs
|
||||
- pyqtgraph_ for real-time charting
|
||||
- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
|
||||
- `apache arrow and parquet`_ for time series history management
|
||||
persistence and sharing
|
||||
- (prototyped) techtonicdb_ for L2 book storage
|
||||
|
||||
potential projects we might integrate with soon,
|
||||
|
||||
- (already prototyped in ) techtonicdb_ for L2 book storage
|
||||
|
||||
.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
|
||||
.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
|
||||
.. _uv: https://docs.astral.sh/uv/
|
||||
.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
|
||||
:target: https://travis-ci.org/pikers/piker
|
||||
.. _trio: https://github.com/python-trio/trio
|
||||
.. _tractor: https://github.com/goodboy/tractor
|
||||
.. _structured concurrency: https://trio.discourse.group/
|
||||
.. _marketstore: https://github.com/alpacahq/marketstore
|
||||
.. _techtonicdb: https://github.com/0b01/tectonicdb
|
||||
.. _Qt: https://www.qt.io/
|
||||
.. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
|
||||
.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
|
||||
.. _apache arrow and parquet: https://arrow.apache.org/faq/
|
||||
.. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
|
||||
.. _techtonicdb: https://github.com/0b01/tectonicdb
|
||||
.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
|
||||
|
||||
|
||||
focus and feats:
|
||||
****************
|
||||
fitting with these tenets, we're always open to new
|
||||
framework/lib/service interop suggestions and ideas!
|
||||
focus and features:
|
||||
*******************
|
||||
- 100% federated: your code, your hardware, your data feeds, your broker fills.
|
||||
- zero web: low latency, native software that doesn't try to re-invent the OS
|
||||
- maximal **privacy**: prevent brokers and mms from knowing your
|
||||
planz; smack their spreads with dark volume.
|
||||
- zero clutter: modal, context oriented UIs that echew minimalism, reduce
|
||||
thought noise and encourage un-emotion.
|
||||
- first class parallelism: built from the ground up on next-gen structured concurrency
|
||||
primitives.
|
||||
- traders first: broker/exchange/asset-class agnostic
|
||||
- systems grounded: real-time financial signal processing that will
|
||||
make any queuing or DSP eng juice their shorts.
|
||||
- non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
|
||||
- data collaboration: every process and protocol is multi-host scalable.
|
||||
- fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
|
||||
|
||||
- **100% federated**:
|
||||
your code, your hardware, your data feeds, your broker fills.
|
||||
fitting with these tenets, we're always open to new framework suggestions and ideas.
|
||||
|
||||
- **zero web**:
|
||||
low latency as a prime objective, native UIs and modern IPC
|
||||
protocols without trying to re-invent the "OS-as-an-app"..
|
||||
|
||||
- **maximal privacy**:
|
||||
prevent brokers and mms from knowing your planz; smack their
|
||||
spreads with dark volume from a VPN tunnel.
|
||||
|
||||
- **zero clutter**:
|
||||
modal, context oriented UIs that echew minimalism, reduce thought
|
||||
noise and encourage un-emotion.
|
||||
|
||||
- **first class parallelism**:
|
||||
built from the ground up on a next-gen structured concurrency
|
||||
supervision sys.
|
||||
|
||||
- **traders first**:
|
||||
broker/exchange/venue/asset-class/money-sys agnostic
|
||||
|
||||
- **systems grounded**:
|
||||
real-time financial signal processing (fsp) that will make any
|
||||
queuing or DSP eng juice their shorts.
|
||||
|
||||
- **non-tina UX**:
|
||||
sleek, powerful keyboard driven interaction with expected use in
|
||||
tiling wms (or maybe even a DDE).
|
||||
|
||||
- **data collab at scale**:
|
||||
every actor-process and protocol is multi-host aware.
|
||||
|
||||
- **fight club ready**:
|
||||
zero interest in adoption by suits; no corporate friendly license,
|
||||
ever.
|
||||
|
||||
building the hottest looking, fastest, most reliable, keyboard
|
||||
friendly FOSS trading platform is the dream; join the cause.
|
||||
building the best looking, most reliable, keyboard friendly trading
|
||||
platform is the dream; join the cause.
|
||||
|
||||
|
||||
a sane install with `uv`
|
||||
************************
|
||||
bc why install with `python` when you can faster with `rust` ::
|
||||
sane install with `poetry`
|
||||
**************************
|
||||
TODO!
|
||||
|
||||
uv lock
|
||||
|
||||
rigorous install on ``nixos`` using ``poetry2nix``
|
||||
**************************************************
|
||||
TODO!
|
||||
|
||||
|
||||
hacky install on nixos
|
||||
**********************
|
||||
``NixOS`` is our core devs' distro of choice for which we offer
|
||||
`NixOS` is our core devs' distro of choice for which we offer
|
||||
a stringently defined development shell envoirment that can be loaded with::
|
||||
|
||||
nix-shell default.nix
|
||||
nix-shell develop.nix
|
||||
|
||||
this will setup the required python environment to run piker, make sure to
|
||||
run::
|
||||
|
||||
pip install -r requirements.txt -e .
|
||||
|
||||
once after loading the shell
|
||||
|
||||
|
||||
start a chart
|
||||
*************
|
||||
run a realtime OHLCV chart stand-alone::
|
||||
install wild-west style via `pip`
|
||||
*********************************
|
||||
``piker`` is currently under heavy pre-alpha development and as such
|
||||
should be cloned from this repo and hacked on directly.
|
||||
|
||||
piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
|
||||
for a development install::
|
||||
|
||||
this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
|
||||
overlayed on the same graph. Use of `piker` without first starting
|
||||
a daemon (`pikerd` - see below) means there is an implicit spawning of the
|
||||
multi-actor-runtime (implemented as a `tractor` app).
|
||||
|
||||
For additional subsystem feats available through our chart UI see the
|
||||
various sub-readmes:
|
||||
|
||||
- order control using a mouse-n-keyboard UX B)
|
||||
- cross venue market-pair (what most call "symbol") search, select, overlay Bo
|
||||
- financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
|
||||
- src-asset derivatives scan for anal, like the infamous "max pain" XO
|
||||
git clone git@github.com:pikers/piker.git
|
||||
cd piker
|
||||
virtualenv env
|
||||
source ./env/bin/activate
|
||||
pip install -r requirements.txt -e .
|
||||
|
||||
|
||||
spawn a daemon standalone
|
||||
*************************
|
||||
we call the root actor-process the ``pikerd``. it can be (and is
|
||||
recommended normally to be) started separately from the ``piker
|
||||
chart`` program::
|
||||
check out our charts
|
||||
********************
|
||||
bet you weren't expecting this from the foss::
|
||||
|
||||
piker -l info -b kraken -b binance chart btcusdt.binance --pdb
|
||||
|
||||
|
||||
this runs the main chart (currently with 1m sampled OHLC) in in debug
|
||||
mode and you can practice paper trading using the following
|
||||
micro-manual:
|
||||
|
||||
``order_mode`` (
|
||||
edge triggered activation by any of the following keys,
|
||||
``mouse-click`` on y-level to submit at that price
|
||||
):
|
||||
|
||||
- ``f``/ ``ctl-f`` to stage buy
|
||||
- ``d``/ ``ctl-d`` to stage sell
|
||||
- ``a`` to stage alert
|
||||
|
||||
|
||||
``search_mode`` (
|
||||
``ctl-l`` or ``ctl-space`` to open,
|
||||
``ctl-c`` or ``ctl-space`` to close
|
||||
) :
|
||||
|
||||
- begin typing to have symbol search automatically lookup
|
||||
symbols from all loaded backend (broker) providers
|
||||
- arrow keys and mouse click to navigate selection
|
||||
- vi-like ``ctl-[hjkl]`` for navigation
|
||||
|
||||
|
||||
you can also configure your position allocation limits from the
|
||||
sidepane.
|
||||
|
||||
|
||||
run in distributed mode
|
||||
***********************
|
||||
start the service manager and data feed daemon in the background and
|
||||
connect to it::
|
||||
|
||||
pikerd -l info --pdb
|
||||
|
||||
the daemon does nothing until a ``piker``-client (like ``piker
|
||||
chart``) connects and requests some particular sub-system. for
|
||||
a connecting chart ``pikerd`` will spawn and manage at least,
|
||||
|
||||
- a data-feed daemon: ``datad`` which does all the work of comms with
|
||||
the backend provider (in this case the ``binance`` cex).
|
||||
- a paper-trading engine instance, ``paperboi.binance``, (if no live
|
||||
account has been configured) which allows for auto/manual order
|
||||
control against the live quote stream.
|
||||
connect your chart::
|
||||
|
||||
*using* an actor-service (aka micro-daemon) manager which dynamically
|
||||
supervises various sub-subsystems-as-services throughout the ``piker``
|
||||
runtime-stack.
|
||||
piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
|
||||
|
||||
now you can (implicitly) connect your chart::
|
||||
|
||||
piker chart btcusdt.spot.binance
|
||||
|
||||
since ``pikerd`` was started separately you can now enjoy a persistent
|
||||
real-time data stream tied to the daemon-tree's lifetime. i.e. the next
|
||||
time you spawn a chart it will obviously not only load much faster
|
||||
(since the underlying ``datad.binance`` is left running with its
|
||||
in-memory IPC data structures) but also the data-feed and any order
|
||||
mgmt states should be persistent until you finally cancel ``pikerd``.
|
||||
enjoy persistent real-time data feeds tied to daemon lifetime. the next
|
||||
time you spawn a chart it will load much faster since the data feed has
|
||||
been cached and is now always running live in the background until you
|
||||
kill ``pikerd``.
|
||||
|
||||
|
||||
if anyone asks you what this project is about
|
||||
*********************************************
|
||||
you don't talk about it; just use it.
|
||||
you don't talk about it.
|
||||
|
||||
|
||||
how do i get involved?
|
||||
|
|
@ -165,15 +166,6 @@ enter the matrix.
|
|||
|
||||
how come there ain't that many docs
|
||||
***********************************
|
||||
i mean we want/need them but building the core right has been higher
|
||||
prio then marketting (and likely will stay that way Bp).
|
||||
|
||||
soo, suck it up bc,
|
||||
|
||||
- no one is trying to sell you on anything
|
||||
- learning the code base is prolly way more valuable
|
||||
- the UI/UXs are intended to be "intuitive" for any hacker..
|
||||
|
||||
we obviously need tonz help so if you want to start somewhere and
|
||||
can't necessarily write "advanced" concurrent python/rust code, this
|
||||
helping document literally anything might be the place for you!
|
||||
suck it up, learn the code; no one is trying to sell you on anything.
|
||||
also, we need lotsa help so if you want to start somewhere and can't
|
||||
necessarily write serious code, this might be the place for you!
|
||||
|
|
|
|||
134
default.nix
134
default.nix
|
|
@ -1,134 +0,0 @@
|
|||
with (import <nixpkgs> {});
|
||||
let
|
||||
glibStorePath = lib.getLib glib;
|
||||
zlibStorePath = lib.getLib zlib;
|
||||
zstdStorePath = lib.getLib zstd;
|
||||
dbusStorePath = lib.getLib dbus;
|
||||
libGLStorePath = lib.getLib libGL;
|
||||
freetypeStorePath = lib.getLib freetype;
|
||||
qt6baseStorePath = lib.getLib qt6.qtbase;
|
||||
fontconfigStorePath = lib.getLib fontconfig;
|
||||
libxkbcommonStorePath = lib.getLib libxkbcommon;
|
||||
xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
|
||||
|
||||
qtpyStorePath = lib.getLib python312Packages.qtpy;
|
||||
pyqt6StorePath = lib.getLib python312Packages.pyqt6;
|
||||
pyqt6SipStorePath = lib.getLib python312Packages.pyqt6-sip;
|
||||
rapidfuzzStorePath = lib.getLib python312Packages.rapidfuzz;
|
||||
qdarkstyleStorePath = lib.getLib python312Packages.qdarkstyle;
|
||||
|
||||
xorgLibX11StorePath = lib.getLib xorg.libX11;
|
||||
xorgLibxcbStorePath = lib.getLib xorg.libxcb;
|
||||
xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
|
||||
xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
|
||||
xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
|
||||
xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
|
||||
xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
|
||||
in
|
||||
stdenv.mkDerivation {
|
||||
name = "piker-qt6-uv";
|
||||
buildInputs = [
|
||||
# System requirements.
|
||||
glib
|
||||
zlib
|
||||
dbus
|
||||
zstd
|
||||
libGL
|
||||
freetype
|
||||
qt6.qtbase
|
||||
libgcc.lib
|
||||
fontconfig
|
||||
libxkbcommon
|
||||
|
||||
# Xorg requirements
|
||||
xcb-util-cursor
|
||||
xorg.libxcb
|
||||
xorg.libX11
|
||||
xorg.xcbutilwm
|
||||
xorg.xcbutilimage
|
||||
xorg.xcbutilerrors
|
||||
xorg.xcbutilkeysyms
|
||||
xorg.xcbutilrenderutil
|
||||
|
||||
# Python requirements.
|
||||
python312Full
|
||||
python312Packages.uv
|
||||
python312Packages.qdarkstyle
|
||||
python312Packages.rapidfuzz
|
||||
python312Packages.pyqt6
|
||||
python312Packages.qtpy
|
||||
];
|
||||
src = null;
|
||||
shellHook = ''
|
||||
set -e
|
||||
|
||||
# Set the Qt plugin path
|
||||
# export QT_DEBUG_PLUGINS=1
|
||||
|
||||
QTBASE_PATH="${qt6baseStorePath}/lib"
|
||||
QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
|
||||
QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
|
||||
|
||||
LIB_GCC_PATH="${libgcc.lib}/lib"
|
||||
GLIB_PATH="${glibStorePath}/lib"
|
||||
ZSTD_PATH="${zstdStorePath}/lib"
|
||||
ZLIB_PATH="${zlibStorePath}/lib"
|
||||
DBUS_PATH="${dbusStorePath}/lib"
|
||||
LIBGL_PATH="${libGLStorePath}/lib"
|
||||
FREETYPE_PATH="${freetypeStorePath}/lib"
|
||||
FONTCONFIG_PATH="${fontconfigStorePath}/lib"
|
||||
LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
|
||||
|
||||
XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
|
||||
XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
|
||||
XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
|
||||
XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
|
||||
XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
|
||||
XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
|
||||
XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
|
||||
XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
|
||||
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
|
||||
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
|
||||
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
|
||||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.12/site-packages"
|
||||
QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.12/site-packages"
|
||||
QTPY_PATH="${qtpyStorePath}/lib/python3.12/site-packages"
|
||||
PYQT6_PATH="${pyqt6StorePath}/lib/python3.12/site-packages"
|
||||
PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.12/site-packages"
|
||||
|
||||
PATCH="$PATCH:$RPDFUZZ_PATH"
|
||||
PATCH="$PATCH:$QDRKSTYLE_PATH"
|
||||
PATCH="$PATCH:$QTPY_PATH"
|
||||
PATCH="$PATCH:$PYQT6_PATH"
|
||||
PATCH="$PATCH:$PYQT6_SIP_PATH"
|
||||
|
||||
export PATCH
|
||||
|
||||
# Install deps
|
||||
uv lock
|
||||
|
||||
'';
|
||||
}
|
||||
|
|
@ -30,7 +30,8 @@ from types import ModuleType
|
|||
from typing import (
|
||||
Any,
|
||||
Iterator,
|
||||
Generator
|
||||
Generator,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import pendulum
|
||||
|
|
@ -59,8 +60,10 @@ from ..clearing._messages import (
|
|||
BrokerdPosition,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.data._symcache import SymbologyCache
|
||||
from ..log import get_logger
|
||||
from piker.log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.data._symcache import SymbologyCache
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
|
@ -493,6 +496,17 @@ class Account(Struct):
|
|||
|
||||
_mktmap_table: dict[str, MktPair] | None = None,
|
||||
|
||||
only_require: list[str]|True = True,
|
||||
# ^list of fqmes that are "required" to be processed from
|
||||
# this ledger pass; we often don't care about others and
|
||||
# definitely shouldn't always error in such cases.
|
||||
# (eg. broker backend loaded that doesn't yet supsport the
|
||||
# symcache but also, inside the paper engine we don't ad-hoc
|
||||
# request `get_mkt_info()` for every symbol in the ledger,
|
||||
# only the one for which we're simulating against).
|
||||
# TODO, not sure if there's a better soln for this, ideally
|
||||
# all backends get symcache support afap i guess..
|
||||
|
||||
) -> dict[str, Position]:
|
||||
'''
|
||||
Update the internal `.pps[str, Position]` table from input
|
||||
|
|
@ -535,11 +549,32 @@ class Account(Struct):
|
|||
if _mktmap_table is None:
|
||||
raise
|
||||
|
||||
required: bool = (
|
||||
only_require is True
|
||||
or (
|
||||
only_require is not True
|
||||
and
|
||||
fqme in only_require
|
||||
)
|
||||
)
|
||||
# XXX: caller is allowed to provide a fallback
|
||||
# mktmap table for the case where a new position is
|
||||
# being added and the preloaded symcache didn't
|
||||
# have this entry prior (eg. with frickin IB..)
|
||||
mkt = _mktmap_table[fqme]
|
||||
if (
|
||||
not (mkt := _mktmap_table.get(fqme))
|
||||
and
|
||||
required
|
||||
):
|
||||
raise
|
||||
|
||||
elif not required:
|
||||
continue
|
||||
|
||||
else:
|
||||
# should be an entry retreived somewhere
|
||||
assert mkt
|
||||
|
||||
|
||||
if not (pos := pps.get(bs_mktid)):
|
||||
|
||||
|
|
@ -656,7 +691,7 @@ class Account(Struct):
|
|||
def write_config(self) -> None:
|
||||
'''
|
||||
Write the current account state to the user's account TOML file, normally
|
||||
something like ``pps.toml``.
|
||||
something like `pps.toml`.
|
||||
|
||||
'''
|
||||
# TODO: show diff output?
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@ from trio_typing import TaskStatus
|
|||
from pendulum import (
|
||||
from_timestamp,
|
||||
)
|
||||
from rapidfuzz import process as fuzzy
|
||||
import numpy as np
|
||||
import tractor
|
||||
|
||||
|
|
@ -110,7 +111,6 @@ class AggTrade(Struct, frozen=True):
|
|||
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
|
||||
) -> AsyncGenerator[NoBsWs, dict]:
|
||||
|
||||
# TODO: match syntax here!
|
||||
|
|
@ -221,8 +221,6 @@ def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
|||
}
|
||||
|
||||
|
||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||
# code?!
|
||||
@acm
|
||||
async def open_history_client(
|
||||
mkt: MktPair,
|
||||
|
|
@ -465,8 +463,6 @@ async def stream_quotes(
|
|||
):
|
||||
init_msgs: list[FeedInit] = []
|
||||
for sym in symbols:
|
||||
mkt: MktPair
|
||||
pair: Pair
|
||||
mkt, pair = await get_mkt_info(sym)
|
||||
|
||||
# build out init msgs according to latest spec
|
||||
|
|
@ -515,6 +511,7 @@ async def stream_quotes(
|
|||
|
||||
# start streaming
|
||||
async for typ, quote in msg_gen:
|
||||
|
||||
# period = time.time() - last
|
||||
# hz = 1/period if period else float('inf')
|
||||
# if hz > 60:
|
||||
|
|
@ -550,7 +547,7 @@ async def open_symbol_search(
|
|||
)
|
||||
|
||||
# repack in fqme-keyed table
|
||||
byfqme: dict[str, Pair] = {}
|
||||
byfqme: dict[start, Pair] = {}
|
||||
for pair in pairs.values():
|
||||
byfqme[pair.bs_fqme] = pair
|
||||
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ from piker._cacheables import (
|
|||
)
|
||||
from piker.log import get_logger
|
||||
from piker.data.validate import FeedInit
|
||||
from piker.types import Struct # NOTE, this is already a `tractor.msg.Struct`
|
||||
from piker.types import Struct
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
match_from_pairs,
|
||||
|
|
@ -98,18 +98,9 @@ class KucoinMktPair(Struct, frozen=True):
|
|||
def size_tick(self) -> Decimal:
|
||||
return Decimal(str(self.quoteMinSize))
|
||||
|
||||
callauctionFirstStageStartTime: None|float
|
||||
callauctionIsEnabled: bool
|
||||
callauctionPriceCeiling: float|None
|
||||
callauctionPriceFloor: float|None
|
||||
callauctionSecondStageStartTime: float|None
|
||||
callauctionThirdStageStartTime: float|None
|
||||
|
||||
enableTrading: bool
|
||||
feeCategory: int
|
||||
feeCurrency: str
|
||||
isMarginEnabled: bool
|
||||
makerFeeCoefficient: float
|
||||
market: str
|
||||
minFunds: float
|
||||
name: str
|
||||
|
|
@ -119,10 +110,11 @@ class KucoinMktPair(Struct, frozen=True):
|
|||
quoteIncrement: float
|
||||
quoteMaxSize: float
|
||||
quoteMinSize: float
|
||||
st: bool
|
||||
symbol: str # our bs_mktid, kucoin's internal id
|
||||
feeCategory: int
|
||||
makerFeeCoefficient: float
|
||||
takerFeeCoefficient: float
|
||||
tradingStartTime: float|None
|
||||
st: bool
|
||||
|
||||
|
||||
class AccountTrade(Struct, frozen=True):
|
||||
|
|
@ -404,13 +396,7 @@ class Client:
|
|||
pairs: dict[str, KucoinMktPair] = {}
|
||||
fqmes2mktids: bidict[str, str] = bidict()
|
||||
for item in entries:
|
||||
try:
|
||||
pair = pairs[item['name']] = KucoinMktPair(**item)
|
||||
except TypeError as te:
|
||||
raise TypeError(
|
||||
'`KucoinMktPair` and reponse fields do not match ??\n'
|
||||
f'{KucoinMktPair.fields_diff(item)}\n'
|
||||
) from te
|
||||
pair = pairs[item['name']] = KucoinMktPair(**item)
|
||||
fqmes2mktids[
|
||||
item['name'].lower().replace('-', '')
|
||||
] = pair.name
|
||||
|
|
|
|||
|
|
@ -1,49 +0,0 @@
|
|||
piker.clearing
|
||||
______________
|
||||
trade execution-n-control subsys for both live and paper trading as
|
||||
well as algo-trading manual override/interaction across any backend
|
||||
broker and data provider.
|
||||
|
||||
avail UIs
|
||||
*********
|
||||
|
||||
order ctl
|
||||
---------
|
||||
the `piker.clearing` subsys is exposed mainly though
|
||||
the `piker chart` GUI as a "chart trader" style UX and
|
||||
is automatically enabled whenever a chart is opened.
|
||||
|
||||
.. ^TODO, more prose here!
|
||||
|
||||
the "manual" order control features are exposed via the
|
||||
`piker.ui.order_mode` API and can pretty much always be
|
||||
used (at least) in simulated-trading mode, aka "paper"-mode, and
|
||||
the micro-manual is as follows:
|
||||
|
||||
``order_mode`` (
|
||||
edge triggered activation by any of the following keys,
|
||||
``mouse-click`` on y-level to submit at that price
|
||||
):
|
||||
|
||||
- ``f``/ ``ctl-f`` to stage buy
|
||||
- ``d``/ ``ctl-d`` to stage sell
|
||||
- ``a`` to stage alert
|
||||
|
||||
|
||||
``search_mode`` (
|
||||
``ctl-l`` or ``ctl-space`` to open,
|
||||
``ctl-c`` or ``ctl-space`` to close
|
||||
) :
|
||||
|
||||
- begin typing to have symbol search automatically lookup
|
||||
symbols from all loaded backend (broker) providers
|
||||
- arrow keys and mouse click to navigate selection
|
||||
- vi-like ``ctl-[hjkl]`` for navigation
|
||||
|
||||
|
||||
position (pp) mgmt
|
||||
------------------
|
||||
you can also configure your position allocation limits from the
|
||||
sidepane.
|
||||
|
||||
.. ^TODO, explain and provide tut once more refined!
|
||||
|
|
@ -653,7 +653,11 @@ class Router(Struct):
|
|||
flume = feed.flumes[fqme]
|
||||
first_quote: dict = flume.first_quote
|
||||
book: DarkBook = self.get_dark_book(broker)
|
||||
book.lasts[fqme]: float = float(first_quote['last'])
|
||||
|
||||
if not (last := first_quote.get('last')):
|
||||
last: float = flume.rt_shm.array[-1]['close']
|
||||
|
||||
book.lasts[fqme]: float = float(last)
|
||||
|
||||
async with self.maybe_open_brokerd_dialog(
|
||||
brokermod=brokermod,
|
||||
|
|
@ -716,7 +720,7 @@ class Router(Struct):
|
|||
subs = self.subscribers[sub_key]
|
||||
|
||||
sent_some: bool = False
|
||||
for client_stream in subs:
|
||||
for client_stream in subs.copy():
|
||||
try:
|
||||
await client_stream.send(msg)
|
||||
sent_some = True
|
||||
|
|
@ -1010,10 +1014,14 @@ async def translate_and_relay_brokerd_events(
|
|||
status_msg.brokerd_msg = msg
|
||||
status_msg.src = msg.broker_details['name']
|
||||
|
||||
await router.client_broadcast(
|
||||
status_msg.req.symbol,
|
||||
status_msg,
|
||||
)
|
||||
if not status_msg.req:
|
||||
# likely some order change state?
|
||||
await tractor.pause()
|
||||
else:
|
||||
await router.client_broadcast(
|
||||
status_msg.req.symbol,
|
||||
status_msg,
|
||||
)
|
||||
|
||||
if status == 'closed':
|
||||
log.info(f'Execution for {oid} is complete!')
|
||||
|
|
|
|||
|
|
@ -653,6 +653,7 @@ async def open_trade_dialog(
|
|||
# in) use manually constructed table from calling
|
||||
# the `.get_mkt_info()` provider EP above.
|
||||
_mktmap_table=mkt_by_fqme,
|
||||
only_require=list(mkt_by_fqme),
|
||||
)
|
||||
|
||||
pp_msgs: list[BrokerdPosition] = []
|
||||
|
|
|
|||
|
|
@ -104,15 +104,14 @@ def get_app_dir(
|
|||
# `tractor`) with the testing dir and check for it whenever we
|
||||
# detect `pytest` is being used (which it isn't under normal
|
||||
# operation).
|
||||
# if "pytest" in sys.modules:
|
||||
# import tractor
|
||||
# actor = tractor.current_actor(err_on_no_runtime=False)
|
||||
# if actor: # runtime is up
|
||||
# rvs = tractor._state._runtime_vars
|
||||
# import pdbp; pdbp.set_trace()
|
||||
# testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
|
||||
# assert testdirpath.exists(), 'piker test harness might be borked!?'
|
||||
# app_name = str(testdirpath)
|
||||
if "pytest" in sys.modules:
|
||||
import tractor
|
||||
actor = tractor.current_actor(err_on_no_runtime=False)
|
||||
if actor: # runtime is up
|
||||
rvs = tractor._state._runtime_vars
|
||||
testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
|
||||
assert testdirpath.exists(), 'piker test harness might be borked!?'
|
||||
app_name = str(testdirpath)
|
||||
|
||||
if platform.system() == 'Windows':
|
||||
key = "APPDATA" if roaming else "LOCALAPPDATA"
|
||||
|
|
|
|||
|
|
@ -360,7 +360,7 @@ async def open_autorecon_ws(
|
|||
|
||||
'''
|
||||
JSONRPC response-request style machinery for transparent multiplexing
|
||||
of msgs over a `NoBsWs`.
|
||||
of msgs over a NoBsWs.
|
||||
|
||||
'''
|
||||
|
||||
|
|
@ -377,44 +377,23 @@ async def open_jsonrpc_session(
|
|||
url: str,
|
||||
start_id: int = 0,
|
||||
response_type: type = JSONRPCResult,
|
||||
msg_recv_timeout: float = float('inf'),
|
||||
# ^NOTE, since only `deribit` is using this jsonrpc stuff atm
|
||||
# and options mkts are generally "slow moving"..
|
||||
#
|
||||
# FURTHER if we break the underlying ws connection then since we
|
||||
# don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
|
||||
# `_reconnect_forever()`, the jsonrpc "transport pipe" get's
|
||||
# broken and never restored with wtv init sequence is required to
|
||||
# re-establish a working req-resp session.
|
||||
|
||||
# request_type: Optional[type] = None,
|
||||
# request_hook: Optional[Callable] = None,
|
||||
# error_hook: Optional[Callable] = None,
|
||||
) -> Callable[[str, dict], dict]:
|
||||
'''
|
||||
Init a json-RPC-over-websocket connection to the provided `url`.
|
||||
|
||||
A `json_rpc: Callable[[str, dict], dict` is delivered to the
|
||||
caller for sending requests and a bg-`trio.Task` handles
|
||||
processing of response msgs including error reporting/raising in
|
||||
the parent/caller task.
|
||||
|
||||
'''
|
||||
# NOTE, store all request msgs so we can raise errors on the
|
||||
# caller side!
|
||||
req_msgs: dict[int, dict] = {}
|
||||
|
||||
async with (
|
||||
trio.open_nursery() as tn,
|
||||
open_autorecon_ws(
|
||||
url=url,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
) as ws
|
||||
trio.open_nursery() as n,
|
||||
open_autorecon_ws(url) as ws
|
||||
):
|
||||
rpc_id: Iterable[int] = count(start_id)
|
||||
rpc_results: dict[int, dict] = {}
|
||||
|
||||
async def json_rpc(
|
||||
method: str,
|
||||
params: dict,
|
||||
) -> dict:
|
||||
async def json_rpc(method: str, params: dict) -> dict:
|
||||
'''
|
||||
perform a json rpc call and wait for the result, raise exception in
|
||||
case of error field present on response
|
||||
|
|
@ -491,10 +470,15 @@ async def open_jsonrpc_session(
|
|||
'params': _,
|
||||
}:
|
||||
log.debug(f'Recieved\n{msg}')
|
||||
# if request_hook:
|
||||
# await request_hook(request_type(**msg))
|
||||
|
||||
case {
|
||||
'error': error
|
||||
}:
|
||||
# if error_hook:
|
||||
# await error_hook(response_type(**msg))
|
||||
|
||||
# retreive orig request msg, set error
|
||||
# response in original "result" msg,
|
||||
# THEN FINALLY set the event to signal caller
|
||||
|
|
@ -513,6 +497,6 @@ async def open_jsonrpc_session(
|
|||
case _:
|
||||
log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
|
||||
|
||||
tn.start_soon(recv_task)
|
||||
n.start_soon(recv_task)
|
||||
yield json_rpc
|
||||
tn.cancel_scope.cancel()
|
||||
n.cancel_scope.cancel()
|
||||
|
|
|
|||
|
|
@ -386,8 +386,6 @@ def ldshm(
|
|||
open_annot_ctl() as actl,
|
||||
):
|
||||
shm_df: pl.DataFrame | None = None
|
||||
tf2aids: dict[float, dict] = {}
|
||||
|
||||
for (
|
||||
shmfile,
|
||||
shm,
|
||||
|
|
@ -528,17 +526,16 @@ def ldshm(
|
|||
new_df,
|
||||
step_gaps,
|
||||
)
|
||||
|
||||
# last chance manual overwrites in REPL
|
||||
# await tractor.pause()
|
||||
await tractor.pause()
|
||||
assert aids
|
||||
tf2aids[period_s] = aids
|
||||
|
||||
else:
|
||||
# allow interaction even when no ts problems.
|
||||
assert not diff
|
||||
await tractor.pause()
|
||||
# assert not diff
|
||||
|
||||
await tractor.pause()
|
||||
log.info('Exiting TSP shm anal-izer!')
|
||||
|
||||
if shm_df is None:
|
||||
log.error(
|
||||
|
|
|
|||
|
|
@ -161,13 +161,7 @@ class NativeStorageClient:
|
|||
|
||||
def index_files(self):
|
||||
for path in self._datadir.iterdir():
|
||||
if (
|
||||
path.is_dir()
|
||||
or
|
||||
'.parquet' not in str(path)
|
||||
# or
|
||||
# path.name in {'borked', 'expired',}
|
||||
):
|
||||
if path.name in {'borked', 'expired',}:
|
||||
continue
|
||||
|
||||
key: str = path.name.rstrip('.parquet')
|
||||
|
|
|
|||
|
|
@ -44,10 +44,8 @@ import trio
|
|||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from pendulum import (
|
||||
Interval,
|
||||
DateTime,
|
||||
Duration,
|
||||
duration as mk_duration,
|
||||
from_timestamp,
|
||||
)
|
||||
import numpy as np
|
||||
|
|
@ -216,8 +214,7 @@ async def maybe_fill_null_segments(
|
|||
# pair, immediately stop backfilling?
|
||||
if (
|
||||
start_dt
|
||||
and
|
||||
end_dt < start_dt
|
||||
and end_dt < start_dt
|
||||
):
|
||||
await tractor.pause()
|
||||
break
|
||||
|
|
@ -265,7 +262,6 @@ async def maybe_fill_null_segments(
|
|||
except tractor.ContextCancelled:
|
||||
# log.exception
|
||||
await tractor.pause()
|
||||
raise
|
||||
|
||||
null_segs_detected.set()
|
||||
# RECHECK for more null-gaps
|
||||
|
|
@ -353,7 +349,7 @@ async def maybe_fill_null_segments(
|
|||
|
||||
async def start_backfill(
|
||||
get_hist,
|
||||
def_frame_duration: Duration,
|
||||
frame_types: dict[str, Duration] | None,
|
||||
mod: ModuleType,
|
||||
mkt: MktPair,
|
||||
shm: ShmArray,
|
||||
|
|
@ -383,23 +379,22 @@ async def start_backfill(
|
|||
update_start_on_prepend: bool = False
|
||||
if backfill_until_dt is None:
|
||||
|
||||
# TODO: per-provider default history-durations?
|
||||
# -[ ] inside the `open_history_client()` config allow
|
||||
# declaring the history duration limits instead of
|
||||
# guessing and/or applying the same limits to all?
|
||||
#
|
||||
# -[ ] allow declaring (default) per-provider backfill
|
||||
# limits inside a [storage] sub-section in conf.toml?
|
||||
#
|
||||
# NOTE, when no tsdb "last datum" is provided, we just
|
||||
# load some near-term history by presuming a "decently
|
||||
# large" 60s duration limit and a much shorter 1s range.
|
||||
# TODO: drop this right and just expose the backfill
|
||||
# limits inside a [storage] section in conf.toml?
|
||||
# when no tsdb "last datum" is provided, we just load
|
||||
# some near-term history.
|
||||
# periods = {
|
||||
# 1: {'days': 1},
|
||||
# 60: {'days': 14},
|
||||
# }
|
||||
|
||||
# do a decently sized backfill and load it into storage.
|
||||
periods = {
|
||||
1: {'days': 2},
|
||||
60: {'years': 6},
|
||||
}
|
||||
period_duration: int = periods[timeframe]
|
||||
update_start_on_prepend: bool = True
|
||||
update_start_on_prepend = True
|
||||
|
||||
# NOTE: manually set the "latest" datetime which we intend to
|
||||
# backfill history "until" so as to adhere to the history
|
||||
|
|
@ -421,6 +416,7 @@ async def start_backfill(
|
|||
f'backfill_until_dt: {backfill_until_dt}\n'
|
||||
f'last_start_dt: {last_start_dt}\n'
|
||||
)
|
||||
|
||||
try:
|
||||
(
|
||||
array,
|
||||
|
|
@ -430,58 +426,37 @@ async def start_backfill(
|
|||
timeframe,
|
||||
end_dt=last_start_dt,
|
||||
)
|
||||
|
||||
except NoData as _daterr:
|
||||
orig_last_start_dt: datetime = last_start_dt
|
||||
gap_report: str = (
|
||||
f'EMPTY FRAME for `end_dt: {last_start_dt}`?\n'
|
||||
f'{mod.name} -> tf@fqme: {timeframe}@{mkt.fqme}\n'
|
||||
f'last_start_dt: {orig_last_start_dt}\n\n'
|
||||
f'bf_until: {backfill_until_dt}\n'
|
||||
)
|
||||
# EMPTY FRAME signal with 3 (likely) causes:
|
||||
#
|
||||
# 1. range contains legit gap in venue history
|
||||
# 2. history actually (edge case) **began** at the
|
||||
# value `last_start_dt`
|
||||
# 3. some other unknown error (ib blocking the
|
||||
# history-query bc they don't want you seeing how
|
||||
# they cucked all the tinas.. like with options
|
||||
# hist)
|
||||
#
|
||||
if def_frame_duration:
|
||||
# decrement by a duration's (frame) worth of time
|
||||
# as maybe indicated by the backend to see if we
|
||||
# can get older data before this possible
|
||||
# "history gap".
|
||||
last_start_dt: datetime = last_start_dt.subtract(
|
||||
seconds=def_frame_duration.total_seconds()
|
||||
# 3 cases:
|
||||
# - frame in the middle of a legit venue gap
|
||||
# - history actually began at the `last_start_dt`
|
||||
# - some other unknown error (ib blocking the
|
||||
# history bc they don't want you seeing how they
|
||||
# cucked all the tinas..)
|
||||
if dur := frame_types.get(timeframe):
|
||||
# decrement by a frame's worth of duration and
|
||||
# retry a few times.
|
||||
last_start_dt.subtract(
|
||||
seconds=dur.total_seconds()
|
||||
)
|
||||
gap_report += (
|
||||
f'Decrementing `end_dt` and retrying with,\n'
|
||||
f'def_frame_duration: {def_frame_duration}\n'
|
||||
f'(new) last_start_dt: {last_start_dt}\n'
|
||||
log.warning(
|
||||
f'{mod.name} -> EMPTY FRAME for end_dt?\n'
|
||||
f'tf@fqme: {timeframe}@{mkt.fqme}\n'
|
||||
'bf_until <- last_start_dt:\n'
|
||||
f'{backfill_until_dt} <- {last_start_dt}\n'
|
||||
f'Decrementing `end_dt` by {dur} and retry..\n'
|
||||
)
|
||||
log.warning(gap_report)
|
||||
# skip writing to shm/tsdb and try the next
|
||||
# duration's worth of prior history.
|
||||
continue
|
||||
|
||||
else:
|
||||
# await tractor.pause()
|
||||
raise DataUnavailable(gap_report)
|
||||
|
||||
# broker says there never was or is no more history to pull
|
||||
except DataUnavailable as due:
|
||||
message: str = due.args[0]
|
||||
except DataUnavailable:
|
||||
log.warning(
|
||||
f'Provider {mod.name!r} halted backfill due to,\n\n'
|
||||
|
||||
f'{message}\n'
|
||||
|
||||
f'fqme: {mkt.fqme}\n'
|
||||
f'timeframe: {timeframe}\n'
|
||||
f'last_start_dt: {last_start_dt}\n'
|
||||
f'bf_until: {backfill_until_dt}\n'
|
||||
f'NO-MORE-DATA in range?\n'
|
||||
f'`{mod.name}` halted history:\n'
|
||||
f'tf@fqme: {timeframe}@{mkt.fqme}\n'
|
||||
'bf_until <- last_start_dt:\n'
|
||||
f'{backfill_until_dt} <- {last_start_dt}\n'
|
||||
)
|
||||
# UGH: what's a better way?
|
||||
# TODO: backends are responsible for being correct on
|
||||
|
|
@ -490,54 +465,34 @@ async def start_backfill(
|
|||
# to halt the request loop until the condition is
|
||||
# resolved or should the backend be entirely in
|
||||
# charge of solving such faults? yes, right?
|
||||
# if timeframe > 1:
|
||||
# await tractor.pause()
|
||||
return
|
||||
|
||||
time: np.ndarray = array['time']
|
||||
assert (
|
||||
time[0]
|
||||
array['time'][0]
|
||||
==
|
||||
next_start_dt.timestamp()
|
||||
)
|
||||
|
||||
assert time[-1] == next_end_dt.timestamp()
|
||||
|
||||
expected_dur: Interval = last_start_dt - next_start_dt
|
||||
diff = last_start_dt - next_start_dt
|
||||
frame_time_diff_s = diff.seconds
|
||||
|
||||
# frame's worth of sample-period-steps, in seconds
|
||||
frame_size_s: float = len(array) * timeframe
|
||||
recv_frame_dur: Duration = (
|
||||
from_timestamp(array[-1]['time'])
|
||||
-
|
||||
from_timestamp(array[0]['time'])
|
||||
)
|
||||
if (
|
||||
(lt_frame := (recv_frame_dur < expected_dur))
|
||||
or
|
||||
(null_frame := (frame_size_s == 0))
|
||||
# ^XXX, should NEVER hit now!
|
||||
):
|
||||
expected_frame_size_s: float = frame_size_s + timeframe
|
||||
if frame_time_diff_s > expected_frame_size_s:
|
||||
|
||||
# XXX: query result includes a start point prior to our
|
||||
# expected "frame size" and thus is likely some kind of
|
||||
# history gap (eg. market closed period, outage, etc.)
|
||||
# so just report it to console for now.
|
||||
if lt_frame:
|
||||
reason = 'Possible GAP (or first-datum)'
|
||||
else:
|
||||
assert null_frame
|
||||
reason = 'NULL-FRAME'
|
||||
|
||||
missing_dur: Interval = expected_dur.end - recv_frame_dur.end
|
||||
log.warning(
|
||||
f'{timeframe}s-series {reason} detected!\n'
|
||||
f'fqme: {mkt.fqme}\n'
|
||||
f'last_start_dt: {last_start_dt}\n\n'
|
||||
f'recv interval: {recv_frame_dur}\n'
|
||||
f'expected interval: {expected_dur}\n\n'
|
||||
|
||||
f'Missing duration of history of {missing_dur.in_words()!r}\n'
|
||||
f'{missing_dur}\n'
|
||||
'GAP DETECTED:\n'
|
||||
f'last_start_dt: {last_start_dt}\n'
|
||||
f'diff: {diff}\n'
|
||||
f'frame_time_diff_s: {frame_time_diff_s}\n'
|
||||
)
|
||||
# await tractor.pause()
|
||||
|
||||
to_push = diff_history(
|
||||
array,
|
||||
|
|
@ -612,27 +567,23 @@ async def start_backfill(
|
|||
# long-term storage.
|
||||
if (
|
||||
storage is not None
|
||||
and
|
||||
write_tsdb
|
||||
and write_tsdb
|
||||
):
|
||||
log.info(
|
||||
f'Writing {ln} frame to storage:\n'
|
||||
f'{next_start_dt} -> {last_start_dt}'
|
||||
)
|
||||
|
||||
# NOTE, always drop the src asset token for
|
||||
# always drop the src asset token for
|
||||
# non-currency-pair like market types (for now)
|
||||
#
|
||||
# THAT IS, for now our table key schema is NOT
|
||||
# including the dst[/src] source asset token. SO,
|
||||
# 'tsla.nasdaq.ib' over 'tsla/usd.nasdaq.ib' for
|
||||
# historical reasons ONLY.
|
||||
if mkt.dst.atype not in {
|
||||
'crypto',
|
||||
'crypto_currency',
|
||||
'fiat', # a "forex pair"
|
||||
'perpetual_future', # stupid "perps" from cex land
|
||||
}:
|
||||
# for now, our table key schema is not including
|
||||
# the dst[/src] source asset token.
|
||||
col_sym_key: str = mkt.get_fqme(
|
||||
delim_char='',
|
||||
without_src=True,
|
||||
|
|
@ -737,7 +688,7 @@ async def back_load_from_tsdb(
|
|||
last_tsdb_dt
|
||||
and latest_start_dt
|
||||
):
|
||||
backfilled_size_s: Duration = (
|
||||
backfilled_size_s = (
|
||||
latest_start_dt - last_tsdb_dt
|
||||
).seconds
|
||||
# if the shm buffer len is not large enough to contain
|
||||
|
|
@ -960,8 +911,6 @@ async def tsdb_backfill(
|
|||
f'{pformat(config)}\n'
|
||||
)
|
||||
|
||||
# concurrently load the provider's most-recent-frame AND any
|
||||
# pre-existing tsdb history already saved in `piker` storage.
|
||||
dt_eps: list[DateTime, DateTime] = []
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(
|
||||
|
|
@ -972,6 +921,7 @@ async def tsdb_backfill(
|
|||
timeframe,
|
||||
config,
|
||||
)
|
||||
|
||||
tsdb_entry: tuple = await load_tsdb_hist(
|
||||
storage,
|
||||
mkt,
|
||||
|
|
@ -1000,25 +950,6 @@ async def tsdb_backfill(
|
|||
mr_end_dt,
|
||||
) = dt_eps
|
||||
|
||||
first_frame_dur_s: Duration = (mr_end_dt - mr_start_dt).seconds
|
||||
calced_frame_size: Duration = mk_duration(
|
||||
seconds=first_frame_dur_s,
|
||||
)
|
||||
# NOTE, attempt to use the backend declared default frame
|
||||
# sizing (as allowed by their time-series query APIs) and
|
||||
# if not provided try to construct a default from the
|
||||
# first frame received above.
|
||||
def_frame_durs: dict[
|
||||
int,
|
||||
Duration,
|
||||
]|None = config.get('frame_types', None)
|
||||
if def_frame_durs:
|
||||
def_frame_size: Duration = def_frame_durs[timeframe]
|
||||
assert def_frame_size == calced_frame_size
|
||||
else:
|
||||
# use what we calced from first frame above.
|
||||
def_frame_size = calced_frame_size
|
||||
|
||||
# NOTE: when there's no offline data, there's 2 cases:
|
||||
# - data backend doesn't support timeframe/sample
|
||||
# period (in which case `dt_eps` should be `None` and
|
||||
|
|
@ -1049,7 +980,7 @@ async def tsdb_backfill(
|
|||
partial(
|
||||
start_backfill,
|
||||
get_hist=get_hist,
|
||||
def_frame_duration=def_frame_size,
|
||||
frame_types=config.get('frame_types', None),
|
||||
mod=mod,
|
||||
mkt=mkt,
|
||||
shm=shm,
|
||||
|
|
|
|||
|
|
@ -616,18 +616,6 @@ def detect_price_gaps(
|
|||
# ])
|
||||
...
|
||||
|
||||
# TODO: probably just use the null_segs impl above?
|
||||
def detect_vlm_gaps(
|
||||
df: pl.DataFrame,
|
||||
col: str = 'volume',
|
||||
|
||||
) -> pl.DataFrame:
|
||||
|
||||
vnull: pl.DataFrame = w_dts.filter(
|
||||
pl.col(col) == 0
|
||||
)
|
||||
return vnull
|
||||
|
||||
|
||||
def dedupe(
|
||||
src_df: pl.DataFrame,
|
||||
|
|
@ -638,6 +626,7 @@ def dedupe(
|
|||
|
||||
) -> tuple[
|
||||
pl.DataFrame, # with dts
|
||||
pl.DataFrame, # gaps
|
||||
pl.DataFrame, # with deduplicated dts (aka gap/repeat removal)
|
||||
int, # len diff between input and deduped
|
||||
]:
|
||||
|
|
@ -650,22 +639,19 @@ def dedupe(
|
|||
'''
|
||||
wdts: pl.DataFrame = with_dts(src_df)
|
||||
|
||||
deduped = wdts
|
||||
|
||||
# remove duplicated datetime samples/sections
|
||||
deduped: pl.DataFrame = wdts.unique(
|
||||
# subset=['dt'],
|
||||
subset=['time'],
|
||||
maintain_order=True,
|
||||
)
|
||||
|
||||
# maybe sort on any time field
|
||||
if sort:
|
||||
deduped = deduped.sort(by='time')
|
||||
wdts = wdts.sort(by='time')
|
||||
# TODO: detect out-of-order segments which were corrected!
|
||||
# -[ ] report in log msg
|
||||
# -[ ] possibly return segment sections which were moved?
|
||||
|
||||
# remove duplicated datetime samples/sections
|
||||
deduped: pl.DataFrame = wdts.unique(
|
||||
subset=['dt'],
|
||||
maintain_order=True,
|
||||
)
|
||||
|
||||
diff: int = (
|
||||
wdts.height
|
||||
-
|
||||
|
|
|
|||
229
pyproject.toml
229
pyproject.toml
|
|
@ -15,119 +15,140 @@
|
|||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
requires = ["poetry-core"]
|
||||
build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[project]
|
||||
# ------ - ------
|
||||
|
||||
[tool.ruff.lint]
|
||||
# https://docs.astral.sh/ruff/settings/#lint_ignore
|
||||
ignore = []
|
||||
|
||||
# https://docs.astral.sh/ruff/settings/#lint_per-file-ignores
|
||||
"piker/ui/qt.py" = [
|
||||
"E402",
|
||||
'F401', # unused imports (without __all__ or blah as blah)
|
||||
# "F841", # unused variable rules
|
||||
]
|
||||
# ignore-init-module-imports = false
|
||||
|
||||
# ------ - ------
|
||||
|
||||
[tool.poetry]
|
||||
name = "piker"
|
||||
version = "0.1.0a0dev0"
|
||||
version = "0.1.0.alpha0.dev0"
|
||||
description = "trading gear for hackers"
|
||||
authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
|
||||
requires-python = ">=3.12, <3.13"
|
||||
license = "AGPL-3.0-or-later"
|
||||
authors = ["Tyler Goodlet <goodboy_foss@protonmail.com>"]
|
||||
license = "AGPLv3"
|
||||
readme = "README.rst"
|
||||
keywords = [
|
||||
"async",
|
||||
"trading",
|
||||
"finance",
|
||||
"quant",
|
||||
"charting",
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 3 - Alpha",
|
||||
"License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
|
||||
"Operating System :: POSIX :: Linux",
|
||||
"Programming Language :: Python :: Implementation :: CPython",
|
||||
"Programming Language :: Python :: 3 :: Only",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Intended Audience :: Financial and Insurance Industry",
|
||||
"Intended Audience :: Science/Research",
|
||||
"Intended Audience :: Developers",
|
||||
"Intended Audience :: Education",
|
||||
]
|
||||
dependencies = [
|
||||
"async-generator >=1.10, <2.0.0",
|
||||
"attrs >=23.1.0, <24.0.0",
|
||||
"bidict >=0.22.1, <0.23.0",
|
||||
"colorama >=0.4.6, <0.5.0",
|
||||
"colorlog >=6.7.0, <7.0.0",
|
||||
"ib-insync >=0.9.86, <0.10.0",
|
||||
"numba >=0.59.0, <0.60.0",
|
||||
"numpy >=1.25, <2.0",
|
||||
"polars >=0.18.13, <0.19.0",
|
||||
"pygments >=2.16.1, <3.0.0",
|
||||
"rich >=13.5.2, <14.0.0",
|
||||
"tomli >=2.0.1, <3.0.0",
|
||||
"tomli-w >=1.0.0, <2.0.0",
|
||||
"trio-util >=0.7.0, <0.8.0",
|
||||
"trio-websocket >=0.10.3, <0.11.0",
|
||||
"typer >=0.9.0, <1.0.0",
|
||||
"rapidfuzz >=3.5.2, <4.0.0",
|
||||
"pdbp >=1.5.0, <2.0.0",
|
||||
"trio >=0.24, <0.25",
|
||||
"pendulum >=3.0.0, <4.0.0",
|
||||
"httpx >=0.27.0, <0.28.0",
|
||||
"cryptofeed >=2.4.0, <3.0.0",
|
||||
"pyarrow >=17.0.0, <18.0.0",
|
||||
"websockets ==12.0",
|
||||
"msgspec",
|
||||
"tractor",
|
||||
"asyncvnc",
|
||||
"tomlkit",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
uis = [
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#optional-dependencies
|
||||
# TODO: make sure the levenshtein shit compiles on nix..
|
||||
# rapidfuzz = {extras = ["speedup"], version = "^0.18.0"}
|
||||
"rapidfuzz >=3.2.0, <4.0.0",
|
||||
"qdarkstyle >=3.0.2, <4.0.0",
|
||||
"pyqt6 >=6.7.0, <7.0.0",
|
||||
"pyqtgraph",
|
||||
# ------ - ------
|
||||
|
||||
# for consideration,
|
||||
# - 'visidata'
|
||||
[tool.poetry.dependencies]
|
||||
async-generator = "^1.10"
|
||||
attrs = "^23.1.0"
|
||||
bidict = "^0.22.1"
|
||||
colorama = "^0.4.6"
|
||||
colorlog = "^6.7.0"
|
||||
cython = "^3.0.0"
|
||||
greenback = "^1.1.1"
|
||||
ib-insync = "^0.9.86"
|
||||
msgspec = "^0.18.0"
|
||||
numba = "^0.59.0"
|
||||
numpy = "^1.25"
|
||||
polars = "^0.18.13"
|
||||
pygments = "^2.16.1"
|
||||
python = ">=3.11, <3.13"
|
||||
rich = "^13.5.2"
|
||||
# setuptools = "^68.0.0"
|
||||
tomli = "^2.0.1"
|
||||
tomli-w = "^1.0.0"
|
||||
trio-util = "^0.7.0"
|
||||
trio-websocket = "^0.10.3"
|
||||
typer = "^0.9.0"
|
||||
rapidfuzz = "^3.5.2"
|
||||
pdbp = "^1.5.0"
|
||||
trio = "^0.24"
|
||||
pendulum = "^3.0.0"
|
||||
httpx = "^0.27.0"
|
||||
|
||||
# TODO: add an `--only daemon` group for running non-ui / pikerd
|
||||
# service tree in distributed mode B)
|
||||
# https://docs.astral.sh/uv/concepts/projects/dependencies/#optional-dependencies
|
||||
]
|
||||
[tool.poetry.dependencies.tractor]
|
||||
develop = true
|
||||
git = 'https://github.com/goodboy/tractor.git'
|
||||
branch = 'asyncio_debugger_support'
|
||||
# path = "../tractor"
|
||||
|
||||
[dependency-groups]
|
||||
# TODO: a toolset that makes debugging a `pikerd` service (tree) easy
|
||||
# to hack on directly using more or less the local env:
|
||||
[tool.poetry.dependencies.asyncvnc]
|
||||
git = 'https://github.com/pikers/asyncvnc.git'
|
||||
branch = 'main'
|
||||
|
||||
[tool.poetry.dependencies.tomlkit]
|
||||
develop = true
|
||||
git = 'https://github.com/pikers/tomlkit.git'
|
||||
branch = 'piker_pin'
|
||||
# path = "../tomlkit/"
|
||||
|
||||
[tool.poetry.group.uis]
|
||||
optional = true
|
||||
[tool.poetry.group.uis.dependencies]
|
||||
# https://python-poetry.org/docs/managing-dependencies/#dependency-groups
|
||||
# TODO: make sure the levenshtein shit compiles on nix..
|
||||
# rapidfuzz = {extras = ["speedup"], version = "^0.18.0"}
|
||||
rapidfuzz = "^3.2.0"
|
||||
qdarkstyle = ">=3.0.2"
|
||||
pyqtgraph = { git = 'https://github.com/pikers/pyqtgraph.git' }
|
||||
|
||||
# ------ - ------
|
||||
pyqt6 = "^6.7.0"
|
||||
|
||||
[tool.poetry.group.dev]
|
||||
optional = true
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
# testing / CI
|
||||
pytest = "^6.0.0"
|
||||
elasticsearch = "^8.9.0"
|
||||
xonsh = "^0.14.2"
|
||||
prompt-toolkit = "3.0.40"
|
||||
|
||||
# console ehancements and eventually remote debugging
|
||||
# extras/helpers.
|
||||
# TODO: add a toolset that makes debugging a `pikerd` service
|
||||
# (tree) easy to hack on directly using more or less the local env:
|
||||
# - xonsh + xxh
|
||||
# - rsyscall + pdbp
|
||||
# - actor runtime control console like BEAM/OTP
|
||||
#
|
||||
# console ehancements and eventually remote debugging extras/helpers.
|
||||
# use `uv --dev` to enable
|
||||
dev = [
|
||||
"pytest >=6.0.0, <7.0.0",
|
||||
"elasticsearch >=8.9.0, <9.0.0",
|
||||
"xonsh >=0.14.2, <0.15.0",
|
||||
"prompt-toolkit ==3.0.40",
|
||||
"cython >=3.0.0, <4.0.0",
|
||||
"greenback >=1.1.1, <2.0.0",
|
||||
"ruff>=0.9.6",
|
||||
|
||||
# ------ - ------
|
||||
|
||||
# TODO: add an `--only daemon` group for running non-ui / pikerd
|
||||
# service tree in distributed mode B)
|
||||
# https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies
|
||||
# [tool.poetry.group.daemon.dependencies]
|
||||
|
||||
[tool.poetry.scripts]
|
||||
piker = 'piker.cli:cli'
|
||||
pikerd = 'piker.cli:pikerd'
|
||||
ledger = 'piker.accounting.cli:ledger'
|
||||
|
||||
|
||||
[project]
|
||||
keywords=[
|
||||
"async",
|
||||
"trading",
|
||||
"finance",
|
||||
"quant",
|
||||
"charting",
|
||||
]
|
||||
classifiers=[
|
||||
'Development Status :: 3 - Alpha',
|
||||
"License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
|
||||
'Operating System :: POSIX :: Linux',
|
||||
"Programming Language :: Python :: Implementation :: CPython",
|
||||
"Programming Language :: Python :: 3 :: Only",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
'Intended Audience :: Financial and Insurance Industry',
|
||||
'Intended Audience :: Science/Research',
|
||||
'Intended Audience :: Developers',
|
||||
'Intended Audience :: Education',
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
piker = "piker.cli:cli"
|
||||
pikerd = "piker.cli:pikerd"
|
||||
ledger = "piker.accounting.cli:ledger"
|
||||
|
||||
[tool.hatch.build.targets.sdist]
|
||||
include = ["piker"]
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
include = ["piker"]
|
||||
|
||||
[tool.uv.sources]
|
||||
pyqtgraph = { git = "https://github.com/pikers/pyqtgraph.git" }
|
||||
asyncvnc = { git = "https://github.com/pikers/asyncvnc.git", branch = "main" }
|
||||
tomlkit = { git = "https://github.com/pikers/tomlkit.git", branch ="piker_pin" }
|
||||
msgspec = { git = "https://github.com/jcrist/msgspec.git" }
|
||||
tractor = { path = "../tractor", editable = true }
|
||||
|
|
|
|||
93
ruff.toml
93
ruff.toml
|
|
@ -1,93 +0,0 @@
|
|||
# from default `ruff.toml` @
|
||||
# https://docs.astral.sh/ruff/configuration/
|
||||
|
||||
# Exclude a variety of commonly ignored directories.
|
||||
exclude = [
|
||||
".bzr",
|
||||
".direnv",
|
||||
".eggs",
|
||||
".git",
|
||||
".git-rewrite",
|
||||
".hg",
|
||||
".ipynb_checkpoints",
|
||||
".mypy_cache",
|
||||
".nox",
|
||||
".pants.d",
|
||||
".pyenv",
|
||||
".pytest_cache",
|
||||
".pytype",
|
||||
".ruff_cache",
|
||||
".svn",
|
||||
".tox",
|
||||
".venv",
|
||||
".vscode",
|
||||
"__pypackages__",
|
||||
"_build",
|
||||
"buck-out",
|
||||
"build",
|
||||
"dist",
|
||||
"node_modules",
|
||||
"site-packages",
|
||||
"venv",
|
||||
]
|
||||
|
||||
# Same as Black.
|
||||
line-length = 88
|
||||
indent-width = 4
|
||||
|
||||
# Assume Python 3.9
|
||||
target-version = "py312"
|
||||
|
||||
# ------ - ------
|
||||
# TODO, stop warnings around `anext()` builtin use?
|
||||
# tool.ruff.target-version = "py310"
|
||||
|
||||
|
||||
[lint]
|
||||
# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default.
|
||||
# Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
|
||||
# McCabe complexity (`C901`) by default.
|
||||
select = ["E4", "E7", "E9", "F"]
|
||||
ignore = []
|
||||
ignore-init-module-imports = false
|
||||
|
||||
[lint.per-file-ignores]
|
||||
"piker/ui/qt.py" = [
|
||||
"E402",
|
||||
'F401', # unused imports (without __all__ or blah as blah)
|
||||
# "F841", # unused variable rules
|
||||
]
|
||||
|
||||
# Allow fix for all enabled rules (when `--fix`) is provided.
|
||||
fixable = ["ALL"]
|
||||
unfixable = []
|
||||
|
||||
# Allow unused variables when underscore-prefixed.
|
||||
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
|
||||
|
||||
[format]
|
||||
# Use single quotes in `ruff format`.
|
||||
quote-style = "single"
|
||||
|
||||
# Like Black, indent with spaces, rather than tabs.
|
||||
indent-style = "space"
|
||||
|
||||
# Like Black, respect magic trailing commas.
|
||||
skip-magic-trailing-comma = false
|
||||
|
||||
# Like Black, automatically detect the appropriate line ending.
|
||||
line-ending = "auto"
|
||||
|
||||
# Enable auto-formatting of code examples in docstrings. Markdown,
|
||||
# reStructuredText code/literal blocks and doctests are all supported.
|
||||
#
|
||||
# This is currently disabled by default, but it is planned for this
|
||||
# to be opt-out in the future.
|
||||
docstring-code-format = false
|
||||
|
||||
# Set the line length limit used when formatting code snippets in
|
||||
# docstrings.
|
||||
#
|
||||
# This only has an effect when the `docstring-code-format` setting is
|
||||
# enabled.
|
||||
docstring-code-line-length = "dynamic"
|
||||
Loading…
Reference in New Issue