the FOSS computational trading platform (for non-tinas..) https://github.com/pikers/piker
 
 
Go to file
Gud Boi a1048c847b Add vlm-based "smart" OHLCV de-duping & bar validation
Using `claude`, add a `.tsp._dedupe_smart` module that attemps "smarter"
duplicate bars by attempting to distinguish between erroneous bars
partially written during concurrent backfill race conditions vs.
**actual** data quality issues from historical providers.

Problem:
--------
Concurrent writes (live updates vs. backfilling) can result in create
duplicate timestamped ohlcv vars with different values. Some
potential scenarios include,

- a market live feed is cancelled during live update resulting in the
  "last" datum being partially updated with all the ticks for the
  time step.
- when the feed is rebooted during charting, the backfiller will not
  finalize this bar since rn it presumes it should only fill data for
  time steps not already in the tsdb storage.

Our current naive  `.unique()` approach obvi keeps the incomplete bar
and a "smarter" approach is to compare the provider's final vlm
amount vs. the maybe-cancelled tsdb's bar; a higher vlm value from
the provider likely indicates the cancelled-during-live-write and
**not** a datum discrepancy from said data provider.

Analysis (with `claude`) of `zecusdt` data revealed:
- 1000 duplicate timestamps
- 999 identical bars (pure duplicates from 2022 backfill overlap)
- 1 volume-monotonic conflict (live partial vs backfill complete)

A soln from `claude` -> `tsp._dedupe_smart.dedupe_ohlcv_smart()`
which:
- sorts by vlm **before** deduplication and keep the most complete
  bar based on vlm monotonicity as well as the following OHLCV
  validation assumptions:
  * volume should always increase
  * high should be non-decreasing,
  * low should be non-increasing
  * open should be identical
- Separates valid race conditions from provider data quality issues
  and reports and returns both dfs.

Change summary by `claude`:
- `.tsp._dedupe_smart`: new module with validation logic
- `.tsp.__init__`: expose `dedupe_ohlcv_smart()`
- `.storage.cli`: integrate smart dedupe, add logging for:
  * duplicate counts (identical vs monotonic races)
  * data quality violations (non-monotonic, invalid OHLC ranges)
  * warnings for provider data issues
- Remove `assert not diff` (duplicates are valid now)

Verified on `zecusdt`: correctly keeps index 3143645
(volume=287.777) over 3143644 (volume=140.299) for
conflicting 2026-01-16 18:54 UTC bar.

`claude`'s Summary of reasoning
-------------------------------
- volume monotonicity is critical: a bar's volume only increases
  during its time window.
- a backfilled bar should always have volume >= live updated.
- violations indicate any of:
  * Provider data corruption
  * Non-OHLCV aggregation semantics
  * Timestamp misalignment

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-01-21 22:20:43 -05:00
.github/workflows Drop elastic search container build for now since we're also skipping the test 2023-05-26 16:50:53 -04:00
config Bump `brokers.toml`, update ib and deribit sections 2026-01-07 13:39:34 -05:00
dockering Bump ib-container docs and compose file 2026-01-07 13:39:34 -05:00
docs Drop marketstore and tina install from readme, add polars and apache! 2023-06-27 13:41:47 -04:00
examples Spurious first-draft of EG collapsing 2026-01-06 22:27:58 -05:00
notes_to_self Mk a `notes_to_self/` move orig file `ideas.rst' 2026-01-06 19:33:23 -05:00
piker Add vlm-based "smart" OHLCV de-duping & bar validation 2026-01-21 22:20:43 -05:00
scripts Add example mtr prober from `mtrpacket` 2023-08-10 11:49:09 -04:00
snippets Add `pexpect`-based `pdbp`-REPL offline helper 2026-01-21 22:20:43 -05:00
tests Draft a gt-one-`.fqme`-in-txns/account-file test 2026-01-07 13:41:07 -05:00
.gitignore Ignore snippets dir 2022-03-01 12:36:32 -05:00
LICENSE Relicense to AGPLv3 2020-06-16 00:12:19 -04:00
MANIFEST.in Slap in brokers.toml template if none exists 2021-05-22 15:11:39 -04:00
README.rst Fix `uv sync` flags in readme 2026-01-14 15:27:13 -05:00
default.nix Update `default.nix` (from @nt) for py313 2026-01-02 12:36:19 -05:00
develop.nix Muck with `develop.nix` to try and hack it with `poetry` venv, go py3.11 2023-09-22 16:39:54 -04:00
flake.lock Bump `flake.lock`, seemly nicely minimized B) 2026-01-02 12:36:05 -05:00
flake.nix Skip `ruff` dev-dep on nix(os) overlays 2026-01-02 12:36:39 -05:00
poetry.lock Attempt to generate .uis deps free lock file 2023-09-07 14:17:01 -04:00
pyproject.toml Another little `pyproject` rejig 2026-01-14 15:29:26 -05:00
pytest.ini Ensure actual pp is sent to ems 2023-02-25 18:59:40 -05:00
ruff.toml Drop variable regex from `ruff.toml` 2026-01-02 12:38:36 -05:00
tags Start a manual `tags` file for internal refs 2026-01-06 19:33:23 -05:00
uv.lock Update `pdbp` to latest release 2026-01-14 09:49:50 -05:00

README.rst

piker

trading gear for hackers

gh_actions

piker is a broker agnostic, next-gen FOSS toolset and runtime for real-time computational trading targeted at hardcore Linux users .

we use much bleeding edge tech including (but not limited to):

potential projects we might integrate with soon,

  • (already prototyped in ) techtonicdb for L2 book storage

focus and feats:

fitting with these tenets, we're always open to new framework/lib/service interop suggestions and ideas!

  • 100% federated: your code, your hardware, your data feeds, your broker fills.
  • zero web: low latency as a prime objective, native UIs and modern IPC protocols without trying to re-invent the "OS-as-an-app"..
  • maximal privacy: prevent brokers and mms from knowing your planz; smack their spreads with dark volume from a VPN tunnel.
  • zero clutter: modal, context oriented UIs that echew minimalism, reduce thought noise and encourage un-emotion.
  • first class parallelism: built from the ground up on a next-gen structured concurrency supervision sys.
  • traders first: broker/exchange/venue/asset-class/money-sys agnostic
  • systems grounded: real-time financial signal processing (fsp) that will make any queuing or DSP eng juice their shorts.
  • non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms (or maybe even a DDE).
  • data collab at scale: every actor-process and protocol is multi-host aware.
  • fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.

building the hottest looking, fastest, most reliable, keyboard friendly FOSS trading platform is the dream; join the cause.

a sane install with uv

bc why install with python when you can faster with rust :

uv sync

# ^ astral's docs,
# https://docs.astral.sh/uv/concepts/projects/sync/

include all GUIs (ex. for charting):

uv sync --group uis

AND with all our normal hacking tools:

uv sync --dev

AND if you want to try WIP integrations:

uv sync --all-groups

Ensure you can run the root-daemon:

uv run pikerd [-l info --pdb]

install on nix(os)

NixOS is our core devs' distro of choice for which we offer a stringently defined development shell envoirment that can currently be applied in one of 2 ways:

# ONLY if running on X11
nix-shell default.nix

Or if you prefer flakes style and a modern DE:

# ONLY if also running on Wayland
nix develop  # for default bash
nix develop -c uv run xonsh  # for @goodboy's preferred sh B)

start a chart

run a realtime OHLCV chart stand-alone:

[uv run] piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken

# ^^^ iff you haven't activated the py-env,
# - https://docs.astral.sh/uv/concepts/projects/run/
#
# in order to create an explicit virt-env see,
# - https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment
# - https://docs.astral.sh/uv/pip/environments/
#
# use $UV_PROJECT_ENVIRONMENT to select any non-`.venv/`
# as the venv sudir in the repo's root.
# - https://docs.astral.sh/uv/reference/environment/#uv_project_environment

this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes overlayed on the same graph. Use of piker without first starting a daemon (pikerd - see below) means there is an implicit spawning of the multi-actor-runtime (implemented as a tractor app).

For additional subsystem feats available through our chart UI see the various sub-readmes:

  • order control using a mouse-n-keyboard UX B)
  • cross venue market-pair (what most call "symbol") search, select, overlay Bo
  • financial-signal-processing (piker.fsp) write-n-reload to sub-chart BO
  • src-asset derivatives scan for anal, like the infamous "max pain" XO

spawn a daemon standalone

we call the root actor-process the pikerd. it can be (and is recommended normally to be) started separately from the piker chart program:

pikerd -l info --pdb

the daemon does nothing until a piker-client (like piker chart) connects and requests some particular sub-system. for a connecting chart pikerd will spawn and manage at least,

  • a data-feed daemon: datad which does all the work of comms with the backend provider (in this case the binance cex).
  • a paper-trading engine instance, paperboi.binance, (if no live account has been configured) which allows for auto/manual order control against the live quote stream.

using an actor-service (aka micro-daemon) manager which dynamically supervises various sub-subsystems-as-services throughout the piker runtime-stack.

now you can (implicitly) connect your chart:

piker chart btcusdt.spot.binance

since pikerd was started separately you can now enjoy a persistent real-time data stream tied to the daemon-tree's lifetime. i.e. the next time you spawn a chart it will obviously not only load much faster (since the underlying datad.binance is left running with its in-memory IPC data structures) but also the data-feed and any order mgmt states should be persistent until you finally cancel pikerd.

if anyone asks you what this project is about

you don't talk about it; just use it.

how do i get involved?

enter the matrix.

how come there ain't that many docs

i mean we want/need them but building the core right has been higher prio then marketting (and likely will stay that way Bp).

soo, suck it up bc,

  • no one is trying to sell you on anything
  • learning the code base is prolly way more valuable
  • the UI/UXs are intended to be "intuitive" for any hacker..

we obviously need tonz help so if you want to start somewhere and can't necessarily write "advanced" concurrent python/rust code, this helping document literally anything might be the place for you!