104 lines
4.1 KiB
Markdown
104 lines
4.1 KiB
Markdown
# Financial Signal Processing
|
|
|
|
## Model the Clock First
|
|
|
|
Market inputs can be quote events, trades, book changes, fixed-time bars, or
|
|
revised historical bars. Name both event time and processing time. For
|
|
irregular events, either use time-aware recurrences or define the resampling
|
|
and interpolation policy explicitly.
|
|
|
|
Never let backfilled information leak into a value presented as historically
|
|
tradable. Distinguish corrected analysis history from the signal values that a
|
|
live strategy could have observed.
|
|
|
|
## Useful Technique Families
|
|
|
|
### Online Moments and Robust Statistics
|
|
|
|
Use Welford/Chan recurrences for stable online moments, compensated summation
|
|
for long accumulations, and mergeable summaries when partitioning actors.
|
|
Rolling median, quantiles, MAD, and winsorized measures are useful under heavy
|
|
tails, but exact rolling order statistics need more state than moments.
|
|
|
|
### Time-Aware Exponential Filters
|
|
|
|
For irregular event intervals, derive decay from elapsed time:
|
|
|
|
```text
|
|
alpha(dt) = 1 - exp(-dt / tau)
|
|
y_t = y_prev + alpha(dt) * (x_t - y_prev)
|
|
```
|
|
|
|
Record initialization, session reset, gap, and backfill behavior. A fixed
|
|
sample alpha is valid only after an explicit regularization policy.
|
|
|
|
### State-Space Models
|
|
|
|
Kalman and robust state-space filters fit latent price, spread, volatility,
|
|
and lead/lag estimates. Prefer square-root or Joseph-form covariance updates
|
|
when conditioning is poor. State revisions after backfill require replay from
|
|
a checkpoint or a bounded smoother; silently prepending observations without
|
|
replaying state is incorrect.
|
|
|
|
### Point Processes and Microstructure
|
|
|
|
Trade and quote arrivals are events, not merely sampled amplitudes. Consider
|
|
intensity, duration, signed order flow, imbalance, spread, queue change, and
|
|
self-excitation models. Validate against provider-specific aggregation and
|
|
duplicate/out-of-order behavior before interpreting a statistic.
|
|
|
|
### Volatility and Covariance
|
|
|
|
Realized variance, bipower variation, EW covariance, and range estimators can
|
|
be updated online. Asynchronous cross-market covariance needs synchronization
|
|
or estimators designed for non-synchronous observations; naive row alignment
|
|
creates lead/lag and Epps-effect artifacts.
|
|
|
|
### Spectral and Multiscale Methods
|
|
|
|
FFT methods assume a regular grid and a window. Declare detrending, tapering,
|
|
overlap, normalization, and latency. For tick data, resample deliberately or
|
|
use irregular-time methods. Wavelets and multiresolution filters can separate
|
|
horizons, but boundary handling and causal delay must be visible to strategy
|
|
code.
|
|
|
|
### Change and Anomaly Detection
|
|
|
|
CUSUM, Page-Hinkley, sequential likelihood ratios, and robust z-scores are
|
|
cheap online tools. Calibrate false alarms under dependence and regime shifts;
|
|
do not treat IID thresholds as market guarantees.
|
|
|
|
## Numerical Implementation Order
|
|
|
|
1. Establish a scalar reference with explicit state and semantics.
|
|
2. Vectorize historical bootstrap with NumPy.
|
|
3. Keep realtime updates as O(1) recurrences where possible.
|
|
4. Use Numba for measured numeric kernels with stable dtypes and no Python
|
|
object traffic.
|
|
5. Use Polars or Arrow for columnar historical transforms and interchange,
|
|
not automatically for each tick.
|
|
6. Partition by independent market/operator state before adding threads or a
|
|
new distributed runtime.
|
|
|
|
## Existing Stack
|
|
|
|
- NumPy 2.x: canonical dense and structured-array kernels.
|
|
- Numba: compiled CPU loops and recurrences that do not vectorize cleanly.
|
|
- Polars: parallel/lazy columnar history preparation and validation.
|
|
- PyArrow: columnar interchange, storage, and ingest boundaries.
|
|
- Tractor/Trio: structured distributed execution and lifecycle.
|
|
- PyQtGraph: visible-range rendering, not a compute scheduler.
|
|
|
|
## Algorithm Acceptance Criteria
|
|
|
|
Every new FSP should define:
|
|
|
|
- input fields, clock, ordering, duplicate, gap, and revision policy;
|
|
- output dtype, units, identity, and parameterization;
|
|
- warm-up length and initialization bias;
|
|
- causal latency and strategy-visible publication point;
|
|
- historical/realtime equivalence tolerance;
|
|
- reset and session-boundary behavior;
|
|
- complexity, state size, and representative throughput;
|
|
- replay behavior when historical source data changes.
|