Online covariance and precision estimation
Project description
precise
A collection of incremental estimators for covariance, precision, correlation, portfolios and ensembles.
TLDR: "Just a pile of functions that forecast covariance in online fashion"
The running_empirical_covariance colab notebook illustrates the style. To see all the other online methods of covariance estimation supplied here, run the cov skaters manifest notebook. Or to look at Elo ratings, run the elo_ratings_and_urls.
Install
pip install precise
or for latest:
pip install git+https://github.com/microprediction/precise.git
M6 Financial forecasting contest utilities
You could use this library to enter the M6 Financial Forecasting competition:
- Pick a cov estimator (i.e. a "cov skater"), if you wish
- Pick a portfolio generator, if you wish
- Pick extra shrinkage params, if you wish
- Pick love and hate ticker lists, if you wish
See precise/examples_m6 and register at the m6 competition. See disclaimer below.
Covariance skaters
Similar in style to skaters used in the timemachines package, this package may be thought of as a collection of covariance prediction functions taking one vector at a time, and also the prior state, and spitting out a prediction mean vector x, a prediction covariance x_cov, and a posterior state whose interpretation is the responsibility of the skater, not the caller.
from precise.skatertools.syntheticdata.miscellaneous import create_correlated_dataset
from precise.skaters.covariance.runemmp import run_emp_pcov_d0 # <-- Running empirical population covariance
from pprint import pprint
if __name__=='__main__':
ys = create_correlated_dataset(n=500)
s = {}
for y in ys:
x, x_cov, s = run_emp_pcov_d0(s=s, y=y)
pprint(x_cov)
See /examples_basic_usage. And yes, this mildly unusual convention requires the caller to maintain state from one call to the next: See the timemachines faq for justification of this style.
Elo ratings
As noted, see the elo_ratings_and_urls.
Browsing for skaters
You can hunt for skaters other than run_emp_pcov_d0 in precise/skaters/covariance. There are some location utilities in precise/whereami.
Interpreting skater names
Examples:
Skater name | Location | Meaning |
---|---|---|
buf_huber_pcov_d1_a1_b2_n50 | skaters/covariance/bufhuber | Applies an approach that exploits Huber pseudo-means to a buffer of data of length 50 in need of differencing once, with generalized Huber loss parameters a=1, b=2. |
buf_sk_ld_pcov_d0_n100 | skaters/covariance/bufsk | Applies sk-learn's implementation of Ledoit-Wolf to stationary buffered data of length 100 |
ewa_pm_emp_scov_r01 | skaters/covariance/ewapartial | Performs an incremental, recency-weighted sample covariance estimate that exploits partial moments. Uses a memory parameter r=0.01 |
Broad calculation style categories
Shorthand | Interpretation | Incremental ? |
---|---|---|
buf | Performs classical batch calculation on a fixed window of data each time | No |
win | Performs incremental fixed window calculation. | Yes |
run | Running calculation weighing all observations equally | Yes |
ewa | Running calculation weighing recent observations more | Yes |
Methodology hints (can be combined)
Shorthand | Inspiration |
---|---|
emp | "Empirical" (not shrunk or augmented) |
lz | Le-Zhong variable-by-variable updating |
lw | Ledoit-Wolf |
pm | Partial moments |
huber | Generalized Huber pseudo-mean |
oas | Oracle approximating shrinkage. |
gl | Graphical Lasso |
mcd | Minimum covariance determinant |
Intended main target (more than one may be produced in the state)
Shorthand | Intent |
---|---|
scov | Sample covariance |
pcov | Population covariance |
spre | Inverse of sample covariance |
ppre | Inverse of population covariance |
Differencing hints:
Shorthand | Intent |
---|---|
d0 | For use on stationary, ideally IID data |
d1 | For use on data that is iid after taking one difference |
Stand-alone covariance utilities
- The covariance/statefunctions are illustrated by the example running_oas_covariance.
- State covariatnce/statemutations do things like ensuring both covariance and precision matrices exist in the state. Or for instance: s = both_cov(s) ensures both sample and population covariances are present.
- Some /covariance/datascatterfunctions
- The /covariance/datafunctions take data and produce covariance functions.
- The /covariance/covfunctions manipulate 2d cov arrays.
Portfolios, ensembles & mixture of experts
Too fluid to document currently. See the portfolio directories in skaters.
Miscellaneous remarks
- Here is some related, and potentially related, literature.
- This is a piece of the microprediction project, should you ever care to cite the same. The uses include mixtures of experts models for time-series analysis, buried in timemachines somewhere.
- If you just want univariate calculations, and don't want numpy as a dependency, there is momentum. However if you want univariate forecasts of the variance of something, as distinct from mere online calculations of the same, I would suggest checking the time-series elo ratings and the "special" category in particular.
- The name of this package refers to precision matrices, not numerical precision. This isn't a source of high precision covariance calculations per se. The intent is more in forecasting future realized covariance. Perhaps I'll include some more numerically stable methods from this survey to make the name more fitting. Pull requests are welcome!
- The intent is that methods are parameter free. However some not-quite autonomous methods admit a few parameters (the factories). A few might even use just one additional scalar parameter r with a space-filling curve convention - somewhat akin to the tuning of skaters explained here in the timemachines package).
Disclaimer
Not investment advice.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.