Comparing Sequential Forecasters
Project description
ComparingForecasters
Code accompanying our paper, Comparing Sequential Forecasters (arXiv preprint), where we derive time-uniform, distribution-free, and non-asymptotic confidence sequences (CS), as well as e-processes, for comparing forecasts on sequential data.
Installation
Requires Python 3.7+.
From pip
:
pip install --upgrade pip
pip install --upgrade comparecast
From source:
git clone https://github.com/yjchoe/ComparingForecasters
cd ComparingForecasters
pip install --upgrade pip
pip install -e .
Data Sources
See data/README.md
.
Sample Usage
Python
import comparecast as cc
# Generate/retrieve synthetic data
data = cc.data_utils.synthetic.get_data("default", size=1000)
# Calculate forecasts
forecasters = ["k29_poly3", "laplace", "always_0.5"]
data = cc.forecast(data, forecasters, out_file="data/test.csv")
cc.plot_forecasts(data, forecasters, plots_dir="plots/test")
# Compare forecasts using confidence sequences & e-values
lcbs, ucbs, evalues = cc.compare_forecasts(
data,
"k29_poly3",
"laplace",
scoring_rule="brier",
alpha=0.05,
boundary_type="mixture",
v_opt=10,
compute_evalues=True,
)
# Draw a comparison plot
results, axes = cc.plot_comparison(
data,
"k29_poly3",
"laplace",
scoring_rule="brier",
alpha=0.05,
boundary_type="mixture",
compare_baselines=("h", "acs"),
plot_width=True,
plots_dir="plots/test",
)
Command Line Interface
# Generate synthetic data and forecasts
python forecast.py -d default -n 1000 -f all \
-o forecasts/test.csv -p plots/test
# Compare forecasts and plot results
python plot_comparisons.py -d forecasts/test.csv -p k29_poly3 -q laplace \
--compare-baselines h acs --plot-width -o plots/test
# Compare 538 and vegas forecasters
python plot_comparisons.py -d forecasts/mlb_2010_2019.csv \
-p fivethirtyeight -q vegas \
--compare-baselines acs --plot-width -o plots/mlb_2010_2019
Experiments
nb_comparecast_synthetic.ipynb
: Experiments on synthetic data and forecasts. Includes comparison with other methods (fixed-time CI and asymptotic CS). Section 5.1 in our paper.nb_comparecast_scoringrules.ipynb
: Experiments on synthetic data and forecasts using different scoring rules. Section 5.1 in our paper.nb_comparecast_baseball.ipynb
: Experiments on Major League Baseball forecasts, leading up to the 2019 World Series. Section 5.2 in our paper.nb_comparecast_weather.ipynb
: Experiments on postprocessing methods for ensemble weather forecasts. Includes e-value comparison with Henzi & Ziegel (2021). Section 5.3 in our paper.nb_comparecast_weather_eda.ipynb
: Exploratory plots on the ensemble weather forecast dataset. Section 5.3 in our paper.nb_iid_mean.ipynb
: Comparison of uniform boundaries on the mean of IID data. Partly reproduces Figure 1 from Howard et al. (2021). Appendix C in our paper.
License
MIT
Authors
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
comparecast-0.2.0.tar.gz
(31.6 kB
view hashes)
Built Distribution
Close
Hashes for comparecast-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edb3a184dbb9dd3ae60bb5e01f1ef92359efc31f5d10e81940ad30e288c6bdf4 |
|
MD5 | 225344ef8b182ab4a1d71dd99feede1e |
|
BLAKE2b-256 | 74ede9a2a035bf0c3aa24d781fdd78dd0bff2febc62dc29e58b47d424a52d623 |