GridShedWatch

A benchmark for early warning of grid load sheds, scored the way forecasters score weather warnings.

Website: https://gotham29.github.io/gridshedwatch (built from this repo by scripts/build_site.py).

Since 2020, US grid operators have had to cut power for lack of supply eight times. The public was warned beforehand in four of them, and in none of the four during Winter Storm Elliott. Plenty of methods claim to detect grid trouble early, but there has been no shared test of whether a warning would actually have come in time. GridShedWatch is that test: every one of those eight load sheds, public hourly data, and one fixed scoring procedure that any method can be run through.

The National Weather Service scores every warning it issues on three numbers read together: how many events were warned in advance, how much lead time people got, and how often warnings were false alarms. GridShedWatch asks the same of grid warnings, at two levels: a watch (about one a month per grid: something unusual is building) and a warning (two a year: act).

What's in it

Results, v1.0

All five reference methods are in the scores track; no alerts-track entries yet. Mean over 5 seeds for the LSTM and HTM. [a-b]: 5th-95th percentile with events resampled. Operators: public notices in the 7 days before onset, scored as watches.

Every method except Chronos's watches catches far more than chance, and none leads at both levels. The differences are by kind of emergency: the simple rules caught all three Uri events days ahead, the LSTM and HTM caught the Elliott events hours ahead (when the public got no warning), and only the operators warned California. With eight events, differences of about one event are not meaningful. Per-event results: results/v1.0/<method>_events.csv.

Quick start

pip install gridshedwatch                # or, to work on the code:
git clone https://github.com/gotham29/gridshedwatch && cd gridshedwatch
pip install -e ".[test]"
gridshedwatch info                       # the eight events and their timelines
gridshedwatch score --oprule             # score the built-in operator-style rule
python examples/my_method.py             # scores track template: writes my_scores/
gridshedwatch score my_scores --name "My method"
python examples/my_alerts.py             # alerts track template: writes my_alerts.csv
gridshedwatch score-alerts my_alerts.csv --name "My rule"

Add your method

There are two ways in.

Scores track (the controlled comparison). Your method says, for every grid, every hour and each of the five signals, how unusual that signal is now, as a score in [0, 1], using data up to that hour only. Write the scores in the submission layout (my_scores/s<seed>/<grid>.csv, see gridshedwatch/submission.py) and run gridshedwatch score. The shared rules, the label-free budget step and the scoring are done for you, so your method cannot be tuned toward the eight events. Template: examples/my_method.py.

Alerts track (any method). If your method decides on its own when to raise a watch or a warning (a rule, a classifier, a weather-based model, a human forecaster), list when each alert started:

grid,level,start_utc,seed
tx,watch,2021-02-11 03:00,0

and run gridshedwatch score-alerts alerts.csv --name "My method". It is scored exactly like the scores track, at the same budget (median grid at most 12 watches and 2 warnings a year, no grid above 3 times that). Format: gridshedwatch/alerts.py; template: examples/my_alerts.py; worked examples: the five reference methods' alerts in results/v1.0/alerts/, which reproduce the table above.

• Scores track: use the five signals provided (and 2019 for calibration). Alerts track: any data available at the time of each alert. Methods that use data beyond the five signals are listed separately.
• If your method is random, submit several seeds; results are averaged.
• To add a result to the table, open a pull request with your scores or alerts (or a link to them), the code that produced them, the output of --json, and short answers on the data used and, for the alerts track, which events you looked at while developing (docs/scoring.md). Every result is added, including ones that beat the reference methods.

Reproducing v1.0

python -m pytest -q                      # rebuilds the rule and z-score results from the data
python scripts/fetch_reference.py        # reference scores for LSTM, Chronos, HTM (about 100 MB)
python -m pytest -q                      # now also checks those three against results/v1.0/
python scripts/reproduce_v1.py           # writes results/v1.0/ from scratch
docker build -t gridshedwatch . && docker run --rm gridshedwatch

The LSTM and Chronos can be rerun with gridshedwatch baseline lstm|chronos -o DIR (needs pip install -e ".[lstm]" or ".[chronos]"). Neural-network training is not bit-identical across platforms and library versions, so a rerun gives close but not identical scores; the reference scores are the official v1.0 results. HTM's scores come from htm-monitor, scored here like any other submission.

Limits

• Eight events, seven from two storms (Uri and Elliott). One storm of a new kind could change the picture.
• Three events were used to design the shared rules; five were held out and scored once.
• Reference methods see only five signals from EIA-930: no weather, outage or market data. The alerts track is open to methods that use more.
• The status quo row covers operators' public notices; their notices to the industry and their false alarms in ordinary years are not yet compiled.

GridShedWatch grows as new events happen: each load shed that meets the rule is added in a new version and scored with the rules frozen in v1.0.

Citation and license

DOI: 10.5281/zenodo.23626223 (archived on Zenodo with every release). See CITATION.cff. Code: MIT. Event list, timelines, documentation and results: CC BY 4.0. EIA-930 data: U.S. public domain (DATA_LICENSE.md).