Experiment 83: tail calibration, on experiment 48
Every stored forecast of experiment 48 is too narrow in both tails, and weekly retraining has the lowest pinball loss at both levels. We built a tail-calibration table for the studies code and ran it on the stored demand forecasts of experiment 48. The stack is the demand model fitted once, with a correction from its own recent errors. This experiment first also built a single-series curtailment-cost metric. We withdrew that metric, because curtailment binds at a substation, so a metric that scores each meter alone against its own history does not measure it.
Results (exploratory, 20 demand series pooled, every lead and run from 2025-07-02)
- Coverage: coverage of the 0.05 quantile is 0.066 to 0.085 and of the 0.95 quantile is 0.891 to 0.927, so every forecast is too narrow in both tails. The target coverage is the level itself.
- Pinball loss: weekly retraining has the lowest pinball loss at both levels.
- Levels not delivered: the 0.01 and 0.99 rows read "not delivered", because the stored sets hold nine quantiles from 0.05 to 0.95.
Caveats and next steps
- Not scored: experiment 64's stored sets carry no
actualcolumn, so they were not scored here. - Next: write the 0.01 and 0.99 quantiles in new experiments.
Reproduce with uv run python ml-experiments/00083-*/e00083_report.py. The printed report is in
data/ml-experiments/00083-curtailment-cost-and-tail-calibration-worked-example/report.md.