Skip to content

Differentiable physics: do the neural network's corrections help the worst days that no label explains?

The neural network removes 22.6% of the excess loss on the 37 worst days that no label explains, but those days are not hot or cold extremes. Only 4 of the 37 days lie in the outer 10% of their substation's temperature range, fewer than the 7 or 8 expected for random days (the 8th percentile of 1,000 random draws). Prediction 1 passes and prediction 2 fails, so the result is partly supported. The neural network with forecast inputs removes 22.6% of the excess and the one without removes 3.9%, so the gain depends on the forecast inputs.

This experiment re-scores stored forecasts and fits nothing. It reads the stored tables of experiment 172 (the worst days) and the stored predictions of experiment 201 (the neural network), The worst days are the 72 days with the highest excess at the five decision substations (D5, D10, D11, D14, and D16). A day is labelled if it carries a switching event, a holiday, a cold snap, a flagged reading, or an Economy 7 step. A day's score is the mean quantile score of its day-ahead rows (leads 15 to 39 hours), so "points" below are points of the quantile score, in percentage points of effective capacity. A day's excess is its score minus its substation's median day score. Arm D is the best-guess convex demand forecast of experiment 155. Rebuilding experiment 172's day table from the stored rows gives all 1,392 days with a maximum score difference of 0.0.

Temperature percentile of the 37 unlabelled worst days, and the share in the outer 10% of temperature in 1,000 draws of 37 ordinary days.

Share of arm D's excess loss removed on the unlabelled and labelled worst days by four stored neural networks, with the range over three seeds.

Five unlabelled worst days: observed power, arm D, and arm D plus the neural network.

Results

  • Prediction 1 (the network cuts the unlabelled days' excess loss by at least 20%): passed. Arm 2 of experiment 201 removes 22.6% (seeds 19.8%, 25.9%, 22.3%). The day score falls from 4.93 to 4.29 points. On the 35 labelled worst days it removes 11.8% (7.03 to 6.46 points), and on all 1,392 days 33.4% of excess.
  • The gain is larger than on random days: the real gain lies above all 1,000 draws. The mean gain per day on the unlabelled worst days is 0.64 points, against 0.21 points (5th to 95th percentile 0.03 to 0.42) over 1,000 draws of 37 ordinary days matched on substation and calendar month.
  • Prediction 2 (at least half the days at temperature extremes): failed. Four of the 37 days (1 below the 10th percentile and 3 above the 90th) lie in the outer 10% of the substation's days before 2025-07-01, against a mean of 20.1% (5th to 95th percentile 10.8% to 29.7%) over the random draws; the real share sits at the 8th percentile of the draws. The 35 labelled days are more extreme (8 of 35), and the unlabelled days are milder than average (mean 11.3 degrees Celsius against 11.5 for all days).
  • Only the neural networks with forecast inputs remove more than 4% of the excess. Arm 0 (no forecast input) removes 3.9% of the unlabelled days' excess, so choosing days for a high arm D error accounts for about 4%. Arm 4 (arm 2 plus recent power) removes 26.6%, and arm 6 (arm D's outputs only) removes 23.2%. Across all 1,392 days arm 2 removes 33.4% of the excess, a larger share than on the unlabelled worst days.
  • The three seeds straddle the 20% threshold. Three training runs of arm 2 that differ only in random seed span 19.8% to 25.9% on the unlabelled days, so the 20% threshold is within the seed spread.

Checks

  • The earlier control found 89.2% of its injected days, so shares are approximate. Experiment 172's control inserted known unusual days into the data and recovered 89.2% of them.
  • The temperature definition does not change the finding. The headline uses the ensemble-mean temperature that arm D used. With ERA5 day temperature, as in experiment 172, the unlabelled days average 11.5 degrees Celsius.
  • The verdict is descriptive, because the sample is 37 days. The unit is one day and the sample is 37 days, so the shares carry no interval beyond the random draws.

Caveats

  • Days chosen for a high arm D error tend to score better under a different forecast. Arm 0 removes 3.9%, which indicates the size of that effect, and the 22.6% is read against it.
  • The result covers the five decision substations together, not each one. It was not split by substation.

Experiments this builds on

Experiments 172, 174, 175, 200, 201, and 209.