Skip to content

ML experiments

Each folder here is one small experiment that tests one hypothesis about improving the production forecast, and the experiments are grouped into themes. A theme has three pages: a Summary (where one has been written) that states the answer so far, an Experiments page that lists every experiment tried, and a Research plan that lists what is planned next. The folders are named <id>-<slug>, where the id is a five-digit integer that orders the experiments by the order in which they were started. A folder holds result.md and its figures.

Themes

How an experiment is recorded

The front matter of result.md is the only place that records an experiment's theme, verdict, and status, and the theme pages build their tables from it. The Experiments table of each theme, the planned-experiments table of each Research plan, and the candidates table below are generated from the front matter when the documentation is built, so adding a finished experiment means writing its result.md and editing nothing else. The ml-experiment skill (.claude/skills/ml-experiment/SKILL.md) lists the front matter fields and their allowed values, including the optional kind and promoted_to fields of the research default rule.

uv run python scripts/mkdocs_hooks/experiment_tables.py --check validates every front matter block, and the documentation build runs the same check. uv run python scripts/mkdocs_hooks/experiment_tables.py --theme <theme> prints a theme's Experiments table as markdown.

The research default

The research default rule fixes one forecast per target family and one test for adding a change to it. The log of versions lists each family's versions, for example "Primary demand research default 1". An experiment with kind: change in its front matter is measured against the current version, and the rule says when a win is adopted.

Data processing

The data-processing page lists the processing that the experiments showed the data needs. Each entry says whether it applies to experiments, to production, or to both.

Background

The background pages state what earlier experiments established about the data sets and the scoring rules, and a planner reads them before planning an experiment.

Glossary

The glossary defines the terms that the experiment pages and the theme Summary pages use without redefining. Each definition links the experiment or code that fixes its meaning.

Planned experiments and their order

Each theme's Research plan page holds its own priority table and the detailed proposals for that theme. A planned experiment has a result.md with status planned, committed before any code runs. An idea that has no folder yet is marked "proposed" in the priority table and carries a tentative id, P1, P2, and so on, which the proposal sections use as headings. A proposal receives its numeric id when its plan is committed, and the tentative id then disappears. The design under test is the staged probabilistic forecast, in which stage 1 turns weather into power, stage 2 calibrates the result, uses lagged power, and blends weather products, and a coupling step restores coherent ensemble members.

The sample resolves gains of roughly 0.3 points of capacity or more, and no smaller gains. A point of capacity is 1% of a series' capacity. The sample is 16 Primary substations, 9 generator series, and one validation year with 12 months. The Research plan pages flag the experiments whose planned contrasts the sample cannot resolve.

Candidates for the next release

These experiments are flagged candidate and are not yet adopted. The flags are proposals for the maintainer to confirm. The maintainer decides which candidates to port into a new model release. The release log records which experiments each release adopted.

Experiment Theme Verdict Take-home message
00357 wind-default-2-monthly-head-longer-retest generator-forecasting partly supported Retraining Wind farms research default 2's head every month improves the forecast slightly, by about 0.04 to 0.05 points over the scored year and about 0.10 points in the latest 2.5 months, and no scope shows harm to the score. The scored-year interval still touches zero, so the change is a candidate and is not adopted. The upper tail stays too heavy (1.7% to 2.0% of observations above the 0.99 quantile), so retraining does not fix it.
00348 wind-plant-mdn-upper-tail generator-forecasting partly supported The frozen physics plant with a mixture head stays about 0.23 points of capacity better than the research default over days 1 to 10, but none of the four tail fixes inside the head brings the top quantile into calibration without losing score at that quantile, and none of their intervals excludes zero. Blending the plant's residual band with the default's XGBoost quantiles gains less (0.15 points) but is the arm of the five whose gain is clear and whose 0.99 quantile is no worse, so that recipe, not a better head, is the next step.
00347 wind-physics-hybrid-plant-then-mdn generator-forecasting partly supported A physics model of each wind farm, run on every ensemble member and corrected by a small neural network, forecasts 0.27 points of capacity (2.4%) better than the research default over leads of 0.5 hours to 15 days, and 0.23 points over true days 1 to 10, but the neural network given only the plant's median and the farm already gets 83% of the gain, and the upper tail is still over-confident. The next step is to fix the upper tail (the neural network caps its quantiles at capacity, and 0.9% of observations exceed it) before any plant-based arm is adopted.
00346 wind-research-default-1-offset-table-refit-cadence generator-forecasting partly supported Refitting the wind forecast's offset table every month on recent runs makes the forecast a little better (a 0.4% to 1.1% lower quantile score), using only data the service already stores. It does not fix the upper quantiles, which still miss in 1.7% of half-hours against 1% intended. So the change between the training years and the scored year is a minor cause, and the upper-tail miss needs another explanation. A monthly job that refits the table on all earlier daily runs passes the adoption conditions and is a candidate.
00345 pooled-solar-wind-mixture-density-network metered-solar-farms partly supported One neural network trained on all six solar farms forecasts each farm as well as a separate network on the physical plant model (within 0.8%), so a production forecast could use one neural network per fuel. Adding wind farms to the same network does not help either fuel and makes solar 0.7% to 2.3% worse, so solar and wind are better trained separately. In this experiment, a wind farm with no history scored 18% worse at day 1 when trained on public wind farms alone than when trained on other metered wind farms.
00339 wind-research-default-1-adapter-plus-mixture-head generator-forecasting partly supported A small neural-network head that re-draws the physics-plus-adapter wind forecast's quantiles gains 0.27 points of capacity (2.4%) on the farms' normal half-hours and keeps that gain on the replicate months, but the same head given no weather summaries does as well, so the gain is a recalibration that a simpler correction partly reproduces. A simple rule that predicts the idle floor at low wind recovers none of the head's loss on the idle-band readings, and training the head on those readings removes the gain.
00337 primary-demand-standard-architecture-168-to-240-hours production-reference-and-best-guess partly supported The standard architecture (a neural network that corrects the forecast of the convex stack, with a learned input adapter and a mixture-density output) forecast Primary demand at 7 to 10 days ahead with a lower error than the current research default over the 12 months scored. Most of the gain comes from the mixture-density output and from two Primaries (D10 and D16). The 12 months of data cannot establish the gain, so the standard architecture is a candidate for demand across NGED's 1-to-10-day range, not yet a replacement.
00336 wind-research-default-1-below-cut-in-zeroing generator-forecasting partly supported Setting the wind forecast, or the training target, to zero below the turbines' cut-in speed does not improve the forecast, because the farms still produce small real output there. Leaving the low-wind rows out of training gains 0.12 points of capacity (about 1% of the default's score) on the kept rows, but mostly by lifting the quantiles upward, which a constant upward shift may reproduce (not yet tested).
00327 lsc-label-free-detect-mask-train-pipeline lsc-detectors partly supported A forecast of the normal running arrangement recovers faster after a load shift when the readings that a label-free detector flags are treated as missing: during and after injected shifts the 95% pinball loss falls from 1.79 to 1.04 points of capacity. On ordinary days the masked forecast is no worse, and slightly better. Without one site (D11) the gain shrinks to 0.30 points, and retraining the demand model on masked power adds nothing beyond masking the filter.
00326 wind-research-default-1-change-arms generator-forecasting supported Feeding XGBoost the neural adapter's corrected wind speed lowers the wind forecast's score by 0.10 points of capacity, about 0.9% of the default's score, at 1 to 10 day leads (99.75% interval -0.19 to -0.01). The change is negative at every farm and at short and long leads. The three other changes were not separable from zero. Only a port that puts the adapter in the production service can use the gain, so the maintainer decides.
00322 primary-demand-default-2-daylight-only-heuristic-blend-and-sunshine-graft-replication production-reference-and-best-guess partly supported At 7 to 10 days the blend with the manual heuristic gains about 4% at all hours, and about 2.4% when it is used only by day, which is the version with no harm on the coldest winter nights. Counting how many of the 51 ensemble members are cold finds 72% of those nights (the ensemble mean finds 41%), and a hard switch on that count keeps the whole 4% with almost no cold-night harm, but that switch was not pre-registered. The sunshine response of the depth-1 XGBoost refit explains about half of the refit's advantage on the worst April-to-June half-hours in both years, so a flexible sunshine term in the convex fit is worth testing.
00321 metered-farm-lower-tail-recalibration-and-tail-head metered-solar-farms partly supported The solar-farm forecast's lowest quantile was already close to correct: only 1.2% to 1.4% of actual values fall strictly below the 0.01 quantile, because the earlier 3.3% counted the 3.0% of half-hours that read exactly zero. A recalibration table, a zero-inflated head, and a tail head each lower the own-history score by 0.04 points or less (the unshrunk table raises it by 0.03 to 0.05 points), so none is adopted. A shrunk recalibration table does lower the score by about 0.05 points for a farm with no history, and two heads lower the 0.99 pinball loss by 3.7% to 8.5% while covering the 0.99 quantile only 98.3% to 98.9% of the time.
00320 primary-demand-default-2-night-aware-heuristic-blend-and-night-band-width production-reference-and-best-guess partly supported A blend with the manual heuristic still improves the 7 to 10 day Primary demand forecast by about 4%, and the harm it does on the coldest winter nights cannot be removed by switching it off when the weather forecast says the night will be cold, because at 7 to 10 days that forecast spots only between one and two cold nights in three. Narrowing the night prediction bands helps one Primary (D10, by 0.25 points on night half-hours), leaves the other five unchanged or slightly worse, and makes the cold-night tails worse.
00319 primary-demand-default-1-solar-summary-and-heuristic-blend-at-7-to-10-days production-reference-and-best-guess partly supported The demand forecast for 7 to 10 days ahead improves by about 2% when its solar term also uses the spread between the 51 weather-forecast members, and the gain is largest on sunny days (6.6%), where the forecast had been worse than the manual heuristic that National Grid Electricity Distribution (NGED) staff use. Blending in the manual heuristic improves the forecast by 4.5%, but the blend worsens the coldest winter nights beyond the agreed tolerance, so only the solar change is adopted.
00318 metered-farm-mdn-head-more-features metered-solar-farms partly supported Giving the solar-farm head more inputs does not improve its overall score at any lead from 1 to 10 days, so the head's inputs are not what limits the forecast. The extra clock and cloud-spread inputs do lower the loss at the 0.99 quantile (the tail that matters for curtailment) by about 5% at days 3 to 7, and none of the changes fixes the lower tail, where 3.3% to 3.7% of actual values fall below the 0.01 quantile, where 1% is expected. Training one head for all leads makes the network with no plant nearly as good as the plant-based head at days 8 to 10 (0.4% worse), so the plant adds little there.
00314 primary-demand-gated-or-pinball-band-head calibration-and-uncertainty partly supported Experiment 311's neural band head was stopped by harm on sunny daylight half-hours. Keeping the current bands on those half-hours fixes that harm, and the gated head scores 0.041 points better than the current Primary demand forecast bands (research default 1, lower is better). The gain is still almost entirely one substation, D10: without it the gain is 0.015 points and its interval touches zero, so the gated head is a candidate and not an adoption. Training the head on the quantile score directly gave no improvement.
00313 metered-farm-pure-xgboost-and-mdn-vs-plant metered-solar-farms partly supported A forecast that sees only weather and the calendar, with no physical plant model anywhere, scores almost as well as the plant-based forecast for the six solar farms, up to 10 days ahead, even when the farm has no history. The plant's added value is small, between 0% and 3.4%, and its intervals include zero except for XGBoost. For wind, the same kind of network also matches the best plant-based forecast, so a plant model is not needed to be competitive on these farms.
00311 primary-demand-mixture-density-band-head calibration-and-uncertainty partly supported A neural network that sets the forecast bands from the weather state scores 0.037 points better than the current Primary demand bands over the scored year, and 0.034 better in the year before, but the gain comes almost entirely from one substation (D10) and the new bands are worse on sunny daylight half-hours. The band head is a candidate, not an adoption; the next test should keep the current bands on sunny daylight half-hours or train the head on the quantile score directly.
00310 metered-farm-long-lead-scoring metered-solar-farms partly supported The assembled solar-farm forecast still beats the seasonal-average forecast (climatology) at every lead up to 10 days, by 39% at day 1 and 10% at days 8 to 10, so NGED can use it across its whole horizon of 1 to 10 days ahead. A plant-fed XGBoost model loses its skill by day 8, so a simple first production forecast should be the mixture-density neural network on the plant, which is the best single family at every lead, and the blend of the GAM, MDN, and XGBoost adds more than 1% only at days 1 and 2.
00300 metered-farm-best-guess-assembly metered-solar-farms partly supported The assembled solar-farm forecast (a physical power model, two correction stages, and a blend with XGBoost) scores 3.3% better than plant-fed XGBoost (XGBoost on the plant forecast, its ensemble spread, the unit mean, and the availability ratio, with no weather features of its own) on the quantile score, which averages the pinball loss over 11 quantile levels, when each farm trains on its own history. It is 43% better at the 0.99 quantile, which bounds how much output NGED may need to curtail. With no history for a farm, the gain over XGBoost is smaller (2.0%) and below the 3% we predicted, so the physical model helps less than we expected when a farm's history is missing.
00287 metered-farm-spread-binned-plant-heads metered-solar-farms supported On top of the spread-binned solar-farm power model, an additive model gives a better forecast band than a neural network that adjusts each quantile, so the explainable model is the candidate. A neural network that outputs a three-component Gaussian mixture scores 1.3% better than the additive model in own history, mostly in the upper tail, but not clearly in zero history, so it needs one more test before it is preferred. A blend of the additive model, a neural network, and XGBoost scores 0.8% better than XGBoost alone and fixes XGBoost's weak 0.99 quantile.
00285 wind-effective-wind-adapters generator-forecasting partly supported The physics model of a wind farm matched XGBoost once a small neural network adjusted the wind speed it takes as input, so XGBoost's lead can be reproduced by a smooth correction of wind speed. On the ensemble weather, mapping each member's wind speed to ERA5's helped by 0.31 points but fixed only a small part of the ensemble's low bias, and a lookup table of power offsets by lead day helped more.
00270 metered-farm-xgboost-vs-plant-diagnostic metered-solar-farms partly supported XGBoost beats the solar-farm power model by about 3% in the quantile score even when it sees only the power model's own weather inputs, and the gain is in the width of the forecast band (the 0.05 and 0.95 quantiles), not in the median forecast. The same gain comes from giving the power model a forecast band that depends on the ensemble spread or the clear-sky index, and that band also beats XGBoost at the 0.99 quantile. Lagged power, calendar columns, and extra weather variables add little in own history.
00262 wind-aifs-members-and-climatology-by-lead generator-forecasting partly supported Adding the AIFS ensemble's wind forecasts to the ECMWF ensemble's improves the wind forecast at 3 to 6 days ahead, by about 5% of the quantile score, but does not improve the high-power tail that matters for curtailment. Climatology and a small neural network that picks the mixing weights add nothing once the table of offsets has corrected the common low bias. The AIFS arms also need testing on a second year, because one year of Tuesday runs sets the weights.
00260 metered-farm-quantile-combination metered-solar-farms partly supported Blending the three stored forecasters with a separate weight at each quantile level scores 2.1% better than H5 alone, and every one of the six farms improves, because the neural network without the plant is best at the top quantile and H5 is best at the bottom one. About a quarter of the gain comes from averaging the five seeds, not from the weights. The rolling adjustment of the outer quantiles corrects the coverage of both tails but raises the error by 1.2%.
00259 wind-plant-plus-quantile-offset-head generator-forecasting partly supported A correction learned from 15 months of daily ensemble forecasts lowered the wind forecast's quantile score by 4.5% to 5.5% on the three metered wind farms, but the neural network and the ensemble's wind summaries were not what earned it. The 51 ensemble members' median sits too low: the observed power was below the physics core's median on 34% of rows, not 50%. A lookup table with one offset per lead day, quantile, and farm recovers 92% of the gain. The open question is where that low bias comes from.
00257 metered-farm-fair-plant-arm metered-solar-farms supported The neural network was not needed. Giving the solar-farm power model the lagged power and the spread of the ensemble members, through a plain linear fit, recovers about 90% of the neural network's gain and scores within half a percent of it. The linear fit is no worse at the 0.99 quantile (the 99% interval of the gap includes zero) and is simpler to serve, so it is the metered-farm candidate.
00256 wind-temperature-density-shear-columns generator-forecasting partly supported Giving the XGBoost wind forecast the air temperature helps a little, about 0.7% of the score, and the evidence just misses the bar we set beforehand. Columns that carry the time of day and the season help more, and their gain sits in daytime and summer rows. That pattern is not explained by the air density we could test (a function of temperature alone) or by shear, and it is consistent with a correction for biases in the ensemble wind at certain hours and seasons. A repeat on other months would show whether the time-of-day and season columns still help.
00251 dp-final-best-guess-assembly-clip-and-coast-against-bg2 production-reference-and-best-guess supported Over the one year we can score, the full best-guess demand forecast (arm D, a small neural network that corrects its median, and a two-number summary of the ensemble's solar response) beats BG2 by 0.136 points of quantile score (5%; lower is better) and the manual heuristic by 0.818 points, and it beats the manual heuristic on every headline metric we scored, including curtailment cost. The gain over BG2 is a winter gain: the stack is level with BG2 in summer and worse in April to June, and at D5 on sunny hours it is clearly worse than BG2. Adding the 60% solar clip at D5 helps only when the neural-network head is left out at D5, so the next work is the solar term at D5, not another stacked part.
00243 metered-farm-architecture-comparison metered-solar-farms partly supported A small neural network sitting on top of a solar-farm power model forecasts a metered solar farm about as well as XGBoost and improves on the power model alone by 4.4% in the quantile score, when the network outputs the whole forecast band. Networks that adjust a single number (the scale, a correction, or the sunlight fed to the power model) gain under 1%. The hybrid is far better than XGBoost at the upper tail (0.99 quantile, 29% lower pinball loss), which matters for curtailment. The neural network without physics does better still at that tail (32% lower pinball loss than XGBoost).
00240 no-xgboost-forecast-sprinkle-part-2 production-reference-and-best-guess partly supported For solar farms, the best way to add XGBoost to the physical plant forecast is to let XGBoost predict the spread around the plant's mean, not to correct the mean: that forecast scores 4.3% better than the plant alone and 1.6% better than full XGBoost, and wins at the upper tail that matters for curtailment. Correcting the mean with XGBoost made the quantile score 0.18 points worse, about as much as a model trained on shuffled residuals. For wind, experiment 230's XGBoost stage on the physics core closes 71% of the gap to full XGBoost at 12 to 36 hours, and 44% over all leads.
00237 dp-two-number-solar-summary-with-clip differentiable-physics-embedded-generation supported At D5, in one year of data, summarising the ensemble weather members as a mean and a standard deviation keeps at least about 90% of its gain when the solar output is clipped at 60% of the solar capacity held for D5 (110%, interval 91% to 141%). On the sunniest third of daylight rows the point estimate of the remaining gain is about half (interval -12% to 88%). A post hoc arm using the standard deviation alone kept the gain there, which suggests testing that form.
00234 no-xgboost-forecast-audit-part-1 production-reference-and-best-guess partly supported An XGBoost-free forecast of Primary substation demand scores 7.5% better than BG2, the XGBoost best guess, at 12 to 36 hours ahead, on one year and six substations. The XGBoost-free solar and wind forecasts score 2.8% and 5.0% worse, and lose at the 0.01, 0.05, and 0.95 quantiles. Only the solar plant model's 0.99 quantile beats XGBoost, and that is the tail that matters for curtailment. The next experiment tests small XGBoost components inside the XGBoost-free solar and wind forecasts.
00228 dp-bg-rolling-origin-stack-against-bg2 production-reference-and-best-guess partly supported Over the one year we can score, the full best-guess stack (arm D, the neural network head, and the solar summary) beats BG2 by 0.136 points of quantile score (5% of BG2's 2.715; lower is better), and most of the gain comes from November to March. From April to June 2026 the stack is worse than BG2, and arm D without the neural network head is worse than BG2 in summer. The year-long gain is only just separable from zero, and we cannot test other years until BG2 is re-run for them.
00227 lsc-fewer-false-alarms-and-softer-use-part-2 lsc-detectors partly supported Keeping only the learned score's most confident alarms gives a mask that finds 21 of 35 real load shifts with 5 alarms per match, recovers most of the forecast gain after a load shift, and does no measurable harm on ordinary days. A gentler use of the alarm is not needed at that setting. The result rests on 11 load shifts after which the gain is measured, so the next step is to confirm it on more load shifts before porting it.
00220 dp-bg-stage4-assembly-with-drop-one differentiable-physics-forecast-engine partly supported Adding two parts to the best-guess demand forecast lowers its quantile score by 0.136 points (5%) of substation effective capacity. The parts are a small neural network that corrects the forecast's median, and a two-number summary of the 51 ensemble members' solar response. Each part's gain is almost exactly what it achieves alone, so the two parts remove different errors. The network's own 11 quantiles reached 0.904 coverage of the 95% quantile on cold nights without the hand-fitted band fixes, and scored 0.048 points better than the corrected forecast overall, mostly at one substation. The coast-and-widen rule for missing readings was not scored with the stack.
00214 dp-pv-clip-ceiling-as-of-and-screen differentiable-physics-embedded-generation partly supported Capping the solar output that the demand forecast subtracts from one Primary substation (D5) at 60% of its solar capacity lowers the forecast error on sunny days by about 0.13 MW. Almost all of that gain (97%) remains when the cap is chosen using only data from before each monthly retraining. The five other Primaries tested show no clear gain, so the cap should be treated as specific to D5 for now.
00211 dp-boosted-tree-calibrator-and-input-ablations differentiable-physics-forecast-engine supported A gradient-boosted tree model given the same inputs as the neural network, and no recent errors, lowers the error score of our best day-ahead forecast by about as much as the network does (0.098 against 0.100 points). The interval is wide, so a loss of up to 0.03 points against the network is not ruled out. The gain comes mostly from two inputs, the forecast level and the weather-data era.
00205 dp-gamk-gap-robustness-coast-or-frozen-gam differentiable-physics-forecast-engine inconclusive When readings stop for 3, 6, 24, or 72 hours, keep running the Kalman filter, which corrects the convex demand model using its recent forecast errors, without feeding it new readings, and widen the quantile bands. At each of those four gap lengths that forecast scored 0.25 to 0.44 points of quantile score better than the convex demand model with no correction (a quantile score is an error measure in percent of the substation's effective capacity, and lower is better; the gain is 8% to 14% of the uncorrected model's score). At 72 hours the 99.4% bootstrap interval for the gain runs from 0.04 to 0.48 points, so the gain may be small. The data cannot show whether dropping only the filter beats coasting after a gap of about 72 hours: at every gap length the difference between the two forecasts is smaller than the variation between months. The forecast with the filter, winter night offset, and recipe R was still 0.025 points worse than its no-gap score 48 hours after a 24-hour gap ended. The filter alone was within 0.02 points of its no-gap score 24 hours after the gap ended.
00202 dp-ens-two-number-rolling-replay differentiable-physics-forecast-engine supported A weather layer that stores two numbers per half-hour (the mean and spread of the 51 ensemble members' solar-generation estimates) and refits its two scaling factors every month, using only runs that had finished before the month began, lowers the daylight quantile score (the score for the forecast's uncertainty bands) by 2.3% at five substations over one year, which is as much as running all 51 members. This confirms experiment 193's result under production-style timing, so the next step is to build the two columns into the weather layer.
00195 switching-neural-plus-detector-features lsc-detectors partly supported A learned score on the features that RO-NC and the pair alarm already compute finds 22 to 24 of the 35 real transfers over 1 day, against 18 or 19 for RO-NC, at the same number of unexplained alarms. A plain logistic regression finds 22 even without any look-ahead feature, so most of the gain is in the features, and the extra transfers are small ones (under 1 MW each). The next step is to test whether the extra alarms help the gated mask, which this experiment did not run.
00193 dp-ens-two-number-solar-summary differentiable-physics-forecast-engine supported Summarising the 51 ensemble weather members' solar estimates by their mean and standard deviation lowers the daylight quantile score by about 2% (98% of the gain from running all 51 members, 93% in the replicate months), at the five substations scored over one year. The weather layer could store two columns per half-hour instead of 51, and the 10th and 90th percentiles add nothing measurable. A follow-up experiment could test the two numbers in the live weather layer.
00187 dp-ens-members-in-stack differentiable-physics-forecast-engine supported Running each of the 51 ensemble weather members through the solar part of the demand model still improves the daylight forecast bands by about 2% on top of the full best-guess stack, and night is untouched. About two thirds of the gain needs that day's own members, and two numbers per half-hour (the mean and spread of the members' solar responses) carry nearly all of it, so a cheaper two-number version is the next thing to test.
00180 dp-sunny-daylight-fix-d5-solar-shape differentiable-physics-embedded-generation supported D5's net power flattens like an export limit: capping the solar output the forecast subtracts from demand at 60% of the capacity held for D5 cuts the best GAM's sunny-daylight loss to BG2 from 0.20 MW to 0.07 MW, and moves the two placebo substations, D10 and D16, by under 0.01 MW on sunny half-hours. A different panel tilt or azimuth does nothing measurable. D5 is the only one of the six substations whose site table marks an export clip, so the next question is whether the 60% ceiling is stable from year to year.
00167 dp-full-history-gam-24-months-or-all-years-with-drift differentiable-physics-long-history-and-drift supported Adding a drift term (a slow adjustment to the demand level) improved the forecast at 11 of 12 quarterly refits. The quantile score, where lower is better, fell by 0.40 points when training on the last 24 months and by 0.55 points when training on all years back to late 2019. Training on all years instead of 24 months adds only 0.10 points on top of a drift term, and gives no gain without one, so extra years of history help little.
00160 calibration-band-width-fix-on-stage-2-winner calibration-and-uncertainty supported The 90% prediction interval (from the 5% to the 95% quantile) of experiment 155's best forecast was equally wide on lead days 2 to 5, because its stored spread table has only four lead bands. Interpolating the table between the band centres removes the equal days and raises the quantile score by 0.002 points, below the pre-registered ceiling of 0.01, but it narrows the interval at 24 to 72 hours and widens it from 72 hours on, so one substation loses 0.009. It does not fix the cold-night upper tail: the forecast exceeds the 95% quantile on about 15% of cold-night rows, against 5% for a calibrated forecast.
00158 dp-partly-pooled-gam-shapes-heating-and-calendar-curves differentiable-physics-pooling-across-substations partly supported In this test on six Primary substations, fitting the demand model's shared curves (heating, lighting, daily profile, holidays) across all six let a substation with only 3 months of history score 3.01 points instead of 3.64 (lower is better), within 0.17 points of the same substation with 24 months of its own history. The gain was largest in winter and on Christmas rows. For a substation with 24 months of history, pooling changed the score by less than 0.02 points, which is too small to justify making each forecast depend on five other substations' data. The test cannot say whether pooling the heating curve alone would help a 3-month substation.
00155 dp-bg-stage2-assembly-night-offset-and-error-correction differentiable-physics-forecast-engine inconclusive Adding recipe R, which subtracts 0.75 times the mean error of the last 14 days by half-hour, lowers the stage 1 assembly's score by about 0.04 points (1.3%), and the winter night offset adds nothing once R is in. The assembly with R scores 2.754 against BG2's 2.780, which is better by 0.026 points with an interval too wide to rank the two. The pre-registered placebo control failed, so the formal verdict is inconclusive, but the choice of winning arm (R alone) does not depend on it.
00148 switching-end-rule-lookahead-ro-nc lsc-detectors supported On injected transfers, a live service needs only 6 hours of data after the candidate end to fix the end of a switching interval, with no loss of recall against the offline rule. The call then comes a median 30 hours after the true end, and 24 of those hours are the step statistic's own window, so a longer lookahead of 24 or 48 hours only delays the call to 48 or 72 hours without finding more. The 35 real transfers give the same 18 found at every lookahead from 1 hour up. A lookahead of 0 hours is wrong in six transfers in ten, because the end is called about 8 hours early.
00147 causal-live-end-rule-for-ro-nc lsc-detectors supported The end of a switching interval can be called from data available at the time, with no loss of recall against the offline rule, by waiting until the step statistic falls to 0.8 of its running peak. The call comes a median 29 hours after the true end. Ending the interval at the first sign of a return step is wrong in about six transfers in ten, because it fires about 8 hours early. Merging overlapping alarms on one Primary substation cuts the share of transfers reported as 2 or more intervals from 0.57 to 0.03.
00145 differentiable-physics-analog-day-error-correction-on-gam-k differentiable-physics-forecast-engine small gain Averaging the errors of the 20 most similar past days (same day type, within 13 weeks of the same date, last year's days included) beat recipe R (the 0.75-weighted mean error of the last 14 days) by 0.041 points at day D+1, the calendar day after the forecast origin (about 1.7% of R's score of 2.436). That gain is larger than the 0.011 points that the weekday term of experiment 134 gained. Matching on forecast temperature helped by only 0.005 points, and copying the error of the same holiday last year made holidays worse, so the next step is a day-type profile with a season window and no correction copied from an earlier holiday.
00121 differentiable-physics-mh-multi-hinge-heating-curve-and-night-step differentiable-physics-demand-calendar-and-weather inconclusive Adding night-time levels indexed on the UTC clock lowers the day-ahead error on winter nights by about 0.23 points after the recent-error filter, at all 5 substations in both winters, and the forecast then shows the 01:00 jump in demand that the current forecast misses on cold winter nights. Five heating bends instead of one gain only 0.016 points, within noise. The synthetic tests could not show that a step this small can be detected, so the night result needs a test on weather forecasts and an audit before it enters the model.
00111 differentiable-physics-6-ens-forecast-engine-against-bg2 differentiable-physics-forecast-engine partly supported Driven by real ensemble weather forecasts, the physics-based demand model with a filter on its recent errors (GAM-K) scored 2.80 points against 2.78 for the production recipe (BG2) at five Primary substations. The 0.02 difference cannot be told from zero (-0.109 to +0.157), and the comparison is not like for like. GAM-K scored 0.13 points better than lagless XGBoost with the same filter (-0.305 to +0.014, 4 of 5 substations better), so the verdict is partly supported. The filter's gain is the firmly supported part: 0.15 points, 0.39 in the first day, with the sign held in the replicate months. Giving lagged XGBoost the physics model's forecast as a starting point changed nothing measurable. The known-answer gate passed 22 of 22 readings on its second run, after a sign error in the gate's own test data (both runs are kept). The departures were that the XGBoost compute estimate was 20 times too low, so the margin arms refitted monthly; lagless XGBoost for the scored year reused SIMP1's refits and had no spread factor; and the confirmation comparison was not run. GAM-K is a candidate for Flexpectation v2's roughly 1,300 substations on the measured filter gain, with four caveats: five substations over one year, a short XGBoost history, the undecided XGBoost comparison, and a margin that gave no gain.
00102 simp1-simplifying-the-bg2-best-guess-before-production production-reference-and-best-guess partly supported Removing the three recent-power inputs from the best-guess XGBoost forecast makes it worse at Primary substations (by 0.36 points of capacity once one substation's commissioning months are cut, and most of that comes from one substation whose demand level fell in March 2026) and at supply points (by 0.28). The wind farm forecasts lost nothing measurable and the solar farm forecasts little. The blend of 5 models can shrink to 3, and the solar offset lowers the score by 0.18 points at midday in summer at the 5 substations that use it.
00094 explicit-heating-offset-primary-demand demand-inputs small gain The heating term lowered the quantile score by 0.143 points of capacity in cold hours at leads up to 48 hours (98.75% interval -0.203 to -0.034), under the 0.3 threshold, and by 0.022 over all rows (-0.042 to +0.000).
00087 bg2-best-guess-assembled-with-drop-one-ablations production-reference-and-best-guess partly supported BG2 beat the previous best forecast at the 16 Primary substations by 0.283 points (98.75% interval -0.425 to -0.146) and at the 4 bulk and grid supply points by 0.634 (-1.111 to -0.279).
00082 sw2-18-19-combined-refit-offset-and-same-sign-count lsc-detectors supported Using the refit offset and the same-sign neighbour count together found 0.770 of the moved power on injected transfers, against 0.656 for the benchmark detector. The gain of 0.114 is about the sum of the gains of 0.056 and 0.050 from each change alone. On the 35 labelled real transfers of more than 1 day the combined detector found 12 and the benchmark 8, which is a weak signal on a small sample, and transfers lasting 15.6 to 60 days gain nothing.
00081 r11-control-member-second-validation-year-demand retraining-and-training-history supported C1, weekly minus the stack: -0.240 points (99.17% interval -0.360 to -0.145).
00078 pvdis-5-current-capacity-vintages solar-disaggregation small gain Re-estimating the solar capacity before each forecast run lowered the error of experiment 64's best forecast by only 0.018 points of capacity (98.75% interval -0.036 to -0.004), under the 0.05 threshold, and by 0.033 (-0.086 to +0.012, an interval that includes zero) at leads under 48 hours.
00071 sw2-18-refit-offset-and-gap-bridging lsc-detectors supported The detector ignored every alarm within 24 hours of the monthly refit of its forecasting model, which lowered recall by about 5% of the moved power. Lifting that rule recovers the loss, and the second residual model added to replace the ignored alarms adds nothing on injected transfers.
00070 sw2-19-same-sign-neighbour-count lsc-detectors supported Counting the neighbours that step the same way as the alarming substation found 0.048 more of the moved power on injected transfers than the single-series detector, at the same number of unexplained alarms. The 35 labelled real transfers are too few to confirm the gain (7 found, against 8 for the benchmark), and the gain does not extend to transfers lasting 15.6 to 60 days.
00064 pvdis-5-net-demand-as-demand-minus-forecast-solar solar-disaggregation supported Giving the XGBoost model the forecast solar output as a starting offset, with a weight fitted on the training window, beat the best guess d0 by 0.152 points of capacity (98.75% interval -0.199 to -0.111; leads under 48 hours -0.272, -0.420 to -0.145), and each of the four arms beat its own placebo twin.
00049 vp1-screening-reference-under-p-roll-demand retraining-and-training-history supported B1, the blend minus its best member: -0.076 points (99.17% interval -0.135 to -0.023).
00048 weekly-retraining-confirmation-and-extension-demand-generators retraining-and-training-history supported C1, weekly retraining minus the stack: -0.20 points (99.44% interval -0.40 to -0.035).
00035 lead-weighted-quantile-blend-of-stored-models calibration-and-uncertainty supported Averaging the quantiles of three stored demand models lowered the Primary demand error by 0.10 points of capacity against the best single model (98.33% interval -0.19 to -0.03), so the estimate clears the 0.05 threshold for moving it into production. For solar and wind the gains were too small to resolve on 6 solar farms and 3 wind farms.
00034 retraining-cadence-and-cheap-proxies-demand retraining-and-training-history inconclusive C1, monthly retraining minus the stack: -0.20 points (99.375% interval -0.44 to +0.08, 95% interval -0.38 to -0.01).
00033 best-guess-architecture-vs-production production-reference-and-best-guess supported The best guess lowered the quantile score below production's by 4.89 points of capacity for Primary demand (98.33% interval -5.66 to -4.06), 3.81 for solar (-5.07 to -2.64), and 6.27 for wind (-7.65 to -5.28), each wholly below its threshold.
00030 lagged-power-feature-sets-primary-demand demand-inputs inconclusive C3, all 13 lags minus N1 (seed 0 only): -0.01 points of capacity on the quantile score (99.375% interval -0.45 to +0.43).
00024 pooling-groupings-demand-xgboost hierarchy-and-pooling supported Pooling the 16 Primary substations into 3 or 4 XGBoost models (with the series identifier) scored 0.15 points of capacity worse than 20 per-series models (99.44% interval +0.03 to +0.26), inside the 0.30 margin; one model for all 20 series was 0.29 worse, and dropping the identifier cost 0.8 to 1.0.
00020 solar-wind-stack-cams-stage0-spread-margin calibration-and-uncertainty partly supported For wind, a starting guess from the turbine power curve lowered the error by 0.055 points of capacity (99.58% interval -0.110 to -0.007), and adding a spread factor made the total 0.071 (-0.133 to -0.014); the spread factor's extra -0.016 (-0.036 to +0.004) is not detectable. For solar, no change helped detectably, and the full stack was 0.035 points higher with an interval that includes zero (-0.075 to +0.116).
00019 primary-demand-stack-vs-manual-heuristic production-reference-and-best-guess supported C1, the full stack minus manual_heuristic, quantile score over all leads: -0.56 points of capacity (99.44% interval -0.93 to -0.26).
00017 training-window-and-retraining-rule-drift-demand retraining-and-training-history supported C9, the 14-day correction against A: -1.52 points (99.44% interval -1.95 to -1.13).
00016 pooled-mixture-calibration-solar-wind calibration-and-uncertainty partly supported For solar, one spread factor brought the 5% to 95% band from covering 94.3% to 95.3% of cases down to 89.5% to 91.8%, with no detectable change in accuracy. For wind, no planned correction fixed the first two days, and a spread factor on two lead groups, added after seeing the results, brought every lead band to 90.7% to 92.5% coverage and has not been tested on another year.
00014 cams-trained-stage1-with-stage0-ens-correction-demand weather-products-and-blending inconclusive C2, stage 0 minus raw ENS for the CAMS-trained arm, Primary quantile score over all leads: -0.067 points of capacity (97.5% interval -0.094 to -0.039), better than the planned -0.03.
00006 stage1a-demand-input-feature-sweep demand-inputs partly supported P1, hour and weekday against the weather block alone, Primary substations: -0.55 (99.4% interval -0.71 to -0.41) on the selection year and -0.46 (-0.61 to -0.30) on the confirmation year, the largest single input but about a third of the predicted size.
00002 stage2-calibrator-and-mixture-vs-ens-quantile calibration-and-uncertainty partly supported Pooling the ERA5-trained solar and wind quantiles over the 51 weather-forecast members beat a model trained on archived weather forecasts: wind by 0.62 points of capacity (95% interval -0.99 to -0.26) and solar by 0.24 (-0.36 to -0.06), with the gain in the first six days. A second calibration model on top did not improve on the pooling and made solar worse by 0.40 points (+0.26 to +0.61). The result rests on 6 solar farms, 3 wind farms, and one validation year.
00001 era5-trained-stage1a-vs-ens-trained weather-products-and-blending supported P1 (revised), s1_memberavg minus ens_trained, all leads: wind -1.70 (-2.33 to -1.08), solar -0.57 (-0.83 to -0.32); material up to day 6, not beyond.

What the experiments taught us so far

  • Retrain the Primary demand model weekly, and do not add a 14-day error correction. Weekly retraining without the correction scored 0.20 points of capacity below the best-guess stack, whose frozen model has the correction (99.44% interval -0.40 to -0.035), and a 7-day correction on the weekly model raised the score by 0.16 (-0.06 to +0.36), unresolved (00048). Experiment 34 found the 14-day correction stops helping once the model is retrained, and hurts at weekly and daily cadence. The 14-day correction (00019, 00021) was measured on a frozen model, and the retraining results supersede it for Primary demand.
  • The 14-day correction on a frozen model is a historical result. It lowered the quantile score of N1, a model frozen at its training end, by 0.99 points (00019). At 4 bulk and grid supply point series it did not lower the score (00021), and at the supply points in experiment 87 adding it raised the score by 0.305 points.
  • A weather product as accurate as ERA5 would lower the Primary demand score by 1.9%. ERA5 weather in place of ENS weather lowered the quantile score by 0.10 points, nearly all from shortwave radiation and none up to 2 days ahead (00022).
  • Check for calendar memorisation. When the same valid time repeats across many ENS runs, a model with calendar columns can memorise it. Give each experiment a climatology row, and group folds by valid time.
  • Choose a row set that covers every weekday. The 53 Tuesday runs of 00001 and 00002 put every case of the first day on a Tuesday or Wednesday, which does not suit demand.
  • Training on ERA5 weather helps solar and wind, but a longer history does not add to it, and for demand a long history hurt. The solar and wind gain appeared even when the ERA5-trained model saw only the ENS-era months, so the gain comes from the weather it was trained on and not from the years. For demand, training from 2019 was worse than training from 2023, probably because the customer mix and embedded generation drifted.
  • Put recent power in the stage 2 calibrator, and calibrate before judging it. For Primary substations, the calibrator with recent-origin inputs beat a stage 1b correction, and a lag-free calibrator is worth about 1 point of capacity at short leads (00005).
  • The term and return-to-work columns showed no detectable gain on top of the Christmas column for Primary demand. Over the three validation years, the return and term columns moved the Primary error by +0.06 points from 2 to 20 January (99.2% interval -0.17 to +0.30), +0.20 at Easter (-0.19 to +0.59), and -0.29 over the half terms (-0.68 to +0.07), and Both was not 0.2 points better than a calendar shifted by 17 days (00011). The arm with the full history and the COVID indicator scored 0.26 points higher on the whole year than the base arm.
  • A change that helps the point forecast of solar or wind need not help the pooled mixture's CRPS. CAMS-trained solar lowered the member-average error by 0.14 points (00012) and the mixture's CRPS by 0.009 (00020), and a stage 0 mapping widened the member spread by a quarter. A power-curve base margin lowered the wind CRPS by 0.055 (00020).
  • Attach the forecast solar term to the XGBoost model as a base margin, with a weight fitted on the training window and zero beyond 10 days. The base-margin arm lowered the Primary demand quantile score by 0.152 points (98.75% interval -0.199 to -0.111) against the best guess d0 and 0.256 against d1l, where the fixed-coefficient shift of 00046 lowered d1l by 0.056, and subtracting the forecast solar from a model trained on gross demand was not shown to match the best guess (00064).
  • Do not expect a zero start to help the production XGBoost models. Experiment 78 found that a zero base margin lowered its reference model d0 by 0.071 points, but on the production recipe the zero start changed the Primary score by -0.017 (-0.072 to +0.054) and the solar score by -0.010 (-0.036 to +0.028), and worsened wind models without a power-curve margin by 0.049 (00080).
  • Refresh the capacity table that feeds the base margin once per season-year. A table refreshed that often captured almost all of the gain from a current capacity (00078).