Calibration and uncertainty: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00375 meta-model-over-worker-forecasts-at-all-primaries | The XGBoost meta-model of experiment 373, which weights BG2's five members, the GAM stack, and BG2 by context, scores better than the fixed 50/50 blend of the GAM stack and BG2 across the 16 Primaries, so one meta-model architecture works for any substation. | not supported | not-for-production | A second-level XGBoost model that weights the stored worker forecasts by context scored worse than a fixed 50/50 blend of the GAM stack and BG2 across 14 Primaries (0.107 points of capacity higher; the blend is lower at 10 of 14). One meta-model architecture therefore does not work for every substation as a replacement for the blend; its one clear gain is in sunny April-to-June half-hours. The data cover one scored year and leads up to 168 hours. | primary |
| 00373 d8-meta-model-weights-worker-forecasts | At D8 (Marsh Lane), an XGBoost meta-model that sees the worker forecasts' quantiles, recent power, lead time, calendar, and each worker's recent error can weight the workers by context, and so scores better than the best single family chosen out of sample. | not supported | not-for-production | A second-level XGBoost model that weights BG2's five members and the GAM stack by context scored no better than a fixed 50/50 blend at substation D8 (0.03 points of capacity lower, an interval that includes zero), so serving every family for every substation is not justified on this evidence. It scored 0.47 points lower on holidays and the Christmas period (a post hoc slice of 18 dates), but it kept most of that gain without calendar inputs, so the cause is unknown. This is one substation and 9 scored months. | primary |
| 00369 d8-cross-site-step-rate-against-stack-gap | The best-guess stack trails BG2 most at the Primaries that show the most generator-like steps, so D8's gap is part of a pattern across Primaries and not only a property of D8's residual. If the hypothesis holds, a step detector's rate is a cheap flag for the Primaries where the stack's uncertainty bands need widening. | inconclusive | not-for-production | The test was inconclusive. A simple step counter at a fixed 1.5 MW does not flag the Primaries where the best-guess forecast trails the production-style forecast, but a counter scaled to each site's capacity does correlate with the gap (post hoc, 14 sites). Shuffled copies of the readings show as many steps as the real ones either way, so the counter mostly measures how volatile demand is, and a step-based explanation needs a detector that survives its placebo. | also relevant |
| 00368 d8-conditional-band-head-on-gam-stack-residuals | One band head fitted on every Primary's as-of residuals of the GAM stack, and conditioned on lead, hour, weather, the site, and the stack's recent residuals, closes at least half of D8's gap to BG2 in quantile score, because it widens D8's band where D8's errors are large and does not shrink the other Primaries' bands at long lead the way experiment 354's D8-only band did. | not supported | not-for-production | A single band head fitted on every Primary's own residuals improves D8's forecast band by only 0.18 points of capacity beyond experiment 354's D8-only band, which is 22% of the remaining gap to BG2, so band work alone cannot close D8's gap. The remaining gap is mostly in the median forecast. A small mixture density network head is harmless at the other Primaries (0.08 points better), but the mixture head is no better at D8. | primary |
| 00367 d8-kalman-transient-off-and-adaptive-short-lead | The Kalman transient adds error at D8 at leads under 24 hours (experiment 364: +0.444 points). A transient whose gain is scaled by the substation's own trailing out-of-sample benefit, with no per-site switch, recovers much of that loss without hurting D2 or D4. | partly supported | not-for-production | The Kalman filter's short-lived correction costs D8 about 0.4 points of quantile score at leads under 24 hours, and turning the correction off recovers that, but the same step hurts D4, where the correction helps. The simple rule we tried, which scales the correction by each substation's own recent benefit, recovers 0.31 points at D8 and still hurts D4, so it is not safe as it stands. The next step is a more responsive gain rule, tested with the same placebos. | also relevant |
| 00364 d8-stage-blame-and-analogue-recent-power | At D8 (Marsh Lane), one stage of the GAM stack (the base GAM, the Kalman filter's level or transient, the winter night offset, recipe R, or the own-residual band) adds error that BG2's members do not carry, and the analogue member's lead over the GAM stack comes from its use of recent power. | partly supported | not-for-production | One part of the stack does hurt D8's short-range forecast: the Kalman filter's short-lived correction adds 0.44 points of quantile score at leads under 24 hours and does not help beyond. Over the full 1-to-10-day horizon the three correction stages after the Kalman level add only 0.19 of the stack's 0.745-point gap to BG2, so most of the gap sits in the plain GAM or the way it is fitted. BG2's analogue member (a forecast built from the demand on similar past days) owes about a third of its lead to recent power. The next step is to test the short-range Kalman transient at D8, and to compare the XGBoost members' inputs with the GAM's. | also relevant |
| 00363 d8-segment-level-term-in-the-gam-stack | D8 (Marsh Lane) is a Primary substation whose demand carries on/off steps of about 4 MW and multi-day drops of about 2.7 MW. Experiment 361 removed or masked the detected steps in the training target and the score did not move. Giving the stack one penalised level term for each stretch between detected steps, fitted jointly with the weather and calendar terms, lets those terms be estimated within stretches, so D8's quantile score falls against experiment 354's best arm. | not supported | not-for-production | Giving the forecast for D8 a separate level for each stretch between detected steps makes the forecast worse, not better, and also worsens the forecasts at two Primaries that have no engine. The detected steps do not carry a level that a monthly refit can extrapolate. One untested option is to estimate the generator's state at forecast time instead of fitting its past levels. | also relevant |
| 00361 d8-step-catalogue-detector-injection-and-gam-stack | D8 (Marsh Lane) is a Primary substation whose demand carries on/off steps of about 4 MW and multi-day drops of about 2.7 MW that look like a dispatchable generator. A plain, as-of step detector finds such components, and a GAM stack fitted on a training target with the detected steps removed scores better at D8 than the same stack fitted on the raw target, with the removed component returning through the own-residual band as the chance that the generator is running. | not supported | not-for-production | A plain step detector finds injected 3 MW and 4 MW generator-like spells most of the time and 2 MW spells about half the time. Removing the steps it finds from D8's training target does not improve D8's forecast, so this test gives no support for building a particle filter that separates the generator. A detector that rejects ordinary demand swings, and a longer record of the generator's own output, would be the next things to try. | also relevant |
| 00359 d8-gam-bg2-quantile-blend | At D8 (Marsh Lane), the GAM stack carries information that BG2 lacks, so averaging the two forecasts' quantiles scores better than BG2 alone. | not supported | not-for-production | At D8 (a substation whose BG2 forecast beats the GAM stack, a demand model fitted to the substation's history), averaging the two forecasts' quantiles did not beat BG2 alone on the last 6 months of the year. The 50-50 blend scored 0.04 points of capacity above BG2, an interval that includes zero, and a 25% weight on the stack lowered the score by 0.06 points, also within noise. At two of three other substations without a gas engine, the 50-50 blend scored 0.10 to 0.12 points below the stack alone. | also relevant |
| 00356 wind-default-2-blend-and-monthly-refit-retest | Wind farms research default 2 (experiment 347's arm y2) gains 0.234 points over Wind farms research default 1 at true days 1 to 10 but puts 2.28% of observations above q99 over every lead (2.15% at true days 1 to 10). Either a per-quantile, per-lead-band blend of the two defaults (u2), experiment 348's arm z5 (u1), a monthly offset layer on default 2 (u4), or a monthly retrained head (u3) lowers the 0.99 pinball loss and the share above q99 without losing the gain. | not supported | not-for-production | Blending Wind farms research default 2 with research default 1, or refitting it every month, does not fix its upper tail: the share of observations above the 0.99 quantile only falls from 2.15% to between 1.77% and 2.04%, against 1% intended, and no arm clearly beats default 2 over days 1 to 10. The blend's small gain (0.05 points) is the same with shuffled weights, so it is averaging, not learning. The top quantile needs new information, not a blend or a refit; the monthly retrained head is the one arm worth a longer retest. | also relevant |
| 00355 d8-weekly-refit-and-median-band-swap | (H1) Refitting D8's stack weekly instead of monthly is not the main reason BG2 beats the stack at D8. (H2) Most of the stack's remaining gap to BG2 sits in the band, not the median. | partly supported | not-for-production | Refitting D8's forecast every week instead of every month lowers its score by only 0.045 points of capacity, so BG2's weekly refit is not why it beats the stack at D8. The remaining 0.79-point gap is not mostly the band or mostly the median: BG2's band on the stack's median closes 14% of it and BG2's median under the stack's band closes 27%. The next step is to look at how BG2's median and spread are fitted together rather than at either part alone. | also relevant |
| 00354 d8-own-residual-bands-and-longer-fit-history | (Fix 1) At D8 (Marsh Lane), a forecast band built from the quantiles of D8's own past residuals scores much better than the band the stack borrows from six other Primaries, and costs little at Primaries with no gas engine. (Fix 3) A GAM fitted on D8's history since 2023-09, with a free level step per hour band at 2025-04-01, scores better than the GAM fitted from 2025-04 only, and the step is needed. | partly supported | not-for-production | A forecast band built from D8's own recent errors lowers D8's score by 2.3 points of capacity and closes about 60% of the gap to BG2, with no change to the median forecast. The same rule raises the score by 1% to 6% at three Primaries that have no engine, so it should apply to D8 only until a version that widens with lead time exists. A longer fit history helps only in the first three scored months, when the stored fits had 3 to 5 months of data, and a level step for the April 2025 change adds nothing measurable. | also relevant |
| 00348 wind-plant-mdn-upper-tail | Experiment 347's best arm (y2: Stage 3 frozen physics plant, 51 ensemble members through it, a three-component Gaussian mixture head on the summary of the resulting power values) beats Wind farms research default 1 by 0.234 points over true days 1 to 10 but is over-confident in the upper tail (2.3% of observations above q99 over every lead, and a 0.99 pinball loss up to 0.084 worse in experiment 347's report). One of five tail fixes (z1 tail-weighted loss, z2 capped quantiles with a point mass at the plant's scale, z3 monotone increments above q90, z4 z1 plus z2, z5 the solar recipe of a plant residual band blended with the default's XGBoost quantiles) brings the share above q99 under 1.5% and the 0.99 pinball loss to within the tolerance, while keeping the headline gain. | partly supported | candidate | The frozen physics plant with a mixture head stays about 0.23 points of capacity better than the research default over days 1 to 10, but none of the four tail fixes inside the head brings the top quantile into calibration without losing score at that quantile, and none of their intervals excludes zero. Blending the plant's residual band with the default's XGBoost quantiles gains less (0.15 points) but is the arm of the five whose gain is clear and whose 0.99 quantile is no worse, so that recipe, not a better head, is the next step. | also relevant |
| 00347 wind-physics-hybrid-plant-then-mdn | A physics plant model of each metered wind farm (a smooth cubic speed-to-power curve with cut-in, rated, and cut-out speeds, a direction-dependent loss, and priors that make the untrained model already sensible), fitted on ERA5 before 2024-04-01 and frozen, turns each of the 51 ensemble members into a power value. A three-component Gaussian mixture-density network (MDN) fed the summary of those 51 values, the lead, and the farm then scores within 0.15 points of Wind farms research default 1 on the kept rows at 1 to 10 days, without a neural adapter. A second plant fitted after masking the blocks where the farm fell short of the first plant (a healthy-farm plant) lets the head learn the shortfall. | partly supported | candidate | A physics model of each wind farm, run on every ensemble member and corrected by a small neural network, forecasts 0.27 points of capacity (2.4%) better than the research default over leads of 0.5 hours to 15 days, and 0.23 points over true days 1 to 10, but the neural network given only the plant's median and the farm already gets 83% of the gain, and the upper tail is still over-confident. The next step is to fix the upper tail (the neural network caps its quantiles at capacity, and 0.9% of observations exceed it) before any plant-based arm is adopted. | also relevant |
| 00346 wind-research-default-1-offset-table-refit-cadence | Experiment 342 found that the default's offset table, fitted once on the 442 daily runs of 2024-04-01 to 2025-06-16, leaves the upper tail too low in the scored year (observed power above q99 in 1.8% of half-hours against 1% intended) although it fits the hindcast, and blamed drift between the hindcast years and the scored year. The ensemble table holds daily runs up to 2026-10-24, so the table can be refitted each month on runs that had already finished. A table refitted monthly on a trailing 6-month window (x2) scores lower than the table fitted once. | partly supported | candidate | Refitting the wind forecast's offset table every month on recent runs makes the forecast a little better (a 0.4% to 1.1% lower quantile score), using only data the service already stores. It does not fix the upper quantiles, which still miss in 1.7% of half-hours against 1% intended. So the change between the training years and the scored year is a minor cause, and the upper-tail miss needs another explanation. A monthly job that refits the table on all earlier daily runs passes the adoption conditions and is a candidate. | also relevant |
| 00344 wind-research-default-1-speed-dependent-offsets | Experiment 340 found that a constant upward shift of the wind quantiles gains 0.09 to 0.12 points on the kept rows but worsens the rows below cut-in by 0.51 to 0.78, and that the default's median is too low at 8 to 15 m/s and too high below 5 m/s. An offset that depends on the ensemble-mean 100 m wind speed at forecast time lifts the quantiles where the power curve is steep and leaves them alone where the forecast is near zero, so it keeps the gain without the low-wind harm. The starting point is Wind farms research default 1, the three metered wind farms W1 to W3 are scored, and the lead range is 1 to 10 days. | not supported | not-for-production | Letting the wind forecast's quantile offsets depend on the forecast wind speed gains only about 0.02 points of capacity when fitted on the same retrospective forecasts (hindcast runs) that fit the quantile offset table, and the gain cannot be told from zero. Fitted on the six months just before the scored year (July 2025 to June 2026), the same speed-dependent correction gains 0.146 points and does not worsen the half-hours where the forecast wind speed is below the turbines' cut-in speed. The speed-dependent bias is therefore real, and it changes from year to year. Refitting the correction on recent months is the untested option that could capture it. | also relevant |
| 00342 wind-research-default-1-upper-tail-offsets-and-head | The upper quantiles of Wind farms research default 1 are too low: in experiment 324 the observed power exceeded q99 in 1.8% of half-hours (1% nominal) and q95 in 6.7% (5% nominal). Three corrections fitted on the hindcast can fix that without harming the headline score. x1 is a table of offsets for q90 to q99 that also depends on the ensemble-mean wind speed. x2 is a small boosted-tree head, fed the default's quantiles and summaries of the ensemble members' wind speed, that adds offsets to q90, q95, and q99. x3 lets the share of ensemble members above the farm's saturation speed set a probability mass at the farm's scale. | not supported | not-for-production | The wind forecast's upper quantiles miss too often (observed power exceeds q99 in 1.8% of half-hours against 1% intended), and corrections fitted on past forecasts do not fix that: the offsets keyed by wind speed, the tail head, and the mass at the farm's cap all leave the share at 1.8% to 1.9% and change the upper-tail loss by about 1% or less. A head that sees the ensemble members does no better than one that sees only the default's quantiles, so in these 3 farms the members add no information about the upper tail. The remaining miss is consistent with the scored year differing from the hindcast years, which the next experiment could attack by refitting more often or by training on the most recent months. | also relevant |
| 00340 wind-research-default-1-constant-shift-then-low-wind-drop | Experiment 336 found that dropping the ERA5 training rows below the turbines' cut-in speed (x2) gains 0.117 points on the kept rows, and its audit traced about 85% of the gain to a constant upward shift of the quantiles. If that is right, a constant shift fitted on the 442-run hindcast (arm S) reproduces most of the gain without changing the training rows. Arm S+X2, which also drops the low-wind rows, then adds little beyond the shift. Wind farms research default 1 is the starting point, the three metered wind farms W1 to W3 are scored, and the lead range is 1 to 10 days. | not supported | not-for-production | A single upward shift of the wind quantiles, fitted on past forecasts, does nothing, because the forecast's offset table already fits those runs. Moving the quantiles up by 0.8 to 1.2 points of capacity does reproduce most of the gain of dropping low-wind training rows, but it also worsens the forecast at low winds by 0.5 to 0.8 points, the same harm that stops X2 being adopted. An untested option that could lift the forecast at 8 to 15 m/s and leave it alone below 5 m/s is a correction that depends on wind speed. | also relevant |
| 00339 wind-research-default-1-adapter-plus-mixture-head | A three-component Gaussian mixture head on the hindcast, fed the quantiles of the physics core plus adapter and the ensemble members' summaries of speed, direction, and temperature (experiment 324's W4), lowers the quantile score of Wind farms research default 1 on the kept rows of the three metered wind farms W1 to W3 at 1 to 10 day leads. The head loses on the idle half-hours because it is trained on kept rows only, and the maintainer's view is that a forecast-time rule (predict the idle floor where the ensemble-mean wind speed is below cut-in) needs no idle detector. Three arms are tested: x1 the head as in experiment 324, x2 x1 plus the rule, x3 the head trained on all hindcast half-hours. | partly supported | candidate | A small neural-network head that re-draws the physics-plus-adapter wind forecast's quantiles gains 0.27 points of capacity (2.4%) on the farms' normal half-hours and keeps that gain on the replicate months, but the same head given no weather summaries does as well, so the gain is a recalibration that a simpler correction partly reproduces. A simple rule that predicts the idle floor at low wind recovers none of the head's loss on the idle-band readings, and training the head on those readings removes the gain. | also relevant |
| 00338 wind-adapter-column-without-neural-network | The gain of experiment 326's arm (c), the neural adapter's effective wind speed as an extra XGBoost column, can be kept without a neural network: a lookup table of speed scale by direction sector, speed ratio bin, and temperature bin (x1) keeps at least half of the gain, and a small monotone gradient-boosted tree (x2) keeps at least 80%. | not supported | not-for-production | A lookup table of wind-speed corrections keeps almost none (5%) of the gain that the neural adapter gave the wind forecast, and boosted trees keep about half (0.051 of 0.099 points of capacity, scored at 1 to 10 day leads). Neither passes the adoption rule, so the neural adapter of experiment 326 is the only arm tested that does, and the maintainer decides whether it belongs in the production service. A non-neural column would need to reduce the physics-based wind-power model's error on ERA5 as much as the neural network does. Boosted trees with more inputs might approach that. | also relevant |
| 00337 primary-demand-standard-architecture-168-to-240-hours | Experiment 330 found that the standard architecture (STD: the frozen convex stack's mean, a two-layer input adapter, and a three-component mixture-density output) scores 0.053 points better than Primary demand research default 3 at 0.5 to 168 hours, but National Grid Electricity Distribution (NGED) needs 1 to 10 days. At 168 to 240 hours STD is again within 0.03 points of the default, so demand can share the standard architecture across the full range. | partly supported | candidate | The standard architecture (a neural network that corrects the forecast of the convex stack, with a learned input adapter and a mixture-density output) forecast Primary demand at 7 to 10 days ahead with a lower error than the current research default over the 12 months scored. Most of the gain comes from the mixture-density output and from two Primaries (D10 and D16). The 12 months of data cannot establish the gain, so the standard architecture is a candidate for demand across NGED's 1-to-10-day range, not yet a replacement. | also relevant |
| 00336 wind-research-default-1-below-cut-in-zeroing | The maintainer proposes that, for wind, rows where the wind speed is below the turbines' cut-in speed carry only noise, so zeroing them lowers the quantile score of Wind farms research default 1 on the kept rows of the three metered wind farms W1 to W3 at 1 to 10 day leads. Three readings are tested, each added alone: x1 sets the forecast to zero power whenever the ensemble-mean 100 m speed is below the farm's cut-in speed, x2 drops the ERA5 training rows below cut-in from the XGBoost model and the offset table, and x3 sets the training target to zero on those rows. At most one is adopted, in the order x1, x2, x3. | partly supported | candidate | Setting the wind forecast, or the training target, to zero below the turbines' cut-in speed does not improve the forecast, because the farms still produce small real output there. Leaving the low-wind rows out of training gains 0.12 points of capacity (about 1% of the default's score) on the kept rows, but mostly by lifting the quantiles upward, which a constant upward shift may reproduce (not yet tested). | also relevant |
| 00335 primary-demand-default-3-tail-calibration-at-0-5-to-240-hours | The 0.01, 0.05, 0.95, and 0.99 quantiles of Primary demand research default 3 miss their nominal rates past 96 hours (experiments 317 and 320 found 2.1% to 2.9% of observations below q01 and a 1.8% upper tail at 7 to 10 days). A calibration fitted per 24-hour lead band and per group (season, day or night, cold winter night) on the forecast errors of earlier origins brings the tails to nominal at every lead from 0.5 to 240 hours without worsening the quantile score. | not supported | not-for-production | Primary demand research default 3 is already close to calibrated at 1 to 10 days: its 5% to 95% band covers 89.3% to 90.9% and 1.1% to 1.95% of observations fall below q01. Calibrations fitted on earlier months move the tail shares only part of the way, and none is worth adopting. A table fitted in-sample on the same year would fix the tails and slightly improve the score, which shows the table can describe that year but not that the miss repeats across years, so testing the miss on a second year is the next step. | primary |
| 00334 primary-demand-other-neural-architectures | The maintainer and National Grid Electricity Distribution (NGED) value the explainability of the generalised additive model (GAM) with a Kalman filter that makes up Primary demand research default 3. A different architecture that gains decisively is the only reason to leave it. Four neural networks that replace the whole stack are tested against the default: a small mixture-density network (MDN) on the served inputs (M1), the same network given only an explicit unmetered-solar signal (M2), a gated-recurrent-unit encoder-decoder network with an MDN head (R), and a temporal fusion transformer (T). | not supported | not-for-production | Replacing the generalised additive model and Kalman filter with a neural network did not help. The best neural network (a recurrent network that reads the past week of power) trails the current research default by 0.04 points, the other three neural networks trail it by more, and none beats it, so the explainable model stays. The neural networks score worst on cold winter nights, and only the one given an explicit solar signal scores lower than the default on sunny daylight, by a margin that could be chance. A small neural network on top of the default, as in experiment 330, is a better place for neural parts than a replacement. | also relevant |
| 00330 primary-demand-standard-architecture-arm | The maintainer prefers one standard architecture (a physics core, an optional neural input adapter, an optional mixture-density output head) across demand, wind, and solar forecasts, unless a different architecture gains strongly. Primary demand has the physics core already (the convex stack with a Kalman filter). One small network, trained jointly to correct the median and to give a mixture-density band, fed through a learned two-layer adapter of the ensemble summaries, scores within 0.03 points of Primary demand research default 3, so demand can share the standard architecture at little or no loss. | partly supported | not-for-production | A single neural network with a learned input adapter and a mixture-density output, placed on top of the frozen convex stack, forecasts Primary demand at least as well as the current research default from 0.5 to 168 hours, so demand does not need a different architecture from wind and solar at those leads. The gain comes from the mixture-density output (the median-only version gains nothing), and most of it comes from one Primary, D10. The 7-to-10-day range is untested, and the cold-night and sunny-daylight tails are too uncertain to call the architecture safe. | also relevant |
| 00329 primary-demand-default-3-member-solar-distribution-at-7-to-10-days | At 168 to 240 hours the 51 ensemble members have little skill for individual days, so the forecast centres on average sunshine and the half-hours that turn out sunny have realised demand below the median. The weakness of Primary demand research default 3 on sunny daylight (5.07 against the manual heuristic's 4.82 in experiment 317, for the convex chain before any blend) is therefore a fault of the predictive distribution (the lower tail on realised-sunny half-hours), not of the median. Replacing the two-number Gaussian summary of the members' solar responses with the 51 responses themselves (X1), mixing them with a climatology of the response (X2), or adding the members of the two previous daily runs (X3) fixes the distribution. | not supported | not-for-production | The 7-to-10-day weakness on sunny half-hours no longer exists in the research demand forecast: the two-number solar summary and the daylight-only blend with the manual heuristic already took it from 5.2% worse than the heuristic to 6.6% better, so no new distribution method is needed there. What remains is one coastal Primary (D5), where the forecast is 11% worse than the heuristic, which points back at the solar term and not at the width of the bands. | also relevant |
| 00327 lsc-label-free-detect-mask-train-pipeline | A forecast of the normal running arrangement (NRA) can be trained and corrected on power with the load shifts set to missing, where a label-free detector finds the load shifts. The detector reads only how unusual the observed power is inside the predictive bands of a demand model that uses no lagged power, and no NGED switching label enters any threshold, mask, or training row. | partly supported | candidate | A forecast of the normal running arrangement recovers faster after a load shift when the readings that a label-free detector flags are treated as missing: during and after injected shifts the 95% pinball loss falls from 1.79 to 1.04 points of capacity. On ordinary days the masked forecast is no worse, and slightly better. Without one site (D11) the gain shrinks to 0.30 points, and retraining the demand model on masked power adds nothing beyond masking the filter. | also relevant |
| 00326 wind-research-default-1-change-arms | Each of four changes, added alone to Wind farms research default 1, lowers the quantile score on the kept rows of the three metered wind farms W1 to W3 at 1 to 10 day leads: (a) 2 m temperature and an air-density column as XGBoost inputs, (b) hour of day and day of year as XGBoost inputs, (c) the ERA5-trained neural adapter's effective wind speed as an extra XGBoost column, and (d) one wind-speed scale factor, fitted on the hindcast to the implied wind, applied to every member's 100 m speed before the XGBoost model. At most one is adopted, in the order c, b, a, d. | supported | candidate | Feeding XGBoost the neural adapter's corrected wind speed lowers the wind forecast's score by 0.10 points of capacity, about 0.9% of the default's score, at 1 to 10 day leads (99.75% interval -0.19 to -0.01). The change is negative at every farm and at short and long leads. The three other changes were not separable from zero. Only a port that puts the adapter in the production service can use the gain, so the maintainer decides. | also relevant |
| 00324 wind-farm-solar-cross-breeds-long-run | The solar-farm findings (a mixture-density network (MDN) fed by a physical plant forecast beats the plant alone, lead features help it, and one pooled head across leads matches separate heads per lead band) carry over to the three metered wind farms W1 to W3, when the plant is the physics core with the experiment 285 wind-speed adapter and the weather is the 51-member ensemble. | partly supported | not-for-production | A small neural network that corrects the quantiles of the wind farms' physics-based forecast lowers the quantile score by 0.27 points of capacity against XGBoost plus a lookup table of offsets, on the half-hours when the farm is not idle, at days 1 to 2 and at days 3 to 10. The gain disappears when the idle half-hours are included. The upper tail gets no better. The network needs the farm's own forecast history to learn from. | also relevant |
| 00322 primary-demand-default-2-daylight-only-heuristic-blend-and-sunshine-graft-replication | (Stages 1 and 2) In experiment 320 the heuristic blend's gain at 168 to 240 hours sits on sunny daylight (-0.236 points) and in April to October, while its harm sits on cold winter nights, which are night half-hours. Confining the blend of experiment 319 to half-hours with the sun above 5 degrees (D1) keeps the gain with no harm on cold winter nights by construction, and a night weight that is a function of the share of the 51 ensemble members below 3 degrees C (D3) recovers the nights where the heuristic helps without harming cold ones. (Stage 3, a diagnostic) Experiment 316's sunshine graft, which recovered 0.36 of the depth-1 XGBoost refit's advantage on the worst April-to-June half-hours, replicates in April and June 2025. | partly supported | candidate | At 7 to 10 days the blend with the manual heuristic gains about 4% at all hours, and about 2.4% when it is used only by day, which is the version with no harm on the coldest winter nights. Counting how many of the 51 ensemble members are cold finds 72% of those nights (the ensemble mean finds 41%), and a hard switch on that count keeps the whole 4% with almost no cold-night harm, but that switch was not pre-registered. The sunshine response of the depth-1 XGBoost refit explains about half of the refit's advantage on the worst April-to-June half-hours in both years, so a flexible sunshine term in the convex fit is worth testing. | also relevant |
| 00321 metered-farm-lower-tail-recalibration-and-tail-head | Metered solar farms research default 1 (the blend assembled by experiment 300, scored to 10 days by experiment 310) lets the actual output fall below its 0.01 quantile 3.3% to 3.7% of the time, and no mixture-density head tried in experiment 318 changes that. Default plus a recalibration table, a zero-inflated lower tail, or a tail-targeted head brings the share of actual values strictly below the 0.01 quantile close to 1% at leads of 1 to 10 days, without raising the quantile score and without losing experiment 318's 0.99 gain. | partly supported | candidate | The solar-farm forecast's lowest quantile was already close to correct: only 1.2% to 1.4% of actual values fall strictly below the 0.01 quantile, because the earlier 3.3% counted the 3.0% of half-hours that read exactly zero. A recalibration table, a zero-inflated head, and a tail head each lower the own-history score by 0.04 points or less (the unshrunk table raises it by 0.03 to 0.05 points), so none is adopted. A shrunk recalibration table does lower the score by about 0.05 points for a farm with no history, and two heads lower the 0.99 pinball loss by 3.7% to 8.5% while covering the 0.99 quantile only 98.3% to 98.9% of the time. | also relevant |
| 00320 primary-demand-default-2-night-aware-heuristic-blend-and-night-band-width | Primary demand research default 2 has two weak spots. (1) At 168 to 240 hours a blend with the manual heuristic lowers the score by 4.5%, but a single weight per lead band and season harms cold winter nights, where the convex chain is the better forecast. Default plus B1 (the blend switched off on half-hours forecast to be cold winter nights) or default plus B2 (a chain weight that is a logistic function of the ensemble-mean temperature, per lead band and day or night) keeps the gain without the harm. (2) The prediction bands are too wide at night, most clearly at one Primary (D10, whose band heads gained 0.144 points, 88% of it at night). Default plus N (a width multiplier on the quantile offsets from the median, fitted per Primary and per night or day block from the previous 90 days) removes that excess width at every Primary. | partly supported | candidate | A blend with the manual heuristic still improves the 7 to 10 day Primary demand forecast by about 4%, and the harm it does on the coldest winter nights cannot be removed by switching it off when the weather forecast says the night will be cold, because at 7 to 10 days that forecast spots only between one and two cold nights in three. Narrowing the night prediction bands helps one Primary (D10, by 0.25 points on night half-hours), leaves the other five unchanged or slightly worse, and makes the cold-night tails worse. | also relevant |
| 00319 primary-demand-default-1-solar-summary-and-heuristic-blend-at-7-to-10-days | Primary demand research default 1 has two weak spots at 168 to 240 hours: sunny daylight half-hours, where it scores 5.2% worse than the manual heuristic, and April to October, where it beats climatology by only 4%. Default plus x1 (the two-number solar summary S, the mean and standard deviation of the 51 ensemble members' solar responses, extended from 0.5 to 168 hours out to 240 hours) lowers the sunny-daylight score at 168 to 240 hours. Default plus x2 (a blend of the convex chain's median with the manual heuristic's, with the weight fitted per 24-hour lead band and season group) lowers the pooled score at 168 to 240 hours, mostly in April to October. | partly supported | candidate | The demand forecast for 7 to 10 days ahead improves by about 2% when its solar term also uses the spread between the 51 weather-forecast members, and the gain is largest on sunny days (6.6%), where the forecast had been worse than the manual heuristic that National Grid Electricity Distribution (NGED) staff use. Blending in the manual heuristic improves the forecast by 4.5%, but the blend worsens the coldest winter nights beyond the agreed tolerance, so only the solar change is adopted. | also relevant |
| 00318 metered-farm-mdn-head-more-features | Giving the mixture-density head of Metered solar farms research default 1 the forecast lead (lead in whole days), the hour of day and day of year as sine and cosine pairs, and the ensemble spread of the shortwave irradiance lowers its quantile score, mostly beyond 36 hours and in the lower tail, and one head trained on all lead bands beats one head per band. | partly supported | candidate | Giving the solar-farm head more inputs does not improve its overall score at any lead from 1 to 10 days, so the head's inputs are not what limits the forecast. The extra clock and cloud-spread inputs do lower the loss at the 0.99 quantile (the tail that matters for curtailment) by about 5% at days 3 to 7, and none of the changes fixes the lower tail, where 3.3% to 3.7% of actual values fall below the 0.01 quantile, where 1% is expected. Training one head for all leads makes the network with no plant nearly as good as the plant-based head at days 8 to 10 (0.4% worse), so the plant adds little there. | also relevant |
| 00317 primary-demand-default-1-at-7-to-10-days | The ensemble weather already reaches 10 days, so the convex model of Primary demand research default 1 can be driven to 240 hours, and with a spread table fitted per 24-hour lead band its quantile score stays better than the manual heuristic and climatology through day 9 and ties climatology on day 10. | partly supported | not-for-production | The demand forecast we use for research still has a lower error than a table of typical demand by month and time of day (climatology) at 9 to 10 days ahead, by about 11%, because the temperature forecast keeps some skill there. Its advantage over NGED's manual heuristic shrinks to 5% to 9% beyond 7 days, and the 95% interval of that advantage includes zero. The advantage sits in winter and at two of the six substations. The forecast is useful out to 10 days on cold winter nights and no better than the heuristic on sunny daylight half-hours. | also relevant |
| 00316 primary-demand-night-level-and-single-input-terms | (Stage 0) The gain of the band heads of experiments 311 and 314 at D10, an anonymised Primary substation, is mostly a night-time level error of Primary demand research default 1 and not a better description of the uncertainty. (Stage 1) The April-to-June gap between arm D and BG2 at D5 and D9, two anonymised Primaries, sits mainly in arm D's response to sunshine and not in its recent-power or calendar terms. | inconclusive | not-for-production | The band heads help at one Primary substation mainly by narrowing its night-time bands, not by correcting a level error: a night level offset reproduces only 14% of the gain, with a wide interval. For the April-to-June gap to BG2, a post hoc test points at the sunshine response of the best-guess model (0.36 of the gap recovered on the worst half-hours, but only 0.19 at one of the two Primaries), and a differently fitted surrogate in the audit did not reproduce it, so the ranking of the families is unreliable. | also relevant |
| 00314 primary-demand-gated-or-pinball-band-head | Experiment 311's mixture-density band head lowered the quantile score of Primary demand research default 1 but harmed sunny daylight half-hours, and its gain rested on D10. Keeping the default's bands on sunny daylight half-hours (gate), training the head directly on the 11-level pinball loss (quantile network), or both, removes the harm and leaves a gain that does not depend on one Primary. | partly supported | candidate | Experiment 311's neural band head was stopped by harm on sunny daylight half-hours. Keeping the current bands on those half-hours fixes that harm, and the gated head scores 0.041 points better than the current Primary demand forecast bands (research default 1, lower is better). The gain is still almost entirely one substation, D10: without it the gain is 0.015 points and its interval touches zero, so the gated head is a candidate and not an adoption. Training the head on the quantile score directly gave no improvement. | primary |
| 00312 lsc-how-good-must-a-detector-be-against-the-band-head | A load shift detector that is far from perfect can still beat the best forecast that uses no detector, so a detector need not be near-perfect to be worth building. The one detector-based arm in experiment 297 that beat that forecast was the one that subtracts the true megawatts from the readings during a shift and adds them back to the forecast. This experiment degrades that arm one axis at a time to find where it stops beating the no-detector forecast. It also asks whether the surprise of recent readings against GAM_ONLY's residual bands (band position), or perfect knowledge of a shift, added as inputs to the no-detector quantile head, improves the head. This is question 3 of the LSC decision program, redefined by the Discuss ruling of 2026-10-23. | not supported | not-for-production | A load shift detector does not need to be near-perfect to improve on the best forecast that uses no detector: with the true megawatts added back, finding half the shifts, learning of each up to 3 days late, or misjudging the megawatts by half still lowers curtailment cost against that forecast. The detector tested earlier (RO-NC) finds only a third of the shifts, below the break-even of one half (measured with every other property perfect). The surprise of recent readings against the forecast's bands (band position) found more, 37% of shifts on even months, but still under one half and with about 90% of its alarms false, so the next step is a detector that finds more shifts. | also relevant |
| 00311 primary-demand-mixture-density-band-head | Primary demand research default 1 (the GAM stack A+H+S of experiments 220 and 251) gets its bands from a smooth spread table and a cold-day widening of the upper offsets, around a median shifted by the neural head. A mixture-density neural network, which reads the weather state and may be skewed, gives bands with a lower quantile score, and the gain sits in the protected tail groups. | partly supported | candidate | A neural network that sets the forecast bands from the weather state scores 0.037 points better than the current Primary demand bands over the scored year, and 0.034 better in the year before, but the gain comes almost entirely from one substation (D10) and the new bands are worse on sunny daylight half-hours. The band head is a candidate, not an adoption; the next test should keep the current bands on sunny daylight half-hours or train the head on the quantile score directly. | primary |
| 00304 metered-farm-node-sum-tail | For the solar farms that share a bulk supply point, adding each farm's quantiles overstates the upper tail of the farms' summed output, adding independent bands understates it, and summing the 51 ensemble members' farm forecasts member by member gives calibrated 0.95 and 0.99 quantiles, because the ensemble carries the farms' shared dependence on cloud. | not supported | not-for-production | The band for the combined output of the three farms beneath one substation comes out too wide if the weather forecast's 51 members are summed member by member, because the members' forecasts for the three farms move together almost perfectly while the farms' actual forecast errors are less closely linked. That band is almost the same as the band from adding each farm's quantiles. A band that assumes the farms' errors are unrelated is too narrow. A band from a Gaussian copula with a correlation fitted on the errors of the other 11 months comes closest to the stated probability. Only 5 half-hours in the scored year exceed 0.75 times a substation's maximum export, so this experiment cannot say what the band widths mean for curtailment cost. | also relevant |
| 00303 wind-adapter-in-production-setting | Experiment 285 found that, with ERA5 as the weather, a small neural adapter in front of the physics core of a wind farm scored better than XGBoost. On the 51 ensemble members in the scored year, an adapter fine-tuned on the ensemble hindcast beats XGBoost plus experiment 259's offset table (the benchmark that matters), because the adapter corrects the wind speed itself and the table corrects only the power. | partly supported | not-for-production | A small neural correction of the wind speed, fitted to ERA5 and run on the 51 ensemble members, does about as well as XGBoost with a lookup table of offsets (0.09 points better, too little to justify porting). Fitting the correction on the one year of ensemble hindcasts made the forecast worse, probably because that year differs from the scored year; we did not test the cause. | also relevant |
| 00301 metered-farm-outage-state-band | A small availability state (online, partly down, offline) estimated from the last 6 to 48 hours of a solar farm's power against its plant forecast improves the forecast band's lower tail during and after an outage. | inconclusive | not-for-production | Solar-farm outages are too rare in the scored year to test an availability state: five of the six farms have at most 34 outage half-hours, and the sixth has 165. Too few half-hours remain to separate a 10% gain from chance, so the experiment stopped before fitting anything. If outages matter, the next test needs a longer period, such as the full history of farms A, B, and F. | also relevant |
| 00300 metered-farm-best-guess-assembly | The assembled candidate (spread-binned plant P2, a generalised additive model (GAM) band on the plant's residuals, a mixture-density head (MDN), blended per quantile with XGBoost) scores better than plant-fed XGBoost on the quantile score and at the 0.99 quantile, and its advantage is larger for a farm with no history because the plant carries what the missing history cannot. | partly supported | candidate | The assembled solar-farm forecast (a physical power model, two correction stages, and a blend with XGBoost) scores 3.3% better than plant-fed XGBoost (XGBoost on the plant forecast, its ensemble spread, the unit mean, and the availability ratio, with no weather features of its own) on the quantile score, which averages the pinball loss over 11 quantile levels, when each farm trains on its own history. It is 43% better at the 0.99 quantile, which bounds how much output NGED may need to curtail. With no history for a farm, the gain over XGBoost is smaller (2.0%) and below the 3% we predicted, so the physical model helps less than we expected when a farm's history is missing. | also relevant |
| 00297 lsc-need-a-detector-no-detector-arms-against-the-mask | A forecast that never learns which readings are load shifts can match a forecast that uses a detector, because its bands and its recent-power correction can be made to tell a load shift from ordinary variation by themselves. This is question 2 of the LSC decision program. | not supported | not-for-production | A forecast with no detector that recalibrates its bands and reads how far the recent-power correction has moved scores better than the RO-NC detector near load shifts, so with RO-NC a detector is not shown to be needed. Perfect knowledge with the true megawatts added back still lowers curtailment cost by 6.5% (3.9 to 9.2) against that forecast, so the next step is to run the better LSC detector (LR_b20) on the same simulated shifts and judge it against a forecast that has the recalibrated bands. | also relevant |
| 00294 regime-aware-model-at-d8-engine-state-term | D8's metered power excludes a gas engine's output (experiment 194) while the engine, which runs in spells (experiment 291), takes net load below what the weather and calendar predict. A single model trained across those spells has no way to know which half-hours the engine ran, so its forecast and its bands carry the engine as noise. A model that keeps the calendar, weather, and solar shapes shared across the whole history but adds one regime-specific term, an offset per half-hour the engine runs (non-negative, taking power off net load), forecasts D8 better than the single model. The offset is estimated from the data, and its size is not assumed. | not supported | not-for-production | Telling the forecast when D8's gas engine is running would make it much better when the state is read at the half-hour being forecast, but the detector builds that state from the very reading being forecast. With only the state of the last settled day, the engine term made the forecast 10% worse; trained on that stale state, it gained only 3%, no better than widening the bands from past residuals. A forecast of whether the engine runs, or NGED's own record of its running hours, would be the data to test next. | also relevant |
| 00293 metered-farm-plant-tilt-and-clip | The six metered farms fit best at a 15 to 20 degree tilt (experiment 282), while the spread-binned plant (P2 of experiment 270) assumes 35 degrees and caps its forecast at the 0.995 quantile of the tuning months' output. Replacing the fleet tilt with 20 degrees and the hard cap with a higher, soft cap lowers P2's April to July under-prediction and its 0.99 pinball loss. | not supported | not-for-production | Changing the solar power model's panel tilt from 35 to 20 degrees fixes most of its winter over-prediction on clear days and improves the lowest 1% quantile of the forecast by 4%, but it leaves the overall forecast band and the 99% quantile unchanged and slightly worsens summer. A higher or softer cap on the power does nothing, so the shape of the power model is not what limits the upper tail of the forecast band. A tilt between 20 and 35 degrees, or a tilt that varies with the season, is the next tilt to test. | also relevant |
| 00292 metered-farm-ens-neighbourhood | A day-ahead cloud field is often right in shape and wrong in position by tens of kilometres, so summaries of the ensemble's irradiance over the hexagonal cells around a metered solar farm carry information that the farm's own cell lacks. Giving H5 (the plant plus a neural network that outputs all 11 quantile offsets) those summaries lowers its quantile score at 12 to 36 hour leads. | not supported | not-for-production | At the six metered solar farms tested, the ensemble's irradiance on the cells around a farm adds almost nothing to the day-ahead forecast: the quantile score improves by 0.36%, against the 2% the plan needed, and the gain is not robust to dropping one farm. Adjacent cells, 15 km apart, differ from the farm's own cell by only about 3% of the irradiance, so the ensemble mean is too smooth between them to show where a cloud edge will fall. A weather source of a different kind is the next experiment to run, rather than more processing of the same ensemble. | also relevant |
| 00288 pooled-head-on-public-wind-farms | A neural head that adds 11 quantile offsets to the XGBoost wind forecast (R0) and was trained on dozens of public onshore wind farms (metered output from Elexon's Balancing Mechanism Reporting Service) lowers the quantile score of the three metered wind farms (W1 to W3) against R0 alone, and a head trained on the pool beats the same head trained on W1 to W3 only, because experiment 259's head had too few farm-years to learn a correction that transfers. A second hypothesis: XGBoost keeps improving as the training history grows from 12 to 48 months while the physics core is flat after 24 months, and a pool of public farms replaces the missing history. | not supported | not-for-production | Training the correction on 40 public wind farms did not help the forecasts for the three NGED wind farms. Applied directly, it made them worse, because the public farms' capacity, turbine rating, and hub height let the head memorise each farm. Without those inputs it did no harm, and after a short fine-tune on the three farms' own data it gained about as much as a simple offset table fitted on those farms, so the public farms added nothing measurable. | also relevant |
| 00287 metered-farm-spread-binned-plant-heads | On top of the spread-binned plant (experiment 270's P2), an additive model of the residuals gives a band as good as a neural head with monotone quantile output, so the additive model is the explainable candidate and the neural head adds nothing worth porting. | supported | candidate | On top of the spread-binned solar-farm power model, an additive model gives a better forecast band than a neural network that adjusts each quantile, so the explainable model is the candidate. A neural network that outputs a three-component Gaussian mixture scores 1.3% better than the additive model in own history, mostly in the upper tail, but not clearly in zero history, so it needs one more test before it is preferred. A blend of the additive model, a neural network, and XGBoost scores 0.8% better than XGBoost alone and fixes XGBoost's weak 0.99 quantile. | also relevant |
| 00284 metered-farm-tail-head-past-weather | Experiment 275 found that adding years of past-weather rows lowered the 0.99 pinball loss of the neural head H5 by 4.6% while its overall score moved by 0.47%. A small neural network trained with a loss restricted to the tail quantiles (0.01, 0.02, 0.05, 0.95, 0.98, 0.99), on the same past-weather plus ensemble rows with an era flag, lowers the 0.01 and 0.99 pinball loss of a metered solar farm's forecast below that of XGBoost trained on the same rows. | partly supported | not-for-production | A neural network trained only to forecast the extreme edges of a solar farm's output band does beat XGBoost at the 0.99 quantile (the upper edge that matters for curtailment) by 30%, but the existing H5 network does slightly better still, and most of the gain comes from building the band around the physical power model and not from tail-only training. The years of past-weather rows add under 1% at the 0.99 quantile, and at the 0.01 quantile the share of half-hours below it is outside 0.5% to 1.5% at three farms (marginally at A, because of exact zeros at D, and clearly only at E). | also relevant |
| 00281 economy7-night-step-smooth-in-temperature | A forecast of primary substation demand whose night level is a set of free levels for each UTC night half-hour (experiment 121's NS-U), and then a step whose size is a monotone function of the cold, scores better in the Economy 7 step hours and on cold winter nights than the same forecast with only the winter night offset (experiment 155), when the weather input is the ensemble-mean forecast and not past weather. Economy 7 is the tariff that switches storage heaters on shortly after 00:30 UTC. Experiment 121 found a gain of 0.233 points on winter nights on past weather and asked for a test on weather forecasts. | inconclusive | not-for-production | Fitting the Economy 7 night levels inside the demand model, and not adding a winter night offset afterwards, lowered the day-ahead error by about 0.02 to 0.03 points of rated demand over all rows in both years and by about 0.04 points on winter nights, but the gain in the two hours after the step starts could not be shown: it changes sign between the two years and depends on two or three substations. Letting the step grow with the cold added almost nothing over free night levels. The Economy 7 term does not explain the night-time gap at D10 in April to June. | also relevant |
| 00275 metered-farm-long-history-neural-head | Experiment 243 fitted H5 (the solar-farm power model plus a neural network that outputs all 11 quantile offsets) on about 17 months of ensemble forecasts. Adding each farm's rows built from past weather (Copernicus Atmosphere Monitoring Service irradiance and ERA5 temperature, back to 2019-09) as extra training rows, with a flag that marks them, lowers H5's quantile score on the ensemble-driven scored year, and the neural head gains more from the extra rows than XGBoost does. | not supported | not-for-production | Training the neural network that corrects a metered solar farm's power forecast on nearly six years of past-weather rows, as well as on 17 months of ensemble forecasts, lowers the quantile score by only 0.5%, far under the 2% we predicted. The same rows improve XGBoost by 1.7%, so the neural network is not the better way to use extra history. The extra rows do lower the loss at the 99% quantile (the upper edge of the forecast band, which matters for curtailment) by 4.6%. | also relevant |
| 00273 metered-farm-sorted-quantile-rescore | Some stored forecasts of experiments 224, 243, 248, 257, and 260 have quantiles that cross (a lower quantile above a higher one). Sorting the 11 quantiles in each row can only lower the pinball loss, so the stored scores are pessimistic for the arms that cross, and the arms that cross most gain most. | not supported | not-for-production | The stored forecasts of seven metered-farm experiments have no crossing quantiles, because every fit script already sorted the 11 quantiles before writing them. Sorting the quantiles again therefore changes no score and no verdict. The 26,344 crossing rows found in experiment 268 are not farm forecasts: they belong to three generators that are not solar farms. Future heads should still be monotone by construction. | also relevant |
| 00270 metered-farm-xgboost-vs-plant-diagnostic | XGBoost beats the plant (the solar-farm power model of experiments 218 and 224) on the six metered farms because of inputs the plant never sees, namely the farm's own lagged power and the ensemble spread, and not because of a better map from the ensemble-mean weather to power. Calendar columns and extra weather variables add little once those two are present. | partly supported | candidate | XGBoost beats the solar-farm power model by about 3% in the quantile score even when it sees only the power model's own weather inputs, and the gain is in the width of the forecast band (the 0.05 and 0.95 quantiles), not in the median forecast. The same gain comes from giving the power model a forecast band that depends on the ensemble spread or the clear-sky index, and that band also beats XGBoost at the 0.99 quantile. Lagged power, calendar columns, and extra weather variables add little in own history. | also relevant |
| 00265 dp-gas-engine-bands-prior-and-visibility-switch | Widening a Primary's forecast bands from its own past residuals (experiment 264, arm 2b) has coverage far from nominal in the first quarter, when there is no history, and after a change in the site's behaviour. A prior taken from the other Primaries' residuals, shrunk towards the site's own trailing residuals as they accrue, fixes the first quarter. A causal switch on the share of large drops in the trailing 30 days returns the widening to normal when D8's engine becomes visible in the readings (around April 2026). Together they hold D8's q05-to-q95 coverage within 5 points of 0.90 in every quarter, without raising the cost in quantile score, and without harming Primaries that have no engine. | not supported | not-for-production | In this year of forecasts, the other five Primaries' residuals did not predict D8's errors, so a starting estimate from them left D8's bands too narrow in the first quarter, and returning to normal widening after the engine's drops faded made them too narrow again. Widening from D8's own history (experiment 264) remained the best of the arms tried. We tested no other fix for the first quarter, and none of the other five Primaries had a heavy lower tail. | also relevant |
| 00264 dp-gas-engine-asymmetric-bands-at-d8 | D8's forecast bands are too narrow in both directions, and part of the lower-tail miss is a 6 MW gas engine (probably three units of about 2 MW) that runs in spells and cannot be detected. A spread that widens as-of, using only residuals already resolved at the forecast origin, and that widens most in the direction the engine pushes (net demand falls), gives honest q05 to q95 bands at D8 without raising the quantile score at the Primaries that have no engine. | not supported | not-for-production | A forecast band that learns from the residuals it has already seen narrows most of the gap at D8: the quantile score falls by 9% to 19% and the share of readings inside the 90% band rises from 42% to between 80% and 87%. The extra gain of the engine mixture over simple widening is not evidence of an engine, because the mixture fits a running engine at two substations that have none. Plain two-sided widening with an earlier residual history is the option these results support, and NGED's own output for the engine would be the data to test an engine model. | also relevant |
| 00262 wind-aifs-members-and-climatology-by-lead | For the three metered wind farms (W1 to W3), a lead-dependent mixture of the ECMWF ensemble's member forecasts, the AIFS ensemble's member forecasts (both run through experiment 230's ERA5-trained XGBoost wind model A0r), and climatology lowers the quantile score and the 0.99 pinball loss against the ECMWF members alone, after both are corrected by a table of offsets per lead day, quantile, and farm; and a small neural gate that reads the two ensembles' spread and disagreement chooses better weights than weights by lead alone. | partly supported | candidate | Adding the AIFS ensemble's wind forecasts to the ECMWF ensemble's improves the wind forecast at 3 to 6 days ahead, by about 5% of the quantile score, but does not improve the high-power tail that matters for curtailment. Climatology and a small neural network that picks the mixing weights add nothing once the table of offsets has corrected the common low bias. The AIFS arms also need testing on a second year, because one year of Tuesday runs sets the weights. | also relevant |
| 00261 wind-low-bias-decomposition | The low bias in the three metered wind farms' forecast (observed power below the physics core's median on 34% of rows, not 50%) comes mostly from the ensemble's weather input, not from the core, the farms, or the mixing of the 51 members. | partly supported | not-for-production | The wind forecast's low median comes mostly from the ensemble's weather: its 10 m and 100 m wind speeds are slower than ERA5's, and the same physics core fed ERA5 wind has 52.8% of rows below its median. Mixing the 51 members adds about a tenth of the shift, and training the physics core and the XGBoost model on 9 months of ensemble forecasts removes between a fifth and a third of the gap. The fix to test next is a correction of the ensemble's wind speeds before the model reads them. | also relevant |
| 00260 metered-farm-quantile-combination | The three best solar-farm forecasters of experiment 243 win at different quantile levels (the neural network without physics at 0.99, H5 at 0.01), so convex per-level weights fitted on earlier months beat H5 alone, and a rolling adjustment of the outer levels brings their coverage to the nominal level without hurting the quantile score. | partly supported | candidate | Blending the three stored forecasters with a separate weight at each quantile level scores 2.1% better than H5 alone, and every one of the six farms improves, because the neural network without the plant is best at the top quantile and H5 is best at the bottom one. About a quarter of the gain comes from averaging the five seeds, not from the weights. The rolling adjustment of the outer quantiles corrects the coverage of both tails but raises the error by 1.2%. | also relevant |
| 00259 wind-plant-plus-quantile-offset-head | For the three metered wind farms (W1 to W3), a small neural network that reads the 51-member mixture of a wind model and the ensemble's wind summaries, and adds an offset to each of 11 quantiles, lowers the quantile score against the wind model alone (R1, the physics core), matches the XGBoost best guess (R0), and lowers the pinball loss at the 0.99 quantile, once it can learn from daily runs of the ensemble forecast over 15 months (about 440 runs) instead of the 13 to 40 Tuesday runs that experiment 221's head saw. | partly supported | candidate | A correction learned from 15 months of daily ensemble forecasts lowered the wind forecast's quantile score by 4.5% to 5.5% on the three metered wind farms, but the neural network and the ensemble's wind summaries were not what earned it. The 51 ensemble members' median sits too low: the observed power was below the physics core's median on 34% of rows, not 50%. A lookup table with one offset per lead day, quantile, and farm recovers 92% of the gain. The open question is where that low bias comes from. | also relevant |
| 00257 metered-farm-fair-plant-arm | The 4.4% gain of H5 (the plant plus a neural network that outputs all 11 quantile offsets) over the plant alone in experiment 243 came from the inputs that H5 saw and the plant did not (the lagged power and the ensemble spread), so a convex linear quantile regression of the 11 offsets on those same inputs (arm P2) recovers most of it, and the neural network is not needed. | supported | candidate | The neural network was not needed. Giving the solar-farm power model the lagged power and the spread of the ensemble members, through a plain linear fit, recovers about 90% of the neural network's gain and scores within half a percent of it. The linear fit is no worse at the 0.99 quantile (the 99% interval of the gap includes zero) and is simpler to serve, so it is the metered-farm candidate. | also relevant |
| 00252 gam-stack-at-eight-more-primaries | The best-guess GAM stack (GAM-K with the heating split by day type, shared filter noise, pinned solar capacity, ensemble-mean weather, winter night offset, and the 60% PV clip) scores within 0.05 points of BG2 over leads 0.5 to 168 hours at D2, D3, D4, D8, D12, D19, D21, and D22, so that every substation card with a BG2 line can also show a best-guess line and example weeks. | not supported | not-for-production | The best-guess demand forecast (the stack) does not transfer unchanged to the eight primary substations whose meters record apparent power. Without a solar term it is at most 0.05 points worse than BG2 (the production-style forecast) at only two of them. Fitting a solar term at each monthly refit puts five of the eight at or below BG2 on the point estimate, so those five cards can show a best-guess line, while D8, D12, and D22 should show none. | also relevant |
| 00251 dp-final-best-guess-assembly-clip-and-coast-against-bg2 | The full best-guess stack of experiment 220 (arm D, the neural-network head, and the two-number solar summary) beats BG2, the manual heuristic, climatology, and a weak production-style XGBoost model over the scored year of 2025-07 to 2026-06. Adding the hand-set 60% clip at D5 lowers the D5 error on sunny half-hours when the neural-network head is absent, and adds nothing measurable when the head is present, because experiment 232 found the head cancels most of the clip's gain. | supported | candidate | Over the one year we can score, the full best-guess demand forecast (arm D, a small neural network that corrects its median, and a two-number summary of the ensemble's solar response) beats BG2 by 0.136 points of quantile score (5%; lower is better) and the manual heuristic by 0.818 points, and it beats the manual heuristic on every headline metric we scored, including curtailment cost. The gain over BG2 is a winter gain: the stack is level with BG2 in summer and worse in April to June, and at D5 on sunny hours it is clearly worse than BG2. Adding the 60% solar clip at D5 helps only when the neural-network head is left out at D5, so the next work is the solar term at D5, not another stacked part. | also relevant |
| 00250 dp-monotone-network-between-the-solar-forecast-and-the-gam | A small neural network that is monotone in the explicit solar forecast, saturates, and reads the clear-sky index, the hour, and the clip cap maps the solar forecast to the solar output that the metered series shows, and so closes more of D5's sunny-daylight gap to BG2 than the hard clip, because it can learn how the saturation depends on the weather. | not supported | not-for-production | A small neural network that sits between the solar forecast and the demand model improves the sunny-day forecast at D5 by about 0.03 MW on top of the hard clip, which is less than half of what is left to close. It does not replace the clip, so the final forecast keeps the clip alone. | also relevant |
| 00244 dp-deep-ensemble-heads-cold-night-bands | On cold nights the demand forecast's error is partly something the correction network does not know (epistemic uncertainty, as opposed to the weather forecast's own uncertainty, which experiment 192 found the 51 ensemble members' temperatures do not signal). The disagreement between 20 independently trained copies of the pooled correction network therefore predicts the absolute cold-night error, and widening the bands by a non-decreasing function of that disagreement lifts cold-night q95 coverage to 0.90. | inconclusive | not-for-production | The disagreement between 20 independently trained correction networks does not tell us how wrong the forecast will be on cold nights (correlation 0.00), so it cannot set the width of the cold-night bands: scaling the bands by it lifts the 95% quantile's coverage by 0.006, the same as a shuffled spread does. The check that the spread reacts to an unusual input failed for a realistic 4 C error, so the experiment cannot say more. The cold-night shortfall is therefore unlikely to be fixed by training more networks; the experiment cannot say what would fix it. | also relevant |
| 00243 metered-farm-architecture-comparison | A solar-farm power model (the plant) as the base prediction, with a small neural network that adapts its inputs, its output, or both, forecasts a metered solar farm 12 to 36 hours ahead better than the plant alone and better than XGBoost, and keeps the plant's advantage on a farm it has never seen, while a neural network with no physics does not beat XGBoost on six farms. | partly supported | candidate | A small neural network sitting on top of a solar-farm power model forecasts a metered solar farm about as well as XGBoost and improves on the power model alone by 4.4% in the quantile score, when the network outputs the whole forecast band. Networks that adjust a single number (the scale, a correction, or the sunlight fed to the power model) gain under 1%. The hybrid is far better than XGBoost at the upper tail (0.99 quantile, 29% lower pinball loss), which matters for curtailment. The neural network without physics does better still at that tail (32% lower pinball loss than XGBoost). | also relevant |
| 00242 wind-physics-core-part-3-member-tails | For the three metered wind farms (W1, W2, W3), running each of the 51 ensemble members through an ERA5-trained wind model and mixing the 51 forecasts gives better 0.01 and 0.99 quantiles than running the model once on the ensemble mean of the weather inputs and applying the same spread factor, because the model is non-linear near cut-in and rated wind, so a model of the mean weather is not the mean of the model. | supported | not-for-production | Keep running all 51 ensemble members through the wind model. Running the ensemble mean once raises the error at the 99% quantile by a factor of 2.1 for the physics model and 3.3 for XGBoost, probably because averaging the members removes the high winds that produce full output, and the single spread factor tried here does not repair that. The physics model and XGBoost differ by under 0.01 points on the tails, so the choice between them does not matter here. | also relevant |
| 00228 dp-bg-rolling-origin-stack-against-bg2 | The best-guess stack (arm D with experiment 169's bands, plus the network head of experiment 201) beats BG2 in most monthly origins and at most substations, and its advantage is larger in winter than in summer. | partly supported | candidate | Over the one year we can score, the full best-guess stack (arm D, the neural network head, and the solar summary) beats BG2 by 0.136 points of quantile score (5% of BG2's 2.715; lower is better), and most of the gain comes from November to March. From April to June 2026 the stack is worse than BG2, and arm D without the neural network head is worse than BG2 in summer. The year-long gain is only just separable from zero, and we cannot test other years until BG2 is re-run for them. | also relevant |
| 00225 dp-network-heads-on-unlabelled-worst-days | Experiments 172, 174, and 175 found that the worst 5% of substation-days carry 41.7% of the excess loss of arm D (the best-guess convex forecast of experiment 155), and that 37 of the 72 worst days at the five decision substations carry none of the labels (switching event, holiday, cold snap, flagged reading, Economy 7 step). Experiments 200 and 201 found that a neural network corrects arm D mainly on very hot and very cold days. The unlabelled worst days therefore sit at temperature extremes, and the neural network cuts their loss. | partly supported | not-for-production | The neural network does cut the loss on the 37 worst days that no label explains, by about a fifth, and cuts it by more than on the 35 days that do carry a label. The unlabelled worst days are not at temperature extremes, so a neural network aimed at the hottest and coldest days would not target them. What helps on them is the forecast input: a neural network without one removes 4%. | also relevant |
| 00224 metered-farm-dayahead-two-number | Passing each of the 51 ensemble members through a solar farm's plant model gives the farm's day- ahead output a spread that the ensemble-mean forecast does not have, and the mean and standard deviation of the 51 plant responses keep most of that information, as they did for Primary substations in experiment 193. A two-number plant forecast therefore forecasts a metered solar farm 12 to 36 hours ahead about as well as XGBoost, and gives calibrated tails. | partly supported | not-for-production | The 51 members of the weather ensemble help a single solar farm's day-ahead forecast only a little: passing each member through a solar-farm power model lowers the quantile score by 1.8% against feeding the model their mean, and two numbers (their mean and spread) carry 95% of that. An XGBoost model on the same weather scores 2.8% better overall, and the solar-farm model beats it only at the 0.99 quantile, the upper tail of the forecast, which matters for curtailment (switching a generator off when the electricity network is overloaded). | also relevant |
| 00222 dp-network-before-filter-order | Experiments 200 and 201 correct arm D after the Kalman filter. Their gain is mostly a systematic error of the convex model at some temperatures and loads, not a recent-error effect, and that systematic error currently sits in the filter's level state. A network that reads only the convex model's own output and inputs, and corrects the convex model before the filter, leaves the filter a cleaner residual stream and so beats the same kind of network applied after the filter. | inconclusive | not-for-production | The gain of the network after the filter does not come from the convex model's own systematic error as seen in its past residuals. A network trained on those residuals learned patterns that do not carry to the next year, and when it was stopped on a later year it learned nothing and left the score unchanged. The test could not settle the order question, because the network failed to recover a known injected error. | also relevant |
| 00221 wind-correction-head-on-stored-predictions | A small neural-network head, fed the nine quantiles of the wind best guess (experiment 33's ERA5-trained mixture over the 51 ENS members, with the power-curve base margin and the spread factor) plus the ensemble mean and spread of the 100 m wind speed and the direction sector, lowers the quantile score at lead days 1 and 2 against the best guess by at least 0.1 points of capacity. The head is pooled over the three farms and trained on earlier months only. | inconclusive | not-for-production | Training a small neural network to correct our wind-farm forecast did not help at 1 to 2 days ahead: the best version scored 0.05 points better and the version that also sees the weather forecast scored 0.18 points worse, with intervals that include zero. With only three farms and 9 scored months, the network may learn the months it trains on and narrow the forecast band too far on the next month. The stored forecast's band was already near the best width. A different kind of correction, or many more farms, is the next thing to try. | also relevant |
| 00220 dp-bg-stage4-assembly-with-drop-one | The three parts added to the best-guess demand forecast since experiment 170 remove mostly different errors, so their gains add. The parts are the two-number solar summary of the 51 ensemble members (experiments 193 and 202), the pooled correction network on arm D's mean (experiments 200, 201, and 209), and the coast-and-widen rule for missing readings (experiment 205). The solar summary changes only daylight bands, and the network corrects arm D's median from its own output, the calendar, and temperature, so the two act on different rows and different error. The network may remove what the winter night offset and recipe R removed (both correct arm D's mean by time of day), and it may make the bands' cold-night widening unnecessary. | partly supported | candidate | Adding two parts to the best-guess demand forecast lowers its quantile score by 0.136 points (5%) of substation effective capacity. The parts are a small neural network that corrects the forecast's median, and a two-number summary of the 51 ensemble members' solar response. Each part's gain is almost exactly what it achieves alone, so the two parts remove different errors. The network's own 11 quantiles reached 0.904 coverage of the 95% quantile on cold nights without the hand-fitted band fixes, and scored 0.048 points better than the corrected forecast overall, mostly at one substation. The coast-and-widen rule for missing readings was not scored with the stack. | also relevant |
| 00216 dp-network-shape-on-arm-d-flexible-temperature-tails | The neural network of experiments 200 and 201 lowers the quantile score of arm D (the best-guess convex forecast of experiment 155) by 0.09 to 0.125 points, mainly by raising the median on very hot and very cold days. That correction is a shape the convex model could carry. The convex model's only temperature terms are one-sided heating hinges at 15.5 degrees Celsius, so it has no term for cooling load and no extra slope in the cold tail. Extra hinge columns in the tails of the site's own temperature range, with a weaker ridge penalty on those columns, recover at least half of the network's gain without a network. | inconclusive | not-for-production | Adding extra temperature terms for the hottest and coldest days to the convex demand model recovers only a fifth to a quarter of the neural network's improvement, and the recovery was not confirmed in a second year. The neural network's correction is spread over all temperatures instead of sitting at the extremes, and a smooth function of temperature explains under 1% of it, so the neural network uses an input other than temperature. The formal verdict is capped at inconclusive because a placebo check narrowly failed, but the shortfall does not depend on that check. | also relevant |
| 00213 calibration-online-band-adaptation-from-coverage-misses | Moving a scale on each quantile offset of experiment 169's arm A up after a reading falls outside that quantile, and down after a reading falls inside it, in the style of adaptive conformal inference (Gibbs and Candes, 2021), lifts the cold-night q95 coverage to 0.90 or more without a second fitted spread model, because a run of misses on a cold spell widens the band before the spell ends. | inconclusive | not-for-production | Letting the forecast widen or narrow its bands after each recent miss does not fix the under-coverage on cold nights: it lifts the 95% quantile's coverage from 0.896 to 0.906 in the scored year and lowers it in the other months, and it worsens the all-row, bank-holiday, and Christmas scores. A fixed widening of the upper quantiles does as well, so the cold-night gap is a shortfall that a signal arriving after the misses cannot learn in time, and the next step is to correct the mean forecast on the coldest days before any further work on the bands. | primary |
| 00211 dp-boosted-tree-calibrator-and-input-ablations | The 0.065 points by which the network without error inputs (experiment 200) beats the best lookup table (experiment 209) comes from a smooth or higher-order dependence on a few inputs, so a gradient-boosted tree model fitted with the pinball loss on the same inputs captures it, and a boosted-tree calibrator can replace the neural network. One-at-a-time removal of the network's inputs shows which input carries the gain. | supported | candidate | A gradient-boosted tree model given the same inputs as the neural network, and no recent errors, lowers the error score of our best day-ahead forecast by about as much as the network does (0.098 against 0.100 points). The interval is wide, so a loss of up to 0.03 points against the network is not ruled out. The gain comes mostly from two inputs, the forecast level and the weather-data era. | also relevant |
| 00209 dp-pinball-table-vs-network-head-on-arm-d | The gain that experiment 200's network head and its blind copy get over arm D (0.100 to 0.125 points) is a recalibration that a table without hidden layers can reproduce. The table gives arm D's median a shift that depends on site, half-hour, day of week, lead band, and arm D's forecast level by an ensemble-era flag, and scales arm D's quantile offsets by site, lead band, and six-hour block. It is fitted by gradient descent on the pinball loss over the 11 quantiles, on the same rows as the network. A second arm adds a temperature-extremes term (a median lift below -3 degrees Celsius and above 24 degrees Celsius, from experiment 201's finding). | not supported | not-for-production | A lookup table of corrections to our best day-ahead forecast lowers its error score by only 0.03 to 0.04 points, against 0.10 points for a neural network given similar inputs and no recent errors. This table design therefore does not match the network, and the cause is not yet isolated. A temperature-extremes term lowers the score by under 0.001 points. | also relevant |
| 00205 dp-gamk-gap-robustness-coast-or-frozen-gam | When power readings go missing, a Kalman filter that coasts (keeps running with no update, so the transient decays and the level holds) beats the frozen convex model (GAM) alone for short gaps and loses to it for long gaps, so a crossover gap length exists between 6 and 72 hours. The mechanism is that the level is a random walk, so a short gap leaves it nearly right, while a long gap lets it drift away from the demand. Widening the quantile bands by the filter's own growth in predictive variance keeps the 5% and 95% quantiles calibrated at the gap lengths where coasting is still better than the frozen model. | inconclusive | candidate | When readings stop for 3, 6, 24, or 72 hours, keep running the Kalman filter, which corrects the convex demand model using its recent forecast errors, without feeding it new readings, and widen the quantile bands. At each of those four gap lengths that forecast scored 0.25 to 0.44 points of quantile score better than the convex demand model with no correction (a quantile score is an error measure in percent of the substation's effective capacity, and lower is better; the gain is 8% to 14% of the uncorrected model's score). At 72 hours the 99.4% bootstrap interval for the gain runs from 0.04 to 0.48 points, so the gain may be small. The data cannot show whether dropping only the filter beats coasting after a gap of about 72 hours: at every gap length the difference between the two forecasts is smaller than the variation between months. The forecast with the filter, winter night offset, and recipe R was still 0.025 points worse than its no-gap score 48 hours after a 24-hour gap ended. The filter alone was within 0.02 points of its no-gap score 24 hours after the gap ended. | also relevant |
| 00202 dp-ens-two-number-rolling-replay | Experiment 193 found that two numbers per half-hour (the mean and the standard deviation of the 51 ensemble members' solar responses) lower the daylight quantile score by 2.20% in the scored year. Its two fitted factors per lead band were fitted on interleaved months (the odd months of 2024-07 to 2025-06), some of them later than the replicate months they were applied to, and its numbers were held in float64. A replay of the live weather layer, which stores the two numbers as float32 columns computed from the one ensemble run published before each origin and refits the factors each month on history that has completed before that month, keeps most of the gain. | supported | candidate | A weather layer that stores two numbers per half-hour (the mean and spread of the 51 ensemble members' solar-generation estimates) and refits its two scaling factors every month, using only runs that had finished before the month began, lowers the daylight quantile score (the score for the forecast's uncertainty bands) by 2.3% at five substations over one year, which is as much as running all 51 members. This confirms experiment 193's result under production-style timing, so the next step is to build the two columns into the weather layer. | also relevant |
| 00201 dp-gam-residual-mlp | A small neural network that reads the convex model's own design features, and later the calendar, the ensemble-mean weather, and lagged power, can learn an interaction the additive convex model lacks, and so lowers the quantile score of the best-guess convex forecast (arm D of experiment 155) when it adds a correction to the median and rescales the band widths. | partly supported | not-for-production | A small neural network that reads the convex demand model's forecast lowers that forecast's quantile score by about 0.09 points (3%), and adding weather, calendar, or recent power does not help further. The neural network corrects an under-forecast on very hot and very cold days. That points to a missing temperature term in the convex demand model. The correction also scores worse on bank holidays and, in point estimate, on Christmas. | also relevant |
| 00200 dp-pooled-correction-head-on-kalman-state | Experiment 142's small network failed its known-answer gate with about 1,100 samples from one year at five substations. The same kind of network, trained on rows pooled across many Primary substations and years, learns to correct arm D's forecast from the recent errors, the calendar, the lead, and the ensemble-mean weather, and so beats recipe R (subtract 0.75 times the mean error of the last 14 days) at day D+1. | partly supported | not-for-production | A small neural network trained on arm D's own past errors lowers the day-ahead score by 0.07 to 0.12 points at five substations, but a copy that sees no recent error gets most of that gain. The gain comes from correcting how arm D's error depends on its forecast level and the calendar, and a pinball-loss table comes within 0.01 to 0.02 points of the network on one year of data. | also relevant |
| 00193 dp-ens-two-number-solar-summary | Experiment 187 found that passing each of the 51 ensemble members through the solar term lowers the daylight quantile score of the full best-guess stack by 2.23%, and that a Gaussian with each row's own mean and spread of the 51 responses recovers 95% of that gain. Two numbers per half-hour (the mean and the standard deviation of the members' solar responses) therefore carry nearly all of the gain, so a serving path can store two columns instead of running 51 passes. | supported | candidate | Summarising the 51 ensemble weather members' solar estimates by their mean and standard deviation lowers the daylight quantile score by about 2% (98% of the gain from running all 51 members, 93% in the replicate months), at the five substations scored over one year. The weather layer could store two columns per half-hour instead of 51, and the 10th and 90th percentiles add nothing measurable. A follow-up experiment could test the two numbers in the live weather layer. | also relevant |
| 00192 dp-ens-members-temperature-cold-nights | Experiment 187 found that passing the 51 ECMWF ensemble members through the solar term of the demand model lowers the daylight quantile score by 2.2%. The demand response to heating is also nonlinear (a hinge at 15.5 degrees C), and the ensemble mean smooths away the cold tail. Passing each member's temperature through the temperature terms therefore lowers the quantile score on cold winter nights. | not supported | not-for-production | Running each ensemble member's temperature through the demand model gave no detectable improvement in the forecast bands on cold winter nights, because the 51 members' temperatures differ too little to signal the day's error. The cold-night bands are already too narrow. The cause of the cold-night error is still unknown; widening the bands directly, and testing wind, night cloud cover, and switching as inputs, are the candidates. | also relevant |
| 00187 dp-ens-members-in-stack | Experiment 184 found that running each of the 51 ensemble members through the solar term of the convex demand model lowers the daylight quantile score by 2.2%, but its verdict was capped at inconclusive because a monotone gate failed on rows where the unit PV shape itself falls as irradiance rises. With a gate that matches that shape, the same arm passes, and its gain lies where the best-guess stack leaves the largest sunny-daylight error, at the substation D5. |
supported | candidate | Running each of the 51 ensemble weather members through the solar part of the demand model still improves the daylight forecast bands by about 2% on top of the full best-guess stack, and night is untouched. About two thirds of the gain needs that day's own members, and two numbers per half-hour (the mean and spread of the members' solar responses) carry nearly all of it, so a cheaper two-number version is the next thing to test. | also relevant |
| 00184 dp-ens-daylight-calibration | Experiment 169 found that passing the ensemble's spread to the width of the forecast's bands adds nothing when scored over all half-hours. The benefit may be confined to daylight half-hours, especially sunny and partly cloudy ones, where the clouds that the 51 ensemble members disagree about change the output of embedded solar generation, and so the demand. Passing each member's irradiance through the convex model's solar term (arm M), or using the spread of the members' irradiance as a fraction of clear-sky to set the band width in daylight (arm S), calibrates the daylight bands better than the existing bands (arm A). | inconclusive | not-for-production | Running each of the 51 ensemble weather members through the solar part of the demand model improves the daylight forecast bands by about 2%, mostly in the lower tail and on partly cloudy half-hours, while night is untouched. Placebos that widen the bands the same way but without the real members gain little, so most of the benefit is information about cloud. The next step is a production-cost check of the member pass and a test of its gain on top of the best-guess forecast. | also relevant |
| 00176 calibration-band-shape-smooth-in-lead | The q05 to q95 band of the combined forecast (experiment 169's arm A) is not smooth along the lead because each cell of the spread table (hour band, calendar quarter, one of four lead bands) is a quantile of a few hundred correlated residuals, and neighbouring cells are fitted independently. Replacing the table with offsets that are a non-decreasing piecewise-linear function of lead (fix P), shrinking each cell towards the table pooled across substations (fix S), or both (fix PS), gives a band whose median width never falls from one lead day to the next, without worsening the quantile score or the cold-night coverage. | inconclusive | not-for-production | D5's forecast band already widens steadily with lead time on the rows we score, and the dips appear only when switching-event rows are added, so these results give no reason to smooth the stored table. A smooth widening curve does not reduce accuracy but does not remove the dips either. Pulling noisy table cells towards a pooled table helps D10 and D14, hurts D5, and leaves cold-night coverage slightly low, so shrinkage needs testing on more substations before anyone adds shrinkage to the production forecast. | primary |
| 00173 dp-christmas-more-years-and-sites | The convex demand model with a Kalman filter on recent errors and recipe R (the mean error of the last 14 days by half-hour, held to 0.75 of its size), which scored 1.13 times an ordinary weekday over 24 December to 2 January at five substations in two Christmases (experiment 170), keeps a Christmas-to-weekday error ratio under 1.3 in every Christmas back to 2019-20 and at 14 or more of 16 Primary substations, and does not over- or under-forecast 25 and 26 December by more than 5% of that day's demand. | inconclusive | not-for-production | Where the demand model had a winter of training data, the error over 24 December to 2 January was 0.97 to 1.22 times an ordinary weekday in all four Christmases from 2022-23 to 2025-26, so no special-day term is justified. The pre-registered run failed because the training rows of Christmas 2020-21 and 2021-22 (and of one substation in 2023-24) held little or no winter, since the lockdown window is left out of training. A forecast for a substation with under 180 days of usable history should be flagged, not trusted. | also relevant |
| 00172 dp-error-budget-worst-days | The error of the stage 2 assembly (experiment 155's arm D) is concentrated on a small number of days that the project already knows how to label. The worst 5% of days, ranked by their pinball loss above the site's median day, carry at least 30% of the total excess loss, and at least half of those days carry one of six labels: a switching event, a metering error or flagged reading, a cold snap, an Economy 7 step, a holiday, or a combination. | inconclusive | not-for-production | A small number of bad days carries a large share of the forecast's error: the worst 5% of days carry about 42% of the excess loss, and about half of those days coincide with a switching event, a holiday, or a cold snap. Switching events are the largest group, on days that carry 29% of the excess loss, so a forecast that improves only ordinary days leaves most of the loss in place. Cold snaps carry 5% of the excess in this mild scored winter, so this year cannot show whether a cold-night term pays. The formal verdict is inconclusive because one control narrowly failed. | also relevant |
| 00170 dp-bg-stage3-drift-and-holiday-scoring | The convex demand model with a Kalman filter on recent errors, ensemble-mean weather, the heating split by day type, one shared filter noise setting, and the pinned solar capacity (experiment 136's arm A2) improves further when it gets a drift term (experiment 167's D1) on top of the two post-forecast corrections of experiment 155's arm D (a winter night offset and recipe R's 14-day mean error profile). The mechanism is that the drift term removes a slow demand-level error that the monthly refit on 24 months of history leaves in, and neither the offset (winter nights only) nor recipe R (a 14-day profile held to 0.75 of its size) removes that error. Training on every year of history adds a further gain, because the drift term and the pinned capacity let the old years teach the heating and calendar terms. | refuted | not-for-production | A slowly varying demand level (a drift term) improved the forecast by 0.40 points in experiment 167 but worsened it by 0.03 points once the forecast has the Kalman filter on recent errors (lower scores are better), and training on all years of history did not change that. The filter already appears to remove the slow level error that the drift term fixes, because without the filter the drift term improves the score by about 0.2 points. The drift term should not be added to experiment 155's arm D. Christmas (24 December to 2 January) scored 1.13 times an ordinary weekday at the same half-hours, below the 1.5 times predicted, and recipe R improved the Christmas score by 0.04 points. | also relevant |
| 00169 calibration-combined-fixes-and-ensemble-spread | The spread across the 51 ensemble members of the temperature and of the downward shortwave radiation at a substation predicts how wide the forecast's residuals are, beyond the hour band, calendar quarter, lead, and the cold-day widening of experiment 164. A convex quantile model that scales the quantile offsets by a non-decreasing function of those two spreads lifts the cold-night q95 coverage of the combined forecast to 0.90 or more. | partly supported | not-for-production | Combining the smooth spread table with the cold-day widening reproduces its 0.896 coverage of the 95% quantile on cold nights, and adding the spread of the ensemble weather forecast (how much its 51 members disagree) lifts that to 0.907. The lift comes from widening the 95% and 99% quantiles by 4% and 10%, which the same model gives with its spread terms removed, so the ensemble spread is not what lifts cold-night coverage. A later experiment could test a bias correction of the mean forecast on the coldest days, and judge cold nights over more than one winter. | primary |
| 00164 calibration-cold-night-tail-under-coverage | The q95 coverage of experiment 160's smooth-table forecast on cold nights (day mean temperature below 2 °C) stays near 85% because the spread table is too narrow when it is cold, not because the mean forecast is biased. Widening the quantiles on days the ensemble forecasts as cold, by a factor fitted on the tuning months, lifts cold-night q95 coverage to 90% or more without worsening the all-row score. | refuted | not-for-production | On cold nights the 95% quantile of the best forecast is exceeded on about 15% of readings, and mostly because its prediction interval is too narrow, not because its mean forecast is biased. Widening only the upper quantiles on days the weather forecast calls cold reduces that share to about 10% without changing the all-hours score, but no fix reached the pre-registered 90% coverage in the scored year, and only 10 cold days back that year's figure. The coldest forecasts (0 °C or below) are also biased low, so correcting the mean forecast on those days is the next experiment to run. | primary |
| 00161 dp-bsp-top-down-against-bottom-up | A bulk supply point (BSP) is forecast better bottom-up than top-down. The top-down forecast is the best-guess GAM-K chain (experiment 155's arm D) applied to the BSP's reading as a stand-alone series. The bottom-up forecast adds the arm D forecasts of the BSP's child Primary substations that report real power, each times a weight learned from earlier readings, to arm D's forecast of the remainder (the BSP's reading minus those children's weighted readings). The mechanism is that the child Primaries' own forecasts are better than the BSP's chain can infer from the BSP's sum, because each child has its own heating, solar, and calendar response and its own recent errors, and the sum is not exact (losses, unmetered load, children that report apparent power), so the remainder and the weights absorb the difference. | inconclusive | not-for-production | Forecasting a BSP from its children's forecasts plus a remainder did not beat forecasting the BSP as a stand-alone series at either BSP: the bottom-up score was 0.6% worse at D13 and 0.5% worse at D17. The likely reason is that the children this test could use (three real-power Primaries per BSP) cover 42% and 56% of the BSP's capacity, and the BSP's error lives in the rest. The oracle control also missed its pre-registered threshold, which was set too high. The verdict is inconclusive because both intervals include zero. |
also relevant |
| 00160 calibration-band-width-fix-on-stage-2-winner | The q05 to q95 band of the stage 2 winning arm (experiment 155's arm D with arm A's spread table) widens in steps along the lead, because the spread table has only four lead bands (up to 24 hours, 48, 144, and 168), fitted separately per hour band and calendar quarter. Interpolating each hour band by quarter cell between the lead-band centres, forcing the width to be non-decreasing in lead, or both, removes the steps without worsening the quantile score. | supported | candidate | The 90% prediction interval (from the 5% to the 95% quantile) of experiment 155's best forecast was equally wide on lead days 2 to 5, because its stored spread table has only four lead bands. Interpolating the table between the band centres removes the equal days and raises the quantile score by 0.002 points, below the pre-registered ceiling of 0.01, but it narrows the interval at 24 to 72 hours and widens it from 72 hours on, so one substation loses 0.009. It does not fix the cold-night upper tail: the forecast exceeds the 95% quantile on about 15% of cold-night rows, against 5% for a calibrated forecast. | primary |
| 00155 dp-bg-stage2-assembly-night-offset-and-error-correction | A GAM-K forecast (the convex demand model with a Kalman filter on recent errors, experiment 111) that already has the heating split by day type, one shared filter noise setting, the pinned solar capacity, and the ensemble mean as its weather input (experiment 136's arm A2) improves further when two post-forecast corrections are added: a winter night offset (experiment 137) and the 14-day mean error profile of recipe R (experiment 115). The mechanism is that the two corrections remove different errors, a level error that repeats on winter nights and a half-hour-of-day shape error that repeats day to day, so their gains add. The assembled arm then scores below experiment 111's GAM-K and BG2 (experiment 87) over leads 0.5 to 168 hours, and its tails on cold nights and on holidays are no worse than the uncorrected arm's. | inconclusive | candidate | Adding recipe R, which subtracts 0.75 times the mean error of the last 14 days by half-hour, lowers the stage 1 assembly's score by about 0.04 points (1.3%), and the winter night offset adds nothing once R is in. The assembly with R scores 2.754 against BG2's 2.780, which is better by 0.026 points with an interval too wide to rank the two. The pre-registered placebo control failed, so the formal verdict is inconclusive, but the choice of winning arm (R alone) does not depend on it. | also relevant |
| 00142 differentiable-physics-sequence-model-on-gam-k-residuals | A small neural network that reads GAM-K's last 14 days of residuals, the calendar, and the target day's forecast weather predicts the day-ahead residual better than the hand-built Kalman filter with the 14-day error profile (recipe R of experiment 115), and gives better tails when it is trained with the pinball loss at the 11 quantiles. | inconclusive | not-for-production | A small neural network that reads the last 14 days of GAM-K's errors did not learn to adapt to changes in them: when we added a known step or a Saturday-evening shift to real errors, it recovered none of it, so we did not score it on the scored year. Nothing was scored, so this result does not compare it with the hand-built Kalman filter and recipe R. | also relevant |
| 00136 dp-bg-stage1-assembly-on-the-ensemble-mean | A convex demand model with a Kalman filter on recent errors (GAM-K, experiment 111), given experiment 114's heating split by day type (T-all) and experiment 133's single shared filter noise setting (POOL), and driven by the ensemble mean of the ECMWF ensemble (ENS) instead of its 51 members, forecasts Primary demand 0.5 to 168 hours ahead at least as well as BG2 (experiment 87) and better than experiment 111's GAM-K. The mechanism is that the three gains came from different parts of the model (the hinge, the filter noise, the weather input) and so add, and that experiment 125 found that the members' own spread adds no tail skill, so one forecast on the mean loses nothing. | inconclusive | not-for-production | Putting everything that has helped so far into the convex demand model and driving it from the ensemble mean gives a forecast that scores the same as the production recipe (BG2) and as experiment 111's GAM-K, and the experiment cannot tell any of the three apart. Using the ensemble mean makes the score about 0.04 points worse than using all 51 members, and the heating split and the shared filter noise improve it by about 0.05, so the assembly did not add up to the hoped-for gain. | also relevant |
| 00133 differentiable-physics-vk-learned-noise-and-observation-variance-in-gam-k | On GAM-K's filter (a level and a transient per hour band, per substation), noise settings chosen per substation by day-ahead score lower the day-ahead quantile score below the fixed settings of experiment 111, which were chosen by marginal likelihood on 184 non-contiguous days. An online observation variance in the manner of VIKING (de Vilmarest and Wintenberger, 2022), used to scale the spread, then improves the coverage of the 0.95 quantile on cold nights, which experiment 125 found at 73% to 78%, because recent large errors widen the spread. The mechanism under test is that the size of recent proxy-driven residuals predicts the size of the next day's error on cold nights. | inconclusive | not-for-production | Choosing the filter's noise settings for each substation by how well they forecast the next day did not improve the forecast on later months: the gain of 0.044 points on the months used to choose shrank to 0.009 on the scored year, inside the noise. A single setting shared by every substation did at least as well as the per-substation settings, so tuning the filter's noise per substation looks unnecessary, although the grid did not find a minimum. Letting the observation variance follow recent errors did not widen the forecast where it matters, on cold nights, where the 0.95 quantile still covered only 86% to 88% of readings. | also relevant |
| 00125 differentiable-physics-n8-ens-members-through-gam-k-and-the-spread-around-them | Placing GAM-K's residual quantiles, fitted on a forecast driven by the analysis proxy (so they hold model and filter error and almost no weather error), around each of the 51 ENS members gives tails closer to nominal than today's table, which is fitted on ensemble-mean residuals, and the gain is largest at long leads and on cold winter nights, where the hinge turns a temperature error into a large demand error. The comparison is against GAM-K re-tuned on the same factor grid, so a plain scale error cannot pass for a better design. | inconclusive | not-for-production | Putting the forecast-error table around each of the 51 weather members, and fitting it on analysis-driven forecasts so that it holds no weather error, did not improve the tails of GAM-K's demand forecast. The weather members' spread is small against the model's own error, so the weather error was counted twice in the code but is too small to matter. The next step is the cold winter nights, where demand exceeds the 95% quantile about a quarter of the time in every arm. The verdict is inconclusive, because no arm reached 14 of 16 calibration cells. | also relevant |
| 00119 differentiable-physics-enc-weather-patch-encoder-on-ens | A small neural network that reads the ENS grid boxes around a Primary, and not only the one box GAM-K reads, lowers GAM-K's quantile score a little on average and more on the cold, showery, and badly forecast days NGED cares about, either as a correction to GAM-K's quantiles or as a replacement for its single-box temperature and irradiance. | inconclusive | not-for-production | A small neural network that reads the ensemble weather around a Primary lowers GAM-K's quantile score by 0.062 points of effective capacity in one scored year and 0.035 in the other (both statistically significant, 5 of 5 substations better), and the gain is smaller in winter (0.041 and 0.027 points) but never a loss. About two-thirds of the gain survives when the network reads the wrong date's weather (0.043 and 0.022), so it is probably a recalibration of GAM-K's quantiles with the lead, the hour, and GAM-K's own forecast, and the surrounding boxes add nothing to the centre box (patch minus centre box: -0.002 and +0.003 points). The known-answer gate failed, so the verdict is inconclusive, and the next step is a plain recalibration of GAM-K's quantiles with the centre box's extra fields (dew point, pressure, precipitation, and member spread), not a patch encoder. | also relevant |
| 00117 differentiable-physics-h2-graph-prior-on-pooled-convex-parameters | A penalty on the differences between interconnected Primaries' parameters (heating slopes, PV capacity as a share of effective capacity, the profile's scale, and the holiday coefficients) narrows those parameters' month-block bootstrap intervals without breaking their coverage when neighbouring Primaries are alike, and worsens the full-history quantile score by at most 0.03 points. | refuted | not-for-production | At the weight tested, tying the parameters of interconnected substations together did not improve the forecast: in the scored year it worsened the quantile score by 0.020 points on all days (the interval spans zero) and by 0.173 points in winter evening peaks (the replicate year gave -0.021). On synthetic substations with a known truth it narrowed only heating-slope intervals, and the 80% intervals of PV capacity held the truth 17% of the time against 78% without the tie. A tie on heating slopes alone worsened the real score by 0.006 and 0.001 points, an exploratory lead and not a result. | also relevant |
| 00107 differentiable-physics-rolling-origin-history-and-tails | Refitted quarterly from 2023-07 to 2026-04 and scored on the next 3 months, the convex model's two history contrasts (all years against 12 months, drift against none) keep one sign at almost every origin. A second claim, labelled separately as C5 and not part of the verdict: a level-dependent spread brings its 95% and 99% quantiles within 1 point of nominal coverage without worsening the quantile score. | partly supported | not-for-production | Giving the convex demand model a drift term lowered its score at 10 of 12 quarterly refits, by 0.45 points when trained on all years (interval 0.23 to 0.68). In a post hoc split, about three-fifths of that size is one substation (D10) at the 2023-24 refits, and the 2025-26 refits give 0.30, as in experiment 104. All years without drift scored worse than 12 months at 8 of 12 refits, and all years with drift against 12 months has no stable answer: winter-quarter refits favour more years, summer ones are level. The spread that depends on the forecast level made the 95% and 99% quantiles worse, so the tail coverage still needs a different method. The verdict is partly supported, because only the drift contrast is stable; a second spread (the conformal spread) is not interpreted, because it was built with a flaw. | also relevant |
| 00100 drop1-random-lag-dropout-widens-intervals | An XGBoost quantile model for Primary substation demand that is trained with labelled switching half-hours removed from its targets and with its lagged power inputs set to missing at random intervals, and that has those lags set to missing when they lie inside a labelled switching event, widens its 5% to 95% band on the half-hours whose lags are missing, so that the band's coverage moves from 0.784 towards the nominal 0.90, without widening the band elsewhere. | inconclusive | not-for-production | Training the demand forecast with lags hidden at random did not make its band wider on half-hours where a lag is missing: on real rows the band covered 0.80 (hidden in 10% of training rows) to 0.84 (30%) of actuals, against a nominal 0.90. Hiding lags in 30% of rows did remove the accuracy loss that masking alone causes on injected transfers (-0.02 points, 95% interval -0.35 to +0.38 against the plain model). Hiding lags in 10% of rows left +0.26 (98.75% interval -0.15 to +0.85), not separable from zero or from the loss. An input that tells the XGBoost model a lag is missing, or a separate XGBoost model of the quantile spread, are untested routes to a wider band. | primary |
| 00099 crps1-crps-type-loss-over-ens-members-primary-demand-pilot | Training the Primary-substation quantile model on the continuous ranked probability score (CRPS) of the pooled ensemble forecast, with several ensemble members as separate rows, scores lower on the quantile score and covers the tails closer to nominal than training the same rows with the pinball loss, because the pinball loss makes every member's row learn the whole predictive spread and pooling the 51 members then counts the weather uncertainty twice. | small gain | not-for-production | The CRPS loss beat the pinball loss on the same 6 members per run by 0.039 points (-0.057 to -0.018). | primary |
| 00097 cal1-post-hoc-recalibration-of-bg2-quantiles-tails | A post-hoc recalibration of the stored quantiles of the best-guess forecast (BG2, experiment 87), fitted on earlier months only, brings the coverage of the 0.01, 0.05, 0.95, and 0.99 quantiles to within 0.01 of nominal in the 12 to 36 hour lead band of each class, without making the centre of the distribution worse. | refuted | not-for-production | The stored best-guess quantiles cover the lower tail too widely (q05 covers 0.035) and the upper tail slightly too narrowly (q95 covers 0.938), at 12 to 36 hours for Primary substations, and no post-hoc recalibration fitted on earlier months moved coverage by more than its own interval. | primary |
| 00080 bm0-zero-base-margin-xgboost-quantile-models | XGBoost starts a quantile model from a base score that it estimates from the training labels, one value for each of the nine quantile levels (experiment 78 found 0.403 at the 5% level to 0.899 at the 95% level for one series, with 500 boosting rounds and no early stopping). Replacing that start with zero (a base margin of zero in every row, which is the same as a base score of zero) lowers the quantile score of the production-style models for Primary substation demand (retrained weekly), bulk supply point and grid supply point demand (retrained weekly), solar farms, and wind farms. The gap is not a symptom of too few boosting rounds, so four times as many rounds, or early stopping, does not remove it. | inconclusive | not-for-production | Starting XGBoost from zero did not lower the quantile score of any production-style XGBoost model: Primary substations -0.017 points of capacity (98.75% interval -0.072 to +0.054), bulk and grid supply points +0.006 (-0.091 to +0.092), the frozen Primary stack -0.033 (-0.084 to +0.031), solar farms -0.010 (-0.036 to +0.028). | primary |
| 00035 lead-weighted-quantile-blend-of-stored-models | For Primary substation demand, solar farms, and wind farms, averaging the quantiles of two or three forecast models that experiments 19, 20, 30, and 33 already scored (N1 is the ENS-trained single-stage XGBoost model, and the stack is N1 with the 14-day correction and a spread factor of 1.15), with one weight per lead band chosen only on earlier months, lowers the quantile score below the best single member by more than the threshold set for each technology (0.05 points of capacity for Primary demand and solar, 0.10 for wind). | supported | candidate | Averaging the quantiles of three stored demand models lowered the Primary demand error by 0.10 points of capacity against the best single model (98.33% interval -0.19 to -0.03), so the estimate clears the 0.05 threshold for moving it into production. For solar and wind the gains were too small to resolve on 6 solar farms and 3 wind farms. | primary |
| 00020 solar-wind-stack-cams-stage0-spread-margin | For the 6 solar farms and 3 wind farms, stacking the candidate changes of experiments 00012, 00014, and 00016 (a CAMS-trained stage 1a and a stage 0 quantile mapping for solar, a power-curve base margin for wind, and a fitted spread factor for both) gives a lower continuous ranked probability score (CRPS) than the pooled ERA5-trained mixture of experiment 00002. | partly supported | candidate | For wind, a starting guess from the turbine power curve lowered the error by 0.055 points of capacity (99.58% interval -0.110 to -0.007), and adding a spread factor made the total 0.071 (-0.133 to -0.014); the spread factor's extra -0.016 (-0.036 to +0.004) is not detectable. For solar, no change helped detectably, and the full stack was 0.035 points higher with an interval that includes zero (-0.075 to +0.116). | primary |
| 00016 pooled-mixture-calibration-solar-wind | For the 6 solar farms and 3 wind farms, a correction fitted on earlier validation months (a scalar spread factor, a spread factor per lead band, a location-and-scale calibrator, or a free quantile calibrator) brings the 5% to 95% band of the pooled ERA5-trained mixture closer to 90% coverage in every lead band, without raising the continuous ranked probability score (CRPS) by more than 0.05 points of capacity. | partly supported | candidate | For solar, one spread factor brought the 5% to 95% band from covering 94.3% to 95.3% of cases down to 89.5% to 91.8%, with no detectable change in accuracy. For wind, no planned correction fixed the first two days, and a spread factor on two lead groups, added after seeing the results, brought every lead band to 90.7% to 92.5% coverage and has not been tested on another year. | primary |
| 00002 stage2-calibrator-and-mixture-vs-ens-quantile | A shared per-technology calibrator and a learned blend, fed by ERA5-trained stage 1a quantiles applied to ENS weather, give a lower continuous ranked probability score than the pooled stage 1a mixture and than a quantile model trained on ENS weather. | partly supported | candidate | Pooling the ERA5-trained solar and wind quantiles over the 51 weather-forecast members beat a model trained on archived weather forecasts: wind by 0.62 points of capacity (95% interval -0.99 to -0.26) and solar by 0.24 (-0.36 to -0.06), with the gain in the first six days. A second calibration model on top did not improve on the pooling and made solar worse by 0.40 points (+0.26 to +0.61). The result rests on 6 solar farms, 3 wind farms, and one validation year. | primary |