Curtailment cost: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00344 wind-research-default-1-speed-dependent-offsets | Experiment 340 found that a constant upward shift of the wind quantiles gains 0.09 to 0.12 points on the kept rows but worsens the rows below cut-in by 0.51 to 0.78, and that the default's median is too low at 8 to 15 m/s and too high below 5 m/s. An offset that depends on the ensemble-mean 100 m wind speed at forecast time lifts the quantiles where the power curve is steep and leaves them alone where the forecast is near zero, so it keeps the gain without the low-wind harm. The starting point is Wind farms research default 1, the three metered wind farms W1 to W3 are scored, and the lead range is 1 to 10 days. | not supported | not-for-production | Letting the wind forecast's quantile offsets depend on the forecast wind speed gains only about 0.02 points of capacity when fitted on the same retrospective forecasts (hindcast runs) that fit the quantile offset table, and the gain cannot be told from zero. Fitted on the six months just before the scored year (July 2025 to June 2026), the same speed-dependent correction gains 0.146 points and does not worsen the half-hours where the forecast wind speed is below the turbines' cut-in speed. The speed-dependent bias is therefore real, and it changes from year to year. Refitting the correction on recent months is the untested option that could capture it. | also relevant |
| 00342 wind-research-default-1-upper-tail-offsets-and-head | The upper quantiles of Wind farms research default 1 are too low: in experiment 324 the observed power exceeded q99 in 1.8% of half-hours (1% nominal) and q95 in 6.7% (5% nominal). Three corrections fitted on the hindcast can fix that without harming the headline score. x1 is a table of offsets for q90 to q99 that also depends on the ensemble-mean wind speed. x2 is a small boosted-tree head, fed the default's quantiles and summaries of the ensemble members' wind speed, that adds offsets to q90, q95, and q99. x3 lets the share of ensemble members above the farm's saturation speed set a probability mass at the farm's scale. | not supported | not-for-production | The wind forecast's upper quantiles miss too often (observed power exceeds q99 in 1.8% of half-hours against 1% intended), and corrections fitted on past forecasts do not fix that: the offsets keyed by wind speed, the tail head, and the mass at the farm's cap all leave the share at 1.8% to 1.9% and change the upper-tail loss by about 1% or less. A head that sees the ensemble members does no better than one that sees only the default's quantiles, so in these 3 farms the members add no information about the upper tail. The remaining miss is consistent with the scored year differing from the hindcast years, which the next experiment could attack by refitting more often or by training on the most recent months. | also relevant |
| 00340 wind-research-default-1-constant-shift-then-low-wind-drop | Experiment 336 found that dropping the ERA5 training rows below the turbines' cut-in speed (x2) gains 0.117 points on the kept rows, and its audit traced about 85% of the gain to a constant upward shift of the quantiles. If that is right, a constant shift fitted on the 442-run hindcast (arm S) reproduces most of the gain without changing the training rows. Arm S+X2, which also drops the low-wind rows, then adds little beyond the shift. Wind farms research default 1 is the starting point, the three metered wind farms W1 to W3 are scored, and the lead range is 1 to 10 days. | not supported | not-for-production | A single upward shift of the wind quantiles, fitted on past forecasts, does nothing, because the forecast's offset table already fits those runs. Moving the quantiles up by 0.8 to 1.2 points of capacity does reproduce most of the gain of dropping low-wind training rows, but it also worsens the forecast at low winds by 0.5 to 0.8 points, the same harm that stops X2 being adopted. An untested option that could lift the forecast at 8 to 15 m/s and leave it alone below 5 m/s is a correction that depends on wind speed. | also relevant |
| 00339 wind-research-default-1-adapter-plus-mixture-head | A three-component Gaussian mixture head on the hindcast, fed the quantiles of the physics core plus adapter and the ensemble members' summaries of speed, direction, and temperature (experiment 324's W4), lowers the quantile score of Wind farms research default 1 on the kept rows of the three metered wind farms W1 to W3 at 1 to 10 day leads. The head loses on the idle half-hours because it is trained on kept rows only, and the maintainer's view is that a forecast-time rule (predict the idle floor where the ensemble-mean wind speed is below cut-in) needs no idle detector. Three arms are tested: x1 the head as in experiment 324, x2 x1 plus the rule, x3 the head trained on all hindcast half-hours. | partly supported | candidate | A small neural-network head that re-draws the physics-plus-adapter wind forecast's quantiles gains 0.27 points of capacity (2.4%) on the farms' normal half-hours and keeps that gain on the replicate months, but the same head given no weather summaries does as well, so the gain is a recalibration that a simpler correction partly reproduces. A simple rule that predicts the idle floor at low wind recovers none of the head's loss on the idle-band readings, and training the head on those readings removes the gain. | also relevant |
| 00336 wind-research-default-1-below-cut-in-zeroing | The maintainer proposes that, for wind, rows where the wind speed is below the turbines' cut-in speed carry only noise, so zeroing them lowers the quantile score of Wind farms research default 1 on the kept rows of the three metered wind farms W1 to W3 at 1 to 10 day leads. Three readings are tested, each added alone: x1 sets the forecast to zero power whenever the ensemble-mean 100 m speed is below the farm's cut-in speed, x2 drops the ERA5 training rows below cut-in from the XGBoost model and the offset table, and x3 sets the training target to zero on those rows. At most one is adopted, in the order x1, x2, x3. | partly supported | candidate | Setting the wind forecast, or the training target, to zero below the turbines' cut-in speed does not improve the forecast, because the farms still produce small real output there. Leaving the low-wind rows out of training gains 0.12 points of capacity (about 1% of the default's score) on the kept rows, but mostly by lifting the quantiles upward, which a constant upward shift may reproduce (not yet tested). | also relevant |
| 00335 primary-demand-default-3-tail-calibration-at-0-5-to-240-hours | The 0.01, 0.05, 0.95, and 0.99 quantiles of Primary demand research default 3 miss their nominal rates past 96 hours (experiments 317 and 320 found 2.1% to 2.9% of observations below q01 and a 1.8% upper tail at 7 to 10 days). A calibration fitted per 24-hour lead band and per group (season, day or night, cold winter night) on the forecast errors of earlier origins brings the tails to nominal at every lead from 0.5 to 240 hours without worsening the quantile score. | not supported | not-for-production | Primary demand research default 3 is already close to calibrated at 1 to 10 days: its 5% to 95% band covers 89.3% to 90.9% and 1.1% to 1.95% of observations fall below q01. Calibrations fitted on earlier months move the tail shares only part of the way, and none is worth adopting. A table fitted in-sample on the same year would fix the tails and slightly improve the score, which shows the table can describe that year but not that the miss repeats across years, so testing the miss on a second year is the next step. | also relevant |
| 00332 lsc-curtailment-cost-of-label-free-mask | Treating as missing the readings that a label-free load shift detector flags (arm A of experiment 327) lowers curtailment cost during and after load shifts, against no mask (arm C), without raising it on ordinary half-hours. | not supported | not-for-production | Masking the load shifts that a label-free detector finds lowers curtailment cost by about a third when the forecast is judged against the normal running arrangement. Judged against the power the electricity network actually carried, which is what a curtailment instruction responds to, the mask changes cost by -3.0% with an interval from -24% to +30%, and it costs 1.3% more on ordinary half-hours. Adding the true moved megawatts to the true spans lowered cost during a shift, but adding them to the label-free mask did not, so this result cannot say what a live service needs. | also relevant |
| 00327 lsc-label-free-detect-mask-train-pipeline | A forecast of the normal running arrangement (NRA) can be trained and corrected on power with the load shifts set to missing, where a label-free detector finds the load shifts. The detector reads only how unusual the observed power is inside the predictive bands of a demand model that uses no lagged power, and no NGED switching label enters any threshold, mask, or training row. | partly supported | candidate | A forecast of the normal running arrangement recovers faster after a load shift when the readings that a label-free detector flags are treated as missing: during and after injected shifts the 95% pinball loss falls from 1.79 to 1.04 points of capacity. On ordinary days the masked forecast is no worse, and slightly better. Without one site (D11) the gain shrinks to 0.30 points, and retraining the demand model on masked power adds nothing beyond masking the filter. | also relevant |
| 00326 wind-research-default-1-change-arms | Each of four changes, added alone to Wind farms research default 1, lowers the quantile score on the kept rows of the three metered wind farms W1 to W3 at 1 to 10 day leads: (a) 2 m temperature and an air-density column as XGBoost inputs, (b) hour of day and day of year as XGBoost inputs, (c) the ERA5-trained neural adapter's effective wind speed as an extra XGBoost column, and (d) one wind-speed scale factor, fitted on the hindcast to the implied wind, applied to every member's 100 m speed before the XGBoost model. At most one is adopted, in the order c, b, a, d. | supported | candidate | Feeding XGBoost the neural adapter's corrected wind speed lowers the wind forecast's score by 0.10 points of capacity, about 0.9% of the default's score, at 1 to 10 day leads (99.75% interval -0.19 to -0.01). The change is negative at every farm and at short and long leads. The three other changes were not separable from zero. Only a port that puts the adapter in the production service can use the gain, so the maintainer decides. | also relevant |
| 00321 metered-farm-lower-tail-recalibration-and-tail-head | Metered solar farms research default 1 (the blend assembled by experiment 300, scored to 10 days by experiment 310) lets the actual output fall below its 0.01 quantile 3.3% to 3.7% of the time, and no mixture-density head tried in experiment 318 changes that. Default plus a recalibration table, a zero-inflated lower tail, or a tail-targeted head brings the share of actual values strictly below the 0.01 quantile close to 1% at leads of 1 to 10 days, without raising the quantile score and without losing experiment 318's 0.99 gain. | partly supported | candidate | The solar-farm forecast's lowest quantile was already close to correct: only 1.2% to 1.4% of actual values fall strictly below the 0.01 quantile, because the earlier 3.3% counted the 3.0% of half-hours that read exactly zero. A recalibration table, a zero-inflated head, and a tail head each lower the own-history score by 0.04 points or less (the unshrunk table raises it by 0.03 to 0.05 points), so none is adopted. A shrunk recalibration table does lower the score by about 0.05 points for a farm with no history, and two heads lower the 0.99 pinball loss by 3.7% to 8.5% while covering the 0.99 quantile only 98.3% to 98.9% of the time. | also relevant |
| 00312 lsc-how-good-must-a-detector-be-against-the-band-head | A load shift detector that is far from perfect can still beat the best forecast that uses no detector, so a detector need not be near-perfect to be worth building. The one detector-based arm in experiment 297 that beat that forecast was the one that subtracts the true megawatts from the readings during a shift and adds them back to the forecast. This experiment degrades that arm one axis at a time to find where it stops beating the no-detector forecast. It also asks whether the surprise of recent readings against GAM_ONLY's residual bands (band position), or perfect knowledge of a shift, added as inputs to the no-detector quantile head, improves the head. This is question 3 of the LSC decision program, redefined by the Discuss ruling of 2026-10-23. | not supported | not-for-production | A load shift detector does not need to be near-perfect to improve on the best forecast that uses no detector: with the true megawatts added back, finding half the shifts, learning of each up to 3 days late, or misjudging the megawatts by half still lowers curtailment cost against that forecast. The detector tested earlier (RO-NC) finds only a third of the shifts, below the break-even of one half (measured with every other property perfect). The surprise of recent readings against the forecast's bands (band position) found more, 37% of shifts on even months, but still under one half and with about 90% of its alarms false, so the next step is a detector that finds more shifts. | also relevant |
| 00309 lsc-two-forecast-gap-choosing-without-truth | NGED sees two forecasts and not the true switching state. A forecast with no lagged power (GAM_ONLY) forecasts the normal running arrangement, and a forecast with lags (K0) forecasts what will flow while an abnormal arrangement lasts. A rule that reads only the gap between the two forecasts, how long it has lasted, and a neighbour check can choose between them nearly as well as knowing the truth. This experiment is part 2 of the brief briefs/sw_lsc_two_forecasts_divergence_and_selection_brief.md, and it builds on experiment 308. | not supported | not-for-production | NGED cannot pick between the two forecasts from the gap alone. A rule on the gap, or on the 24-hour mean of measured power minus the no-lag forecast, did no better than always using the no- lag forecast in the injected worlds, and worse than always using the forecast with lags in the real record. An oracle that knows the true switching state gained little, except in the 72 hours after a real shift ended, so the most promising signal to test next is a flag for the end of a shift. The gap is a filtered copy of the measured-power residual, so it adds no information that the measured power lacks. | also relevant |
| 00304 metered-farm-node-sum-tail | For the solar farms that share a bulk supply point, adding each farm's quantiles overstates the upper tail of the farms' summed output, adding independent bands understates it, and summing the 51 ensemble members' farm forecasts member by member gives calibrated 0.95 and 0.99 quantiles, because the ensemble carries the farms' shared dependence on cloud. | not supported | not-for-production | The band for the combined output of the three farms beneath one substation comes out too wide if the weather forecast's 51 members are summed member by member, because the members' forecasts for the three farms move together almost perfectly while the farms' actual forecast errors are less closely linked. That band is almost the same as the band from adding each farm's quantiles. A band that assumes the farms' errors are unrelated is too narrow. A band from a Gaussian copula with a correlation fitted on the errors of the other 11 months comes closest to the stated probability. Only 5 half-hours in the scored year exceed 0.75 times a substation's maximum export, so this experiment cannot say what the band widths mean for curtailment cost. | also relevant |
| 00297 lsc-need-a-detector-no-detector-arms-against-the-mask | A forecast that never learns which readings are load shifts can match a forecast that uses a detector, because its bands and its recent-power correction can be made to tell a load shift from ordinary variation by themselves. This is question 2 of the LSC decision program. | not supported | not-for-production | A forecast with no detector that recalibrates its bands and reads how far the recent-power correction has moved scores better than the RO-NC detector near load shifts, so with RO-NC a detector is not shown to be needed. Perfect knowledge with the true megawatts added back still lowers curtailment cost by 6.5% (3.9 to 9.2) against that forecast, so the next step is to run the better LSC detector (LR_b20) on the same simulated shifts and judge it against a forecast that has the recalibrated bands. | also relevant |
| 00296 lsc-benefit-by-event-class-long-deep-versus-small-shifts | Load shifts of at least 2 days and at least 10% of a Primary substation's load carry most of the benefit of knowing every load shift exactly, and shifts that are shorter than 12 hours or under 10% of load carry little. This is the question behind the "long-and-deep" detector of the LSC decision program. | supported | not-for-production | A detector that finds only load shifts of at least 2 days and at least 10% of a substation's load would deliver about 96% of the upper-tail forecast benefit and about 90% of the curtailment-cost benefit of knowing every shift, if it were perfect. Detecting shifts from 6 hours to 2 days would add 3% to 8% more, and detecting shifts down to 5% of load would add 1% to 2%. The next step is to weight these shares by how often a real long-and-deep detector finds each class, which waits for Part 1 of the long-deep detector experiment. | also relevant |
| 00288 pooled-head-on-public-wind-farms | A neural head that adds 11 quantile offsets to the XGBoost wind forecast (R0) and was trained on dozens of public onshore wind farms (metered output from Elexon's Balancing Mechanism Reporting Service) lowers the quantile score of the three metered wind farms (W1 to W3) against R0 alone, and a head trained on the pool beats the same head trained on W1 to W3 only, because experiment 259's head had too few farm-years to learn a correction that transfers. A second hypothesis: XGBoost keeps improving as the training history grows from 12 to 48 months while the physics core is flat after 24 months, and a pool of public farms replaces the missing history. | not supported | not-for-production | Training the correction on 40 public wind farms did not help the forecasts for the three NGED wind farms. Applied directly, it made them worse, because the public farms' capacity, turbine rating, and hub height let the head memorise each farm. Without those inputs it did no harm, and after a short fine-tune on the three farms' own data it gained about as much as a simple offset table fitted on those farms, so the public farms added nothing measurable. | also relevant |
| 00284 metered-farm-tail-head-past-weather | Experiment 275 found that adding years of past-weather rows lowered the 0.99 pinball loss of the neural head H5 by 4.6% while its overall score moved by 0.47%. A small neural network trained with a loss restricted to the tail quantiles (0.01, 0.02, 0.05, 0.95, 0.98, 0.99), on the same past-weather plus ensemble rows with an era flag, lowers the 0.01 and 0.99 pinball loss of a metered solar farm's forecast below that of XGBoost trained on the same rows. | partly supported | not-for-production | A neural network trained only to forecast the extreme edges of a solar farm's output band does beat XGBoost at the 0.99 quantile (the upper edge that matters for curtailment) by 30%, but the existing H5 network does slightly better still, and most of the gain comes from building the band around the physical power model and not from tail-only training. The years of past-weather rows add under 1% at the 0.99 quantile, and at the 0.01 quantile the share of half-hours below it is outside 0.5% to 1.5% at three farms (marginally at A, because of exact zeros at D, and clearly only at E). | also relevant |
| 00283 lsc-perfect-knowledge-value-to-forecast-and-curtailment | Knowing every load shift exactly (its true start and end) lowers the error of the forecast of the normal running arrangement (NRA) in the 48 hours after a shift, and lowers the curtailment cost of the forecast of observed power in the days around a shift, while leaving ordinary rows unchanged. This is question 1 of the LSC decision program. | partly supported | not-for-production | Perfect knowledge of load shifts lowers the forecast's error most in the 48 hours after a shift ends (23% lower upper-tail score, 95% interval 9% to 39%) and by under 1% over a whole year, because shifts are rare. Curtailment cost against observed power falls by 8% (6% to 10%) near a shift only when the known shift's megawatts are added back to the forecast, and does not fall when recent readings are merely set aside. The next step is to ask how good a detector must be to keep that gain, using the megawatts-added-back arm as the target. | also relevant |
| 00277 lsc-curtailment-metric-on-existing-forecasts | The curtailment-cost metric of experiment 268 (studies.node_curtailment, sorted quantiles) ranks the four existing forecasts as the other metrics do, and can resolve the differences between them well enough to judge the load-shift questions of the decision program. | supported | not-for-production | Over a year of ordinary days, the curtailment metric ranks the four existing forecasts the way the other scores do, and it can tell the GAM stack from BG2 (BG2 costs GBP 7.35 million more on the six Primaries at the 0.5 limit, a figure that ranks forecasts and is not money saved). Inside the windows around the 7 logged load transfers, the GAM stack cannot be separated from BG2, because those windows hold only 926 to 2,923 scored half-hours. Questions about load shifts need injected worlds with many more shifts, not the real record. | primary |
| 00271 lsc-load-shifts-net-out-at-bulk-supply-points | A load shift between two Primary substations under the same bulk supply point (BSP) leaves the power flow at the BSP unchanged, so it barely changes the curtailment cost at the BSP, and only a load shift that crosses from one BSP to the other changes the cost noticeably. The step 7 program brief for the load-shift (LSC) decision program asks this in order to know whether the money question can be asked at BSP level. | partly supported | not-for-production | A load shift between two Primary substations under the same bulk supply point changes the curtailment cost at that bulk supply point by 0.2% at a random time and 0.7% just before days that would be curtailed (interval up to 1.08%), because the two Primaries' forecast errors partly cancel. A load shift across the boundary between the two bulk supply points changes it by 1.4% at a random time and 5.5% just before curtailed days. Only 4 of the 54 hand-filtered real events that involve two Primaries cross the boundary, so the curtailment metric at bulk supply points cannot say whether detecting load shifts is worth money, and that question has to be asked at Primary substations. | also relevant |
| 00268 lsc-curtailment-metric-v2-member-wise-ens | A member-wise input path added to studies.node_curtailment (ensemble members of the node's demand and of each farm's output, paired by member, reduced to the quantiles the rule reads) gives the same avoidable cost as the quantile path when the members are drawn from the same quantile function, and scores nodes that experiment 255 could not (Primaries that connect a farm, and the bulk supply points D13 and D17 with member-wise inputs). | partly supported | not-for-production | The curtailment metric can now take ensemble members for a substation's demand and for each solar farm, and on experiment 255's rows it reproduces the earlier answer. A third of the stored solar-farm forecasts list their quantiles out of order. Sorting them moves the cost of some forecasts by up to 3.7% and leaves every comparison between forecasts with the same sign, but it swaps which of two near-equal forecasts is cheapest. Forecasts that are already quantile sets can be scored now. Forecasts driven by ensemble members need member frames that we did not find in the experiment folders we searched. | primary |
| 00265 dp-gas-engine-bands-prior-and-visibility-switch | Widening a Primary's forecast bands from its own past residuals (experiment 264, arm 2b) has coverage far from nominal in the first quarter, when there is no history, and after a change in the site's behaviour. A prior taken from the other Primaries' residuals, shrunk towards the site's own trailing residuals as they accrue, fixes the first quarter. A causal switch on the share of large drops in the trailing 30 days returns the widening to normal when D8's engine becomes visible in the readings (around April 2026). Together they hold D8's q05-to-q95 coverage within 5 points of 0.90 in every quarter, without raising the cost in quantile score, and without harming Primaries that have no engine. | not supported | not-for-production | In this year of forecasts, the other five Primaries' residuals did not predict D8's errors, so a starting estimate from them left D8's bands too narrow in the first quarter, and returning to normal widening after the engine's drops faded made them too narrow again. Widening from D8's own history (experiment 264) remained the best of the arms tried. We tested no other fix for the first quarter, and none of the other five Primaries had a heavy lower tail. | also relevant |
| 00264 dp-gas-engine-asymmetric-bands-at-d8 | D8's forecast bands are too narrow in both directions, and part of the lower-tail miss is a 6 MW gas engine (probably three units of about 2 MW) that runs in spells and cannot be detected. A spread that widens as-of, using only residuals already resolved at the forecast origin, and that widens most in the direction the engine pushes (net demand falls), gives honest q05 to q95 bands at D8 without raising the quantile score at the Primaries that have no engine. | not supported | not-for-production | A forecast band that learns from the residuals it has already seen narrows most of the gap at D8: the quantile score falls by 9% to 19% and the share of readings inside the 90% band rises from 42% to between 80% and 87%. The extra gain of the engine mixture over simple widening is not evidence of an engine, because the mixture fits a running engine at two substations that have none. Plain two-sided widening with an earlier residual history is the option these results support, and NGED's own output for the engine would be the data to test an engine model. | also relevant |
| 00259 wind-plant-plus-quantile-offset-head | For the three metered wind farms (W1 to W3), a small neural network that reads the 51-member mixture of a wind model and the ensemble's wind summaries, and adds an offset to each of 11 quantiles, lowers the quantile score against the wind model alone (R1, the physics core), matches the XGBoost best guess (R0), and lowers the pinball loss at the 0.99 quantile, once it can learn from daily runs of the ensemble forecast over 15 months (about 440 runs) instead of the 13 to 40 Tuesday runs that experiment 221's head saw. | partly supported | candidate | A correction learned from 15 months of daily ensemble forecasts lowered the wind forecast's quantile score by 4.5% to 5.5% on the three metered wind farms, but the neural network and the ensemble's wind summaries were not what earned it. The 51 ensemble members' median sits too low: the observed power was below the physics core's median on 34% of rows, not 50%. A lookup table with one offset per lead day, quantile, and farm recovers 92% of the gain. The open question is where that low bias comes from. | also relevant |
| 00255 metered-farm-node-curtailment-value | When six metered solar farms beneath two bulk supply points are curtailed by the node-plus-generators rule of studies.node_curtailment, a forecast of each farm's output (the plant, XGBoost, or the plant with a neural head) avoids more curtailment cost than the NGED baseline, which assumes every generator runs at full capacity, and the neural head's lower 0.99 pinball loss does not show in the metric, because the rule uses only a farm's median. | inconclusive | not-for-production | Forecasting each solar farm's output instead of assuming every farm runs at full capacity cut the cost of curtailing at one of two substations (Boston, 40% lower) and raised it at the other (Skegness, 24% higher), so across both the saving is about a quarter, smaller than the uncertainty in the estimate. The neural network's better upper tail does not show in this cost, because the rule reads only each farm's median, and it lowers the cost by 3% against XGBoost, far under the 10% that would justify serving a neural network. Ranking farm forecasts by curtailment cost needs many more substations. | also relevant |
| 00251 dp-final-best-guess-assembly-clip-and-coast-against-bg2 | The full best-guess stack of experiment 220 (arm D, the neural-network head, and the two-number solar summary) beats BG2, the manual heuristic, climatology, and a weak production-style XGBoost model over the scored year of 2025-07 to 2026-06. Adding the hand-set 60% clip at D5 lowers the D5 error on sunny half-hours when the neural-network head is absent, and adds nothing measurable when the head is present, because experiment 232 found the head cancels most of the clip's gain. | supported | candidate | Over the one year we can score, the full best-guess demand forecast (arm D, a small neural network that corrects its median, and a two-number summary of the ensemble's solar response) beats BG2 by 0.136 points of quantile score (5%; lower is better) and the manual heuristic by 0.818 points, and it beats the manual heuristic on every headline metric we scored, including curtailment cost. The gain over BG2 is a winter gain: the stack is level with BG2 in summer and worse in April to June, and at D5 on sunny hours it is clearly worse than BG2. Adding the 60% solar clip at D5 helps only when the neural-network head is left out at D5, so the next work is the solar term at D5, not another stacked part. | also relevant |
| 00213 calibration-online-band-adaptation-from-coverage-misses | Moving a scale on each quantile offset of experiment 169's arm A up after a reading falls outside that quantile, and down after a reading falls inside it, in the style of adaptive conformal inference (Gibbs and Candes, 2021), lifts the cold-night q95 coverage to 0.90 or more without a second fitted spread model, because a run of misses on a cold spell widens the band before the spell ends. | inconclusive | not-for-production | Letting the forecast widen or narrow its bands after each recent miss does not fix the under-coverage on cold nights: it lifts the 95% quantile's coverage from 0.896 to 0.906 in the scored year and lowers it in the other months, and it worsens the all-row, bank-holiday, and Christmas scores. A fixed widening of the upper quantiles does as well, so the cold-night gap is a shortfall that a signal arriving after the misses cannot learn in time, and the next step is to correct the mean forecast on the coldest days before any further work on the bands. | also relevant |
| 00169 calibration-combined-fixes-and-ensemble-spread | The spread across the 51 ensemble members of the temperature and of the downward shortwave radiation at a substation predicts how wide the forecast's residuals are, beyond the hour band, calendar quarter, lead, and the cold-day widening of experiment 164. A convex quantile model that scales the quantile offsets by a non-decreasing function of those two spreads lifts the cold-night q95 coverage of the combined forecast to 0.90 or more. | partly supported | not-for-production | Combining the smooth spread table with the cold-day widening reproduces its 0.896 coverage of the 95% quantile on cold nights, and adding the spread of the ensemble weather forecast (how much its 51 members disagree) lifts that to 0.907. The lift comes from widening the 95% and 99% quantiles by 4% and 10%, which the same model gives with its spread terms removed, so the ensemble spread is not what lifts cold-night coverage. A later experiment could test a bias correction of the mean forecast on the coldest days, and judge cold nights over more than one winter. | also relevant |
| 00164 calibration-cold-night-tail-under-coverage | The q95 coverage of experiment 160's smooth-table forecast on cold nights (day mean temperature below 2 °C) stays near 85% because the spread table is too narrow when it is cold, not because the mean forecast is biased. Widening the quantiles on days the ensemble forecasts as cold, by a factor fitted on the tuning months, lifts cold-night q95 coverage to 90% or more without worsening the all-row score. | refuted | not-for-production | On cold nights the 95% quantile of the best forecast is exceeded on about 15% of readings, and mostly because its prediction interval is too narrow, not because its mean forecast is biased. Widening only the upper quantiles on days the weather forecast calls cold reduces that share to about 10% without changing the all-hours score, but no fix reached the pre-registered 90% coverage in the scored year, and only 10 cold days back that year's figure. The coldest forecasts (0 °C or below) are also biased low, so correcting the mean forecast on those days is the next experiment to run. | also relevant |
| 00160 calibration-band-width-fix-on-stage-2-winner | The q05 to q95 band of the stage 2 winning arm (experiment 155's arm D with arm A's spread table) widens in steps along the lead, because the spread table has only four lead bands (up to 24 hours, 48, 144, and 168), fitted separately per hour band and calendar quarter. Interpolating each hour band by quarter cell between the lead-band centres, forcing the width to be non-decreasing in lead, or both, removes the steps without worsening the quantile score. | supported | candidate | The 90% prediction interval (from the 5% to the 95% quantile) of experiment 155's best forecast was equally wide on lead days 2 to 5, because its stored spread table has only four lead bands. Interpolating the table between the band centres removes the equal days and raises the quantile score by 0.002 points, below the pre-registered ceiling of 0.01, but it narrows the interval at 24 to 72 hours and widens it from 72 hours on, so one substation loses 0.009. It does not fix the cold-night upper tail: the forecast exceeds the 95% quantile on about 15% of cold-night rows, against 5% for a calibrated forecast. | also relevant |
| 00125 differentiable-physics-n8-ens-members-through-gam-k-and-the-spread-around-them | Placing GAM-K's residual quantiles, fitted on a forecast driven by the analysis proxy (so they hold model and filter error and almost no weather error), around each of the 51 ENS members gives tails closer to nominal than today's table, which is fitted on ensemble-mean residuals, and the gain is largest at long leads and on cold winter nights, where the hinge turns a temperature error into a large demand error. The comparison is against GAM-K re-tuned on the same factor grid, so a plain scale error cannot pass for a better design. | inconclusive | not-for-production | Putting the forecast-error table around each of the 51 weather members, and fitting it on analysis-driven forecasts so that it holds no weather error, did not improve the tails of GAM-K's demand forecast. The weather members' spread is small against the model's own error, so the weather error was counted twice in the code but is too small to matter. The next step is the cold winter nights, where demand exceeds the 95% quantile about a quarter of the time in every arm. The verdict is inconclusive, because no arm reached 14 of 16 calibration cells. | also relevant |
| 00084 substation-limit-curtailment-cost-worked-example | The substation-limit curtailment metric (studies.node_curtailment) can run end to end on the pilot's two bulk supply points, Boston and Skegness, and the pilot holds enough half-hours beyond a scaled limit to rank forecasts. This is a worked example of new code, not a test of a forecasting idea. |
inconclusive | not-for-production | Two bulk supply points have 19 half-hours beyond a limit of 0.9 times their earlier maximum export in the scored year, so the pilot proves the arithmetic and cannot rank forecasts at the headline limit. | primary |
| 00083 curtailment-cost-and-tail-calibration-worked-example | A tail-calibration table, built in packages/studies, can rank the stored quantile forecasts of experiment 48 by how well they cover the tails of the forecast distribution. This is a worked example of new code, not a test of a forecasting idea. |
inconclusive | not-for-production | Every forecast is too narrow in both tails: coverage of the 0.05 quantile is 0.066 to 0.085 and of the 0.95 quantile is 0.891 to 0.927. | primary |