Weather products and blending: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00371 d8-monthly-xgboost-input-group-ablation | One input group of monthly XGBoost (the weather inputs the GAM lacks, the three origin lags, the 0.5-year recency weighting, or the 51-member mixture) accounts for most of its 0.64-point lead over the GAM stack at substation D8. | supported | not-for-production | Monthly XGBoost beats the GAM stack at substation D8 mostly because it uses three lagged values of the power measured shortly before the forecast starts: without them its score rises from 7.70 to 8.25, within 0.1 of the GAM stack's 8.34. The five extra weather inputs (0.05) and the ensemble mean (0.01) explain little of the gap, and the recency weighting (0.23 at D8) explains a smaller part and only at D8. The lags also help at two other Primary substations (D4 +0.28, D2 +0.11), so the pre-registered expectation that only D8 would respond failed. A GAM term that uses recent power is the next candidate to test. This is one substation and one scored year. | also relevant |
| 00343 wind-research-default-1-train-on-ensemble-mean | Wind farms research default 1 trains XGBoost on ERA5 and runs it on each of the 51 ensemble members, then corrects with the offset table. The ensemble's 100 m wind speed at the grid cell is lower than at the turbines (a factor of 1.13 closed 78% of the gap to the ERA5 oracle in experiment 307). An XGBoost model trained on the 51-member mean of the ensemble's own history (arm M, 2024-04 to 2025-06, no calendar columns) would learn that bias and need less offset-table correction. Arm MS adds the member standard deviation of the 100 m and 10 m speeds. The three metered wind farms W1 to W3 are scored at 1 to 10 days. | not supported | not-for-production | A wind model trained on the ensemble mean needs almost no offset table (the table changes its score by 0.017 points, against 0.345 for the default), but its forecast is no better than the default's: 0.08 points worse on the headline rows, an amount the data cannot distinguish from zero, and worse at low winds and in the 0.99 quantile. The default's recipe (ERA5 training plus the table) stays; mean training is a simpler recipe that scores about the same. | also relevant |
| 00333 primary-demand-foundation-model-zero-shot | A pre-trained time-series foundation model (Chronos-2, 120 million parameters, Apache-2.0) forecasts a Primary substation's half-hourly demand from the substation's own history. Without weather inputs Chronos-2 cannot match Primary demand research default 3 at leads of 1 to 10 days. With the ensemble-mean temperature, the sun's elevation, and calendar columns passed as covariates, Chronos-2 closes part of the gap, but not all of it. | partly supported | not-for-production | A pre-trained time-series foundation model, used without any training on our data, forecasts Primary demand clearly worse than Primary demand research default 3 at every lead from 0.5 to 240 hours. Giving the foundation model the temperature forecast, the sun's elevation, and the calendar recovers about one sixth of the gap, and makes sunny-daylight half-hours worse. Chronos-2, the one foundation model we tested, is not a candidate to replace the default, and a foundation-model forecast would have to be scored alone against the default before use as a blend member. | also relevant |
| 00329 primary-demand-default-3-member-solar-distribution-at-7-to-10-days | At 168 to 240 hours the 51 ensemble members have little skill for individual days, so the forecast centres on average sunshine and the half-hours that turn out sunny have realised demand below the median. The weakness of Primary demand research default 3 on sunny daylight (5.07 against the manual heuristic's 4.82 in experiment 317, for the convex chain before any blend) is therefore a fault of the predictive distribution (the lower tail on realised-sunny half-hours), not of the median. Replacing the two-number Gaussian summary of the members' solar responses with the 51 responses themselves (X1), mixing them with a climatology of the response (X2), or adding the members of the two previous daily runs (X3) fixes the distribution. | not supported | not-for-production | The 7-to-10-day weakness on sunny half-hours no longer exists in the research demand forecast: the two-number solar summary and the daylight-only blend with the manual heuristic already took it from 5.2% worse than the heuristic to 6.6% better, so no new distribution method is needed there. What remains is one coastal Primary (D5), where the forecast is 11% worse than the heuristic, which points back at the solar term and not at the width of the bands. | also relevant |
| 00325 primary-demand-default-3-sunshine-response-refit-and-restriction | Experiment 323's sunshine response (F1) harmed all daylight half-hours because the Kalman filter's level, the winter night offset, and the median correction were estimated on the errors of a fit without a response and so counted the same error twice (double counting), or because the response acts too widely, at forecast clear-sky index where the damped ensemble-mean forecast index mis-applies the response. Refitting those corrections on the response's own errors (F1r) removes the first cause, and restricting the response to clear-sky index above 0.6 and sun elevation above 15 degrees (F1s) removes the second. | small gain | not-for-production | The sunshine response, once the rest of the forecast chain is refitted around it, no longer makes daylight forecasts worse, but the response improves the overall forecast by under 0.01 points, too little to adopt. The response improves spring daylight at two Primaries (D5 and D9) by about 0.11 points. | also relevant |
| 00323 primary-demand-default-3-flexible-sunshine-response-in-the-convex-fit | Primary demand research default 3 trails the depth-1 XGBoost refit on April-to-June daylight half-hours because its solar term (the pinned capacity times the derated unit shape, with the D5 clip) is too high on clear spring middays. Experiment 322 grafted the refit's sunshine response onto the default and recovered 0.50 of the refit's advantage. A smooth correction in clear-sky index and sun elevation, fitted in the same bounded least-squares fit as the solar term, recovers a useful part of that gain with no model of other inputs. | not supported | not-for-production | Adding a flexible sunshine curve to the solar part of the demand forecast does not recover the gap to the better-performing XGBoost model: it helps the sunniest half-hours but worsens the pooled score by about 0.4%. The advantage seen in experiment 322 is therefore not recovered by adding a smooth curve in clear-sky index and sun elevation on top of the existing level corrections, and most of the harm sits in the level shift. A boosted-tree sunshine term could use more inputs at once, and whether to test it is a decision for the research plan. | also relevant |
| 00319 primary-demand-default-1-solar-summary-and-heuristic-blend-at-7-to-10-days | Primary demand research default 1 has two weak spots at 168 to 240 hours: sunny daylight half-hours, where it scores 5.2% worse than the manual heuristic, and April to October, where it beats climatology by only 4%. Default plus x1 (the two-number solar summary S, the mean and standard deviation of the 51 ensemble members' solar responses, extended from 0.5 to 168 hours out to 240 hours) lowers the sunny-daylight score at 168 to 240 hours. Default plus x2 (a blend of the convex chain's median with the manual heuristic's, with the weight fitted per 24-hour lead band and season group) lowers the pooled score at 168 to 240 hours, mostly in April to October. | partly supported | candidate | The demand forecast for 7 to 10 days ahead improves by about 2% when its solar term also uses the spread between the 51 weather-forecast members, and the gain is largest on sunny days (6.6%), where the forecast had been worse than the manual heuristic that National Grid Electricity Distribution (NGED) staff use. Blending in the manual heuristic improves the forecast by 4.5%, but the blend worsens the coldest winter nights beyond the agreed tolerance, so only the solar change is adopted. | also relevant |
| 00307 wind-farm-power-as-wind-sensor | The wind that the three metered wind farms' own power implies (the fitted physics core's power curve run backwards) is closer to a high-resolution analysis than to ERA5 at 9 to 14 m/s, and at least half of the ensemble's shortfall (observed power below the forecast's median on 34.1% of rows instead of 50%) is in the spread and timing of the members, not in their mean speed. | not supported | not-for-production | The ensemble's low wind-power forecast comes mostly from wind speeds that are too slow, by about 1.1 m/s on average and more at longer leads. Multiplying every member's wind speed by a factor fitted on the 14 months before the scored year closed about 78% of the gap to the ERA5 oracle, while widening the members or shifting them in time closed almost none. The median is slightly better calibrated when the factor is fitted to the wind that the farms' power implies than when it is fitted to ERA5 on the same rows (1.7 percentage points), but the quantile score is not better. | also relevant |
| 00292 metered-farm-ens-neighbourhood | A day-ahead cloud field is often right in shape and wrong in position by tens of kilometres, so summaries of the ensemble's irradiance over the hexagonal cells around a metered solar farm carry information that the farm's own cell lacks. Giving H5 (the plant plus a neural network that outputs all 11 quantile offsets) those summaries lowers its quantile score at 12 to 36 hour leads. | not supported | not-for-production | At the six metered solar farms tested, the ensemble's irradiance on the cells around a farm adds almost nothing to the day-ahead forecast: the quantile score improves by 0.36%, against the 2% the plan needed, and the gain is not robust to dropping one farm. Adjacent cells, 15 km apart, differ from the farm's own cell by only about 3% of the irradiance, so the ensemble mean is too smooth between them to show where a cloud edge will fall. A weather source of a different kind is the next experiment to run, rather than more processing of the same ensemble. | also relevant |
| 00272 metered-farm-weather-adapter-gam-vs-neural | Extra weather variables that the solar-farm power model (the plant) ignores, fed through a learned correction to the plant's irradiance and temperature, close part of the gap between the plant driven by the ensemble weather forecast (about 0.91 of output fitted on sunny half-hours) and the plant driven by observed weather (0.973), and a small neural network does this better than additive curves. | not supported | not-for-production | The extra ensemble weather variables (longwave radiation, humidity, precipitation, pressure, wind, and 500 hPa height) showed no usable information for the solar-farm power model at these six farms, whether they enter through additive curves or a small neural network, and the real variables do no better than placebo copies. A neural network with no plant, given the lagged power and the ensemble spread, scores 2.7% better than the plant but not better than the linear fit of experiment 257, so a correction to the plant's weather is not the route to a better forecast. | also relevant |
| 00270 metered-farm-xgboost-vs-plant-diagnostic | XGBoost beats the plant (the solar-farm power model of experiments 218 and 224) on the six metered farms because of inputs the plant never sees, namely the farm's own lagged power and the ensemble spread, and not because of a better map from the ensemble-mean weather to power. Calendar columns and extra weather variables add little once those two are present. | partly supported | candidate | XGBoost beats the solar-farm power model by about 3% in the quantile score even when it sees only the power model's own weather inputs, and the gain is in the width of the forecast band (the 0.05 and 0.95 quantiles), not in the median forecast. The same gain comes from giving the power model a forecast band that depends on the ensemble spread or the clear-sky index, and that band also beats XGBoost at the 0.99 quantile. Lagged power, calendar columns, and extra weather variables add little in own history. | also relevant |
| 00262 wind-aifs-members-and-climatology-by-lead | For the three metered wind farms (W1 to W3), a lead-dependent mixture of the ECMWF ensemble's member forecasts, the AIFS ensemble's member forecasts (both run through experiment 230's ERA5-trained XGBoost wind model A0r), and climatology lowers the quantile score and the 0.99 pinball loss against the ECMWF members alone, after both are corrected by a table of offsets per lead day, quantile, and farm; and a small neural gate that reads the two ensembles' spread and disagreement chooses better weights than weights by lead alone. | partly supported | candidate | Adding the AIFS ensemble's wind forecasts to the ECMWF ensemble's improves the wind forecast at 3 to 6 days ahead, by about 5% of the quantile score, but does not improve the high-power tail that matters for curtailment. Climatology and a small neural network that picks the mixing weights add nothing once the table of offsets has corrected the common low bias. The AIFS arms also need testing on a second year, because one year of Tuesday runs sets the weights. | also relevant |
| 00246 metered-farm-ens-ukv-icond2-blend | The DWD ICON-D2 regional weather forecast (2 km grid, run every 3 hours, 48 hours ahead) carries information about cloud over a metered solar farm that neither the ECMWF ensemble nor the Met Office UKV carries, so adding it to the ensemble plus UKV lowers the farm's day-ahead error. The evidence against is that at day 1 every single product lost to the ensemble mean for solar in the matched-lead study, and ICON-D2's errors are likely to be correlated with UKV's. Every gain is an upper limit for production, because the UKV archive used is the CEDA research archive. | small gain | not-for-production | Adding the German ICON-D2 weather forecast to the ECMWF ensemble and the Met Office UKV lowers the day-ahead error of a metered solar farm by about 1%, which is under the 1.5% we set as worth a fourth weather source. ICON-D2 is available for only 12 to 29 hours of the day ahead at an 08 UTC forecast time, and the UKV archive used is a research archive, so this is an upper limit for production. | also relevant |
| 00237 dp-two-number-solar-summary-with-clip | The mean and standard deviation of the 51 ensemble members' solar responses lower the daylight quantile score of the demand forecast also when each member is passed through the clipped solar term, because cloud uncertainty and the clip both act at midday only partly on the same rows. | supported | candidate | At D5, in one year of data, summarising the ensemble weather members as a mean and a standard deviation keeps at least about 90% of its gain when the solar output is clipped at 60% of the solar capacity held for D5 (110%, interval 91% to 141%). On the sunniest third of daylight rows the point estimate of the remaining gain is about half (interval -12% to 88%). A post hoc arm using the standard deviation alone kept the gain there, which suggests testing that form. | also relevant |
| 00229 metered-farm-ens-ukv-blend | The Met Office UKV weather model (2 km grid, 54 hours) carries information about cloud over a solar farm that the 51-member ECMWF ensemble lacks, so giving a metered solar farm's forecast both products lowers its day-ahead error against the ensemble alone, and closes part of the gap between the ensemble-driven fit (0.91 of output on sunny half-hours) and the fit with observed weather (0.973) of experiment 218. Every gain is an upper limit for production, because the UKV archive used here is the research archive on CEDA, which differs statistically from the live UKV. | small gain | not-for-production | Adding the Met Office UKV forecast to the ECMWF ensemble lowers the day-ahead error of a metered solar farm by about 1% to 2%. The gain is under the 3% we set as worth a second weather source, and the farms' fit on sunny half-hours moves only about 6% of the way to the fit with observed weather. The UKV archive used here is a research archive, and the live UKV differs, so this result is an upper limit for production; a weather product of a different kind is the option we see for a larger gain. | also relevant |
| 00202 dp-ens-two-number-rolling-replay | Experiment 193 found that two numbers per half-hour (the mean and the standard deviation of the 51 ensemble members' solar responses) lower the daylight quantile score by 2.20% in the scored year. Its two fitted factors per lead band were fitted on interleaved months (the odd months of 2024-07 to 2025-06), some of them later than the replicate months they were applied to, and its numbers were held in float64. A replay of the live weather layer, which stores the two numbers as float32 columns computed from the one ensemble run published before each origin and refits the factors each month on history that has completed before that month, keeps most of the gain. | supported | candidate | A weather layer that stores two numbers per half-hour (the mean and spread of the 51 ensemble members' solar-generation estimates) and refits its two scaling factors every month, using only runs that had finished before the month began, lowers the daylight quantile score (the score for the forecast's uncertainty bands) by 2.3% at five substations over one year, which is as much as running all 51 members. This confirms experiment 193's result under production-style timing, so the next step is to build the two columns into the weather layer. | also relevant |
| 00193 dp-ens-two-number-solar-summary | Experiment 187 found that passing each of the 51 ensemble members through the solar term lowers the daylight quantile score of the full best-guess stack by 2.23%, and that a Gaussian with each row's own mean and spread of the 51 responses recovers 95% of that gain. Two numbers per half-hour (the mean and the standard deviation of the members' solar responses) therefore carry nearly all of the gain, so a serving path can store two columns instead of running 51 passes. | supported | candidate | Summarising the 51 ensemble weather members' solar estimates by their mean and standard deviation lowers the daylight quantile score by about 2% (98% of the gain from running all 51 members, 93% in the replicate months), at the five substations scored over one year. The weather layer could store two columns per half-hour instead of 51, and the 10th and 90th percentiles add nothing measurable. A follow-up experiment could test the two numbers in the live weather layer. | also relevant |
| 00192 dp-ens-members-temperature-cold-nights | Experiment 187 found that passing the 51 ECMWF ensemble members through the solar term of the demand model lowers the daylight quantile score by 2.2%. The demand response to heating is also nonlinear (a hinge at 15.5 degrees C), and the ensemble mean smooths away the cold tail. Passing each member's temperature through the temperature terms therefore lowers the quantile score on cold winter nights. | not supported | not-for-production | Running each ensemble member's temperature through the demand model gave no detectable improvement in the forecast bands on cold winter nights, because the 51 members' temperatures differ too little to signal the day's error. The cold-night bands are already too narrow. The cause of the cold-night error is still unknown; widening the bands directly, and testing wind, night cloud cover, and switching as inputs, are the candidates. | also relevant |
| 00187 dp-ens-members-in-stack | Experiment 184 found that running each of the 51 ensemble members through the solar term of the convex demand model lowers the daylight quantile score by 2.2%, but its verdict was capped at inconclusive because a monotone gate failed on rows where the unit PV shape itself falls as irradiance rises. With a gate that matches that shape, the same arm passes, and its gain lies where the best-guess stack leaves the largest sunny-daylight error, at the substation D5. |
supported | candidate | Running each of the 51 ensemble weather members through the solar part of the demand model still improves the daylight forecast bands by about 2% on top of the full best-guess stack, and night is untouched. About two thirds of the gain needs that day's own members, and two numbers per half-hour (the mean and spread of the members' solar responses) carry nearly all of it, so a cheaper two-number version is the next thing to test. | also relevant |
| 00184 dp-ens-daylight-calibration | Experiment 169 found that passing the ensemble's spread to the width of the forecast's bands adds nothing when scored over all half-hours. The benefit may be confined to daylight half-hours, especially sunny and partly cloudy ones, where the clouds that the 51 ensemble members disagree about change the output of embedded solar generation, and so the demand. Passing each member's irradiance through the convex model's solar term (arm M), or using the spread of the members' irradiance as a fraction of clear-sky to set the band width in daylight (arm S), calibrates the daylight bands better than the existing bands (arm A). | inconclusive | not-for-production | Running each of the 51 ensemble weather members through the solar part of the demand model improves the daylight forecast bands by about 2%, mostly in the lower tail and on partly cloudy half-hours, while night is untouched. Placebos that widen the bands the same way but without the real members gain little, so most of the benefit is information about cloud. The next step is a production-cost check of the member pass and a test of its gain on top of the best-guess forecast. | also relevant |
| 00174 dp-unlabelled-worst-days-weather-shared-or-single-site | Most of the excess loss on the unlabelled worst days of experiment 172 is weather error, and the unlabelled worst days are mostly shared with other Primaries. Replacing the ensemble-mean weather with the estimate of past weather (ERA5 temperature and CAMS irradiance) in arm D's forecast removes at least 50% of the unlabelled days' excess loss, and at least a third of the unlabelled days are also in the worst 5% of days of at least one other Primary. | partly supported | not-for-production | Most of the forecast's error on its worst unlabelled days is not weather error: with past weather in place of the forecast weather, about 69% of the excess loss stays. Other Primaries share these days about as often as random shifting gives, so the data show at most a weak shared cause. The remaining loss could therefore come from events at one substation, such as undetected switching, a change in behaviour, or a metering fault. The next place to look is the substation's own recent readings and the switching records on those days. | also relevant |
| 00125 differentiable-physics-n8-ens-members-through-gam-k-and-the-spread-around-them | Placing GAM-K's residual quantiles, fitted on a forecast driven by the analysis proxy (so they hold model and filter error and almost no weather error), around each of the 51 ENS members gives tails closer to nominal than today's table, which is fitted on ensemble-mean residuals, and the gain is largest at long leads and on cold winter nights, where the hinge turns a temperature error into a large demand error. The comparison is against GAM-K re-tuned on the same factor grid, so a plain scale error cannot pass for a better design. | inconclusive | not-for-production | Putting the forecast-error table around each of the 51 weather members, and fitting it on analysis-driven forecasts so that it holds no weather error, did not improve the tails of GAM-K's demand forecast. The weather members' spread is small against the model's own error, so the weather error was counted twice in the code but is too small to matter. The next step is the cold winter nights, where demand exceeds the 95% quantile about a quarter of the time in every arm. The verdict is inconclusive, because no arm reached 14 of 16 calibration cells. | also relevant |
| 00119 differentiable-physics-enc-weather-patch-encoder-on-ens | A small neural network that reads the ENS grid boxes around a Primary, and not only the one box GAM-K reads, lowers GAM-K's quantile score a little on average and more on the cold, showery, and badly forecast days NGED cares about, either as a correction to GAM-K's quantiles or as a replacement for its single-box temperature and irradiance. | inconclusive | not-for-production | A small neural network that reads the ensemble weather around a Primary lowers GAM-K's quantile score by 0.062 points of effective capacity in one scored year and 0.035 in the other (both statistically significant, 5 of 5 substations better), and the gain is smaller in winter (0.041 and 0.027 points) but never a loss. About two-thirds of the gain survives when the network reads the wrong date's weather (0.043 and 0.022), so it is probably a recalibration of GAM-K's quantiles with the lead, the hour, and GAM-K's own forecast, and the surrounding boxes add nothing to the centre box (patch minus centre box: -0.002 and +0.003 points). The known-answer gate failed, so the verdict is inconclusive, and the next step is a plain recalibration of GAM-K's quantiles with the centre box's extra fields (dew point, pressure, precipitation, and member spread), not a patch encoder. | also relevant |
| 00111 differentiable-physics-6-ens-forecast-engine-against-bg2 | Driven by ECMWF ENS forecasts, the convex DP demand model forecasts Primary demand up to 7 days ahead better than lagless XGBoost given the same recent-level correction, and, used as the base margin of the BG2 recipe's lagged XGBoost, makes the production recipe better like for like. | partly supported | candidate | Driven by real ensemble weather forecasts, the physics-based demand model with a filter on its recent errors (GAM-K) scored 2.80 points against 2.78 for the production recipe (BG2) at five Primary substations. The 0.02 difference cannot be told from zero (-0.109 to +0.157), and the comparison is not like for like. GAM-K scored 0.13 points better than lagless XGBoost with the same filter (-0.305 to +0.014, 4 of 5 substations better), so the verdict is partly supported. The filter's gain is the firmly supported part: 0.15 points, 0.39 in the first day, with the sign held in the replicate months. Giving lagged XGBoost the physics model's forecast as a starting point changed nothing measurable. The known-answer gate passed 22 of 22 readings on its second run, after a sign error in the gate's own test data (both runs are kept). The departures were that the XGBoost compute estimate was 20 times too low, so the margin arms refitted monthly; lagless XGBoost for the scored year reused SIMP1's refits and had no spread factor; and the confirmation comparison was not run. GAM-K is a candidate for Flexpectation v2's roughly 1,300 substations on the measured filter gain, with four caveats: five substations over one year, a short XGBoost history, the undecided XGBoost comparison, and a margin that gave no gain. | also relevant |
| 00057 product-trained-xgboost-solar-farms | For the six metered solar farms, an XGBoost quantile model trained on ENS and given UKV's shortwave radiation, total cloud, and temperature as extra input columns scores lower on the quantile score at lead hours 24 to 54 than the same model trained on ENS alone, with every arm trained on the same window, rows, and seeds. UKV is read from the CEDA archive, which is not the live product, so the contrast is a ceiling read. | inconclusive | not-for-production | Giving UKV forecasts to an ENS-trained solar-farm model changed the quantile score by -0.035 points of capacity at 1 to 2 days ahead (95% interval -0.119 to +0.037), which is too small to resolve. The data cannot show a gain and cannot rule out the 0.10-point gain that would justify a new weather feed. UKV helped the six solar farms far less than it helped the three wind farms. | also relevant |
| 00054 product-trained-xgboost-wind-farms | For the three metered wind farms, an XGBoost quantile model trained on ENS and given UKV's wind as extra input columns scores lower on the quantile score at lead days 1 and 2 than the same model trained on ENS alone, with every arm trained on the same window, rows, and seeds. UKV is read from the CEDA archive, which is not the live product, so the contrast is a ceiling read. | small gain | not-for-production | Giving the Met Office's UKV wind forecast to an ENS-trained wind-farm model lowered the quantile score by about 0.2 points of capacity at 1 to 2 days ahead (95% interval 0.07 to 0.32), and a shuffled-UKV placebo shows the gain is information and not extra columns. The UKV data come from a research archive and not the live feed, so the gain is an upper limit on what a service could get. | also relevant |
| 00053 ukv-vs-ens-solar-anomaly | At the five Primary substations metered in MW that decide experiment 00046's verdict (D9, D10, D11, D14, and D16), the Met Office UKV forecast of the solar anomaly (the output of a generic solar fleet minus its own month-by-half-hour mean), alone or blended with the ECMWF ensemble mean (ENS) using weights fitted on the training window, tracks the CAMS solar anomaly more closely than ENS alone does within the UKV's lead range of 120 hours. | small gain | not-for-production | At 5 Primary substations, a fitted blend of the Met Office's UKV and ENS lowered the cross-validated squared error of the sunshine anomaly by 7.6% over 0 to 54 hours (3.8% to 13.8% in each band), but the validation year cannot confirm it (correlation gain +0.011, 95% interval -0.006 to +0.027). The test was post hoc and its intervals rest on 6 or 7 origin months, so they understate the uncertainty. UKV alone was worse than ENS, and the UKV data are a research archive, so any gain is an upper limit. | primary |
| 00052 more-nwp-products-for-wind-farms | For the three metered wind farms, averaging the quantiles of the experiment 33 wind best guess (the ERA5-trained mixture over the 51 ENS members, with the power-curve margin and the spread factor) with the quantiles of the same ERA5-trained model fed ICON-EU's and IFS 0.25 degree's forecast wind lowers the quantile score at 1 to 2 days ahead, with weights chosen only on earlier months. | inconclusive | not-for-production | Averaging the wind best guess with two extra weather forecasts, ICON-EU and the ECMWF high-resolution forecast, scored 0.055 points of capacity better at lead days 1 and 2, but the 95% interval (-0.200 to +0.072) includes zero. Three wind farms cannot show whether the extra forecasts help, and the blend scored 0.122 worse at days 3 to 6. | also relevant |
| 00050 second-weather-forecast-for-solar-anomaly | At the five Primary substations metered in MW in experiment 00046's verdict set (anonymised series D9, D10, D11, D14, and D16), an anomaly forecast that blends the NOAA GEFS ensemble mean with the ECMWF ENS ensemble mean, with weights fitted on the training window only, tracks the CAMS solar anomaly more closely than ENS alone. Laid into experiment 00046's solar term with its fitted lead-dependent coefficient, the blended anomaly lowers the quantile score by at least 0.1 points of capacity against the same term built from ENS alone. | refuted | not-for-production | The stage 1 gate stopped the experiment: in cross-validation on the training window GEFS's squared error against the CAMS anomaly is 1.109 times ENS's (worse in all 9 held-out months) and the fitted blend's is 1.002 times, against the 0.99 needed. | primary |
| 00022 era5-weather-ceiling-primary-demand | For the 16 Primary substation demand series, giving the winning recipe of experiment 00019 (the single-stage ENS-trained quantile model N1 with a recency half-life of 0.5 years, the 14-day error correction, and the 1.15 spread factor) ERA5 weather in place of ECMWF ensemble (ENS) weather at serving time lowers the quantile score by at least 0.3 points of capacity over all leads. | refuted | not-for-production | ERA5 weather in place of ENS weather lowered the quantile score over all leads by 0.10 points of capacity (99.38% interval -0.16 to -0.04), so the 0.3-point gain the rule asked for is ruled out. | also relevant |
| 00014 cams-trained-stage1-with-stage0-ens-correction-demand | A stage 0 that maps the ECMWF ensemble (ENS) member's shortwave irradiance onto the irradiance of the Copernicus Atmosphere Monitoring Service (CAMS) lowers the Primary substation quantile score of the CAMS-trained stage 1a XGBoost demand model, relative to feeding that model the raw ENS shortwave, because the ENS shortwave is about 7% above CAMS and stage 1a never saw that offset in training. | inconclusive | candidate | C2, stage 0 minus raw ENS for the CAMS-trained arm, Primary quantile score over all leads: -0.067 points of capacity (97.5% interval -0.094 to -0.039), better than the planned -0.03. | primary |
| 00009 cams-trained-stage1a-vs-era5-trained-demand | A stage 1a XGBoost demand model trained with irradiance from the Copernicus Atmosphere Monitoring Service (CAMS) in place of ERA5 shortwave, and applied to the shortwave of ECMWF ensemble (ENS) members, has a lower quantile score at Primary substations than the same model trained on ERA5 shortwave, because the satellite-derived irradiance is closer to the irradiance that the substations' embedded solar generation responded to. | refuted | not-for-production | C1, CAMS-trained minus ERA5-trained, Primary quantile score over all leads: -0.035 points of capacity (97.5% interval -0.070 to +0.001), a sixth of the planned 0.2-point gain. | primary |
| 00007 cams-static-correction-stage1a-residual | Most of the daytime residual of the stage 1a demand model at Primary substations is a fixed error in the ERA5 irradiance input and not a growing embedded-solar capacity, so a correction with fixed coefficients on irradiance from the Copernicus Atmosphere Monitoring Service (CAMS), fitted on one year of out-of-fold residuals, removes most of the residual in the next year. | inconclusive | not-for-production | The static correction removes 46.6% of the next year's daylight squared residual, just below the planned 50%, and a fixed bias alone removes 39.1%. | primary |
| 00003 era5-trained-stage1a-vs-ens-trained-demand | A weather-to-power XGBoost model trained on ERA5 (2019-09 onward) and applied to the 51 ENS members, with the member forecasts averaged, has a lower mean absolute error on electricity demand series than the same columns trained on ENS weather. | not supported | not-for-production | D1, s1_memberavg minus ens_trained, weather only, all leads: +1.37 (+0.68 to +2.17), 22 series; with calendar +1.38 (D2, against the one-row-per-valid-time arm: +1.67); with calendar and lags +0.10 (-0.26 to +0.50). | primary |
| 00001 era5-trained-stage1a-vs-ens-trained | A weather-to-power XGBoost model trained on ERA5 and CAMS (2019-09 onward) and applied to ENS weather has a lower point-forecast error than the same columns trained on ENS weather. | supported | candidate | P1 (revised), s1_memberavg minus ens_trained, all leads: wind -1.70 (-2.33 to -1.08), solar -0.57 (-0.83 to -0.32); material up to day 6, not beyond. | primary |