Skip to content

Studies

This section holds the findings of experiments we have run, written up so the conclusion survives the code that produced it. It complements the other sections: background describes the problem, techniques explains the methods, the roadmap says what we plan to build, and architecture documents what is already built.

A page lands here when a study answers a question that outlives its code. The code behind a page here is deliberately lighter than production code — no Dagster asset, no data contract, and no maintenance promise — and lives in studies/, which merges so that the measurement can be audited and re-run. What the project keeps is the answer, which is why each page states its numbers, the checks the result survived, and the limits on how far it generalises.

Every study here but the solar-BMU census uses only Flexpectation's own data, and its purpose is to inform Flexpectation's choices. Each study scores its weather products or methods only at the generators in Flexpectation's trial area in Lincolnshire. No study here compares results across many regions or climates, so a result may not hold elsewhere.

  • Does a weather product's beam/diffuse split help a PV forecast? — on a 5 km satellite retrieval the published direct-beam field cuts photovoltaic (PV) power error by 1.8% beyond what a separation model recovers from the total irradiance; on a 31 km reanalysis no effect is detected at the setting named before the run; and choosing the better irradiance product matters about 34 times more than the split does. The page also measures the two irradiance products against each other, a gradient-boosted tree against a fitted five-parameter physical PV model, what calibrating that physical model with a tree recovers, and what the per-site error drift says about estimating a generator's effective capacity. It reads NGED's two records of active network management against each other and against the telemetry, and finds the setpoint history the one to build on.

  • Embedded solar behind six Lincolnshire Primary substations is 2.5 to 4.5 MW, and grew 7% to 16% a year — a season-year table of the embedded-solar capacity behind each directional Primary substation in the trial area, built from stored fits with no refit, with the sampling, farm-to-rooftop, and undetected-demand error terms kept separate. Kirton gets no capacity because its demand responds to cloud in a way solar cannot explain.

  • How many Balancing Mechanism Units in Great Britain are solar? — a census of the BMUs whose settled output follows the sun, found because no field in the BMU register says which units are solar. The study finds 12 single-site solar BMUs at 11 sites, 8 of those sites with storage built or planned, reports 26 aggregate BMUs apart, and sets the five published capacity figures side by side.

  • Can solar be separated from an aggregate Balancing Mechanism Unit? — a physical solar plant fitted to four-hour changes in output recovers the solar part of synthetic aggregates of real BMUs beside wind and gas, and finds solar in the supplier BMUs.

  • Can solar be separated from a battery in an aggregate Balancing Mechanism Unit? — on synthetic aggregates of a real solar BMU and a real battery BMU, a joint model of solar and a battery with a state-of-charge limit has a solar error of 14.8% of the solar part's 99th-percentile output, against 27.2% for a solar-only fit. With a demand-like part added the model overshoots the solar capacity, and on the 25 real aggregate BMUs its 1.1 to 1.2 GW is not validated.

  • How well can a primary substation's half-hourly flow reveal an unmetered battery? — a simulated merchant battery that follows the day-ahead price in the estimator's own way is found from about 10% of a primary's 99th-percentile flow, with 3 of 36 blocks flagged falsely. Real public batteries and NGED battery A are not found at any size tested, the intervals of a battery that follows another rule hold the truth far less often than stated, and the screen of eight primaries has no power.

  • How well can an embedded battery's output be forecast, and how many of NGED's batteries have a published plan? — across 35 battery BMUs, no XGBoost forecast tested beats a 56-day trailing climatology day ahead; the real day-ahead price lowers an XGBoost forecast's CRPS by 1.1%, and the planned output of other companies' batteries lowers it by 1.3% 1 hour ahead. The registers list 8 embedded storage BMUs against 176 connected storage rows in NGED's area, and cannot say how many rows the 8 stand for. All numbers are provisional.

  • Do INDDEM and INDGEN line up with the GSP groups of NGED's licence areas? — Elexon's indicated demand and generation series sum the Physical Notifications of every Balancing Mechanism Unit for the national total and 17 transmission boundaries. An issue reaches 15 to 42 hours ahead, the zones recover from the boundaries to within rounding, and the sum of the Physical Notifications reproduces INDDEM to within 5 MW in 125 of 192 sampled half-hours once each interconnector is netted. The series do not map cleanly onto the 14 Grid Supply Point groups: with the shared cycles and the national anomaly removed, the highest correlation between a group's settled take and a zone's INDDEM is 0.58, and 0.32 for an NGED group.

Past weather

Three studies score how well each weather product describes weather that has already happened. The Past-weather overview summarises them, and the Methods page states the methods they share.

  • Which weather product best describes past sunshine? — a satellite retrieval describes past sunshine far better than any weather model tested; among the models, ICON-D2 is best as served but its advantage shrinks within hours of each run, and ICON-EU no longer beats UKV once UKV's hour is rebuilt from its own snapshots. The page says which product each offline consumer — capacity estimation, training history, historical features, and disaggregation — should read.
  • Which weather product best describes past wind? — at the three metered wind farms, the Met Office's UKV and the German weather service's ICON-D2 describe past hub-height wind best of the five products tested and both beat the ERA5 reanalysis, by more from April to September in intervals estimated separately for each half of the year; ICON-EU also beats ERA5; and ICON global has the largest error of the five, about half of its gap to ICON-EU a pair of steps in the wind Open-Meteo's archive serves for it at one generator.
  • Does blending weather products beat the best single weather product? — at the six solar farms and three wind farms, an XGBoost model given several weather products at once beats one given the best single product with its neighbouring hours: by 0.13 points of capacity for solar and 0.48 for wind. The gain comes from the other products' weather rather than from the extra columns, a blend of UKV and ICON-EU that a live service could read beats UKV alone, and an XGBoost blend beats a linear stack of single-product predictions. An XGBoost model given CAMS's split plus SARAH-3's global irradiance, the two satellite retrievals the past-solar study compared, beats CAMS's split with its neighbouring hours by 0.18 points and plain CAMS's split by 0.20 points, on a longer row set from January 2021.
  • Does CEDA's UKV or ERA5 describe past wind and temperature better? — at three wind farms, an XGBoost model given ERA5's wind has a power error 0.125 percentage points of capacity lower than an XGBoost model given the 6-hourly UKV archive from CEDA. The difference is statistically significant and below the planned 0.16-point margin, so the planned rule gives ERA5. The gap grows with the archive's lead, and the gap does not shrink when both XGBoost models read 10 m wind alone. At four Met Office stations the archived UKV's temperature is 0.124 K closer than ERA5's, so the planned rule gives UKV, but the advantage falls from 0.250 K at lead 0 to 0.035 K at lead 5. At six solar farms the choice of temperature moves power error by no more than 0.009 points (95% interval, whole row set).

  • Do CEDA's and Open-Meteo's archives of UKV give the same power forecasts? — at the nine metered farms over 23 months, the two archives agree closely at the start of each run (a mean absolute temperature difference of 0.098 K, and Open-Meteo's wind speed about 3% lower). An XGBoost model of wind power trained on CEDA's archive loses 0.441 points of capacity [+0.286, +0.600] when given Open-Meteo's wind, a level bias that rescaling the speed mostly removes. CEDA's larger power error over all hours (wind +0.270 points once two spans of Open-Meteo's wind are dropped) is mostly its longer leads: at lead 0 the wind gap is +0.030 [-0.043, +0.122]. By the plan's rule the two archives are not mixed. The page does not compare CEDA's archive with the Met Office's live feed.

  • Which ERA5 variables help predict solar farm output? — at six solar farms, an XGBoost model given ERA5's total and layered cloud cover as well as its solar radiation has about 6% lower error, and one given every ERA5 variable has about 12% lower error. ERA5 is a reanalysis, and part of the gain may come from cloud fields that saw later observations than its solar radiation. Whether fetching the 12 variables that only ECMWF's MARS archive serves is worth it is unresolved.

Forecasts

Six studies score power forecasts driven by weather forecasts.

  • How accurate is a power forecast driven by ECMWF ENS at each horizon? — at the 6 solar farms and 3 wind farms, an XGBoost model given the ENS ensemble mean beats every forecast that reads no weather forecast to day 5 for solar and day 7 for wind; from day 7 for solar and day 10 for wind it no longer beats climatology, and by day 14 climatology is ahead, with the ensemble mean adding no statistically significant skill, in a post hoc check, over the same model given no weather at all. The ensemble mean beats the control member and beats training on every member, and rebuilding solar radiation through the clear-sky index beats the straight-line resample the live service uses today at days 0 to 3.
  • Do other weather forecasts beat ECMWF's ensemble at day-ahead lead? — at the 6 solar farms and 3 wind farms, and reading each product as the study does, ECMWF's ensemble mean beats UKV, ICON-EU, and GEFS at matched lead for solar; for wind it beats UKV and GEFS, and ICON-EU cannot be separated from it. A blend of ENS, ICON-EU, and IFS 0.25° lowers the wind error but shows no detectable solar gain at a lead a 09:00 UTC service could use.
  • Which single weather forecast should be added to ECMWF's ensemble mean first? — at the 6 solar farms and 3 wind farms, adding ICON-EU at the optimistic lead (exploratory; an upper bound on a live service's gain) lowers the wind error at day 1 by 0.58 points of capacity [0.48, 0.69]; AIFS Single is the one product the page can name for solar, at day 2, by 0.33 [0.18, 0.48]. Elsewhere AIFS Single and ICON-EU are not separable.
  • Does adding UKV from CEDA to ECMWF's ensemble mean lower the error at lead days 1 to 4? — at the 3 wind farms, adding UKV-CEDA lowers the error at days 1 and 2 by 0.24 to 0.27 and 0.19 to 0.21 points of capacity, at both hyperparameter settings, against both shuffled controls, after the Bonferroni correction, and with any one month dropped. At wind day 3 the gain rests on February 2026. At the 6 solar farms the gain is about 0.1 points at days 1 to 3, unresolved under the planned rule at days 1 and 2. Day 4 is inconclusive.
  • How do Open-Meteo's ensemble-mean products compare for solar and wind power? — over 88 summer days at the 6 solar farms and 3 wind farms, ICON-D2-EPS's ensemble mean gave the lowest power error of the four Open-Meteo ensemble means, 7.4% of capacity for solar against 7.7% for ECMWF ENS's mean, and 5.75% for wind at 10 m, level with ECMWF ENS's mean. The comparison is descriptive and uses stitched series that carry no run time.
  • Does a same-hour power lag improve a day-ahead solar forecast? — at 6 solar plants over 18 months, giving an XGBoost model each plant's power from the same hour two days earlier changed the day-ahead error by −0.001 points of capacity [−0.075, +0.073] against a baseline of 8.93, so none of the five planned comparisons found a gain. The lag recovers little of a planted power loss that weekly statistics or satellite irradiance partly recover, and the gains of those inputs came mostly from one plant's low-output months in the training data. At lead days 10 to 14, blending the baseline with an earlier-years climatology beat every lag arm.
  • Is a daily capacity estimate worth building for day-ahead solar forecasts? — at 6 solar plants over 18 months, dividing power by a Kalman filter's daily estimate of each plant's capacity raised the day-ahead error by 0.255 points of capacity [+0.004, +0.563] against a baseline of 8.93, so the study plan's rule reads "not worth building", although only 3 of 106 plant-months are the low-output kind of loss that an estimator exists to repair. The Kalman filter recovered a planted capacity loss best of 13 estimators but chased the December low in its own level, and a post hoc rule that holds the static capacity until a drop lasts 7 days did not raise the error.
  • Can a language model label a solar plant's capacity changes? — with six deterministic evidence tools, Claude Sonnet 5.5 found 0 of 32 planted one-month losses and the 30% positive control failed its gate, so the planned contrasts are untested. In post hoc rounds with an amended check, it hit 18 of 129 one-month losses (14%) and every one of 12 losses of 60% lasting 90 days, against 13 of 129 and 1 of 12 for a fixed rule.
  • Which IFS forecast variables, or which second forecast, help a solar power forecast? — at the 6 solar farms, no planned step of adding extra variables from ECMWF's 9 km IFS forecast lowers the error at lead days 1 to 3 by a resolved amount, although all of them together lower it by about 0.45 points of capacity. Adding AIFS Single's radiation and temperature lowers the error by about 1 point. The ensemble mean the production service reads gives about the same gain, and AIFS Single adds about 0.13 points to that mean, which the study cannot resolve.
  • Which cheap changes lower the production XGBoost forecast's error? — at 31 NGED series over one validation year, four changes to the production training recipe (quantile training, a lead-time column, leaner wind-farm columns, and one XGBoost model per series type) lower the continuous ranked probability score (CRPS) from 11.3% to 7.9% of capacity, against 7.2% for the manual heuristic. The production ensemble's 10% to 90% band holds the outcome in only 10% of half-hours, and rescaling its spread lowers the CRPS by 1.3 to 1.9 points. The absolute-error objective and early stopping each lower the CRPS by about 0.4 points. Removing power lags from solar and wind farms raised their error.