Metered solar farms: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00345 pooled-solar-wind-mixture-density-network | One mixture-density network (MDN) trained once for all leads on the six metered solar farms and the wind farms together, with a fuel-type indicator and weather inputs common to both fuels, scores as well on each fuel as that fuel's best single-family model, and scores better for a farm with no history. | partly supported | candidate | One neural network trained on all six solar farms forecasts each farm as well as a separate network on the physical plant model (within 0.8%), so a production forecast could use one neural network per fuel. Adding wind farms to the same network does not help either fuel and makes solar 0.7% to 2.3% worse, so solar and wind are better trained separately. In this experiment, a wind farm with no history scored 18% worse at day 1 when trained on public wind farms alone than when trained on other metered wind farms. | primary |
| 00321 metered-farm-lower-tail-recalibration-and-tail-head | Metered solar farms research default 1 (the blend assembled by experiment 300, scored to 10 days by experiment 310) lets the actual output fall below its 0.01 quantile 3.3% to 3.7% of the time, and no mixture-density head tried in experiment 318 changes that. Default plus a recalibration table, a zero-inflated lower tail, or a tail-targeted head brings the share of actual values strictly below the 0.01 quantile close to 1% at leads of 1 to 10 days, without raising the quantile score and without losing experiment 318's 0.99 gain. | partly supported | candidate | The solar-farm forecast's lowest quantile was already close to correct: only 1.2% to 1.4% of actual values fall strictly below the 0.01 quantile, because the earlier 3.3% counted the 3.0% of half-hours that read exactly zero. A recalibration table, a zero-inflated head, and a tail head each lower the own-history score by 0.04 points or less (the unshrunk table raises it by 0.03 to 0.05 points), so none is adopted. A shrunk recalibration table does lower the score by about 0.05 points for a farm with no history, and two heads lower the 0.99 pinball loss by 3.7% to 8.5% while covering the 0.99 quantile only 98.3% to 98.9% of the time. | primary |
| 00318 metered-farm-mdn-head-more-features | Giving the mixture-density head of Metered solar farms research default 1 the forecast lead (lead in whole days), the hour of day and day of year as sine and cosine pairs, and the ensemble spread of the shortwave irradiance lowers its quantile score, mostly beyond 36 hours and in the lower tail, and one head trained on all lead bands beats one head per band. | partly supported | candidate | Giving the solar-farm head more inputs does not improve its overall score at any lead from 1 to 10 days, so the head's inputs are not what limits the forecast. The extra clock and cloud-spread inputs do lower the loss at the 0.99 quantile (the tail that matters for curtailment) by about 5% at days 3 to 7, and none of the changes fixes the lower tail, where 3.3% to 3.7% of actual values fall below the 0.01 quantile, where 1% is expected. Training one head for all leads makes the network with no plant nearly as good as the plant-based head at days 8 to 10 (0.4% worse), so the plant adds little there. | primary |
| 00313 metered-farm-pure-xgboost-and-mdn-vs-plant | A predictor built only from weather and calendar inputs, with no plant model anywhere in the pipeline (not even as an input column), scores close to the plant-based forecast in own history and falls behind it where the plant carries what missing history or a long lead cannot. | partly supported | candidate | A forecast that sees only weather and the calendar, with no physical plant model anywhere, scores almost as well as the plant-based forecast for the six solar farms, up to 10 days ahead, even when the farm has no history. The plant's added value is small, between 0% and 3.4%, and its intervals include zero except for XGBoost. For wind, the same kind of network also matches the best plant-based forecast, so a plant model is not needed to be competitive on these farms. | primary |
| 00310 metered-farm-long-lead-scoring | Experiment 300's candidate (spread-binned plant, additive band, mixture head, blend with XGBoost) still beats XGBoost at every day band from 1 to 10 days ahead, and its margin over climatology shrinks as the lead grows. | partly supported | candidate | The assembled solar-farm forecast still beats the seasonal-average forecast (climatology) at every lead up to 10 days, by 39% at day 1 and 10% at days 8 to 10, so NGED can use it across its whole horizon of 1 to 10 days ahead. A plant-fed XGBoost model loses its skill by day 8, so a simple first production forecast should be the mixture-density neural network on the plant, which is the best single family at every lead, and the blend of the GAM, MDN, and XGBoost adds more than 1% only at days 1 and 2. | primary |
| 00304 metered-farm-node-sum-tail | For the solar farms that share a bulk supply point, adding each farm's quantiles overstates the upper tail of the farms' summed output, adding independent bands understates it, and summing the 51 ensemble members' farm forecasts member by member gives calibrated 0.95 and 0.99 quantiles, because the ensemble carries the farms' shared dependence on cloud. | not supported | not-for-production | The band for the combined output of the three farms beneath one substation comes out too wide if the weather forecast's 51 members are summed member by member, because the members' forecasts for the three farms move together almost perfectly while the farms' actual forecast errors are less closely linked. That band is almost the same as the band from adding each farm's quantiles. A band that assumes the farms' errors are unrelated is too narrow. A band from a Gaussian copula with a correlation fitted on the errors of the other 11 months comes closest to the stated probability. Only 5 half-hours in the scored year exceed 0.75 times a substation's maximum export, so this experiment cannot say what the band widths mean for curtailment cost. | primary |
| 00301 metered-farm-outage-state-band | A small availability state (online, partly down, offline) estimated from the last 6 to 48 hours of a solar farm's power against its plant forecast improves the forecast band's lower tail during and after an outage. | inconclusive | not-for-production | Solar-farm outages are too rare in the scored year to test an availability state: five of the six farms have at most 34 outage half-hours, and the sixth has 165. Too few half-hours remain to separate a 10% gain from chance, so the experiment stopped before fitting anything. If outages matter, the next test needs a longer period, such as the full history of farms A, B, and F. | primary |
| 00300 metered-farm-best-guess-assembly | The assembled candidate (spread-binned plant P2, a generalised additive model (GAM) band on the plant's residuals, a mixture-density head (MDN), blended per quantile with XGBoost) scores better than plant-fed XGBoost on the quantile score and at the 0.99 quantile, and its advantage is larger for a farm with no history because the plant carries what the missing history cannot. | partly supported | candidate | The assembled solar-farm forecast (a physical power model, two correction stages, and a blend with XGBoost) scores 3.3% better than plant-fed XGBoost (XGBoost on the plant forecast, its ensemble spread, the unit mean, and the availability ratio, with no weather features of its own) on the quantile score, which averages the pinball loss over 11 quantile levels, when each farm trains on its own history. It is 43% better at the 0.99 quantile, which bounds how much output NGED may need to curtail. With no history for a farm, the gain over XGBoost is smaller (2.0%) and below the 3% we predicted, so the physical model helps less than we expected when a farm's history is missing. | primary |
| 00293 metered-farm-plant-tilt-and-clip | The six metered farms fit best at a 15 to 20 degree tilt (experiment 282), while the spread-binned plant (P2 of experiment 270) assumes 35 degrees and caps its forecast at the 0.995 quantile of the tuning months' output. Replacing the fleet tilt with 20 degrees and the hard cap with a higher, soft cap lowers P2's April to July under-prediction and its 0.99 pinball loss. | not supported | not-for-production | Changing the solar power model's panel tilt from 35 to 20 degrees fixes most of its winter over-prediction on clear days and improves the lowest 1% quantile of the forecast by 4%, but it leaves the overall forecast band and the 99% quantile unchanged and slightly worsens summer. A higher or softer cap on the power does nothing, so the shape of the power model is not what limits the upper tail of the forecast band. A tilt between 20 and 35 degrees, or a tilt that varies with the season, is the next tilt to test. | primary |
| 00292 metered-farm-ens-neighbourhood | A day-ahead cloud field is often right in shape and wrong in position by tens of kilometres, so summaries of the ensemble's irradiance over the hexagonal cells around a metered solar farm carry information that the farm's own cell lacks. Giving H5 (the plant plus a neural network that outputs all 11 quantile offsets) those summaries lowers its quantile score at 12 to 36 hour leads. | not supported | not-for-production | At the six metered solar farms tested, the ensemble's irradiance on the cells around a farm adds almost nothing to the day-ahead forecast: the quantile score improves by 0.36%, against the 2% the plan needed, and the gain is not robust to dropping one farm. Adjacent cells, 15 km apart, differ from the farm's own cell by only about 3% of the irradiance, so the ensemble mean is too smooth between them to show where a cloud edge will fall. A weather source of a different kind is the next experiment to run, rather than more processing of the same ensemble. | primary |
| 00287 metered-farm-spread-binned-plant-heads | On top of the spread-binned plant (experiment 270's P2), an additive model of the residuals gives a band as good as a neural head with monotone quantile output, so the additive model is the explainable candidate and the neural head adds nothing worth porting. | supported | candidate | On top of the spread-binned solar-farm power model, an additive model gives a better forecast band than a neural network that adjusts each quantile, so the explainable model is the candidate. A neural network that outputs a three-component Gaussian mixture scores 1.3% better than the additive model in own history, mostly in the upper tail, but not clearly in zero history, so it needs one more test before it is preferred. A blend of the additive model, a neural network, and XGBoost scores 0.8% better than XGBoost alone and fixes XGBoost's weak 0.99 quantile. | primary |
| 00286 metered-farm-drift-soiling-injection | A yield index that divides each farm's output by the output of the other farms, on half-hours when the other farms are near clear-sky output, reads a yearly trend more precisely than the monthly yield index of experiment 179, and it can recover a soiling loss that the monthly index cannot. | partly supported | not-for-production | Comparing each metered solar farm with the other farms does not make the yearly trend of its output more precise overall, but of the two indices only the comparison can read soiling, because the plain index shows a false loss slope from weather alone. A loss of 1 %/yr or more is visible at 4 to 5 of the 6 farms, and a loss under about 0.4 %/yr is visible at only 2 farms (C and F). Whether these limits are the best that output alone can give is open: a panel-level or inverter- level record would test it. | primary |
| 00284 metered-farm-tail-head-past-weather | Experiment 275 found that adding years of past-weather rows lowered the 0.99 pinball loss of the neural head H5 by 4.6% while its overall score moved by 0.47%. A small neural network trained with a loss restricted to the tail quantiles (0.01, 0.02, 0.05, 0.95, 0.98, 0.99), on the same past-weather plus ensemble rows with an era flag, lowers the 0.01 and 0.99 pinball loss of a metered solar farm's forecast below that of XGBoost trained on the same rows. | partly supported | not-for-production | A neural network trained only to forecast the extreme edges of a solar farm's output band does beat XGBoost at the 0.99 quantile (the upper edge that matters for curtailment) by 30%, but the existing H5 network does slightly better still, and most of the gain comes from building the band around the physical power model and not from tail-only training. The years of past-weather rows add under 1% at the 0.99 quantile, and at the 0.01 quantile the share of half-hours below it is outside 0.5% to 1.5% at three farms (marginally at A, because of exact zeros at D, and clearly only at E). | primary |
| 00282 pv-plant-curve-audit | The curve that turns sunshine into power in our solar models departs from the metered farms' measured curve by more than 5% of capacity somewhere in the clear-sky range, and the departure is the flat-top level (the inverter limit), not the panel tilt or the temperature loss. | not supported | not-for-production | The curve that turns sunshine into solar power stays within 5 percentage points of capacity at 3 of the 6 metered solar farms and departs from their measured output at the other 3 in different directions, so no single fix to the inverter limit repairs it. The best-guess demand stack assumes panels tilted 35 degrees and the farms fit best at about 20 degrees, which makes the stack over-predict output on clear half-hours in December (measured is 0.66 of the stack, median over farms) and under-predict it from April to July by 5 to 8%. Swapping in the pooled shape of the other farms does not help, so the next experiment should try the 20 degree tilt inside the stack. | primary |
| 00275 metered-farm-long-history-neural-head | Experiment 243 fitted H5 (the solar-farm power model plus a neural network that outputs all 11 quantile offsets) on about 17 months of ensemble forecasts. Adding each farm's rows built from past weather (Copernicus Atmosphere Monitoring Service irradiance and ERA5 temperature, back to 2019-09) as extra training rows, with a flag that marks them, lowers H5's quantile score on the ensemble-driven scored year, and the neural head gains more from the extra rows than XGBoost does. | not supported | not-for-production | Training the neural network that corrects a metered solar farm's power forecast on nearly six years of past-weather rows, as well as on 17 months of ensemble forecasts, lowers the quantile score by only 0.5%, far under the 2% we predicted. The same rows improve XGBoost by 1.7%, so the neural network is not the better way to use extra history. The extra rows do lower the loss at the 99% quantile (the upper edge of the forecast band, which matters for curtailment) by 4.6%. | primary |
| 00273 metered-farm-sorted-quantile-rescore | Some stored forecasts of experiments 224, 243, 248, 257, and 260 have quantiles that cross (a lower quantile above a higher one). Sorting the 11 quantiles in each row can only lower the pinball loss, so the stored scores are pessimistic for the arms that cross, and the arms that cross most gain most. | not supported | not-for-production | The stored forecasts of seven metered-farm experiments have no crossing quantiles, because every fit script already sorted the 11 quantiles before writing them. Sorting the quantiles again therefore changes no score and no verdict. The 26,344 crossing rows found in experiment 268 are not farm forecasts: they belong to three generators that are not solar farms. Future heads should still be monotone by construction. | primary |
| 00272 metered-farm-weather-adapter-gam-vs-neural | Extra weather variables that the solar-farm power model (the plant) ignores, fed through a learned correction to the plant's irradiance and temperature, close part of the gap between the plant driven by the ensemble weather forecast (about 0.91 of output fitted on sunny half-hours) and the plant driven by observed weather (0.973), and a small neural network does this better than additive curves. | not supported | not-for-production | The extra ensemble weather variables (longwave radiation, humidity, precipitation, pressure, wind, and 500 hPa height) showed no usable information for the solar-farm power model at these six farms, whether they enter through additive curves or a small neural network, and the real variables do no better than placebo copies. A neural network with no plant, given the lagged power and the ensemble spread, scores 2.7% better than the plant but not better than the linear fit of experiment 257, so a correction to the plant's weather is not the route to a better forecast. | primary |
| 00270 metered-farm-xgboost-vs-plant-diagnostic | XGBoost beats the plant (the solar-farm power model of experiments 218 and 224) on the six metered farms because of inputs the plant never sees, namely the farm's own lagged power and the ensemble spread, and not because of a better map from the ensemble-mean weather to power. Calendar columns and extra weather variables add little once those two are present. | partly supported | candidate | XGBoost beats the solar-farm power model by about 3% in the quantile score even when it sees only the power model's own weather inputs, and the gain is in the width of the forecast band (the 0.05 and 0.95 quantiles), not in the median forecast. The same gain comes from giving the power model a forecast band that depends on the ensemble spread or the clear-sky index, and that band also beats XGBoost at the 0.99 quantile. Lagged power, calendar columns, and extra weather variables add little in own history. | primary |
| 00260 metered-farm-quantile-combination | The three best solar-farm forecasters of experiment 243 win at different quantile levels (the neural network without physics at 0.99, H5 at 0.01), so convex per-level weights fitted on earlier months beat H5 alone, and a rolling adjustment of the outer levels brings their coverage to the nominal level without hurting the quantile score. | partly supported | candidate | Blending the three stored forecasters with a separate weight at each quantile level scores 2.1% better than H5 alone, and every one of the six farms improves, because the neural network without the plant is best at the top quantile and H5 is best at the bottom one. About a quarter of the gain comes from averaging the five seeds, not from the weights. The rolling adjustment of the outer quantiles corrects the coverage of both tails but raises the error by 1.2%. | primary |
| 00257 metered-farm-fair-plant-arm | The 4.4% gain of H5 (the plant plus a neural network that outputs all 11 quantile offsets) over the plant alone in experiment 243 came from the inputs that H5 saw and the plant did not (the lagged power and the ensemble spread), so a convex linear quantile regression of the 11 offsets on those same inputs (arm P2) recovers most of it, and the neural network is not needed. | supported | candidate | The neural network was not needed. Giving the solar-farm power model the lagged power and the spread of the ensemble members, through a plain linear fit, recovers about 90% of the neural network's gain and scores within half a percent of it. The linear fit is no worse at the 0.99 quantile (the 99% interval of the gap includes zero) and is simpler to serve, so it is the metered-farm candidate. | primary |
| 00255 metered-farm-node-curtailment-value | When six metered solar farms beneath two bulk supply points are curtailed by the node-plus-generators rule of studies.node_curtailment, a forecast of each farm's output (the plant, XGBoost, or the plant with a neural head) avoids more curtailment cost than the NGED baseline, which assumes every generator runs at full capacity, and the neural head's lower 0.99 pinball loss does not show in the metric, because the rule uses only a farm's median. | inconclusive | not-for-production | Forecasting each solar farm's output instead of assuming every farm runs at full capacity cut the cost of curtailing at one of two substations (Boston, 40% lower) and raised it at the other (Skegness, 24% higher), so across both the saving is about a quarter, smaller than the uncertainty in the estimate. The neural network's better upper tail does not show in this cost, because the rule reads only each farm's median, and it lowers the cost by 3% against XGBoost, far under the 10% that would justify serving a neural network. Ranking farm forecasts by curtailment cost needs many more substations. | primary |
| 00248 metered-farm-architecture-ukv-inputs | UKV irradiance and cloud, given as extra inputs to the two best neural heads of experiment 243 (H5, the plant plus 11 quantile offsets, and H2, the plant plus a correction), lower a metered solar farm's quantile score at 12 to 36 hours by more than the 1.8% that experiment 229 found for a blend of ensemble and UKV irradiance, because a neural head can use UKV cloud layers that a blend cannot. | small gain | not-for-production | Giving the neural head on the solar-farm power model the Met Office's UKV irradiance and cloud forecasts lowers the day-ahead error of a metered solar farm by about 1%, the same small gain that XGBoost gets from the same columns. At these six farms, a neural head gained no more from UKV than XGBoost did, so UKV is not worth a second weather source on this evidence. The UKV archive used is a research archive, and the live UKV differs, so the 1% gain is an upper limit. | primary |
| 00246 metered-farm-ens-ukv-icond2-blend | The DWD ICON-D2 regional weather forecast (2 km grid, run every 3 hours, 48 hours ahead) carries information about cloud over a metered solar farm that neither the ECMWF ensemble nor the Met Office UKV carries, so adding it to the ensemble plus UKV lowers the farm's day-ahead error. The evidence against is that at day 1 every single product lost to the ensemble mean for solar in the matched-lead study, and ICON-D2's errors are likely to be correlated with UKV's. Every gain is an upper limit for production, because the UKV archive used is the CEDA research archive. | small gain | not-for-production | Adding the German ICON-D2 weather forecast to the ECMWF ensemble and the Met Office UKV lowers the day-ahead error of a metered solar farm by about 1%, which is under the 1.5% we set as worth a fourth weather source. ICON-D2 is available for only 12 to 29 hours of the day ahead at an 08 UTC forecast time, and the UKV archive used is a research archive, so this is an upper limit for production. | primary |
| 00243 metered-farm-architecture-comparison | A solar-farm power model (the plant) as the base prediction, with a small neural network that adapts its inputs, its output, or both, forecasts a metered solar farm 12 to 36 hours ahead better than the plant alone and better than XGBoost, and keeps the plant's advantage on a farm it has never seen, while a neural network with no physics does not beat XGBoost on six farms. | partly supported | candidate | A small neural network sitting on top of a solar-farm power model forecasts a metered solar farm about as well as XGBoost and improves on the power model alone by 4.4% in the quantile score, when the network outputs the whole forecast band. Networks that adjust a single number (the scale, a correction, or the sunlight fed to the power model) gain under 1%. The hybrid is far better than XGBoost at the upper tail (0.99 quantile, 29% lower pinball loss), which matters for curtailment. The neural network without physics does better still at that tail (32% lower pinball loss than XGBoost). | primary |
| 00240 no-xgboost-forecast-sprinkle-part-2 | A small XGBoost residual model on the plant forecast's median error, an XGBoost spread model for the quantile offsets around it, or both, close at least half of the gap in quantile score between the XGBoost-free metered solar forecast (the plant fed the mean and spread of 51 ensemble members, experiment 224's arm 3) and the full XGBoost forecast (experiment 224's arm 5) at 12 to 36 hour leads. | partly supported | candidate | For solar farms, the best way to add XGBoost to the physical plant forecast is to let XGBoost predict the spread around the plant's mean, not to correct the mean: that forecast scores 4.3% better than the plant alone and 1.6% better than full XGBoost, and wins at the upper tail that matters for curtailment. Correcting the mean with XGBoost made the quantile score 0.18 points worse, about as much as a model trained on shuffled residuals. For wind, experiment 230's XGBoost stage on the physics core closes 71% of the gap to full XGBoost at 12 to 36 hours, and 44% over all leads. | also relevant |
| 00234 no-xgboost-forecast-audit-part-1 | For Primary substation demand, a metered solar farm, and a metered wind farm, the best forecast that uses no XGBoost anywhere scores within 2% of the quantile score of the best forecast that uses XGBoost, at 12 to 36 hour leads. | partly supported | candidate | An XGBoost-free forecast of Primary substation demand scores 7.5% better than BG2, the XGBoost best guess, at 12 to 36 hours ahead, on one year and six substations. The XGBoost-free solar and wind forecasts score 2.8% and 5.0% worse, and lose at the 0.01, 0.05, and 0.95 quantiles. Only the solar plant model's 0.99 quantile beats XGBoost, and that is the tail that matters for curtailment. The next experiment tests small XGBoost components inside the XGBoost-free solar and wind forecasts. | also relevant |
| 00229 metered-farm-ens-ukv-blend | The Met Office UKV weather model (2 km grid, 54 hours) carries information about cloud over a solar farm that the 51-member ECMWF ensemble lacks, so giving a metered solar farm's forecast both products lowers its day-ahead error against the ensemble alone, and closes part of the gap between the ensemble-driven fit (0.91 of output on sunny half-hours) and the fit with observed weather (0.973) of experiment 218. Every gain is an upper limit for production, because the UKV archive used here is the research archive on CEDA, which differs statistically from the live UKV. | small gain | not-for-production | Adding the Met Office UKV forecast to the ECMWF ensemble lowers the day-ahead error of a metered solar farm by about 1% to 2%. The gain is under the 3% we set as worth a second weather source, and the farms' fit on sunny half-hours moves only about 6% of the way to the fit with observed weather. The UKV archive used here is a research archive, and the live UKV differs, so this result is an upper limit for production; a weather product of a different kind is the option we see for a larger gain. | primary |
| 00224 metered-farm-dayahead-two-number | Passing each of the 51 ensemble members through a solar farm's plant model gives the farm's day- ahead output a spread that the ensemble-mean forecast does not have, and the mean and standard deviation of the 51 plant responses keep most of that information, as they did for Primary substations in experiment 193. A two-number plant forecast therefore forecasts a metered solar farm 12 to 36 hours ahead about as well as XGBoost, and gives calibrated tails. | partly supported | not-for-production | The 51 members of the weather ensemble help a single solar farm's day-ahead forecast only a little: passing each member through a solar-farm power model lowers the quantile score by 1.8% against feeding the model their mean, and two numbers (their mean and spread) carry 95% of that. An XGBoost model on the same weather scores 2.8% better overall, and the solar-farm model beats it only at the 0.99 quantile, the upper tail of the forecast, which matters for curtailment (switching a generator off when the electricity network is overloaded). | primary |
| 00218 metered-solar-farm-plant-fit | The six metered solar farms differ in panel tilt, panel azimuth, ratio of direct-current to alternating-current capacity, heat loss, and inverter clipping, so a plant model with those parameters fitted per farm reproduces each farm's output better than the one fleet shape (tilt 35 degrees, facing south, DC:AC ratio 1.15, 0.4% loss per kelvin, cell temperature equal to air temperature plus 25 K times the shape), and explains why the flat top of farms C, D, E, and F falls 6% to 9% from spring to summer (experiment 183). | not supported | not-for-production | Giving each metered solar farm its own panel tilt, azimuth, heat loss, and inverter limit fits its output only 0.003 better than one fleet shape (a fraction of output fitted of 0.953 against 0.950), so the plant is probably not what limits the fit. About 5% of the squared metered output stays unexplained, and the sunshine data is the likelier source. Per-farm plant parameters look unpromising for forecasting; a better sunshine forecast looks more promising, because the same plants fit only 0.91 with the ensemble forecast. | primary |