Differentiable physics: the forecast engine: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00306 summer-gap-solar-interactions-with-temperature-and-day-type | The April-to-June daylight excess of arm D (the convex demand model with a Kalman filter on recent errors) over BG2 (the production-style XGBoost forecast) at the substations D5 and D9 is an additive-structure problem. On cold clear spring days heating demand and rooftop solar output are both large, and an additive model must give the solar term one weight on every day. Experiment 299 found the D5 excess largest on clear midday half-hours (+1.67 points). Letting the solar shape interact with temperature (shape times the heating hinge) and with day type (shape times a weekend indicator) lets the model give those days a different solar weight. | refuted | not-for-production | Letting the rooftop-solar term vary with temperature and with weekdays versus weekends does not close the April-to-June gap to BG2: at the two substations with the gap, the extra columns close about 0% of it. Two linear interaction columns therefore do not explain the gap. The next test is what BG2 uses that the convex demand model does not (item 5 of the summer-gap brief). | also relevant |
| 00305 summer-gap-what-bg2-knows-that-arm-d-does-not | BG2 (experiment 87's production-style XGBoost forecast) beats the convex best-guess model (arm D) at the Primary substations D5 and D9 in April to June mainly through interactions of the sunshine inputs with temperature or with recent power, and not through the main effect of sunshine. | refuted | not-for-production | The production-style forecast does not beat the convex best-guess model in April to June at two Primary substations through interactions of sunshine with temperature or with recent power: forbidding them removes under 3% of its advantage on the half-hours where the convex model trails most, and 16% on all daylight half-hours. A refit allowed no interactions at all keeps 92% of the advantage on the worst half-hours but only 45% on all daylight half-hours, so the worst half-hours come from the shape of single inputs and typical ones partly from other interactions. | also relevant |
| 00299 summer-gap-timing-night-level-and-error-correction | The April-to-June excess of arm D (the convex demand model with a Kalman filter on recent errors) over BG2 (the production-style XGBoost forecast) at the substations D5 and D9 is a step that starts at the March clock change and sits at short leads, which would point at a local-time mismatch or at BG2's use of recent power. D10's night-time level offset of about -2 points (21:00 to 07:30 UTC) is a step on one date, not a smooth drift with temperature. The Kalman filter and the correction head are net harmful at D5 and D9 in April to June. | not supported | not-for-production | The spring excess of the best-guess demand forecast over BG2 grows with sunshine, appearing on clear midday half-hours at D5 and across the whole day at D9. It is not concentrated in the 4 weeks after the clock change or at short leads, so the next test is the solar term's response on clear half-hours at D5. The Kalman filter and the correction heads are not the cause: removing the filter changes little, and at D5 the heads reduce the excess. D10's night-time offset varies partly with month and has no single date on which it changes. | also relevant |
| 00266 summer-gap-diagnostics-solar-term | The best-guess demand stack (experiment 251's arm 1) trails BG2, the production-style XGBoost forecast, in April to June 2026 mainly because of its explicit solar term (the subtracted estimate of unmetered rooftop solar output), and not because of the neural-network head, the two-number summary of the ensemble's solar response, the Kalman filter, or the heating and lighting terms. The same defect explains the loss at D5, an anonymised Primary substation, on sunny half-hours. | partly supported | not-for-production | The best-guess stack trails BG2 in April to June because of the solar term at two of the six substations (D5 and D9 on daylight half-hours) and a night-time error at a third (D10), not because of the neural-network head, the two-number summary, the Kalman filter, or the ensemble sunshine forecast; the head and summary actually reduce the gap. The pinned solar capacity looks too high in those months, not too low, and clear mornings are forecast worse than clear afternoons. Candidate next experiments are the export clip at D9, a morning-and-afternoon solar shape, and a look at D10's night residuals; the stack's added parts need no change for this gap. | also relevant |
| 00251 dp-final-best-guess-assembly-clip-and-coast-against-bg2 | The full best-guess stack of experiment 220 (arm D, the neural-network head, and the two-number solar summary) beats BG2, the manual heuristic, climatology, and a weak production-style XGBoost model over the scored year of 2025-07 to 2026-06. Adding the hand-set 60% clip at D5 lowers the D5 error on sunny half-hours when the neural-network head is absent, and adds nothing measurable when the head is present, because experiment 232 found the head cancels most of the clip's gain. | supported | candidate | Over the one year we can score, the full best-guess demand forecast (arm D, a small neural network that corrects its median, and a two-number summary of the ensemble's solar response) beats BG2 by 0.136 points of quantile score (5%; lower is better) and the manual heuristic by 0.818 points, and it beats the manual heuristic on every headline metric we scored, including curtailment cost. The gain over BG2 is a winter gain: the stack is level with BG2 in summer and worse in April to June, and at D5 on sunny hours it is clearly worse than BG2. Adding the 60% solar clip at D5 helps only when the neural-network head is left out at D5, so the next work is the solar term at D5, not another stacked part. | also relevant |
| 00244 dp-deep-ensemble-heads-cold-night-bands | On cold nights the demand forecast's error is partly something the correction network does not know (epistemic uncertainty, as opposed to the weather forecast's own uncertainty, which experiment 192 found the 51 ensemble members' temperatures do not signal). The disagreement between 20 independently trained copies of the pooled correction network therefore predicts the absolute cold-night error, and widening the bands by a non-decreasing function of that disagreement lifts cold-night q95 coverage to 0.90. | inconclusive | not-for-production | The disagreement between 20 independently trained correction networks does not tell us how wrong the forecast will be on cold nights (correlation 0.00), so it cannot set the width of the cold-night bands: scaling the bands by it lifts the 95% quantile's coverage by 0.006, the same as a shuffled spread does. The check that the spread reacts to an unusual input failed for a realistic 4 C error, so the experiment cannot say more. The cold-night shortfall is therefore unlikely to be fixed by training more networks; the experiment cannot say what would fix it. | primary |
| 00234 no-xgboost-forecast-audit-part-1 | For Primary substation demand, a metered solar farm, and a metered wind farm, the best forecast that uses no XGBoost anywhere scores within 2% of the quantile score of the best forecast that uses XGBoost, at 12 to 36 hour leads. | partly supported | candidate | An XGBoost-free forecast of Primary substation demand scores 7.5% better than BG2, the XGBoost best guess, at 12 to 36 hours ahead, on one year and six substations. The XGBoost-free solar and wind forecasts score 2.8% and 5.0% worse, and lose at the 0.01, 0.05, and 0.95 quantiles. Only the solar plant model's 0.99 quantile beats XGBoost, and that is the tail that matters for curtailment. The next experiment tests small XGBoost components inside the XGBoost-free solar and wind forecasts. | also relevant |
| 00228 dp-bg-rolling-origin-stack-against-bg2 | The best-guess stack (arm D with experiment 169's bands, plus the network head of experiment 201) beats BG2 in most monthly origins and at most substations, and its advantage is larger in winter than in summer. | partly supported | candidate | Over the one year we can score, the full best-guess stack (arm D, the neural network head, and the solar summary) beats BG2 by 0.136 points of quantile score (5% of BG2's 2.715; lower is better), and most of the gain comes from November to March. From April to June 2026 the stack is worse than BG2, and arm D without the neural network head is worse than BG2 in summer. The year-long gain is only just separable from zero, and we cannot test other years until BG2 is re-run for them. | also relevant |
| 00225 dp-network-heads-on-unlabelled-worst-days | Experiments 172, 174, and 175 found that the worst 5% of substation-days carry 41.7% of the excess loss of arm D (the best-guess convex forecast of experiment 155), and that 37 of the 72 worst days at the five decision substations carry none of the labels (switching event, holiday, cold snap, flagged reading, Economy 7 step). Experiments 200 and 201 found that a neural network corrects arm D mainly on very hot and very cold days. The unlabelled worst days therefore sit at temperature extremes, and the neural network cuts their loss. | partly supported | not-for-production | The neural network does cut the loss on the 37 worst days that no label explains, by about a fifth, and cuts it by more than on the 35 days that do carry a label. The unlabelled worst days are not at temperature extremes, so a neural network aimed at the hottest and coldest days would not target them. What helps on them is the forecast input: a neural network without one removes 4%. | primary |
| 00222 dp-network-before-filter-order | Experiments 200 and 201 correct arm D after the Kalman filter. Their gain is mostly a systematic error of the convex model at some temperatures and loads, not a recent-error effect, and that systematic error currently sits in the filter's level state. A network that reads only the convex model's own output and inputs, and corrects the convex model before the filter, leaves the filter a cleaner residual stream and so beats the same kind of network applied after the filter. | inconclusive | not-for-production | The gain of the network after the filter does not come from the convex model's own systematic error as seen in its past residuals. A network trained on those residuals learned patterns that do not carry to the next year, and when it was stopped on a later year it learned nothing and left the score unchanged. The test could not settle the order question, because the network failed to recover a known injected error. | primary |
| 00220 dp-bg-stage4-assembly-with-drop-one | The three parts added to the best-guess demand forecast since experiment 170 remove mostly different errors, so their gains add. The parts are the two-number solar summary of the 51 ensemble members (experiments 193 and 202), the pooled correction network on arm D's mean (experiments 200, 201, and 209), and the coast-and-widen rule for missing readings (experiment 205). The solar summary changes only daylight bands, and the network corrects arm D's median from its own output, the calendar, and temperature, so the two act on different rows and different error. The network may remove what the winter night offset and recipe R removed (both correct arm D's mean by time of day), and it may make the bands' cold-night widening unnecessary. | partly supported | candidate | Adding two parts to the best-guess demand forecast lowers its quantile score by 0.136 points (5%) of substation effective capacity. The parts are a small neural network that corrects the forecast's median, and a two-number summary of the 51 ensemble members' solar response. Each part's gain is almost exactly what it achieves alone, so the two parts remove different errors. The network's own 11 quantiles reached 0.904 coverage of the 95% quantile on cold nights without the hand-fitted band fixes, and scored 0.048 points better than the corrected forecast overall, mostly at one substation. The coast-and-widen rule for missing readings was not scored with the stack. | primary |
| 00216 dp-network-shape-on-arm-d-flexible-temperature-tails | The neural network of experiments 200 and 201 lowers the quantile score of arm D (the best-guess convex forecast of experiment 155) by 0.09 to 0.125 points, mainly by raising the median on very hot and very cold days. That correction is a shape the convex model could carry. The convex model's only temperature terms are one-sided heating hinges at 15.5 degrees Celsius, so it has no term for cooling load and no extra slope in the cold tail. Extra hinge columns in the tails of the site's own temperature range, with a weaker ridge penalty on those columns, recover at least half of the network's gain without a network. | inconclusive | not-for-production | Adding extra temperature terms for the hottest and coldest days to the convex demand model recovers only a fifth to a quarter of the neural network's improvement, and the recovery was not confirmed in a second year. The neural network's correction is spread over all temperatures instead of sitting at the extremes, and a smooth function of temperature explains under 1% of it, so the neural network uses an input other than temperature. The formal verdict is capped at inconclusive because a placebo check narrowly failed, but the shortfall does not depend on that check. | primary |
| 00211 dp-boosted-tree-calibrator-and-input-ablations | The 0.065 points by which the network without error inputs (experiment 200) beats the best lookup table (experiment 209) comes from a smooth or higher-order dependence on a few inputs, so a gradient-boosted tree model fitted with the pinball loss on the same inputs captures it, and a boosted-tree calibrator can replace the neural network. One-at-a-time removal of the network's inputs shows which input carries the gain. | supported | candidate | A gradient-boosted tree model given the same inputs as the neural network, and no recent errors, lowers the error score of our best day-ahead forecast by about as much as the network does (0.098 against 0.100 points). The interval is wide, so a loss of up to 0.03 points against the network is not ruled out. The gain comes mostly from two inputs, the forecast level and the weather-data era. | primary |
| 00209 dp-pinball-table-vs-network-head-on-arm-d | The gain that experiment 200's network head and its blind copy get over arm D (0.100 to 0.125 points) is a recalibration that a table without hidden layers can reproduce. The table gives arm D's median a shift that depends on site, half-hour, day of week, lead band, and arm D's forecast level by an ensemble-era flag, and scales arm D's quantile offsets by site, lead band, and six-hour block. It is fitted by gradient descent on the pinball loss over the 11 quantiles, on the same rows as the network. A second arm adds a temperature-extremes term (a median lift below -3 degrees Celsius and above 24 degrees Celsius, from experiment 201's finding). | not supported | not-for-production | A lookup table of corrections to our best day-ahead forecast lowers its error score by only 0.03 to 0.04 points, against 0.10 points for a neural network given similar inputs and no recent errors. This table design therefore does not match the network, and the cause is not yet isolated. A temperature-extremes term lowers the score by under 0.001 points. | primary |
| 00205 dp-gamk-gap-robustness-coast-or-frozen-gam | When power readings go missing, a Kalman filter that coasts (keeps running with no update, so the transient decays and the level holds) beats the frozen convex model (GAM) alone for short gaps and loses to it for long gaps, so a crossover gap length exists between 6 and 72 hours. The mechanism is that the level is a random walk, so a short gap leaves it nearly right, while a long gap lets it drift away from the demand. Widening the quantile bands by the filter's own growth in predictive variance keeps the 5% and 95% quantiles calibrated at the gap lengths where coasting is still better than the frozen model. | inconclusive | candidate | When readings stop for 3, 6, 24, or 72 hours, keep running the Kalman filter, which corrects the convex demand model using its recent forecast errors, without feeding it new readings, and widen the quantile bands. At each of those four gap lengths that forecast scored 0.25 to 0.44 points of quantile score better than the convex demand model with no correction (a quantile score is an error measure in percent of the substation's effective capacity, and lower is better; the gain is 8% to 14% of the uncorrected model's score). At 72 hours the 99.4% bootstrap interval for the gain runs from 0.04 to 0.48 points, so the gain may be small. The data cannot show whether dropping only the filter beats coasting after a gap of about 72 hours: at every gap length the difference between the two forecasts is smaller than the variation between months. The forecast with the filter, winter night offset, and recipe R was still 0.025 points worse than its no-gap score 48 hours after a 24-hour gap ended. The filter alone was within 0.02 points of its no-gap score 24 hours after the gap ended. | primary |
| 00202 dp-ens-two-number-rolling-replay | Experiment 193 found that two numbers per half-hour (the mean and the standard deviation of the 51 ensemble members' solar responses) lower the daylight quantile score by 2.20% in the scored year. Its two fitted factors per lead band were fitted on interleaved months (the odd months of 2024-07 to 2025-06), some of them later than the replicate months they were applied to, and its numbers were held in float64. A replay of the live weather layer, which stores the two numbers as float32 columns computed from the one ensemble run published before each origin and refits the factors each month on history that has completed before that month, keeps most of the gain. | supported | candidate | A weather layer that stores two numbers per half-hour (the mean and spread of the 51 ensemble members' solar-generation estimates) and refits its two scaling factors every month, using only runs that had finished before the month began, lowers the daylight quantile score (the score for the forecast's uncertainty bands) by 2.3% at five substations over one year, which is as much as running all 51 members. This confirms experiment 193's result under production-style timing, so the next step is to build the two columns into the weather layer. | primary |
| 00201 dp-gam-residual-mlp | A small neural network that reads the convex model's own design features, and later the calendar, the ensemble-mean weather, and lagged power, can learn an interaction the additive convex model lacks, and so lowers the quantile score of the best-guess convex forecast (arm D of experiment 155) when it adds a correction to the median and rescales the band widths. | partly supported | not-for-production | A small neural network that reads the convex demand model's forecast lowers that forecast's quantile score by about 0.09 points (3%), and adding weather, calendar, or recent power does not help further. The neural network corrects an under-forecast on very hot and very cold days. That points to a missing temperature term in the convex demand model. The correction also scores worse on bank holidays and, in point estimate, on Christmas. | primary |
| 00200 dp-pooled-correction-head-on-kalman-state | Experiment 142's small network failed its known-answer gate with about 1,100 samples from one year at five substations. The same kind of network, trained on rows pooled across many Primary substations and years, learns to correct arm D's forecast from the recent errors, the calendar, the lead, and the ensemble-mean weather, and so beats recipe R (subtract 0.75 times the mean error of the last 14 days) at day D+1. | partly supported | not-for-production | A small neural network trained on arm D's own past errors lowers the day-ahead score by 0.07 to 0.12 points at five substations, but a copy that sees no recent error gets most of that gain. The gain comes from correcting how arm D's error depends on its forecast level and the calendar, and a pinball-loss table comes within 0.01 to 0.02 points of the network on one year of data. | primary |
| 00197 dp-kalman-level-adaptation-retest | Experiment 190's known-answer gate could not be passed by a filter with one state per hour band, because the target's band is usually unread since an offset began. Scored only on pairs the filter can know about, (a) a faster per-band filter (A1) recovers at least 50% of a 0.10-size, 16-hour offset on target bands already read, and (b) a filter whose bands share one level while a same-sign run lasts (A3) recovers at least 50% on target bands not yet read, because it moves bands that no reading has reached. | not supported | not-for-production | Making the Kalman filter's level state faster does not fix offsets that last for hours. Faster settings recover a long offset only late, and only by scoring worse on quiet days, and a level shared across the four hour bands recovered under a fifth of the offset, because its run counter cannot tell an offset from noise. No setting on the grid tried recovered an injected offset without scoring worse on quiet days, so level adaptation is dropped. | primary |
| 00193 dp-ens-two-number-solar-summary | Experiment 187 found that passing each of the 51 ensemble members through the solar term lowers the daylight quantile score of the full best-guess stack by 2.23%, and that a Gaussian with each row's own mean and spread of the 51 responses recovers 95% of that gain. Two numbers per half-hour (the mean and the standard deviation of the members' solar responses) therefore carry nearly all of the gain, so a serving path can store two columns instead of running 51 passes. | supported | candidate | Summarising the 51 ensemble weather members' solar estimates by their mean and standard deviation lowers the daylight quantile score by about 2% (98% of the gain from running all 51 members, 93% in the replicate months), at the five substations scored over one year. The weather layer could store two columns per half-hour instead of 51, and the 10th and 90th percentiles add nothing measurable. A follow-up experiment could test the two numbers in the live weather layer. | primary |
| 00192 dp-ens-members-temperature-cold-nights | Experiment 187 found that passing the 51 ECMWF ensemble members through the solar term of the demand model lowers the daylight quantile score by 2.2%. The demand response to heating is also nonlinear (a hinge at 15.5 degrees C), and the ensemble mean smooths away the cold tail. Passing each member's temperature through the temperature terms therefore lowers the quantile score on cold winter nights. | not supported | not-for-production | Running each ensemble member's temperature through the demand model gave no detectable improvement in the forecast bands on cold winter nights, because the 51 members' temperatures differ too little to signal the day's error. The cold-night bands are already too narrow. The cause of the cold-night error is still unknown; widening the bands directly, and testing wind, night cloud cover, and switching as inputs, are the candidates. | primary |
| 00191 dp-price-sensitive-assets-in-the-history | Some of D8's multi-day drops (experiment 159) and the matching shortfalls at its neighbour D14 are output of price-following gas generation (experiment 186), which no weather input explains and no forecast of the price can reach at a lead of days. Letting the convex demand model (GAM-K) see the half-hourly price in its training history, gated by the registered connection date of the 6 MW gas engine, lets it learn a cleaner baseline for the rest of the demand. The forecast then improves even when the price is unknown at forecast time. | not supported | not-for-production | Modelling a price-following asset in the history does not make the forecast better at D8 or D14. A demand model that is given the true price at forecast time, which no live service has here, gains only 0.6% at D8, and the models that use a price available at forecast time lose 0.9% to 4.4%. The gas engine's response is real enough to fit (1.2 to 1.9 MW at D8 within the training rows), so the remaining D8 error comes from causes that these price indicators do not explain, and NGED's half-hourly output for the engine is the data to ask for next. | also relevant |
| 00190 dp-kalman-level-adaptation-gated-boost | The Kalman filter of the convex demand model (GAM-K) has a level state that is a random walk with one shared process-noise setting. A single setting must trade speed on a real level shift against noise on ordinary days. A gate that multiplies the level's process noise by m only while the last k one-step innovations (pooled over the four hour bands) share one sign and their mean exceeds z times the day-before innovation spread gives speed when a shift looks real and keeps the quiet setting otherwise. The mechanism predicts that the gate lowers the excess loss on days where the residual keeps one sign for hours (the 15 smooth-offset days of experiment 175), without harming ordinary days. | inconclusive | not-for-production | A Kalman filter whose level state speeds up when the residual keeps one sign scored the same as the current filter (+0.0002 points), and an always-fast version scored 0.023 points worse. The test cannot say that filter speed is useless: its known-answer gate could not have been passed by any filter with one state per hour band, and a faster filter did recover offsets where they were visible. A fair test needs a redesigned gate and a wider grid of multipliers. | primary |
| 00187 dp-ens-members-in-stack | Experiment 184 found that running each of the 51 ensemble members through the solar term of the convex demand model lowers the daylight quantile score by 2.2%, but its verdict was capped at inconclusive because a monotone gate failed on rows where the unit PV shape itself falls as irradiance rises. With a gate that matches that shape, the same arm passes, and its gain lies where the best-guess stack leaves the largest sunny-daylight error, at the substation D5. |
supported | candidate | Running each of the 51 ensemble weather members through the solar part of the demand model still improves the daylight forecast bands by about 2% on top of the full best-guess stack, and night is untouched. About two thirds of the gain needs that day's own members, and two numbers per half-hour (the mean and spread of the members' solar responses) carry nearly all of it, so a cheaper two-number version is the next thing to test. | primary |
| 00185 dp-price-dfs-national-signals-past-demand | A national signal that experiment 177 found on many of the 37 unlabelled worst days (a high national system price, or a Demand Flexibility Service (DFS) event) explains part of the day-to-day variation in the best-guess forecast's loss over all days of the scored year, not only on the 37 worst days. Such a signal cannot be forecast, but a hindsight covariate may help the model interpret the past. | refuted | not-for-production | The best-guess forecast's bad days are not explained by national system price, Demand Flexibility Service events, national demand, wind, solar, or interconnector flow: each explains under 1% of the day-to-day variation in the forecast's loss, and handing the signal to the model in hindsight makes the score slightly worse. High price days are a little over-represented among the 37 worst days, so the price may mark a few of them, but it does not help the model read the past. The search for causes should move to local sources (outage notices, local events). | primary |
| 00184 dp-ens-daylight-calibration | Experiment 169 found that passing the ensemble's spread to the width of the forecast's bands adds nothing when scored over all half-hours. The benefit may be confined to daylight half-hours, especially sunny and partly cloudy ones, where the clouds that the 51 ensemble members disagree about change the output of embedded solar generation, and so the demand. Passing each member's irradiance through the convex model's solar term (arm M), or using the spread of the members' irradiance as a fraction of clear-sky to set the band width in daylight (arm S), calibrates the daylight bands better than the existing bands (arm A). | inconclusive | not-for-production | Running each of the 51 ensemble weather members through the solar part of the demand model improves the daylight forecast bands by about 2%, mostly in the lower tail and on partly cloudy half-hours, while night is untouched. Placebos that widen the bands the same way but without the real members gain little, so most of the benefit is information about cloud. The next step is a production-cost check of the member pass and a test of its gain on top of the best-guess forecast. | primary |
| 00177 dp-worst-days-other-data-sources | At least a quarter of the 37 unlabelled worst substation-days of experiments 172, 174, and 175 coincide with an event recorded in data the project already holds: a logged switching operation near the substation, a step in a metered generator on the same or an interconnected Primary, a reading-quality fault in the substation's own series, or a repeat of the substation's own bad weekday or date. | refuted | not-for-production | The 37 worst days of the best-guess forecast that carry no label are not explained by any of the six sources we tested: logged switching, reading faults, repeat bad weekdays, Demand Flexibility Service events, and national price spikes each cover them no more often than they cover ordinary days. The three raw NGED files we read hold only time and power. Sources we have not tested include local outage notices, local events, and a location join to the metered generators. | primary |
| 00175 dp-unlabelled-day-shapes | At least half of the 37 unlabelled worst substation-days of experiment 174 are sharp-edged blocks in the forecast's error (a switching-like change in the power a substation draws), and in at least a third of those a substation interconnected with the first shows a step of opposite sign within 2 hours (a load transfer between the two). | refuted | not-for-production | The forecast's worst unlabelled days are roughly half sharp-edged blocks (a switching-like change in the power a substation draws) and 40% smooth offsets (a change in level or behaviour that lasts the day). Interconnected neighbours do not show the opposite step more often than chance would give, and the existing switching detectors fired on none of the block days. The counts point to two separate problems, level changes and blocks that the interconnection table does not explain. The shape rules miss about 40% of injected blocks, so the block share is uncertain. | primary |
| 00174 dp-unlabelled-worst-days-weather-shared-or-single-site | Most of the excess loss on the unlabelled worst days of experiment 172 is weather error, and the unlabelled worst days are mostly shared with other Primaries. Replacing the ensemble-mean weather with the estimate of past weather (ERA5 temperature and CAMS irradiance) in arm D's forecast removes at least 50% of the unlabelled days' excess loss, and at least a third of the unlabelled days are also in the worst 5% of days of at least one other Primary. | partly supported | not-for-production | Most of the forecast's error on its worst unlabelled days is not weather error: with past weather in place of the forecast weather, about 69% of the excess loss stays. Other Primaries share these days about as often as random shifting gives, so the data show at most a weak shared cause. The remaining loss could therefore come from events at one substation, such as undetected switching, a change in behaviour, or a metering fault. The next place to look is the substation's own recent readings and the switching records on those days. | primary |
| 00173 dp-christmas-more-years-and-sites | The convex demand model with a Kalman filter on recent errors and recipe R (the mean error of the last 14 days by half-hour, held to 0.75 of its size), which scored 1.13 times an ordinary weekday over 24 December to 2 January at five substations in two Christmases (experiment 170), keeps a Christmas-to-weekday error ratio under 1.3 in every Christmas back to 2019-20 and at 14 or more of 16 Primary substations, and does not over- or under-forecast 25 and 26 December by more than 5% of that day's demand. | inconclusive | not-for-production | Where the demand model had a winter of training data, the error over 24 December to 2 January was 0.97 to 1.22 times an ordinary weekday in all four Christmases from 2022-23 to 2025-26, so no special-day term is justified. The pre-registered run failed because the training rows of Christmas 2020-21 and 2021-22 (and of one substation in 2023-24) held little or no winter, since the lockdown window is left out of training. A forecast for a substation with under 180 days of usable history should be flagged, not trusted. | primary |
| 00172 dp-error-budget-worst-days | The error of the stage 2 assembly (experiment 155's arm D) is concentrated on a small number of days that the project already knows how to label. The worst 5% of days, ranked by their pinball loss above the site's median day, carry at least 30% of the total excess loss, and at least half of those days carry one of six labels: a switching event, a metering error or flagged reading, a cold snap, an Economy 7 step, a holiday, or a combination. | inconclusive | not-for-production | A small number of bad days carries a large share of the forecast's error: the worst 5% of days carry about 42% of the excess loss, and about half of those days coincide with a switching event, a holiday, or a cold snap. Switching events are the largest group, on days that carry 29% of the excess loss, so a forecast that improves only ordinary days leaves most of the loss in place. Cold snaps carry 5% of the excess in this mild scored winter, so this year cannot show whether a cold-night term pays. The formal verdict is inconclusive because one control narrowly failed. | primary |
| 00170 dp-bg-stage3-drift-and-holiday-scoring | The convex demand model with a Kalman filter on recent errors, ensemble-mean weather, the heating split by day type, one shared filter noise setting, and the pinned solar capacity (experiment 136's arm A2) improves further when it gets a drift term (experiment 167's D1) on top of the two post-forecast corrections of experiment 155's arm D (a winter night offset and recipe R's 14-day mean error profile). The mechanism is that the drift term removes a slow demand-level error that the monthly refit on 24 months of history leaves in, and neither the offset (winter nights only) nor recipe R (a 14-day profile held to 0.75 of its size) removes that error. Training on every year of history adds a further gain, because the drift term and the pinned capacity let the old years teach the heating and calendar terms. | refuted | not-for-production | A slowly varying demand level (a drift term) improved the forecast by 0.40 points in experiment 167 but worsened it by 0.03 points once the forecast has the Kalman filter on recent errors (lower scores are better), and training on all years of history did not change that. The filter already appears to remove the slow level error that the drift term fixes, because without the filter the drift term improves the score by about 0.2 points. The drift term should not be added to experiment 155's arm D. Christmas (24 December to 2 January) scored 1.13 times an ordinary weekday at the same half-hours, below the 1.5 times predicted, and recipe R improved the Christmas score by 0.04 points. | primary |
| 00168 dp-gamk-production-cadence-and-3-5-hour-delay | The GAM-K forecast of experiment 155's arm D (the convex demand model with a Kalman filter on recent errors, the winter night offset, and recipe R) keeps its score when it is issued the way production would issue it. Production refreshes the forecast every 6 hours from the newest ECMWF run, so the weather forecast is 12 to 30 hours old, and the filter reads only power readings that are at least 3.5 hours old. Experiments 136 and 155 issued one forecast a day, 9 hours after the run. The mechanism is that a fresher filter state makes up for an older weather forecast, so the score at a given lead changes little. | inconclusive | not-for-production | Issued every 6 hours from the newest weather forecast with power readings 3.5 hours old, the stage 2 forecast scores about the same as the once-a-day forecast over the next 24 hours (-0.007 points, 99% interval -0.076 to +0.037). The largest loss, 0.073 points, falls between 09:00 and 12:00 UTC, when the new weather run has arrived and the 6-hourly schedule has not yet used it. The pre-registered same-lead contrast read +0.076 but is confounded with the hour of the day, so the formal verdict is inconclusive. | primary |
| 00158 dp-partly-pooled-gam-shapes-heating-and-calendar-curves | A GAM-K forecast (the convex demand model with a Kalman filter on recent errors, experiment 111) fitted jointly across six Primary substations, so that each substation's heating curve, lighting curve, daily profile, holiday effects, and Christmas shape are a fleet mean plus a penalised deviation of its own, forecasts better than the same model fitted to each substation alone, now that the solar capacity is pinned and the filter handles recent switching. The mechanism is that a substation sees only two winters and two Christmases in a 24-month window, so its own curves are noisy, and five other substations carry information about the shapes that physically are shared. Capacity, level, and the band means stay per substation. The gain is largest for a substation with 3 months of history and on Christmas rows, and no pooled arm is worse than the per-site fit by more than 0.02 points on ordinary rows. | partly supported | candidate | In this test on six Primary substations, fitting the demand model's shared curves (heating, lighting, daily profile, holidays) across all six let a substation with only 3 months of history score 3.01 points instead of 3.64 (lower is better), within 0.17 points of the same substation with 24 months of its own history. The gain was largest in winter and on Christmas rows. For a substation with 24 months of history, pooling changed the score by less than 0.02 points, which is too small to justify making each forecast depend on five other substations' data. The test cannot say whether pooling the heating curve alone would help a 3-month substation. | also relevant |
| 00155 dp-bg-stage2-assembly-night-offset-and-error-correction | A GAM-K forecast (the convex demand model with a Kalman filter on recent errors, experiment 111) that already has the heating split by day type, one shared filter noise setting, the pinned solar capacity, and the ensemble mean as its weather input (experiment 136's arm A2) improves further when two post-forecast corrections are added: a winter night offset (experiment 137) and the 14-day mean error profile of recipe R (experiment 115). The mechanism is that the two corrections remove different errors, a level error that repeats on winter nights and a half-hour-of-day shape error that repeats day to day, so their gains add. The assembled arm then scores below experiment 111's GAM-K and BG2 (experiment 87) over leads 0.5 to 168 hours, and its tails on cold nights and on holidays are no worse than the uncorrected arm's. | inconclusive | candidate | Adding recipe R, which subtracts 0.75 times the mean error of the last 14 days by half-hour, lowers the stage 1 assembly's score by about 0.04 points (1.3%), and the winter night offset adds nothing once R is in. The assembly with R scores 2.754 against BG2's 2.780, which is better by 0.026 points with an interval too wide to rank the two. The pre-registered placebo control failed, so the formal verdict is inconclusive, but the choice of winning arm (R alone) does not depend on it. | primary |
| 00152 dp-mva-fold-aware-gam-k-at-mva-only-primaries | At the MVA-only Primaries, the GAM-K forecast driven by the ensemble mean (experiment 136's arm A2) forecasts the MVA reading better when it models the fold: it fits the demand model through the meter (experiment 110's L0), corrects the forecast with a Kalman filter whose state is the signed flow, and builds its quantiles by folding a signed predictive distribution. The gain is concentrated at Stickney (D12), the one trial Primary where reverse flow is the usual state on sunny middays, and the change is harmless at the Primaries without reverse flow. | inconclusive | not-for-production | At Stickney (D12), the one Primary with clear reverse flow, drawing the quantiles around a signed flow and folding them at zero lowered the score by about 0.4 points on sunny middays against the same fit with symmetric quantiles, and it did no harm at the other seven Primaries. The whole-year gain against reading the apparent power as signed power is of similar size, but its interval in the scored year includes zero. The known-answer test on five Primaries with hidden farms passed on the sign at the larger injection and failed at the smaller injection and on the upper tail's coverage, so the experiment does not support a port yet. | also relevant |
| 00145 differentiable-physics-analog-day-error-correction-on-gam-k | Averaging GAM-K's past day-ahead errors over the days most similar to the target day (same day type, with holiday identity for holidays, similar forecast daily mean temperature, and a season window of a few weeks either side of the date) gives a better correction than averaging the last 14 days alike (recipe R of experiment 115) or the same-day-type days of the last few weeks (experiment 134). The reasoning is that demand errors repeat on days that resemble each other. The errors that repeat most, on holidays and on cold days, are the ones that a 14-day average dilutes. | small gain | candidate | Averaging the errors of the 20 most similar past days (same day type, within 13 weeks of the same date, last year's days included) beat recipe R (the 0.75-weighted mean error of the last 14 days) by 0.041 points at day D+1, the calendar day after the forecast origin (about 1.7% of R's score of 2.436). That gain is larger than the 0.011 points that the weekday term of experiment 134 gained. Matching on forecast temperature helped by only 0.005 points, and copying the error of the same holiday last year made holidays worse, so the next step is a day-type profile with a season window and no correction copied from an earlier holiday. | primary |
| 00144 differentiable-physics-weather-forecast-bias-persistence-on-gam-k | The day-ahead error of the ensemble weather forecast (temperature and irradiance) repeats from one day to the next, so the trailing mean of recent weather errors can correct GAM-K's demand forecast beyond what the Kalman filter already does. | refuted | not-for-production | In this test, the day-ahead temperature and irradiance forecasts did not repeat their errors from one day to the next: the correlation between one day's error and the next day's was at most +0.11, against a stop threshold of 0.2. A correction that feeds recent weather errors into the demand forecast therefore has little signal to use, so no demand forecast was fitted. Two caveats apply: wind was not tested, because experiment 111's ensemble store does not hold it, and the reference was a later forecast (the analysis proxy), not observed weather, so a bias against observations is not ruled out. | primary |
| 00143 differentiable-physics-holiday-aware-updating-of-the-gam-k-filter | The Kalman filter of GAM-K lets a holiday's reading error teach it a wrong level, which carries into the days after the holiday; leaving holiday readings out of the filter's update (and nothing else) lowers the day-ahead score on those following days, and does not hurt the holiday targets or the rest of the year. | not supported | not-for-production | Leaving holiday readings out of the day-ahead filter's update does not reliably help the days after a holiday: the effect had the opposite sign after Christmas 2024 and Christmas 2025, and its 99% interval in the scored year (-0.309 to +0.036 points) includes zero. In 10 of 12 holidays, the holiday readings' error had the same sign as the error on the following days, so these readings are not usually contamination. Setting the correction to zero on holiday days, as experiment 118 did, is the action that hurts (+0.62 points on holiday days). Two Christmases give no basis for changing the filter this Christmas. | primary |
| 00142 differentiable-physics-sequence-model-on-gam-k-residuals | A small neural network that reads GAM-K's last 14 days of residuals, the calendar, and the target day's forecast weather predicts the day-ahead residual better than the hand-built Kalman filter with the 14-day error profile (recipe R of experiment 115), and gives better tails when it is trained with the pinball loss at the 11 quantiles. | inconclusive | not-for-production | A small neural network that reads the last 14 days of GAM-K's errors did not learn to adapt to changes in them: when we added a known step or a Saturday-evening shift to real errors, it recovered none of it, so we did not score it on the scored year. Nothing was scored, so this result does not compare it with the hand-built Kalman filter and recipe R. | primary |
| 00141 switching-detector-fed-into-gam-k | Telling GAM-K's Kalman filter when the RO-NC detector has seen a load transfer lowers the forecast's quantile score on days after a transfer begins, because the filter's level then adapts within hours instead of over days. The same signal given as plain features (event age and signed step size) does the same, more weakly. | not supported | not-for-production | Telling the forecast's filter when a load transfer started and ended helped inside the transfer and hurt in the 72 hours after it, because the filter holds the transfer's step over the whole 7-day forecast. The switching detector's alarm arrives 48 hours after the step and did no better than a random placebo. The next test is a correction that decays over the typical transfer duration. | also relevant |
| 00139 differentiable-physics-pv-as-of-capacity-in-gam-k | Feeding the convex model with the Kalman filter (GAM-K, experiment 111) the as-of solar capacity of experiment 129 in place of experiment 45's pinned capacity lowers the quantile score on the sunniest third of half-hours, and does not raise it on the darkest third. The mechanism is that the pinned capacity is a hindsight growth prior, and the as-of capacity is read from each Primary's own power before the forecast origin. | refuted | not-for-production | Replacing the pinned solar capacity with a capacity read from each substation's own power did not improve the forecast, and made sunny half-hours worse by 0.11 points of capacity. A more accurate capacity (the hindsight arm) only matched the pin, so the accuracy of the capacity number is not what limits the solar part of the forecast. This experiment does not locate the remaining sunny-day error; it did not test the ensemble's cloud forecast, the solar shape, the Kalman filter, or the non-solar load. | also relevant |
| 00137 differentiable-physics-e7-economy-7-night-step-term-in-the-forecast | An additive night term, fitted on the base forecast's own past errors as a function of the ENS-mean daily temperature and the UTC clock time, lowers the day-ahead quantile score on the half-hours from 01:00 to 03:00 UTC on cold winter nights, without raising the score elsewhere. The mechanism is that the base GAM-K forecast (experiment 133's shared-noise arm) runs below actual demand from about 01:00 on cold nights (experiment 123), so a correction learned from earlier winters removes a bias the forecast repeats. | inconclusive | not-for-production | The day-ahead forecast runs about 2.5 points too low on cold winter nights from 01:00 UTC, and a correction learned from its own past errors removes that bias and lifts the 0.95 quantile's coverage there from 80% to 93%. An audit showed that a plain winter night offset does as well, so the experiment found a winter night bias, not a step that grows with cold. The verdict is inconclusive: one winter, 365 headline rows, and a failed placebo. | also relevant |
| 00136 dp-bg-stage1-assembly-on-the-ensemble-mean | A convex demand model with a Kalman filter on recent errors (GAM-K, experiment 111), given experiment 114's heating split by day type (T-all) and experiment 133's single shared filter noise setting (POOL), and driven by the ensemble mean of the ECMWF ensemble (ENS) instead of its 51 members, forecasts Primary demand 0.5 to 168 hours ahead at least as well as BG2 (experiment 87) and better than experiment 111's GAM-K. The mechanism is that the three gains came from different parts of the model (the hinge, the filter noise, the weather input) and so add, and that experiment 125 found that the members' own spread adds no tail skill, so one forecast on the mean loses nothing. | inconclusive | not-for-production | Putting everything that has helped so far into the convex demand model and driving it from the ensemble mean gives a forecast that scores the same as the production recipe (BG2) and as experiment 111's GAM-K, and the experiment cannot tell any of the three apart. Using the ensemble mean makes the score about 0.04 points worse than using all 51 members, and the heating split and the shared filter noise improve it by about 0.05, so the assembly did not add up to the hoped-for gain. | primary |
| 00135 differentiable-physics-n10-joint-kalman-and-demand-fit | Fitting the convex demand model (GAM-K) so that its coefficients minimise the day-ahead error after the Kalman correction, instead of the plain error before it, lowers the day-ahead quantile score at 12 to 36 hours below the two-stage fit (fit the demand model, then filter its residuals), without moving the fitted shapes to explain recent-level structure that the filter already tracks. | not supported | not-for-production | We fitted the demand model so that its coefficients minimise the error left after a Kalman filter (a running correction of recent errors) has adjusted the forecast. That made the day-ahead forecast much worse, because the filter can absorb a shift in the slow parts of the demand model, so the fit barely constrains those parts. Tying the fit to the two-stage coefficients removed the damage and also removed any gain: the quantile score was within 0.013 points of the two-stage score, in either direction. The two-stage design (fit the demand model, then filter its residuals) stays. | also relevant |
| 00134 differentiable-physics-same-weekday-error-profile-on-gam-k | Adding a same-weekday component to the 14-day all-days error profile (recipe R of experiment 115) lowers GAM-K's day-ahead quantile score, because demand errors repeat from week to week (Saturdays differ from Tuesdays) and the all-days profile averages that structure away. | not supported | not-for-production | The gain is too small to count: using the same weekday's recent errors, instead of the last 14 days' errors alike, lowers the day-ahead error by about 0.011 points (0.5%) at five Primary substations, and the scored-year interval includes zero. The gain is on Mondays and Fridays and is absent midweek. Grouping days into five day types did better than seven separate weekdays, so day types are the version to test next, with the profile limited to day D+1. | primary |
| 00133 differentiable-physics-vk-learned-noise-and-observation-variance-in-gam-k | On GAM-K's filter (a level and a transient per hour band, per substation), noise settings chosen per substation by day-ahead score lower the day-ahead quantile score below the fixed settings of experiment 111, which were chosen by marginal likelihood on 184 non-contiguous days. An online observation variance in the manner of VIKING (de Vilmarest and Wintenberger, 2022), used to scale the spread, then improves the coverage of the 0.95 quantile on cold nights, which experiment 125 found at 73% to 78%, because recent large errors widen the spread. The mechanism under test is that the size of recent proxy-driven residuals predicts the size of the next day's error on cold nights. | inconclusive | not-for-production | Choosing the filter's noise settings for each substation by how well they forecast the next day did not improve the forecast on later months: the gain of 0.044 points on the months used to choose shrank to 0.009 on the scored year, inside the noise. A single setting shared by every substation did at least as well as the per-substation settings, so tuning the filter's noise per substation looks unnecessary, although the grid did not find a minimum. Letting the observation variance follow recent errors did not widen the forecast where it matters, on cold nights, where the 0.95 quantile still covered only 86% to 88% of readings. | primary |
| 00130 ro-nc-detector-on-the-gam-k-residual | The refit-offset neighbour-count detector (RO-NC) finds more of the moved megawatts (MW) of injected load transfers when its residual is the one-step innovation of the convex demand model with the Kalman filter (GAM-K), than when it is the residual of the lagless XGBoost model of experiment 82. A lower residual noise should lift more of the donor steps of 2 to 4 noise units over the 4.0 threshold. In experiment 82, 401 of the transfers that RO-NC missed had no metered receiver and a donor step of 2 to 4 noise units. | inconclusive | not-for-production | A detector of load transfers found about 2 more of every 100 megawatts moved when it was built on a demand forecast from the convex model without recent power than when it was built on the lagless XGBoost forecast (a difference of +0.023 in the share found, interval -0.001 to +0.046, under the +0.03 bar). It found 18 of the 35 real transfers lasting over 1 day against 12, but the convex model was trained with those real transfers masked, so that count flatters it. Built on the Kalman filter's one-step errors the detector found far less (0.488 of the moved power against 0.772), most likely because the filter follows a transfer's step within hours and so absorbs it, which we did not test. Most of the detector's remaining misses are small steps with no metered receiving substation, so a stronger signal for those transfers is the next idea to try. The maintainer has chosen to treat the gain as an improvement and to build later detector work on the frozen convex residual; that decision lies outside the pre-registered rule, so the verdict stays inconclusive. | also relevant |
| 00128 differentiable-physics-kfd-shared-day-anomaly-and-sparse-q-in-gam-k | Adding to experiment 111's level-and-anomaly filter (GAM-K) one day-level error state shared by all four hour bands, and estimating the level noise sparsely from zero, makes the filter's midday correction track an injected solar-panel-shaped change in real residuals better than GAM-K, and lowers the day-ahead score on ENS. Real residuals carry a day-level error shared across the four hour bands: the daily means of GAM-K's one-step innovations in different hour bands correlate at 0.18 to 0.23 (control P-premise). An earlier diagnosis of experiment 118's residuals gave 0.58 on a different quantity, the daily means of the residuals, which cannot separate a day-level error from a slow drift shared by all bands. | inconclusive | not-for-production | The day-ahead forecast was not scored, because the filter with one shared day-level error state failed its test on injected changes in real recorded errors, and the experiment stopped there. The errors at different times of day do move together (a correlation of about 0.2 between their daily innovations), but a single shared state probably also soaks up errors that belong to one time of day, so it cannot follow a slow, solar-panel-shaped change. Options for a next experiment are a day-level state for each time of day with a shared part, an explicit solar-panel term, and a test with an injected change large enough to detect; the verdict is inconclusive. | primary |
| 00126 differentiable-physics-s1-cheap-screens-added-to-gam-k | Each of six families of cheap additions to the day-ahead convex model GAM-K lowers its score after the Kalman filter, winter first: a free or monotone temperature curve (A-free, A-mono), exponentially smoothed temperatures (S-ewma) or smoothed daily extremes (S-mm), wind chill (C-wind) or apparent temperature (C-hw) inside the heating hinge, a school-term term (D-sch), lighting from solar elevation (E-sol), and a filter whose noise differs by regime (F-mix). A second family of four exploratory screens: a switching Kalman filter with three hidden regimes (G-skf), a learned periodic basis for the year cycle (W-basis), a soft cold/mild mixture of heating slopes (W-mix), and a cross-substation anomaly from weather-station residuals (W-stn). One hypothesis per arm, in "Arms" and "Four exploratory screens". | inconclusive | not-for-production | None of the thirteen cheap additions to the day-ahead GAM-K lowered the winter score by a margin we can trust. E-sol (lighting from solar elevation) came closest, at -0.036 points on dark winter evenings, but its interval (-0.122 to +0.024) is too wide to call, because six of the thirteen screens had a half-width above their 0.03 to 0.05-point margin. Only W-basis (learned year cycle) was clearly worse, in every month, site, and year. F-mix and W-stn looked worse too, but each rests on 2 to 3 calendar months. G-skf found injected load transfers 4 to 7 times as often at 25% and 40% transfers as the plain filter, but failed its own power gate and recovered no faster. | also relevant |
| 00125 differentiable-physics-n8-ens-members-through-gam-k-and-the-spread-around-them | Placing GAM-K's residual quantiles, fitted on a forecast driven by the analysis proxy (so they hold model and filter error and almost no weather error), around each of the 51 ENS members gives tails closer to nominal than today's table, which is fitted on ensemble-mean residuals, and the gain is largest at long leads and on cold winter nights, where the hinge turns a temperature error into a large demand error. The comparison is against GAM-K re-tuned on the same factor grid, so a plain scale error cannot pass for a better design. | inconclusive | not-for-production | Putting the forecast-error table around each of the 51 weather members, and fitting it on analysis-driven forecasts so that it holds no weather error, did not improve the tails of GAM-K's demand forecast. The weather members' spread is small against the model's own error, so the weather error was counted twice in the code but is too small to matter. The next step is the cold winter nights, where demand exceeds the 95% quantile about a quarter of the time in every arm. The verdict is inconclusive, because no arm reached 14 of 16 calibration cells. | primary |
| 00123 differentiable-physics-e7-storage-heating-night-step-estimation | At Primary substations, demand steps up in the half-hours from 00:30 to 02:30 on winter nights, the step grows with cold, and the pooled step over the six decision series is positive at a daily mean of 3 C. At D5, D10, and D16, which were chosen because winter 2025-26 showed the step, it is 5 to 10 points (1% of each series' 99th percentile of demand); at the other decision series it is smaller. | inconclusive | not-for-production | The night step at Primary substations is about 10 points on a 3 C day, where one point is 1% of a substation's near-peak demand, and it starts around 01:00 and looks fixed to UTC rather than the local clock (a check chosen after seeing the data). The rise on cold winter nights is the feature an operator would see: the day-ahead forecast runs 6 to 10 points below actual demand from about 01:00 at D5 and D16 on the two pre-registered winters, and at D10 in winter 2024-25 alone (7.6 points at 01:00); at D10 in winter 2023-24 the baseline is instead 9 to 17 points too high all night. The step appears at 13 of the 14 substations, and its full size rests on summer nights having no step, which the gate cannot test. The verdict is inconclusive because the gate is weaker than it looks: it holds one real test of the interval, and that test under-covers. | also relevant |
| 00122 differentiable-physics-three-winter-backtest-gam-k | On past weather, the convex model with the Kalman filter (GAM-K) forecasts Primary demand one day ahead better than both the plain convex model (GAM) and lagless XGBoost, in each of the three winters 2023-24, 2024-25, and 2025-26, by at least 1 point of mean absolute error. | supported | not-for-production | In all three winters, the convex demand model with the recent-error filter (GAM-K) forecast Primary demand one day ahead 1.8 to 3.1 points of capacity better than the same convex model without the filter, and 2.0 to 3.1 points better than lagless XGBoost. The gain is consistent with the filter removing a level error that a fit frozen for a year carries, and lagless XGBoost has no recent power to correct the same error. GAM-K's filter settings were fitted on the 2023-24 winter, and the convex model's ridge penalty was tuned on a year that contains the 2024-25 winter, so those two gains are partly in-sample. 2025-26 has the least tuning leakage, but it was read in an earlier analysis before this plan, and no winter is clean on both counts. A post hoc check at the audit subtracted each plain model's own winter-mean error at each substation, which no forecaster could know in advance: GAM-K stayed ahead of the plain GAM by 0.63, 0.71, and 0.35 points and ahead of lagless XGBoost by 0.95 to 1.05 points, so 5 of the 6 contrasts would miss the 1-point bar. Most of the gain is therefore level tracking, and this result does not show that GAM-K beats a model that sees recent power. The scoring uses past weather, so the gain under live weather forecasts is unknown. | also relevant |
| 00121 differentiable-physics-mh-multi-hinge-heating-curve-and-night-step | In the convex model of Primary demand in the form Part 6 made the DP's candidate (GAM-K), on past weather, replacing the single fixed 15.5 C heating hinge by five hinges (10, 12, 14, 16, and 18 C) with non-negative slopes, so the heating curve never falls as it gets colder and stays convex, lowers the day-ahead quantile score after the recent-error filter. Second question (NS): GAM-K leaves a 4 to 7 point step at 01:00 on winter nights at three substations (a pattern like storage heating switching on), and a night-step term lowers the score on winter nights without worsening the other rows by more than 0.02 points. Indexed in UTC against local time, the winning indexing is reported. | inconclusive | candidate | Adding night-time levels indexed on the UTC clock lowers the day-ahead error on winter nights by about 0.23 points after the recent-error filter, at all 5 substations in both winters, and the forecast then shows the 01:00 jump in demand that the current forecast misses on cold winter nights. Five heating bends instead of one gain only 0.016 points, within noise. The synthetic tests could not show that a step this small can be detected, so the night result needs a test on weather forecasts and an audit before it enters the model. | also relevant |
| 00119 differentiable-physics-enc-weather-patch-encoder-on-ens | A small neural network that reads the ENS grid boxes around a Primary, and not only the one box GAM-K reads, lowers GAM-K's quantile score a little on average and more on the cold, showery, and badly forecast days NGED cares about, either as a correction to GAM-K's quantiles or as a replacement for its single-box temperature and irradiance. | inconclusive | not-for-production | A small neural network that reads the ensemble weather around a Primary lowers GAM-K's quantile score by 0.062 points of effective capacity in one scored year and 0.035 in the other (both statistically significant, 5 of 5 substations better), and the gain is smaller in winter (0.041 and 0.027 points) but never a loss. About two-thirds of the gain survives when the network reads the wrong date's weather (0.043 and 0.022), so it is probably a recalibration of GAM-K's quantiles with the lead, the hour, and GAM-K's own forecast, and the surrounding boxes add nothing to the centre box (patch minus centre box: -0.002 and +0.003 points). The known-answer gate failed, so the verdict is inconclusive, and the next step is a plain recalibration of GAM-K's quantiles with the centre box's extra fields (dew point, pressure, precipitation, and member spread), not a patch encoder. | primary |
| 00118 differentiable-physics-kfe-kalman-filter-on-the-gam-effects-on-ens | Freezing the convex model's fitted effects and letting a weight on each follow a random walk, with a diagonal noise matrix fitted by likelihood on the tuning months, forecasts day-ahead demand on ENS better than experiment 111's filter of one level and one anomaly per hour band (GAM-K), because heating slopes, PV capacity, profile scale, and holiday effects drift faster than a 24-month fit follows. The fitted noise matrix beats one step variance shared by every weight, so the gain comes from how each effect drifts and not from the weights' freedom alone. | inconclusive | not-for-production | Letting the convex model's PV, heating, profile, and holiday weights drift did not improve the day-ahead forecast over experiment 111's level-and-anomaly filter (0.015 points worse at D+1, interval -0.013 to +0.048), and did no better in winter or on cold days. Switching the correction off on holidays is much worse (+1.24 points) and halving it is not better, so the filter still helps on holidays. The weights cannot be read as measurements, because the known-answer gate failed on real residuals, although two of its bars were mis-set. Keep experiment 111's filter, and the next step is a filter that fixes the PV weight and moves the others; the verdict is inconclusive. | primary |
| 00116 differentiable-physics-nra-edge-flows-between-primaries | Flows on the electricity network's interconnection edges, fitted to a frozen convex model's residuals with a sparse-step penalty, recover injected conserved transfers between MW-metered Primaries; removing them gives a normal running arrangement (NRA) model trained without hand labels that forecasts no worse outside events; and fed causally into the Kalman filter as a known input, they improve the metered forecast during and after a transfer. | inconclusive | not-for-production | The method that looks for load moved between neighbouring substations did not work: with the settings we fixed in advance it found under a quarter of the load we added artificially, and it changed no forecast. Looser settings found more, but they also found load moving between substations that are not connected, probably the slow drifts of demand that our model does not describe. The gate failed for a structural reason: at the settings we fixed in advance, the fit cannot find any transfer shorter than 7 to 9 days, and the longest test transfer lasts under 5 days, so the failure is a limit of the method as specified, not a bug in the solver. The zero flow we first saw on the unconnected pairs was also an artefact of the order the fit visits its paths, so those pairs were never clean. Adding a drift term to the model, or accepting a flow only where the two substations are connected, is the next thing to test. | also relevant |
| 00115 differentiable-physics-w1-diurnal-shape-state-on-ens | GAM-K's remaining day-ahead error on ENS is mostly in the shape of the day inside each hour band, and that shape drifts faster than the convex model's monthly knots follow. A 48-half-hour shape state with zero mean in each hour band, added to the Kalman filter and updated from every reading the telemetry delay allows, lowers GAM-K's day-ahead quantile score at most decision substations, in months that played no part in choosing the idea's form and in the scored year. | inconclusive | not-for-production | Teaching the physics-based forecast the shape of the day from recent readings lowers its day-ahead error by 0.075 points at five Primary substations (about 3%), in two separate periods and more in winter (0.11, a post hoc breakdown), but averaging the last 14 days' errors by half-hour gains as much, so the learned shape state is not needed on this evidence. The shape state's own check failed: it learns an injected change too slowly (0.35 against a floor of 0.4), so the verdict is capped at inconclusive. A faster shape state, tried afterwards and labelled post hoc, passed that check and beat both K0 and the recipe, which a rerun with the check fixed in advance would need to confirm. | primary |
| 00114 differentiable-physics-n3-n4-interactions-and-heating-threshold | Two labelled claims about the convex model of Primary demand in the form Part 6 made the DP's candidate (GAM-K: 24 months, no drift, pinned capacity, Kalman filter), on past weather. N3: interaction terms (heating by day-type group, heating on eight sub-band blocks, lighting by day-type group), each shrunk towards the base by a difference penalty, lower the day-ahead quantile score after the filter. N4: a per-site heating threshold, chosen by profiling the training loss, is recoverable on synthetic substations at real noise, is identified away from 15.5 C at one or more real substations under the per-site printing rule, and lowers the score after the filter. | partly supported | not-for-production | Splitting the convex model's heating terms between the first and second half of each part of the day lowered its day-ahead error after the recent-error filter by about 0.04 points (97.5% interval 0.014 to 0.070), at all 5 substations in both years, but the gain label rests on one substation (D5), and the gain is a temperature-dependent shape of the day more than a heating effect. Letting heating or lighting differ between weekdays and weekends adds nothing, and choosing a heating threshold separately for each substation did not clearly help (0.01 points, within noise). Every number uses past temperature, not forecasts, so the gain needs a test on weather forecasts before GAM-K's convex model changes; the PV panel-orientation mix (N13) was dropped because earlier experiments found it unidentified. |
also relevant |
| 00111 differentiable-physics-6-ens-forecast-engine-against-bg2 | Driven by ECMWF ENS forecasts, the convex DP demand model forecasts Primary demand up to 7 days ahead better than lagless XGBoost given the same recent-level correction, and, used as the base margin of the BG2 recipe's lagged XGBoost, makes the production recipe better like for like. | partly supported | candidate | Driven by real ensemble weather forecasts, the physics-based demand model with a filter on its recent errors (GAM-K) scored 2.80 points against 2.78 for the production recipe (BG2) at five Primary substations. The 0.02 difference cannot be told from zero (-0.109 to +0.157), and the comparison is not like for like. GAM-K scored 0.13 points better than lagless XGBoost with the same filter (-0.305 to +0.014, 4 of 5 substations better), so the verdict is partly supported. The filter's gain is the firmly supported part: 0.15 points, 0.39 in the first day, with the sign held in the replicate months. Giving lagged XGBoost the physics model's forecast as a starting point changed nothing measurable. The known-answer gate passed 22 of 22 readings on its second run, after a sign error in the gate's own test data (both runs are kept). The departures were that the XGBoost compute estimate was 20 times too low, so the margin arms refitted monthly; lagless XGBoost for the scored year reused SIMP1's refits and had no spread factor; and the confirmation comparison was not run. GAM-K is a candidate for Flexpectation v2's roughly 1,300 substations on the measured filter gain, with four caveats: five substations over one year, a short XGBoost history, the undecided XGBoost comparison, and a margin that gave no gain. | primary |