Differentiable physics (demand, calendar, and weather): experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00281 economy7-night-step-smooth-in-temperature | A forecast of primary substation demand whose night level is a set of free levels for each UTC night half-hour (experiment 121's NS-U), and then a step whose size is a monotone function of the cold, scores better in the Economy 7 step hours and on cold winter nights than the same forecast with only the winter night offset (experiment 155), when the weather input is the ensemble-mean forecast and not past weather. Economy 7 is the tariff that switches storage heaters on shortly after 00:30 UTC. Experiment 121 found a gain of 0.233 points on winter nights on past weather and asked for a test on weather forecasts. | inconclusive | not-for-production | Fitting the Economy 7 night levels inside the demand model, and not adding a winter night offset afterwards, lowered the day-ahead error by about 0.02 to 0.03 points of rated demand over all rows in both years and by about 0.04 points on winter nights, but the gain in the two hours after the step starts could not be shown: it changes sign between the two years and depends on two or three substations. Letting the step grow with the cold added almost nothing over free night levels. The Economy 7 term does not explain the night-time gap at D10 in April to June. | primary |
| 00279 dp-heating-curve-monotone-lagged-temperature-and-wind | The heating response of the convex demand model T-all-K (a hinge at 15.5 C on four trailing-mean temperatures, split by day type) is too rigid. A smooth, monotone, non-negative curve on the same temperatures (A1), the same curve also on exponentially weighted and previous-day mean temperatures that carry the building's thermal lag (A2), or a small monotone neural network on three daily mean temperatures and the wind speed (A3), lowers the day-ahead quantile score on cold nights and winter days at the five decision Primaries, and the fitted curve is not a straight line. | not supported | not-for-production | A smooth heating curve, a thermal lag, and wind each leave the day-ahead forecast within about 0.02 points of the straight-line heating response, so the evidence does not justify adding any of them. A small neural network learns a plausible curved heating response and lowers the error on cold nights by 0.13 points, but that gain comes from one cold spell in January 2026, did not appear in the previous year, and the network failed a test on synthetic data, so the substation card keeps the straight line for now. | primary |
| 00198 d8-summer-spikes-cooling-load-and-residual-test | The summer spikes a colleague saw in D8's power (time series 26, Marsh Lane) are air-conditioning load: they rise smoothly with temperature, peak in the afternoon of the hottest days, and repeat every year. The best-guess forecast has a heating hinge but no cooling term, so its summer residual (metered minus forecast) rises with temperature, and a cooling hinge on the ensemble mean temperature fitted on past residuals lowers its summer score. | inconclusive | not-for-production | D8's afternoon power does not rise with temperature, and its large summer departures from expected power are mostly downward shortfalls of 1.5 MW or more, which grow from 18% of summer half-hours in 2022 to 52% in 2025, so the spikes are not air-conditioning load. A cooling term added to the best-guess forecast changed the summer score at the five decision Primaries by 0.0002 points (interval -0.010 to +0.015), but the 4 fit months held too few hot days and the known-answer control failed, so the experiment cannot rule a cooling term out. On afternoons of 25 degrees Celsius or more, metered power exceeds the forecast by 0.6 to 5.1 points on average at each of six Primaries, so a term fitted on the full multi-year history of past weather, pooled across Primaries, is the next test. | primary |
| 00173 dp-christmas-more-years-and-sites | The convex demand model with a Kalman filter on recent errors and recipe R (the mean error of the last 14 days by half-hour, held to 0.75 of its size), which scored 1.13 times an ordinary weekday over 24 December to 2 January at five substations in two Christmases (experiment 170), keeps a Christmas-to-weekday error ratio under 1.3 in every Christmas back to 2019-20 and at 14 or more of 16 Primary substations, and does not over- or under-forecast 25 and 26 December by more than 5% of that day's demand. | inconclusive | not-for-production | Where the demand model had a winter of training data, the error over 24 December to 2 January was 0.97 to 1.22 times an ordinary weekday in all four Christmases from 2022-23 to 2025-26, so no special-day term is justified. The pre-registered run failed because the training rows of Christmas 2020-21 and 2021-22 (and of one substation in 2023-24) held little or no winter, since the lockdown window is left out of training. A forecast for a substation with under 180 days of usable history should be flagged, not trusted. | also relevant |
| 00145 differentiable-physics-analog-day-error-correction-on-gam-k | Averaging GAM-K's past day-ahead errors over the days most similar to the target day (same day type, with holiday identity for holidays, similar forecast daily mean temperature, and a season window of a few weeks either side of the date) gives a better correction than averaging the last 14 days alike (recipe R of experiment 115) or the same-day-type days of the last few weeks (experiment 134). The reasoning is that demand errors repeat on days that resemble each other. The errors that repeat most, on holidays and on cold days, are the ones that a 14-day average dilutes. | small gain | candidate | Averaging the errors of the 20 most similar past days (same day type, within 13 weeks of the same date, last year's days included) beat recipe R (the 0.75-weighted mean error of the last 14 days) by 0.041 points at day D+1, the calendar day after the forecast origin (about 1.7% of R's score of 2.436). That gain is larger than the 0.011 points that the weekday term of experiment 134 gained. Matching on forecast temperature helped by only 0.005 points, and copying the error of the same holiday last year made holidays worse, so the next step is a day-type profile with a season window and no correction copied from an earlier holiday. | also relevant |
| 00143 differentiable-physics-holiday-aware-updating-of-the-gam-k-filter | The Kalman filter of GAM-K lets a holiday's reading error teach it a wrong level, which carries into the days after the holiday; leaving holiday readings out of the filter's update (and nothing else) lowers the day-ahead score on those following days, and does not hurt the holiday targets or the rest of the year. | not supported | not-for-production | Leaving holiday readings out of the day-ahead filter's update does not reliably help the days after a holiday: the effect had the opposite sign after Christmas 2024 and Christmas 2025, and its 99% interval in the scored year (-0.309 to +0.036 points) includes zero. In 10 of 12 holidays, the holiday readings' error had the same sign as the error on the following days, so these readings are not usually contamination. Setting the correction to zero on holiday days, as experiment 118 did, is the action that hurts (+0.62 points on holiday days). Two Christmases give no basis for changing the filter this Christmas. | also relevant |
| 00137 differentiable-physics-e7-economy-7-night-step-term-in-the-forecast | An additive night term, fitted on the base forecast's own past errors as a function of the ENS-mean daily temperature and the UTC clock time, lowers the day-ahead quantile score on the half-hours from 01:00 to 03:00 UTC on cold winter nights, without raising the score elsewhere. The mechanism is that the base GAM-K forecast (experiment 133's shared-noise arm) runs below actual demand from about 01:00 on cold nights (experiment 123), so a correction learned from earlier winters removes a bias the forecast repeats. | inconclusive | not-for-production | The day-ahead forecast runs about 2.5 points too low on cold winter nights from 01:00 UTC, and a correction learned from its own past errors removes that bias and lifts the 0.95 quantile's coverage there from 80% to 93%. An audit showed that a plain winter night offset does as well, so the experiment found a winter night bias, not a step that grows with cold. The verdict is inconclusive: one winter, 365 headline rows, and a failed placebo. | primary |
| 00126 differentiable-physics-s1-cheap-screens-added-to-gam-k | Each of six families of cheap additions to the day-ahead convex model GAM-K lowers its score after the Kalman filter, winter first: a free or monotone temperature curve (A-free, A-mono), exponentially smoothed temperatures (S-ewma) or smoothed daily extremes (S-mm), wind chill (C-wind) or apparent temperature (C-hw) inside the heating hinge, a school-term term (D-sch), lighting from solar elevation (E-sol), and a filter whose noise differs by regime (F-mix). A second family of four exploratory screens: a switching Kalman filter with three hidden regimes (G-skf), a learned periodic basis for the year cycle (W-basis), a soft cold/mild mixture of heating slopes (W-mix), and a cross-substation anomaly from weather-station residuals (W-stn). One hypothesis per arm, in "Arms" and "Four exploratory screens". | inconclusive | not-for-production | None of the thirteen cheap additions to the day-ahead GAM-K lowered the winter score by a margin we can trust. E-sol (lighting from solar elevation) came closest, at -0.036 points on dark winter evenings, but its interval (-0.122 to +0.024) is too wide to call, because six of the thirteen screens had a half-width above their 0.03 to 0.05-point margin. Only W-basis (learned year cycle) was clearly worse, in every month, site, and year. F-mix and W-stn looked worse too, but each rests on 2 to 3 calendar months. G-skf found injected load transfers 4 to 7 times as often at 25% and 40% transfers as the plain filter, but failed its own power gate and recovered no faster. | primary |
| 00123 differentiable-physics-e7-storage-heating-night-step-estimation | At Primary substations, demand steps up in the half-hours from 00:30 to 02:30 on winter nights, the step grows with cold, and the pooled step over the six decision series is positive at a daily mean of 3 C. At D5, D10, and D16, which were chosen because winter 2025-26 showed the step, it is 5 to 10 points (1% of each series' 99th percentile of demand); at the other decision series it is smaller. | inconclusive | not-for-production | The night step at Primary substations is about 10 points on a 3 C day, where one point is 1% of a substation's near-peak demand, and it starts around 01:00 and looks fixed to UTC rather than the local clock (a check chosen after seeing the data). The rise on cold winter nights is the feature an operator would see: the day-ahead forecast runs 6 to 10 points below actual demand from about 01:00 at D5 and D16 on the two pre-registered winters, and at D10 in winter 2024-25 alone (7.6 points at 01:00); at D10 in winter 2023-24 the baseline is instead 9 to 17 points too high all night. The step appears at 13 of the 14 substations, and its full size rests on summer nights having no step, which the gate cannot test. The verdict is inconclusive because the gate is weaker than it looks: it holds one real test of the interval, and that test under-covers. | primary |
| 00122 differentiable-physics-three-winter-backtest-gam-k | On past weather, the convex model with the Kalman filter (GAM-K) forecasts Primary demand one day ahead better than both the plain convex model (GAM) and lagless XGBoost, in each of the three winters 2023-24, 2024-25, and 2025-26, by at least 1 point of mean absolute error. | supported | not-for-production | In all three winters, the convex demand model with the recent-error filter (GAM-K) forecast Primary demand one day ahead 1.8 to 3.1 points of capacity better than the same convex model without the filter, and 2.0 to 3.1 points better than lagless XGBoost. The gain is consistent with the filter removing a level error that a fit frozen for a year carries, and lagless XGBoost has no recent power to correct the same error. GAM-K's filter settings were fitted on the 2023-24 winter, and the convex model's ridge penalty was tuned on a year that contains the 2024-25 winter, so those two gains are partly in-sample. 2025-26 has the least tuning leakage, but it was read in an earlier analysis before this plan, and no winter is clean on both counts. A post hoc check at the audit subtracted each plain model's own winter-mean error at each substation, which no forecaster could know in advance: GAM-K stayed ahead of the plain GAM by 0.63, 0.71, and 0.35 points and ahead of lagless XGBoost by 0.95 to 1.05 points, so 5 of the 6 contrasts would miss the 1-point bar. Most of the gain is therefore level tracking, and this result does not show that GAM-K beats a model that sees recent power. The scoring uses past weather, so the gain under live weather forecasts is unknown. | primary |
| 00121 differentiable-physics-mh-multi-hinge-heating-curve-and-night-step | In the convex model of Primary demand in the form Part 6 made the DP's candidate (GAM-K), on past weather, replacing the single fixed 15.5 C heating hinge by five hinges (10, 12, 14, 16, and 18 C) with non-negative slopes, so the heating curve never falls as it gets colder and stays convex, lowers the day-ahead quantile score after the recent-error filter. Second question (NS): GAM-K leaves a 4 to 7 point step at 01:00 on winter nights at three substations (a pattern like storage heating switching on), and a night-step term lowers the score on winter nights without worsening the other rows by more than 0.02 points. Indexed in UTC against local time, the winning indexing is reported. | inconclusive | candidate | Adding night-time levels indexed on the UTC clock lowers the day-ahead error on winter nights by about 0.23 points after the recent-error filter, at all 5 substations in both winters, and the forecast then shows the 01:00 jump in demand that the current forecast misses on cold winter nights. Five heating bends instead of one gain only 0.016 points, within noise. The synthetic tests could not show that a step this small can be detected, so the night result needs a test on weather forecasts and an audit before it enters the model. | primary |
| 00120 differentiable-physics-sh-convex-daily-shape-library | A convex model in which each Primary's daily profile is a non-negative mix of K = 3 or 4 shared daily shapes per day type, plus a penalised deviation, forecasts Primary demand with 2 years of history at least 0.05 points better than experiment 109's shrinkage towards one fleet-mean profile. That needs the library to beat the per-site profile by about 0.02 points, because the fleet-mean model is 0.031 worse than it; the plan predicts a tie or a small gain, so supported is a long shot. The stability of the shapes and the shape mix of each substation are descriptive. |
inconclusive | not-for-production | In a test with synthetic substations, 5 Primaries were too few for a library of 3 shared daily shapes to recover the shapes we planted, so the check failed on 7 of 10 lines and the verdict is inconclusive. In the scored year the library scored the same as the fleet-mean model and 0.030 points worse than a profile per substation (0.068 worse in winter), and its forecast reproduced a tenth to a half of the Economy 7 night step on cold winter nights. | also relevant |
| 00119 differentiable-physics-enc-weather-patch-encoder-on-ens | A small neural network that reads the ENS grid boxes around a Primary, and not only the one box GAM-K reads, lowers GAM-K's quantile score a little on average and more on the cold, showery, and badly forecast days NGED cares about, either as a correction to GAM-K's quantiles or as a replacement for its single-box temperature and irradiance. | inconclusive | not-for-production | A small neural network that reads the ensemble weather around a Primary lowers GAM-K's quantile score by 0.062 points of effective capacity in one scored year and 0.035 in the other (both statistically significant, 5 of 5 substations better), and the gain is smaller in winter (0.041 and 0.027 points) but never a loss. About two-thirds of the gain survives when the network reads the wrong date's weather (0.043 and 0.022), so it is probably a recalibration of GAM-K's quantiles with the lead, the hour, and GAM-K's own forecast, and the surrounding boxes add nothing to the centre box (patch minus centre box: -0.002 and +0.003 points). The known-answer gate failed, so the verdict is inconclusive, and the next step is a plain recalibration of GAM-K's quantiles with the centre box's extra fields (dew point, pressure, precipitation, and member spread), not a patch encoder. | also relevant |
| 00117 differentiable-physics-h2-graph-prior-on-pooled-convex-parameters | A penalty on the differences between interconnected Primaries' parameters (heating slopes, PV capacity as a share of effective capacity, the profile's scale, and the holiday coefficients) narrows those parameters' month-block bootstrap intervals without breaking their coverage when neighbouring Primaries are alike, and worsens the full-history quantile score by at most 0.03 points. | refuted | not-for-production | At the weight tested, tying the parameters of interconnected substations together did not improve the forecast: in the scored year it worsened the quantile score by 0.020 points on all days (the interval spans zero) and by 0.173 points in winter evening peaks (the replicate year gave -0.021). On synthetic substations with a known truth it narrowed only heating-slope intervals, and the 80% intervals of PV capacity held the truth 17% of the time against 78% without the tie. A tie on heating slopes alone worsened the real score by 0.006 and 0.001 points, an exploratory lead and not a result. | also relevant |
| 00114 differentiable-physics-n3-n4-interactions-and-heating-threshold | Two labelled claims about the convex model of Primary demand in the form Part 6 made the DP's candidate (GAM-K: 24 months, no drift, pinned capacity, Kalman filter), on past weather. N3: interaction terms (heating by day-type group, heating on eight sub-band blocks, lighting by day-type group), each shrunk towards the base by a difference penalty, lower the day-ahead quantile score after the filter. N4: a per-site heating threshold, chosen by profiling the training loss, is recoverable on synthetic substations at real noise, is identified away from 15.5 C at one or more real substations under the per-site printing rule, and lowers the score after the filter. | partly supported | not-for-production | Splitting the convex model's heating terms between the first and second half of each part of the day lowered its day-ahead error after the recent-error filter by about 0.04 points (97.5% interval 0.014 to 0.070), at all 5 substations in both years, but the gain label rests on one substation (D5), and the gain is a temperature-dependent shape of the day more than a heating effect. Letting heating or lighting differ between weekdays and weekends adds nothing, and choosing a heating threshold separately for each substation did not clearly help (0.01 points, within noise). Every number uses past temperature, not forecasts, so the gain needs a test on weather forecasts before GAM-K's convex model changes; the PV panel-orientation mix (N13) was dropped because earlier experiments found it unidentified. |
primary |
| 00113 differentiable-physics-n5-convex-identification-at-realistic-noise | On synthetic substations built from the real calendar and weather, with noise as large and as autocorrelated as the convex model's real residuals, the convex model's 80% month-block bootstrap intervals cover the truth 70 to 90% of the time, separately for each hour band's heating slope and for the latest season-year's PV capacity. Labelled separately, not in the verdict: the day-of-year harmonics and the drift term make those intervals wide, by absorbing the seasonal cycles of temperature and clear-sky PV. | refuted | not-for-production | At real noise levels, the convex model's 80% month-block bootstrap intervals contain the true heating slope and the true rooftop PV capacity 84% of the time each, but a heating slope is weak (the interval is about 38% of the slope wide). The pre-registered verdict is refuted, because with independent noise the capacity interval covers the truth only 52% of the time, and the cause is mainly a bias of about 0.24 points from the drift penalty at its tuned step of 3,000 (at Part 4's step of 30 the bias is 0.02 points and coverage 78%, a post hoc check). The truth here has the model's own form, so the test cannot show the bias that a wrong PV shape causes. | also relevant |
| 00112 differentiable-physics-vp-variable-projection-fit-of-pooled-model | Part 3's pooled DP lost to the per-site DP because gradient descent did not reach the optimum, not because pooling hurts: solving every linear coefficient exactly at each step, with gradient steps only on the non-linear parameters, makes the pooled DP at least as good as the per-site DP on substations with 2 years of history. | inconclusive | not-for-production | Solving the pooled model's linear coefficients exactly did not make its forecasts better. With the interaction neural network on, the exact fit drove the network's last layer and the seasonal and holiday terms to very large, weakly penalised values: it forecast up to 280 points too high on four days of spring 2026 (the weekend of the clock change, Good Friday, and Easter Monday), and it was 0.7 points worse on the other days. With the network off, the exact fit is stable and within 0.03 points of the gradient fit, so the optimiser was not what held the pooled model back; the known-answer test failed, and with the network on the model cannot recover heating even from noise-free data. | also relevant |
| 00110 differentiable-physics-5-embedded-generation-and-capacity | The convex differentiable-physics model can represent the generation side of a Primary substation without making its demand forecast worse: an MVA meter through an observation layer, a flat unmetered generator found where one exists and not invented where none does, and an export-limited PV clip; and its PV capacity stays the same when the routes by which demand terms can imitate solar are closed. | inconclusive | not-for-production | The demand forecast could not yet account for generators connected behind a substation. Rooftop solar capacity fitted from the data was not precise or consistent enough to trust, so the capacity from outside estimates stays in use. A generator-finding component reported a generator at every substation tested, so it stays switched off. A limit on exported solar helped only at the one substation known to export-limit. Treating the meter as apparent power improved the forecast at the 8 substations that report only apparent power by 0.13 points of capacity, but the test of the meter's flow direction failed, so that gain is provisional. | also relevant |
| 00109 differentiable-physics-n0-partially-pooled-convex-model | A convex model whose per-substation coefficients are a fleet mean plus a ridge-penalised per-substation deviation forecasts Primary demand as well as the per-substation convex model when both have 2 years of history, and clearly better when the substation has 4 weeks or 3 months. | inconclusive | not-for-production | The known-answer test of this method failed under its pre-registered definition, and only one of five runs (each on a new seed, runs 2 to 5 being post hoc redefinitions) passes every line, so the verdict is inconclusive. Taking the numbers alone, for the five MW Primaries tested, one convex model shared across substations forecast a new substation with only 4 weeks of history 3.3 points better than a convex model fitted to that substation alone (and 0.8 points better with 3 months), far more than the 0.5 to 2.0 predicted, and scored 0.03 points worse on substations with 2 years of history. On the six held-out Primaries it came within 0.2 points of experiment 105's neural pooled model with 4 weeks (the neural model 0.13 better) and 0.12 better with 3 months, without that model's loss on long histories, which suggests that the loss in experiment 105 came largely from the neural fit. Complete pooling scored 0.07 points worse than partial pooling at 4 weeks in the scored year (interval touching zero) and no differently in the replicate year. The gains should be relied on only after a rerun of the gate. | also relevant |
| 00107 differentiable-physics-rolling-origin-history-and-tails | Refitted quarterly from 2023-07 to 2026-04 and scored on the next 3 months, the convex model's two history contrasts (all years against 12 months, drift against none) keep one sign at almost every origin. A second claim, labelled separately as C5 and not part of the verdict: a level-dependent spread brings its 95% and 99% quantiles within 1 point of nominal coverage without worsening the quantile score. | partly supported | not-for-production | Giving the convex demand model a drift term lowered its score at 10 of 12 quarterly refits, by 0.45 points when trained on all years (interval 0.23 to 0.68). In a post hoc split, about three-fifths of that size is one substation (D10) at the 2023-24 refits, and the 2025-26 refits give 0.30, as in experiment 104. All years without drift scored worse than 12 months at 8 of 12 refits, and all years with drift against 12 months has no stable answer: winter-quarter refits favour more years, summer ones are level. The spread that depends on the forecast level made the 95% and 99% quantiles worse, so the tail coverage still needs a different method. The verdict is partly supported, because only the drift contrast is stable; a second spread (the conformal spread) is not interpreted, because it was built with a flaw. | also relevant |
| 00105 differentiable-physics-3-pooled-and-per-site-demand-model | One differentiable-physics model fitted across all the Primary substations forecasts a substation as well as the same model fitted to that substation alone when both have two years of history, and forecasts clearly better when the substation is new (4 weeks of history) or short of history (3 months). | inconclusive | not-for-production | One neural-network demand model fitted across 15 Primary substations forecasts a substation with only 4 weeks of history far better than a per-site copy of the same model (score 4.04 against 14.38 on the six scored substations), but forecasts a substation with 2 full years of history worse than a per-site copy (3.42 against 3.04 on the five decision substations, worse at all 5). By the pre-registered rule pooling is not yet a success, because that full-history result (C1) is refuted, and a failed known-answer check (gate test 2, injected-farm recovery) caps the verdict at inconclusive. Post hoc, for a new substation the pooled model also beat a per-site convex model fitted to the same 4 weeks (mean absolute error 6.88 against 12.79). | also relevant |
| 00104 differentiable-physics-2-simple-structural-model | A convex demand-minus-solar model, trained on every usable year with a slowly drifting level per hour band, forecasts Primary demand better than the same model trained on the last 12 months. | inconclusive | not-for-production | A simple model of demand minus solar, trained on 12 months, scored 0.25 points better than lagless XGBoost with its default start (post hoc; a lower score is better). Most of that lead is in the forecast spread, because the error of the mean forecast differs by only 0.08 points, and four of the five pre-registered positive controls failed, so the verdict is inconclusive. More years of history did not clearly help: the scored year and the tuning year gave opposite signs for two history-window contrasts, and a first version of this page had the wrong sign for both because its spread carried the previous year's mean bias. | primary |
| 00063 sw2-10-linear-joint-fit-v2-in-miniature | On the 8 Primary substations metered in MW, one linear fit per Primary that holds a weather baseline without radiation inputs, the embedded-solar capacity of experiment 00045 times the CAMS unit PV shape, and one share-of-the-donor's-net-power term per labelled transfer, (C1) lowers the error of the median forecast outside switching on held-out months, in hindsight, below that of the same baseline with radiation inputs by at least 0.35 points of capacity, (S1) recovers an injected solar capacity to within 0.15 of its size, and (S2) recovers an injected transfer share to within 0.05. | inconclusive | not-for-production | C1 is -0.322 points of capacity (-0.437 to -0.207), close to the 0.35-point threshold, but the S1 and S2 gates failed for reasons of test design, so the verdict is inconclusive. | primary |