Differentiable physics: long history and drift: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00170 dp-bg-stage3-drift-and-holiday-scoring | The convex demand model with a Kalman filter on recent errors, ensemble-mean weather, the heating split by day type, one shared filter noise setting, and the pinned solar capacity (experiment 136's arm A2) improves further when it gets a drift term (experiment 167's D1) on top of the two post-forecast corrections of experiment 155's arm D (a winter night offset and recipe R's 14-day mean error profile). The mechanism is that the drift term removes a slow demand-level error that the monthly refit on 24 months of history leaves in, and neither the offset (winter nights only) nor recipe R (a 14-day profile held to 0.75 of its size) removes that error. Training on every year of history adds a further gain, because the drift term and the pinned capacity let the old years teach the heating and calendar terms. | refuted | not-for-production | A slowly varying demand level (a drift term) improved the forecast by 0.40 points in experiment 167 but worsened it by 0.03 points once the forecast has the Kalman filter on recent errors (lower scores are better), and training on all years of history did not change that. The filter already appears to remove the slow level error that the drift term fixes, because without the filter the drift term improves the score by about 0.2 points. The drift term should not be added to experiment 155's arm D. Christmas (24 December to 2 January) scored 1.13 times an ordinary weekday at the same half-hours, below the 1.5 times predicted, and recipe R improved the Christmas score by 0.04 points. | also relevant |
| 00167 dp-full-history-gam-24-months-or-all-years-with-drift | With the PV capacity pinned to the as-of table (explicit PV) and the heating split by day type (T-all), the convex demand model (the GAM) trained on every year of history back to late 2019 with a drift term scores better than the same model trained on the last 24 months, when both are refitted at 12 quarterly origins on past weather (ERA5 and CAMS). The mechanism is that explicit PV removes the changing solar capacity from the target, so the old years teach the heating and calendar terms without the capacity growth that made the all-years fits of experiments 104 and 107 unstable, and the drift term absorbs the remaining slow change in the demand level. | supported | candidate | Adding a drift term (a slow adjustment to the demand level) improved the forecast at 11 of 12 quarterly refits. The quantile score, where lower is better, fell by 0.40 points when training on the last 24 months and by 0.55 points when training on all years back to late 2019. Training on all years instead of 24 months adds only 0.10 points on top of a drift term, and gives no gain without one, so extra years of history help little. | primary |
| 00149 differentiable-physics-full-history-gam-era5-cams | When the convex model's solar term is explicit (a pinned capacity times a unit solar shape in both arms), training on every year of a substation's history back to 2019 gives a lower quantile score than training on the last 24 months, once a drift term lets the model follow the slow level change. The mechanism is that more years pin down the temperature, calendar, and weekday terms better, and the drift term absorbs the level change that made all-years training unstable in experiments 104, 106, and 107. | inconclusive | not-for-production | Training the demand model on every year back to 2019, instead of the last 24 months, lowered its forecast error by only about 3% (0.08 points of substation capacity), and the difference could be chance. The drift term helped much more, lowering the error by 0.38 points on 24 months and 0.50 points on all history, but a post hoc audit found that a plain correction by the mean error of the last 28 training days recovers 86% to 94% of that gain, so the gain is a recent-level correction and the drift form is not shown to beat the simpler fix. We did not test whether explicit heating, switching, or pooled terms would make the longer history help. | primary |
| 00135 differentiable-physics-n10-joint-kalman-and-demand-fit | Fitting the convex demand model (GAM-K) so that its coefficients minimise the day-ahead error after the Kalman correction, instead of the plain error before it, lowers the day-ahead quantile score at 12 to 36 hours below the two-stage fit (fit the demand model, then filter its residuals), without moving the fitted shapes to explain recent-level structure that the filter already tracks. | not supported | not-for-production | We fitted the demand model so that its coefficients minimise the error left after a Kalman filter (a running correction of recent errors) has adjusted the forecast. That made the day-ahead forecast much worse, because the filter can absorb a shift in the slow parts of the demand model, so the fit barely constrains those parts. Tying the fit to the two-stage coefficients removed the damage and also removed any gain: the quantile score was within 0.013 points of the two-stage score, in either direction. The two-stage design (fit the demand model, then filter its residuals) stays. | primary |
| 00128 differentiable-physics-kfd-shared-day-anomaly-and-sparse-q-in-gam-k | Adding to experiment 111's level-and-anomaly filter (GAM-K) one day-level error state shared by all four hour bands, and estimating the level noise sparsely from zero, makes the filter's midday correction track an injected solar-panel-shaped change in real residuals better than GAM-K, and lowers the day-ahead score on ENS. Real residuals carry a day-level error shared across the four hour bands: the daily means of GAM-K's one-step innovations in different hour bands correlate at 0.18 to 0.23 (control P-premise). An earlier diagnosis of experiment 118's residuals gave 0.58 on a different quantity, the daily means of the residuals, which cannot separate a day-level error from a slow drift shared by all bands. | inconclusive | not-for-production | The day-ahead forecast was not scored, because the filter with one shared day-level error state failed its test on injected changes in real recorded errors, and the experiment stopped there. The errors at different times of day do move together (a correlation of about 0.2 between their daily innovations), but a single shared state probably also soaks up errors that belong to one time of day, so it cannot follow a slow, solar-panel-shaped change. Options for a next experiment are a day-level state for each time of day with a shared part, an explicit solar-panel term, and a test with an injected change large enough to detect; the verdict is inconclusive. | also relevant |
| 00122 differentiable-physics-three-winter-backtest-gam-k | On past weather, the convex model with the Kalman filter (GAM-K) forecasts Primary demand one day ahead better than both the plain convex model (GAM) and lagless XGBoost, in each of the three winters 2023-24, 2024-25, and 2025-26, by at least 1 point of mean absolute error. | supported | not-for-production | In all three winters, the convex demand model with the recent-error filter (GAM-K) forecast Primary demand one day ahead 1.8 to 3.1 points of capacity better than the same convex model without the filter, and 2.0 to 3.1 points better than lagless XGBoost. The gain is consistent with the filter removing a level error that a fit frozen for a year carries, and lagless XGBoost has no recent power to correct the same error. GAM-K's filter settings were fitted on the 2023-24 winter, and the convex model's ridge penalty was tuned on a year that contains the 2024-25 winter, so those two gains are partly in-sample. 2025-26 has the least tuning leakage, but it was read in an earlier analysis before this plan, and no winter is clean on both counts. A post hoc check at the audit subtracted each plain model's own winter-mean error at each substation, which no forecaster could know in advance: GAM-K stayed ahead of the plain GAM by 0.63, 0.71, and 0.35 points and ahead of lagless XGBoost by 0.95 to 1.05 points, so 5 of the 6 contrasts would miss the 1-point bar. Most of the gain is therefore level tracking, and this result does not show that GAM-K beats a model that sees recent power. The scoring uses past weather, so the gain under live weather forecasts is unknown. | also relevant |
| 00118 differentiable-physics-kfe-kalman-filter-on-the-gam-effects-on-ens | Freezing the convex model's fitted effects and letting a weight on each follow a random walk, with a diagonal noise matrix fitted by likelihood on the tuning months, forecasts day-ahead demand on ENS better than experiment 111's filter of one level and one anomaly per hour band (GAM-K), because heating slopes, PV capacity, profile scale, and holiday effects drift faster than a 24-month fit follows. The fitted noise matrix beats one step variance shared by every weight, so the gain comes from how each effect drifts and not from the weights' freedom alone. | inconclusive | not-for-production | Letting the convex model's PV, heating, profile, and holiday weights drift did not improve the day-ahead forecast over experiment 111's level-and-anomaly filter (0.015 points worse at D+1, interval -0.013 to +0.048), and did no better in winter or on cold days. Switching the correction off on holidays is much worse (+1.24 points) and halving it is not better, so the filter still helps on holidays. The weights cannot be read as measurements, because the known-answer gate failed on real residuals, although two of its bars were mis-set. Keep experiment 111's filter, and the next step is a filter that fixes the PV weight and moves the others; the verdict is inconclusive. | also relevant |
| 00115 differentiable-physics-w1-diurnal-shape-state-on-ens | GAM-K's remaining day-ahead error on ENS is mostly in the shape of the day inside each hour band, and that shape drifts faster than the convex model's monthly knots follow. A 48-half-hour shape state with zero mean in each hour band, added to the Kalman filter and updated from every reading the telemetry delay allows, lowers GAM-K's day-ahead quantile score at most decision substations, in months that played no part in choosing the idea's form and in the scored year. | inconclusive | not-for-production | Teaching the physics-based forecast the shape of the day from recent readings lowers its day-ahead error by 0.075 points at five Primary substations (about 3%), in two separate periods and more in winter (0.11, a post hoc breakdown), but averaging the last 14 days' errors by half-hour gains as much, so the learned shape state is not needed on this evidence. The shape state's own check failed: it learns an injected change too slowly (0.35 against a floor of 0.4), so the verdict is capped at inconclusive. A faster shape state, tried afterwards and labelled post hoc, passed that check and beat both K0 and the recipe, which a rerun with the check fixed in advance would need to confirm. | also relevant |
| 00112 differentiable-physics-vp-variable-projection-fit-of-pooled-model | Part 3's pooled DP lost to the per-site DP because gradient descent did not reach the optimum, not because pooling hurts: solving every linear coefficient exactly at each step, with gradient steps only on the non-linear parameters, makes the pooled DP at least as good as the per-site DP on substations with 2 years of history. | inconclusive | not-for-production | Solving the pooled model's linear coefficients exactly did not make its forecasts better. With the interaction neural network on, the exact fit drove the network's last layer and the seasonal and holiday terms to very large, weakly penalised values: it forecast up to 280 points too high on four days of spring 2026 (the weekend of the clock change, Good Friday, and Easter Monday), and it was 0.7 points worse on the other days. With the network off, the exact fit is stable and within 0.03 points of the gradient fit, so the optimiser was not what held the pooled model back; the known-answer test failed, and with the network on the model cannot recover heating even from noise-free data. | also relevant |
| 00108 differentiable-physics-shared-anomaly-factor-across-substations | The residuals of the frozen convex demand model carry a component shared across Primaries, and a shared state estimated from the other substations' residuals, available at the forecast origin, lowers the day-ahead quantile score by more than the per-site Kalman filter of Part 4 already does. | inconclusive | not-for-production | The experiment stopped at its known-answer gate, where three of seven tests failed, so it cannot say whether a shared factor improves the forecast. On injected anomalies the shared filter, tuned on 2023-24, carried under 5% of the injected points (the plan asked for 40% or more), and it corrected no better than the per-site filter (2.402 against 2.397 points of mean absolute error over days 2 and 3 of the episodes). Part of the test was mis-specified, so a corrected test would be needed before any real score. | also relevant |
| 00107 differentiable-physics-rolling-origin-history-and-tails | Refitted quarterly from 2023-07 to 2026-04 and scored on the next 3 months, the convex model's two history contrasts (all years against 12 months, drift against none) keep one sign at almost every origin. A second claim, labelled separately as C5 and not part of the verdict: a level-dependent spread brings its 95% and 99% quantiles within 1 point of nominal coverage without worsening the quantile score. | partly supported | not-for-production | Giving the convex demand model a drift term lowered its score at 10 of 12 quarterly refits, by 0.45 points when trained on all years (interval 0.23 to 0.68). In a post hoc split, about three-fifths of that size is one substation (D10) at the 2023-24 refits, and the 2025-26 refits give 0.30, as in experiment 104. All years without drift scored worse than 12 months at 8 of 12 refits, and all years with drift against 12 months has no stable answer: winter-quarter refits favour more years, summer ones are level. The spread that depends on the forecast level made the 95% and 99% quantiles worse, so the tail coverage still needs a different method. The verdict is partly supported, because only the drift contrast is stable; a second spread (the conformal spread) is not interpreted, because it was built with a flaw. | primary |
| 00106 differentiable-physics-4-all-years-drift-and-recent-level | The differentiable-physics model trained on every usable year from 2019-09 with a drift term forecasts Primary demand better than the same model on its best window, and assimilating recent residuals through a Kalman filter makes it better again, by more than a simple moving-average correction does. | partly supported | not-for-production | Correcting each day-ahead demand forecast with the errors the model made up to 3.5 hours earlier lowered the quantile score by 0.61 points (from 2.90 to 2.29) in the scored year and by 1.02 points in the year before, at all five long-history substations. The gain relies on estimates of past weather that a live service would not have, and against lagged XGBoost at the same lead the filtered model scored 2.349 against 2.487, a difference that is not significant. Whether old years of history help is not settled, because the contrast changed sign between the two years, so the verdict is partly supported. | primary |
| 00104 differentiable-physics-2-simple-structural-model | A convex demand-minus-solar model, trained on every usable year with a slowly drifting level per hour band, forecasts Primary demand better than the same model trained on the last 12 months. | inconclusive | not-for-production | A simple model of demand minus solar, trained on 12 months, scored 0.25 points better than lagless XGBoost with its default start (post hoc; a lower score is better). Most of that lead is in the forecast spread, because the error of the mean forecast differs by only 0.08 points, and four of the five pre-registered positive controls failed, so the verdict is inconclusive. More years of history did not clearly help: the scored year and the tuning year gave opposite signs for two history-window contrasts, and a first version of this page had the wrong sign for both because its spread carried the previous year's mean bias. | also relevant |
| 00103 differentiable-physics-1-data-look | The trial area's power records hold enough clean, demand-only history for the series to go ahead as designed: 6 MW Primaries with at least 12 months before July 2025, and 8 MVA Primaries with over 3 site-years each of rows with no sunlight. | supported | not-for-production | The data lets the series go ahead, with three changes: start 13 of the 16 Primaries later than their first record (19.4 months later at D4), fix the lockdown window in advance because the data does not isolate it from drift, and count eligible long-history MW sites at each tuning year's first day (5 at 2024-07-01, so 4 of 5 must improve in Part 2's tuning contrast). The power timestamps show no 30-minute error on either side of the 2026-03-26 fix: at the 6 metered photovoltaic series the median offsets are +0.9 minutes before it and +4.7 after, and the correction is right to within 3 minutes, but offsets at the Primaries are not resolved beyond 0 to +8 minutes. | primary |