Differentiable physics: pooling across substations: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00200 dp-pooled-correction-head-on-kalman-state | Experiment 142's small network failed its known-answer gate with about 1,100 samples from one year at five substations. The same kind of network, trained on rows pooled across many Primary substations and years, learns to correct arm D's forecast from the recent errors, the calendar, the lead, and the ensemble-mean weather, and so beats recipe R (subtract 0.75 times the mean error of the last 14 days) at day D+1. | partly supported | not-for-production | A small neural network trained on arm D's own past errors lowers the day-ahead score by 0.07 to 0.12 points at five substations, but a copy that sees no recent error gets most of that gain. The gain comes from correcting how arm D's error depends on its forecast level and the calendar, and a pinball-loss table comes within 0.01 to 0.02 points of the network on one year of data. | also relevant |
| 00158 dp-partly-pooled-gam-shapes-heating-and-calendar-curves | A GAM-K forecast (the convex demand model with a Kalman filter on recent errors, experiment 111) fitted jointly across six Primary substations, so that each substation's heating curve, lighting curve, daily profile, holiday effects, and Christmas shape are a fleet mean plus a penalised deviation of its own, forecasts better than the same model fitted to each substation alone, now that the solar capacity is pinned and the filter handles recent switching. The mechanism is that a substation sees only two winters and two Christmases in a 24-month window, so its own curves are noisy, and five other substations carry information about the shapes that physically are shared. Capacity, level, and the band means stay per substation. The gain is largest for a substation with 3 months of history and on Christmas rows, and no pooled arm is worse than the per-site fit by more than 0.02 points on ordinary rows. | partly supported | candidate | In this test on six Primary substations, fitting the demand model's shared curves (heating, lighting, daily profile, holidays) across all six let a substation with only 3 months of history score 3.01 points instead of 3.64 (lower is better), within 0.17 points of the same substation with 24 months of its own history. The gain was largest in winter and on Christmas rows. For a substation with 24 months of history, pooling changed the score by less than 0.02 points, which is too small to justify making each forecast depend on five other substations' data. The test cannot say whether pooling the heating curve alone would help a 3-month substation. | primary |
| 00154 differentiable-physics-n0-gate-2-heating-error-against-truth | When the partly pooled convex regression model of experiment 109 is fitted to only 4 winter weeks of one substation, its fitted heating curve is closer to the injected truth than the curves of the no-pooling and complete-pooling fits. The curve is as close to the truth as a fair oracle's curve (within 10%). The model recovers the sign of the substation's own heating offset relative to the fleet's curve. The fleet's shared curve is recovered as well as an average of per-site fits recovers it. | partly supported | not-for-production | On synthetic substations with a known answer, a partly pooled demand model fitted to 4 winter weeks of one substation estimated that substation's heating response with a smaller error than fitting the substation alone (2.8 to 3.6 against 4.1 to 11.5 points of capacity) and than copying the fleet's curve (3.4 to 4.6), on every seed. It recovered the fleet's shared curve about as well as separate fits did. Neither the pooled fit nor the fair oracle recovered the sign of the substation's own heating offset reliably (the pooled fit got the sign right at 6 or more of 8 substations on only 2 of 5 seeds), so on these synthetic fleets and one January, a partly pooled model can share the heating curve but cannot learn a substation's own offset from 4 weeks. | primary |
| 00150 differentiable-physics-n0-gate-rerun-fair-oracle-five-seeds | When the reference estimator estimates every coefficient from the same 4 weeks of rows (so that no coefficient is fixed at its true value), the partially pooled convex model of experiment 109 recovers the injected truth about as well as that reference on most of 5 random seeds, and its tuning rule picks a deviation penalty within one grid step of the best one on most of the 5 seeds. | inconclusive | not-for-production | The rerun of the synthetic known-answer test could not say whether the pooled model shrinks substation deviations correctly, because a control that keeps no deviation at all passed the shrinkage line on 2 of 5 seeds when 4 June weeks were held. Capacity from 4 weeks was within 25% of the truth at 8 of 8 synthetic substations on every seed, the tuning rule picked a penalty within one grid step of the best on 5 of 5 seeds, and the pooled fit's heating error was below a fair reference estimator's on 5 of 5 seeds. The full-history line passed on only 2 of 5 seeds. A post hoc rerun on 4 January weeks separated the controls, so a test that scores heating error against the injected truth in winter should settle the question. | primary |
| 00124 differentiable-physics-h-bsp-closure-audit-and-bsp-hint | A bulk supply point (BSP) reading, fitted jointly with the Primaries beneath it as the sum of their demand terms plus the metered generators plus a penalised remainder, with a tuned weight, limits the damage that a metering error at one Primary does to that Primary's forecast, without raising the Primaries' forecast error when no error is present. | not supported | not-for-production | The bulk supply point reading halved the damage from injected scale errors, but it made forecasts on clean data worse at every weight above zero: by at least 0.385 points of capacity in the corrected re-run, about 17 times the 0.023-point limit (2.355 points or more in the first run). A synthetic system that closes exactly loses clean accuracy the same way, so the loss does not come mainly from a wrong multiplier. The cause is untested. As a detector of metering errors the reading is weak: a 25% scale error lasting 30 days is found in 0% of cases by a fit that includes it and in 50% by a fit trained only on earlier rows, which then false-alarms on 61% of later windows. | primary |
| 00120 differentiable-physics-sh-convex-daily-shape-library | A convex model in which each Primary's daily profile is a non-negative mix of K = 3 or 4 shared daily shapes per day type, plus a penalised deviation, forecasts Primary demand with 2 years of history at least 0.05 points better than experiment 109's shrinkage towards one fleet-mean profile. That needs the library to beat the per-site profile by about 0.02 points, because the fleet-mean model is 0.031 worse than it; the plan predicts a tie or a small gain, so supported is a long shot. The stability of the shapes and the shape mix of each substation are descriptive. |
inconclusive | not-for-production | In a test with synthetic substations, 5 Primaries were too few for a library of 3 shared daily shapes to recover the shapes we planted, so the check failed on 7 of 10 lines and the verdict is inconclusive. In the scored year the library scored the same as the fleet-mean model and 0.030 points worse than a profile per substation (0.068 worse in winter), and its forecast reproduced a tenth to a half of the Economy 7 night step on cold winter nights. | primary |
| 00117 differentiable-physics-h2-graph-prior-on-pooled-convex-parameters | A penalty on the differences between interconnected Primaries' parameters (heating slopes, PV capacity as a share of effective capacity, the profile's scale, and the holiday coefficients) narrows those parameters' month-block bootstrap intervals without breaking their coverage when neighbouring Primaries are alike, and worsens the full-history quantile score by at most 0.03 points. | refuted | not-for-production | At the weight tested, tying the parameters of interconnected substations together did not improve the forecast: in the scored year it worsened the quantile score by 0.020 points on all days (the interval spans zero) and by 0.173 points in winter evening peaks (the replicate year gave -0.021). On synthetic substations with a known truth it narrowed only heating-slope intervals, and the 80% intervals of PV capacity held the truth 17% of the time against 78% without the tie. A tie on heating slopes alone worsened the real score by 0.006 and 0.001 points, an exploratory lead and not a result. | primary |
| 00113 differentiable-physics-n5-convex-identification-at-realistic-noise | On synthetic substations built from the real calendar and weather, with noise as large and as autocorrelated as the convex model's real residuals, the convex model's 80% month-block bootstrap intervals cover the truth 70 to 90% of the time, separately for each hour band's heating slope and for the latest season-year's PV capacity. Labelled separately, not in the verdict: the day-of-year harmonics and the drift term make those intervals wide, by absorbing the seasonal cycles of temperature and clear-sky PV. | refuted | not-for-production | At real noise levels, the convex model's 80% month-block bootstrap intervals contain the true heating slope and the true rooftop PV capacity 84% of the time each, but a heating slope is weak (the interval is about 38% of the slope wide). The pre-registered verdict is refuted, because with independent noise the capacity interval covers the truth only 52% of the time, and the cause is mainly a bias of about 0.24 points from the drift penalty at its tuned step of 3,000 (at Part 4's step of 30 the bias is 0.02 points and coverage 78%, a post hoc check). The truth here has the model's own form, so the test cannot show the bias that a wrong PV shape causes. | also relevant |
| 00112 differentiable-physics-vp-variable-projection-fit-of-pooled-model | Part 3's pooled DP lost to the per-site DP because gradient descent did not reach the optimum, not because pooling hurts: solving every linear coefficient exactly at each step, with gradient steps only on the non-linear parameters, makes the pooled DP at least as good as the per-site DP on substations with 2 years of history. | inconclusive | not-for-production | Solving the pooled model's linear coefficients exactly did not make its forecasts better. With the interaction neural network on, the exact fit drove the network's last layer and the seasonal and holiday terms to very large, weakly penalised values: it forecast up to 280 points too high on four days of spring 2026 (the weekend of the clock change, Good Friday, and Easter Monday), and it was 0.7 points worse on the other days. With the network off, the exact fit is stable and within 0.03 points of the gradient fit, so the optimiser was not what held the pooled model back; the known-answer test failed, and with the network on the model cannot recover heating even from noise-free data. | primary |
| 00110 differentiable-physics-5-embedded-generation-and-capacity | The convex differentiable-physics model can represent the generation side of a Primary substation without making its demand forecast worse: an MVA meter through an observation layer, a flat unmetered generator found where one exists and not invented where none does, and an export-limited PV clip; and its PV capacity stays the same when the routes by which demand terms can imitate solar are closed. | inconclusive | not-for-production | The demand forecast could not yet account for generators connected behind a substation. Rooftop solar capacity fitted from the data was not precise or consistent enough to trust, so the capacity from outside estimates stays in use. A generator-finding component reported a generator at every substation tested, so it stays switched off. A limit on exported solar helped only at the one substation known to export-limit. Treating the meter as apparent power improved the forecast at the 8 substations that report only apparent power by 0.13 points of capacity, but the test of the meter's flow direction failed, so that gain is provisional. | also relevant |
| 00109 differentiable-physics-n0-partially-pooled-convex-model | A convex model whose per-substation coefficients are a fleet mean plus a ridge-penalised per-substation deviation forecasts Primary demand as well as the per-substation convex model when both have 2 years of history, and clearly better when the substation has 4 weeks or 3 months. | inconclusive | not-for-production | The known-answer test of this method failed under its pre-registered definition, and only one of five runs (each on a new seed, runs 2 to 5 being post hoc redefinitions) passes every line, so the verdict is inconclusive. Taking the numbers alone, for the five MW Primaries tested, one convex model shared across substations forecast a new substation with only 4 weeks of history 3.3 points better than a convex model fitted to that substation alone (and 0.8 points better with 3 months), far more than the 0.5 to 2.0 predicted, and scored 0.03 points worse on substations with 2 years of history. On the six held-out Primaries it came within 0.2 points of experiment 105's neural pooled model with 4 weeks (the neural model 0.13 better) and 0.12 better with 3 months, without that model's loss on long histories, which suggests that the loss in experiment 105 came largely from the neural fit. Complete pooling scored 0.07 points worse than partial pooling at 4 weeks in the scored year (interval touching zero) and no differently in the replicate year. The gains should be relied on only after a rerun of the gate. | primary |
| 00105 differentiable-physics-3-pooled-and-per-site-demand-model | One differentiable-physics model fitted across all the Primary substations forecasts a substation as well as the same model fitted to that substation alone when both have two years of history, and forecasts clearly better when the substation is new (4 weeks of history) or short of history (3 months). | inconclusive | not-for-production | One neural-network demand model fitted across 15 Primary substations forecasts a substation with only 4 weeks of history far better than a per-site copy of the same model (score 4.04 against 14.38 on the six scored substations), but forecasts a substation with 2 full years of history worse than a per-site copy (3.42 against 3.04 on the five decision substations, worse at all 5). By the pre-registered rule pooling is not yet a success, because that full-history result (C1) is refuted, and a failed known-answer check (gate test 2, injected-farm recovery) caps the verdict at inconclusive. Post hoc, for a new substation the pooled model also beat a per-site convex model fitted to the same 4 weeks (mean absolute error 6.88 against 12.79). | primary |