Hierarchy and pooling: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00223 dp-bsp-generators-as-children-and-double-count-test | The metered generators that NGED lists under a bulk supply point (BSP) are children of that BSP as much as the Primaries are: the BSP's reading is the Primaries' demand plus the unmetered load minus the generators' output. If the Primaries' readings already include a generator's output, adding that generator to the sum double-counts it. This experiment first tests, from the readings alone, which generators the BSP's remainder carries and which a Primary already carries, and then asks how much the BSP's forecast improves when the generators the BSP carries are summed as children. | inconclusive | not-for-production | The two bulk supply points' readings move with the metered generators that NGED lists under them, and the Primaries' readings show only small coefficients on the same generators. The size of the movement is not the clean one-for-one that adding the generator to the sum needs, so the experiment did not test whether adding the generators improves the bulk supply point forecast. A better test is to separate the unmetered solar and the battery's price-driven dispatch first. | primary |
| 00204 dp-bsp-full-bottom-up-with-signed-mva-children | A bulk supply point (BSP) is forecast better bottom-up than top-down when the sum covers every child Primary. Experiment 161 summed only the three children per BSP that report real power (MW) and have a long history, which reached 42% of D13's effective capacity and 56% of D17's, and the bottom-up forecast scored 0.5% to 0.6% worse than the top-down forecast. This experiment adds the children that report apparent power (MVA). A child's apparent-power reading is the magnitude of its flow, so a child with reverse flow at midday needs a signed flow before it can enter a sum. The fold-aware model of experiment 152 supplies that signed flow as a latent variable. | inconclusive | not-for-production | Adding every child Primary to the sum did not improve the forecast of either bulk supply point: the bottom-up score matched the top-down score (+0.1% at D13, -0.15% at D17), and the intervals include zero. Converting the children that report apparent power to signed flow changed the score by at most 0.03 points, at the one bulk supply point where a child has reverse flow. Even with every child's reading known exactly the BSP's score gains only 12% to 24%, and the children's forecasts gave no detectable gain over the BSP's own chain, so the next gain may lie in forecasting the generators and the unmetered load beneath each BSP. | primary |
| 00161 dp-bsp-top-down-against-bottom-up | A bulk supply point (BSP) is forecast better bottom-up than top-down. The top-down forecast is the best-guess GAM-K chain (experiment 155's arm D) applied to the BSP's reading as a stand-alone series. The bottom-up forecast adds the arm D forecasts of the BSP's child Primary substations that report real power, each times a weight learned from earlier readings, to arm D's forecast of the remainder (the BSP's reading minus those children's weighted readings). The mechanism is that the child Primaries' own forecasts are better than the BSP's chain can infer from the BSP's sum, because each child has its own heating, solar, and calendar response and its own recent errors, and the sum is not exact (losses, unmetered load, children that report apparent power), so the remainder and the weights absorb the difference. | inconclusive | not-for-production | Forecasting a BSP from its children's forecasts plus a remainder did not beat forecasting the BSP as a stand-alone series at either BSP: the bottom-up score was 0.6% worse at D13 and 0.5% worse at D17. The likely reason is that the children this test could use (three real-power Primaries per BSP) cover 42% and 56% of the BSP's capacity, and the BSP's error lives in the rest. The oracle control also missed its pre-registered threshold, which was set too high. The verdict is inconclusive because both intervals include zero. |
also relevant |
| 00158 dp-partly-pooled-gam-shapes-heating-and-calendar-curves | A GAM-K forecast (the convex demand model with a Kalman filter on recent errors, experiment 111) fitted jointly across six Primary substations, so that each substation's heating curve, lighting curve, daily profile, holiday effects, and Christmas shape are a fleet mean plus a penalised deviation of its own, forecasts better than the same model fitted to each substation alone, now that the solar capacity is pinned and the filter handles recent switching. The mechanism is that a substation sees only two winters and two Christmases in a 24-month window, so its own curves are noisy, and five other substations carry information about the shapes that physically are shared. Capacity, level, and the band means stay per substation. The gain is largest for a substation with 3 months of history and on Christmas rows, and no pooled arm is worse than the per-site fit by more than 0.02 points on ordinary rows. | partly supported | candidate | In this test on six Primary substations, fitting the demand model's shared curves (heating, lighting, daily profile, holidays) across all six let a substation with only 3 months of history score 3.01 points instead of 3.64 (lower is better), within 0.17 points of the same substation with 24 months of its own history. The gain was largest in winter and on Christmas rows. For a substation with 24 months of history, pooling changed the score by less than 0.02 points, which is too small to justify making each forecast depend on five other substations' data. The test cannot say whether pooling the heating curve alone would help a 3-month substation. | also relevant |
| 00154 differentiable-physics-n0-gate-2-heating-error-against-truth | When the partly pooled convex regression model of experiment 109 is fitted to only 4 winter weeks of one substation, its fitted heating curve is closer to the injected truth than the curves of the no-pooling and complete-pooling fits. The curve is as close to the truth as a fair oracle's curve (within 10%). The model recovers the sign of the substation's own heating offset relative to the fleet's curve. The fleet's shared curve is recovered as well as an average of per-site fits recovers it. | partly supported | not-for-production | On synthetic substations with a known answer, a partly pooled demand model fitted to 4 winter weeks of one substation estimated that substation's heating response with a smaller error than fitting the substation alone (2.8 to 3.6 against 4.1 to 11.5 points of capacity) and than copying the fleet's curve (3.4 to 4.6), on every seed. It recovered the fleet's shared curve about as well as separate fits did. Neither the pooled fit nor the fair oracle recovered the sign of the substation's own heating offset reliably (the pooled fit got the sign right at 6 or more of 8 substations on only 2 of 5 seeds), so on these synthetic fleets and one January, a partly pooled model can share the heating curve but cannot learn a substation's own offset from 4 weeks. | also relevant |
| 00150 differentiable-physics-n0-gate-rerun-fair-oracle-five-seeds | When the reference estimator estimates every coefficient from the same 4 weeks of rows (so that no coefficient is fixed at its true value), the partially pooled convex model of experiment 109 recovers the injected truth about as well as that reference on most of 5 random seeds, and its tuning rule picks a deviation penalty within one grid step of the best one on most of the 5 seeds. | inconclusive | not-for-production | The rerun of the synthetic known-answer test could not say whether the pooled model shrinks substation deviations correctly, because a control that keeps no deviation at all passed the shrinkage line on 2 of 5 seeds when 4 June weeks were held. Capacity from 4 weeks was within 25% of the truth at 8 of 8 synthetic substations on every seed, the tuning rule picked a penalty within one grid step of the best on 5 of 5 seeds, and the pooled fit's heating error was below a fair reference estimator's on 5 of 5 seeds. The full-history line passed on only 2 of 5 seeds. A post hoc rerun on 4 January weeks separated the controls, so a test that scores heating error against the injected truth in winter should settle the question. | also relevant |
| 00124 differentiable-physics-h-bsp-closure-audit-and-bsp-hint | A bulk supply point (BSP) reading, fitted jointly with the Primaries beneath it as the sum of their demand terms plus the metered generators plus a penalised remainder, with a tuned weight, limits the damage that a metering error at one Primary does to that Primary's forecast, without raising the Primaries' forecast error when no error is present. | not supported | not-for-production | The bulk supply point reading halved the damage from injected scale errors, but it made forecasts on clean data worse at every weight above zero: by at least 0.385 points of capacity in the corrected re-run, about 17 times the 0.023-point limit (2.355 points or more in the first run). A synthetic system that closes exactly loses clean accuracy the same way, so the loss does not come mainly from a wrong multiplier. The cause is untested. As a detector of metering errors the reading is weak: a 25% scale error lasting 30 days is found in 0% of cases by a fit that includes it and in 50% by a fit trained only on earlier rows, which then false-alarms on 61% of later windows. | also relevant |
| 00120 differentiable-physics-sh-convex-daily-shape-library | A convex model in which each Primary's daily profile is a non-negative mix of K = 3 or 4 shared daily shapes per day type, plus a penalised deviation, forecasts Primary demand with 2 years of history at least 0.05 points better than experiment 109's shrinkage towards one fleet-mean profile. That needs the library to beat the per-site profile by about 0.02 points, because the fleet-mean model is 0.031 worse than it; the plan predicts a tie or a small gain, so supported is a long shot. The stability of the shapes and the shape mix of each substation are descriptive. |
inconclusive | not-for-production | In a test with synthetic substations, 5 Primaries were too few for a library of 3 shared daily shapes to recover the shapes we planted, so the check failed on 7 of 10 lines and the verdict is inconclusive. In the scored year the library scored the same as the fleet-mean model and 0.030 points worse than a profile per substation (0.068 worse in winter), and its forecast reproduced a tenth to a half of the Economy 7 night step on cold winter nights. | also relevant |
| 00117 differentiable-physics-h2-graph-prior-on-pooled-convex-parameters | A penalty on the differences between interconnected Primaries' parameters (heating slopes, PV capacity as a share of effective capacity, the profile's scale, and the holiday coefficients) narrows those parameters' month-block bootstrap intervals without breaking their coverage when neighbouring Primaries are alike, and worsens the full-history quantile score by at most 0.03 points. | refuted | not-for-production | At the weight tested, tying the parameters of interconnected substations together did not improve the forecast: in the scored year it worsened the quantile score by 0.020 points on all days (the interval spans zero) and by 0.173 points in winter evening peaks (the replicate year gave -0.021). On synthetic substations with a known truth it narrowed only heating-slope intervals, and the 80% intervals of PV capacity held the truth 17% of the time against 78% without the tie. A tie on heating slopes alone worsened the real score by 0.006 and 0.001 points, an exploratory lead and not a result. | also relevant |
| 00109 differentiable-physics-n0-partially-pooled-convex-model | A convex model whose per-substation coefficients are a fleet mean plus a ridge-penalised per-substation deviation forecasts Primary demand as well as the per-substation convex model when both have 2 years of history, and clearly better when the substation has 4 weeks or 3 months. | inconclusive | not-for-production | The known-answer test of this method failed under its pre-registered definition, and only one of five runs (each on a new seed, runs 2 to 5 being post hoc redefinitions) passes every line, so the verdict is inconclusive. Taking the numbers alone, for the five MW Primaries tested, one convex model shared across substations forecast a new substation with only 4 weeks of history 3.3 points better than a convex model fitted to that substation alone (and 0.8 points better with 3 months), far more than the 0.5 to 2.0 predicted, and scored 0.03 points worse on substations with 2 years of history. On the six held-out Primaries it came within 0.2 points of experiment 105's neural pooled model with 4 weeks (the neural model 0.13 better) and 0.12 better with 3 months, without that model's loss on long histories, which suggests that the loss in experiment 105 came largely from the neural fit. Complete pooling scored 0.07 points worse than partial pooling at 4 weeks in the scored year (interval touching zero) and no differently in the replicate year. The gains should be relied on only after a rerun of the gate. | also relevant |
| 00105 differentiable-physics-3-pooled-and-per-site-demand-model | One differentiable-physics model fitted across all the Primary substations forecasts a substation as well as the same model fitted to that substation alone when both have two years of history, and forecasts clearly better when the substation is new (4 weeks of history) or short of history (3 months). | inconclusive | not-for-production | One neural-network demand model fitted across 15 Primary substations forecasts a substation with only 4 weeks of history far better than a per-site copy of the same model (score 4.04 against 14.38 on the six scored substations), but forecasts a substation with 2 full years of history worse than a per-site copy (3.42 against 3.04 on the five decision substations, worse at all 5). By the pre-registered rule pooling is not yet a success, because that full-history result (C1) is refuted, and a failed known-answer check (gate test 2, injected-farm recovery) caps the verdict at inconclusive. Post hoc, for a new substation the pooled model also beat a per-site convex model fitted to the same 4 weeks (mean absolute error 6.88 against 12.79). | also relevant |
| 00103 differentiable-physics-1-data-look | The trial area's power records hold enough clean, demand-only history for the series to go ahead as designed: 6 MW Primaries with at least 12 months before July 2025, and 8 MVA Primaries with over 3 site-years each of rows with no sunlight. | supported | not-for-production | The data lets the series go ahead, with three changes: start 13 of the 16 Primaries later than their first record (19.4 months later at D4), fix the lockdown window in advance because the data does not isolate it from drift, and count eligible long-history MW sites at each tuning year's first day (5 at 2024-07-01, so 4 of 5 must improve in Part 2's tuning contrast). The power timestamps show no 30-minute error on either side of the 2026-03-26 fix: at the 6 metered photovoltaic series the median offsets are +0.9 minutes before it and +4.7 after, and the correction is right to within 3 minutes, but offsets at the Primaries are not resolved beyond 0 to +8 minutes. | also relevant |
| 00036 hierarchy-aware-bsp-gsp-forecasts | For the two bulk supply points (BSPs) Boston and Skegness and the two grid supply points (GSPs) Walpole and Bicker Fen, forecasts of the series beneath each point, and an N1 model trained on ERA5 and applied to ENS members, lower the quantile score below that of N1 trained on ENS. | inconclusive | not-for-production | Forecasts of the series beneath two bulk supply points added nothing (+0.04 and +0.19 points of capacity), nor did wind and solar columns at two grid supply points; an N1 trained on ERA5 lowered the grid supply point scores by 0.36 and 0.40 with intervals spanning -1.8 to +1.0. | primary |
| 00032 substation-fingerprints-pooled-demand | For demand series of Primary substations, adding a numeric fingerprint of each substation (its location, its mean level and daily and weekly shape over the training window, and its level over the 14 days before the forecast origin) to a by-type pooled XGBoost quantile model closes part of the 0.15-point gap between the pooled model with the series identifier and per-series models, and lets a substation held out of the pool be forecast better than a pooled model with no fingerprint. | inconclusive | not-for-production | Adding numeric substation fingerprints to the pooled model changed the Primary score by at most 0.02 points of capacity, and training-window statistics made a held-out substation worse by 1.39 (inconclusive overall; recent power statistics alone helped it by 0.23, post hoc). | primary |
| 00024 pooling-groupings-demand-xgboost | For demand series of Primary substations, one XGBoost quantile model trained across a group of series (all series, one model per substation type, or one model per size band), given the series identifier as a categorical column, scores no more than 0.3 points of capacity worse than one model per series, so that the number of models can fall from one per series to one per group. | supported | candidate | Pooling the 16 Primary substations into 3 or 4 XGBoost models (with the series identifier) scored 0.15 points of capacity worse than 20 per-series models (99.44% interval +0.03 to +0.26), inside the 0.30 margin; one model for all 20 series was 0.29 worse, and dropping the identifier cost 0.8 to 1.0. | primary |