Load-shift labels in the forecast: experiments
This page lists every experiment tried in the theme, newest first. An experiment that belongs mainly to another theme but bears on this one is marked "also relevant".
| Experiment | Hypothesis | Verdict | Production status | Take-home message | Relevance |
|---|---|---|---|---|---|
| 00283 lsc-perfect-knowledge-value-to-forecast-and-curtailment | Knowing every load shift exactly (its true start and end) lowers the error of the forecast of the normal running arrangement (NRA) in the 48 hours after a shift, and lowers the curtailment cost of the forecast of observed power in the days around a shift, while leaving ordinary rows unchanged. This is question 1 of the LSC decision program. | partly supported | not-for-production | Perfect knowledge of load shifts lowers the forecast's error most in the 48 hours after a shift ends (23% lower upper-tail score, 95% interval 9% to 39%) and by under 1% over a whole year, because shifts are rare. Curtailment cost against observed power falls by 8% (6% to 10%) near a shift only when the known shift's megawatts are added back to the forecast, and does not fall when recent readings are merely set aside. The next step is to ask how good a detector must be to keep that gain, using the megawatts-added-back arm as the target. | also relevant |
| 00258 lsc-mask-out-of-sample-all-primaries-whole-history | Experiment 227 found that LR_b20 (the learned alarm score of experiment 195, at the threshold that leaves 20 unexplained odd-month candidates) with a mask recovers 83.5% of the logged-oracle gain in the 72 hours after a logged load shift and raises the ordinary-day score by -0.007 points (-0.039 to +0.014). That rests on 11 load shifts at 6 Primary substations, 27 months of cached ensemble forecasts, a threshold chosen on the scored rows, and months that were the re-ranker's training negatives. On all 15 Primary series with a residual from the frozen convex model of experiment 130 (arm B), over the whole residual history of about 4.5 years, and scored only on rows the learned score was not trained on, the LR_b20 mask keeps at least half of that gain and keeps its ordinary-day harm inside +/-0.05 points. | inconclusive | not-for-production | On rows it was never trained on, the carried load shift mask recovers about half of the best-case gain after a load shift (47%, with a wide range of 18% to 85%) and changes ordinary days by less than 0.05 points. Over every row of 4.5 years it scores slightly worse than no mask, because a few long alarms over logged closures add more excess score than the mask removes elsewhere. The check against simulated load shifts failed, so the result is inconclusive, and this page gives no support for porting the mask. | also relevant |
| 00249 lsc-self-training-on-consistent-events | The learned load-shift score of experiment 195 is trained only on injected transfers. If it is retrained on injected transfers plus the real events that pass a label-free consistency rule (balanced opposite steps at a metered neighbour, a high round 1 score, and a clean return step), it finds more of the 35 real transfers lasting over 1 day than the round 1 score, at the same 80 unexplained odd-month alarms, because the pseudo-positives show it what real events look like. | refuted | not-for-production | Retraining the learned load-shift score on its own consistent real events found no extra real transfers: the score still finds 24 of 35. Only 38 to 47 real events per fold pass the consistency rule, against about 2,000 injected transfers per fold, so the extra rows probably could not move the score. Self-training may need many more consistent events than 4.5 years of 16 Primary substations provide. | also relevant |
| 00227 lsc-fewer-false-alarms-and-softer-use-part-2 | Experiment 208 found that no real alarm source stays inside the ordinary-day harm limit of 0.05 points, because each raises 104 to 242 alarms for 15 to 20 matched logged load shifts. (a) A source at a higher threshold, or a union of the pair alarm and the learned score that keeps the learned score only where the pair alarm also fires or its score is high, matches enough load shifts with few alarms per match. (b) A use of the alarm that keeps some recent power (soft down-weighting, a pair-sum correction, or a hedge between masked and unmasked forecasts) costs less per false alarm than a mask and keeps most of the gain after a load shift. | partly supported | candidate | Keeping only the learned score's most confident alarms gives a mask that finds 21 of 35 real load shifts with 5 alarms per match, recovers most of the forecast gain after a load shift, and does no measurable harm on ordinary days. A gentler use of the alarm is not needed at that setting. The result rests on 11 load shifts after which the gain is measured, so the next step is to confirm it on more load shifts before porting it. | also relevant |
| 00226 lsc-why-learned-score-mask-lost-part-1 | Experiment 203's learned-score mask (7.31 points of effective capacity on the 72 hours after the 7 logged load shifts, against 6.32 for the pair-alarm mask) loses because the learned alarm is known 2 to 4 days after the step, so its mask starts late on long load shifts. If that is the cause, a mask that takes the pair alarm's onset where the pair alarm fires, and the learned score's coverage elsewhere, keeps the learned score's extra matches without the loss. | not supported | not-for-production | The learned load-shift score finds more real load shifts than the pair alarm, but the extra ones it finds do not improve the forecast and do the harm, so merging the two alarms does not give a mask that is both as good after load shifts and harmless on ordinary days. The alarms that both sources raise are harmless but too few to help. The next step is fewer false alarms per matched load shift, not more matches. | also relevant |
| 00212 switching-detector-masks-in-gam-training | A convex additive model of substation demand (GAM) is trained on readings that include switching transfers, which move load between substations for hours or days. Every earlier GAM in this project trained with the logged and hand-labelled transfers set missing, so nobody has measured what the masking is worth. If masking the detector's own alarms during training does about as well as masking the hand labels, a live service with no labels can train as well as the project does now. | inconclusive | not-for-production | Masking switching transfers when the forecast model is trained barely matters: with no mask the score is only 0.002 points worse on clean rows and 0.003 worse after a logged transfer, and 476 simulated transfers added to the training readings worsen it by just 0.009 points. Training with the detector's alarms masked scores 0.007 points worse than with hand labels masked on clean rows (interval -0.007 to +0.025) and 0.027 worse in winter, so the detector's effect in training is small on average but not resolved from zero. On this record the unmasked fit is already almost as good as the masked fits, so a detector mask has little to improve in training. | also relevant |
| 00208 switching-value-of-a-detector-by-phase-and-quality | Knowing about a switching transfer lowers the forecast error mainly in the 48 hours after the transfer ends and in the days before an alarm would normally arrive, and the benefit shrinks roughly in proportion to the share of transfers missed and to the delay of the alarm. A detector of the quality of the refit-offset neighbour-count detector (RO-NC) recovers well under half of the perfect-knowledge benefit, so the quality it must reach to be worth building is higher than RO-NC's. | not supported | not-for-production | Telling the forecast about a switching transfer lowers its error by about 1.5 points of capacity during the transfer and in the 48 hours after it ends, but by only about 0.005 points averaged over a year, because transfers are rare. A detector that finds about 4 transfers in 10, as RO-NC does, recovers about half of that gain, and none of the real alarm sources tested keeps its ordinary-day error inside the limit. Options for a next step are an alarm source with far fewer false alarms per matched transfer, or a use of the alarm that does not set recent power aside. | also relevant |
| 00203 switching-replay-learned-score-in-masked-forecast | Experiment 195's learned score finds 22 to 24 of the 35 real transfers lasting over 1 day, against 19 for the refit-offset neighbour-count detector (RO-NC). If its alarms drive the gated mask of experiments 163 and 171 (readings set missing from the detector's step stamp until the transfer ends), the forecast scores better after real transfers than the pair-alarm mask does, and the mask still does no harm on ordinary days. The mask ignores recent power inside an alarm, so extra true alarms should help and extra false alarms should harm. | partly supported | not-for-production | Masking the forecast with the alarms of experiment 195's learned score did not lower the error after real switching transfers by a reliable margin, and it did not meet the limit for no harm on ordinary days. The learned alarms arrive 2 to 4 days after a step, so they miss the start of the long transfers that the pair alarm catches. A candidate next step is a mask that takes the pair alarm where it fires and the learned alarm elsewhere. | also relevant |
| 00195 switching-neural-plus-detector-features | A learned score that reads the features the rule-based switching detectors already compute (the 24-hour step statistic of RO-NC, the opposite step at the metered neighbour of the pair alarm, and related quantities), trained only on injected transfers, finds more of the 35 real transfers lasting over 1 day than RO-NC alone at the same number of unexplained alarms, and more than a rule-based union of RO-NC and the pair alarm. | partly supported | candidate | A learned score on the features that RO-NC and the pair alarm already compute finds 22 to 24 of the 35 real transfers over 1 day, against 18 or 19 for RO-NC, at the same number of unexplained alarms. A plain logistic regression finds 22 even without any look-ahead feature, so most of the gain is in the features, and the extra transfers are small ones (under 1 MW each). The next step is to test whether the extra alarms help the gated mask, which this experiment did not run. | also relevant |
| 00191 dp-price-sensitive-assets-in-the-history | Some of D8's multi-day drops (experiment 159) and the matching shortfalls at its neighbour D14 are output of price-following gas generation (experiment 186), which no weather input explains and no forecast of the price can reach at a lead of days. Letting the convex demand model (GAM-K) see the half-hourly price in its training history, gated by the registered connection date of the 6 MW gas engine, lets it learn a cleaner baseline for the rest of the demand. The forecast then improves even when the price is unknown at forecast time. | not supported | not-for-production | Modelling a price-following asset in the history does not make the forecast better at D8 or D14. A demand model that is given the true price at forecast time, which no live service has here, gains only 0.6% at D8, and the models that use a price available at forecast time lose 0.9% to 4.4%. The gas engine's response is real enough to fit (1.2 to 1.9 MW at D8 within the training rows), so the remaining D8 error comes from causes that these price indicators do not explain, and NGED's half-hourly output for the engine is the data to ask for next. | also relevant |
| 00189 switching-neural-onset-end-block-detector | A small neural network (under 50,000 weights) that reads a 7-day window of the convex-model residual of a Primary substation and its connected neighbours, and is trained only on injected transfers and on real stretches without a logged event, finds more of the 35 real transfers lasting over 1 day than the pair alarm of experiment 156 does, and finds as many as RO-NC (the refit-offset neighbour-count detector), at the same number of unexplained alarms, and places the onset to within 3 hours. It can do so only if the injected transfers resemble the real ones, so the injector is fitted to the real events and checked for realism before training. | refuted | not-for-production | A small neural network trained only on injected transfers finds 7 of the 35 real transfers lasting over 1 day, against 18 for the existing rule-based detector, when both the start and the end must be right. It locates the start of 12 transfers (17 after longer training) to within 2 hours, so the learned start alarm is competitive and the end alarm is what fails. The next step is a learned start alarm paired with the rule-based end rule, or a longer input window that holds the end of multi-day transfers. | also relevant |
| 00188 switching-d10-audit-and-onset-location-in-look-back | Part 1. Experiment 171 found that ignoring recent power while a gated pair alarm is active scores 0.030 points of effective capacity worse than no mask on rows outside logged transfers (95% interval -0.011 to +0.074), and that leaving out one substation, D10, moves the estimate to +0.002. If D10's gated alarms are real switching that NGED did not log, those rows are not ordinary days, and treating them as events leaves a no-harm interval inside +/-0.03. Part 2. The alarm for a transfer is known hours to days after the step, but the forecast runs every 6 hours, so at each run the readings since the step are in the look-back. A step-fitting search over those readings finds the onset more precisely than the alarm time, and masking from the found onset gives a smaller forecast error in the first 24 hours after the alarm than masking from the alarm time. | partly supported | not-for-production | Once a switching alarm fires, masking recent power back to the alarm's own step stamp removes almost all the extra error of masking only from the alarm time, and searching the look-back for a more precise onset adds nothing. D10's extra error under the gated mask is not shown to be unlogged switching, since an independent step test confirms only 3 of its 13 unlogged alarm intervals (its recall was not measured), so showing that the gated mask does no harm on ordinary days still needs more months of forecasts. | also relevant |
| 00181 switching-block-detector-and-like-for-like-table | The 17 unlabelled worst days that experiment 175 classed as sharp-edged blocks were missed by the refit-offset neighbour-count detector (RO-NC) and by the opposite-step pair alarm because most of the blocks are shorter than the 24-hour windows those detectors average over and their steps are small in noise units, not because the blocks are invisible. A detector that fits a rectangular block (a step up, a flat level, a step down) at lengths of 6 hours to 5 days finds most of them at the same number of unexplained alarms as RO-NC. | refuted | not-for-production | A detector fitted to rectangular blocks finds none of the 17 unlabelled worst days. It was allowed the same 80 unexplained alarms in odd-numbered months as RO-NC, the existing detector. The days are mostly blocks lasting 12 hours or less and standing 1 to 4 noise units above the residual (one noise unit is the usual spread of the forecast residual), too close to its ordinary variation for an alarm that reads only the residual. The next step is to send NGED the candidate label table, so that NGED can say which of the 17 days were real switching. On the 35 labelled real transfers lasting over 1 day, RO-NC finds 18 on the convex-model residual and 12 on the XGBoost residual. Four of the 6 extra transfers sit at Primaries where the XGBoost panel has no readings, and on the 29 transfers that both residuals cover the counts are 14 and 12. | also relevant |
| 00177 dp-worst-days-other-data-sources | At least a quarter of the 37 unlabelled worst substation-days of experiments 172, 174, and 175 coincide with an event recorded in data the project already holds: a logged switching operation near the substation, a step in a metered generator on the same or an interconnected Primary, a reading-quality fault in the substation's own series, or a repeat of the substation's own bad weekday or date. | refuted | not-for-production | The 37 worst days of the best-guess forecast that carry no label are not explained by any of the six sources we tested: logged switching, reading faults, repeat bad weekdays, Demand Flexibility Service events, and national price spikes each cover them no more often than they cover ordinary days. The three raw NGED files we read hold only time and power. Sources we have not tested include local outage notices, local events, and a location join to the metered generators. | also relevant |
| 00175 dp-unlabelled-day-shapes | At least half of the 37 unlabelled worst substation-days of experiment 174 are sharp-edged blocks in the forecast's error (a switching-like change in the power a substation draws), and in at least a third of those a substation interconnected with the first shows a step of opposite sign within 2 hours (a load transfer between the two). | refuted | not-for-production | The forecast's worst unlabelled days are roughly half sharp-edged blocks (a switching-like change in the power a substation draws) and 40% smooth offsets (a change in level or behaviour that lasts the day). Interconnected neighbours do not show the opposite step more often than chance would give, and the existing switching detectors fired on none of the block days. The counts point to two separate problems, level changes and blocks that the interconnection table does not explain. The shape rules miss about 40% of injected blocks, so the block share is uncertain. | also relevant |
| 00174 dp-unlabelled-worst-days-weather-shared-or-single-site | Most of the excess loss on the unlabelled worst days of experiment 172 is weather error, and the unlabelled worst days are mostly shared with other Primaries. Replacing the ensemble-mean weather with the estimate of past weather (ERA5 temperature and CAMS irradiance) in arm D's forecast removes at least 50% of the unlabelled days' excess loss, and at least a third of the unlabelled days are also in the worst 5% of days of at least one other Primary. | partly supported | not-for-production | Most of the forecast's error on its worst unlabelled days is not weather error: with past weather in place of the forecast weather, about 69% of the excess loss stays. Other Primaries share these days about as often as random shifting gives, so the data show at most a weak shared cause. The remaining loss could therefore come from events at one substation, such as undetected switching, a change in behaviour, or a metering fault. The next place to look is the substation's own recent readings and the switching records on those days. | also relevant |
| 00172 dp-error-budget-worst-days | The error of the stage 2 assembly (experiment 155's arm D) is concentrated on a small number of days that the project already knows how to label. The worst 5% of days, ranked by their pinball loss above the site's median day, carry at least 30% of the total excess loss, and at least half of those days carry one of six labels: a switching event, a metering error or flagged reading, a cold snap, an Economy 7 step, a holiday, or a combination. | inconclusive | not-for-production | A small number of bad days carries a large share of the forecast's error: the worst 5% of days carry about 42% of the excess loss, and about half of those days coincide with a switching event, a holiday, or a cold snap. Switching events are the largest group, on days that carry 29% of the excess loss, so a forecast that improves only ordinary days leaves most of the loss in place. Cold snaps carry 5% of the excess in this mild scored winter, so this year cannot show whether a cold-night term pays. The formal verdict is inconclusive because one control narrowly failed. | also relevant |
| 00171 switching-gated-mask-no-harm-pooled-months | Experiment 166 found that ignoring recent power (a "mask") only while a pair alarm with a balance score of at least 0.7 is active (the "gated mask") scores 0.009 points of effective capacity worse than no mask on the real rows outside logged transfers, with a 95% interval of -0.068 to +0.081 from one year of rows. That interval was too wide to show no harm. Pooling every cached forecast run, 27 months instead of 12, shows whether the gated mask does no harm on ordinary days. | inconclusive | not-for-production | Ignoring recent power while a gated switching alarm is active scores 0.030 points of effective capacity worse over 27 months of rows outside logged transfers (95% interval -0.011 to +0.074), against +0.009 in experiment 166's 12 months. That interval is too wide to show the harm is below 0.03, and the difference comes mostly from one substation, D10. Showing no harm needs more months of forecasts, or an explanation of why D10's gated alarms raise the error. | also relevant |
| 00166 switching-gated-mask-and-soft-mask-while-alarmed | Experiment 165 found that setting recent power missing while a switching alarm is active lowers the error in the 72 hours after a logged transfer ends (13.69 to 6.31 points of effective capacity) but raises it by 0.102 points on rows outside logged transfers, because many alarms are not transfers. Two ways of using the alarm less bluntly keep the gain and remove the harm. (1) A gated mask: mask only when the alarm is a pair alarm of experiment 156 whose opposite steps balance at a metered neighbour with a balance score of at least 0.7, and never for a single-series refit-offset neighbour-count (RO-NC) alarm. (2) A soft mask: keep the readings in the Kalman filter but inflate their observation-noise variance by a factor that rises with the pair alarm's balance score, until the end rule fires. | not supported | not-for-production | Setting recent power aside only when a switching alarm has a balanced opposite step at a metered neighbour (balance score 0.7 or more) keeps 99.6% of the gain after a logged transfer ends (error score 13.69 down to 6.34 points of effective capacity, 7 transfers) and adds only 0.009 points on other rows, against 0.102 for the earlier mask from either alarm. One year of data leaves the interval for the extra error too wide (-0.068 to +0.081) to meet the pre-registered limit of 0.03, so the verdict is not supported. Down-weighting recent power instead of dropping it does not help: the gain and the extra error rise together. | also relevant |
| 00165 switching-drop-recent-power-while-alarmed-all-rows | A forecast built from a frozen generalised additive model (GAM) and a Kalman filter on its recent errors, with the filter's readings set missing from the moment EITHER the pair alarm of experiment 156 OR the refit-offset neighbour-count detector (RO-NC) is known until the end rule of experiments 147 and 148 fires, forecasts better than the same forecast with no mask while an event is running and in the 72 hours after it ends, and does not make the forecast worse on the real rows outside events, because the filter then restarts and the mask only removes readings that contain moved load. | not supported | not-for-production | Setting recent power aside (a mask) while a switching alarm is active, where an alarm flags load moved between substations, helps only after a logged transfer ends: the mask lowers the forecast error score from 13.69 to 6.31 points of the substation's effective capacity in the 72 hours after (7 transfers). On the other rows the mask raises the score by 0.10 points, about 4% more error, which fails the pre-registered limit of +0.01 points on the upper end of the 95% interval. The next step is a mask gated on how reliable the alarm is, or used in winter only, that keeps reading recent power on rows outside transfers. | also relevant |
| 00163 switching-real-event-comparison-gam-stack-against-bg2 | On the logged real load transfers at the six decision Primaries, a forecast built from a frozen generalised additive model (GAM), a Kalman filter on its recent errors, and a mask that sets the readings missing from the moment an alarm is known (the pair alarm of experiment 156, with the end rule of experiments 147 and 148) forecasts the days after a transfer ends better than best guess 2 (BG2), because BG2 takes lagged power as an input and so carries the transfer into its forecast, and the mask does not. The stack costs little on the real rows outside transfers. | partly supported | not-for-production | After a real load transfer ends, a forecast that masks the power readings of the transfer period or ignores recent power has an error score about 40% lower than best guess 2 (an XGBoost forecast that uses recent power) and about 54% lower than the same forecast with no mask (6.32 against 10.09 and 13.69 points of effective capacity). This rests on 7 logged transfers, and two long winter transfers carry most of the gain. On ordinary days the mask scores 0.03 points worse than no mask, and best guess 2 and the stack are indistinguishable over the year, with the stack better in winter and worse in other months. The next step is more logged transfers, or a forecast that drops recent power while the pair alarm or the refit-offset neighbour-count detector (RO-NC) is active. | also relevant |
| 00162 switching-shaped-transfers-donor-demand-and-pv-shapes | A load transfer moves houses, factories, and solar sites, not a constant number of megawatts, so the power it moves follows the donor's demand and solar shapes. Part 1: on the logged real transfers, a donor residual during the transfer is fitted better by a daily shape, -(a x the donor's demand term + b x its solar term) from the convex model (GAM), than by a constant offset, and the fit separates a from b only for transfers that last several days and contain both sunny and dull days. Part 2: an injection whose moved power follows the donor's solar term vanishes at night and is small at midday in winter, so the half-size-reversal end rule and the step threshold of the detectors misfire on it, and a constant-MW injection (and, less so, today's injection, which moves a share of the donor's net power) flatters the detectors. | partly supported | not-for-production | If a transfer removed a daytime load that follows the donor's solar output, or removed solar output itself, the refit-offset neighbour-count detector (RO-NC) would find about half of them instead of 91%, and the live end rule would cover 22% of the true interval instead of 62%. The pair alarm loses far less once a step may start at first light. The 25 logged real transfers cannot say whether they move solar output, because a daily shape fits them no better than it fits quiet stretches. A realistic mix of houses and small solar sites is nearer the share-of-net-power injection of earlier experiments, which the detectors find as easily as a constant transfer. | also relevant |
| 00159 switching-d8-multi-day-drops-recurrence-and-report | Descriptive, with no hypothesis test beyond the prediction below. D8 is a Primary substation whose power drops by 3 to 5 MW for days at a time while the convex demand model's forecast does not follow. The experiment lists every such drop, asks whether the drops recur on a pattern, and asks whether a connected metered neighbour steps up by a similar amount at the same time (a load transfer) or does not (a customer load change or an outage). | partly supported | not-for-production | D8's multi-day drops are 29 episodes since 2022, nearly all since mid-2023, in which D8's power runs about 2.7 megawatts below the demand regression model's expectation for 2 to 19 days. The three connected metered neighbours step up with the drops at a rate not clearly above their rate at random times, so the drops do not clearly look like load moved onto them, and they show no detectable weekday, hour, bank-holiday, or seasonal pattern. Most of the switching detector's alarms on D8 are away from the drops (10 of 52 are within 6 hours of an edge), so the detector does not find the drops. The next step is to ask NGED which load moved or which switching took place, starting with the six drops that have no logged closure. | also relevant |
| 00156 switching-faster-onset-alarm-from-opposite-steps | RO-NC (the refit-offset neighbour-count detector) learns of a load transfer about 48 hours after it starts, because its step statistic needs 24 hours of data after the step and its peak test needs 24 hours more. A transfer between two metered substations makes opposite-signed steps of similar size at the two, while weather, Economy 7, and holidays make same-signed steps. An alarm that fires when one node steps by k noise units and a connected metered node steps the opposite way by at least k/2, measured over a window of 2, 4, or 6 hours, therefore learns of the transfer within hours instead of days, at the same rate of real false alarms as RO-NC. A mask forecast (readings from the alarm onset set missing until the end rule fires) driven by this alarm then scores better than the same mask driven by RO-NC, because the mask starts days sooner. | partly supported | not-for-production | An alarm built from opposite steps at a metered neighbour is known 3 to 9 hours after a load transfer starts, instead of 48, and masking readings from it closes 95% of the forecast-error gap on the transfers it finds. It finds few transfers, though: 38% of simulated transfers and 5 of 35 real ones. On all rows after a transfer begins, no mask beats a forecast that ignores recent power, so the next step is a forecast that drops recent power while a transfer is suspected, not a better mask. | also relevant |
| 00153 latent-nra-state-in-the-gam-k-filter | Experiment 141's two-regime filter lets the level follow a load transfer, and 141 scored observed power, so neither tested the forecast of the normal running arrangement (NRA). If the filter instead holds the NRA as its own slowly moving state and routes the transfer into a separate, fast offset state that is switched on by the causal alarm and left out of the forecast, the forecast of the NRA moves towards the true NRA series during and after a transfer, and stays where it was when no transfer happens. | partly supported | not-for-production | Giving the transfer its own offset state in the Kalman filter forecasts the normal running arrangement as well as the filter that ignores power does, and no better than simply dropping the readings inside a detected alarm. Dropping them is a smaller change, because it needs no second filter state on the serving path, so the extra state is not worth building. The no-change test on real alarms also failed: most real alarms coincide with genuine level shifts, and on them the new filter moved the median forecast by about 4 points of capacity. | primary |
| 00151 nra-forecast-rescored-against-the-true-nra-series | The detector-driven correction of experiment 141 was scored against observed power, which rewards a filter for following a load transfer. Aim 2 of the switching work is a forecast of the normal running arrangement (NRA), the power the substation would have drawn had no transfer happened. Scored against that true NRA series, a forecast whose filter subtracts the megawatts (MW) of each detected transfer from its input closes a useful share of the gap between the filter that ignores transfers and the filter fed the true NRA series, once the alarm has been raised. The two-regime filter of experiment 141 closes little, because it still follows the transfer. | partly supported | not-for-production | Once the detector has raised its alarm, taking the transfer out of the recent power closes most of the gap to a forecast fed the true normal-running-arrangement series, but ignoring recent power altogether closes nearly as much. Subtracting the detected megawatts is fragile (it rests on a few transfers at one substation), and masking the readings is steadier. Counting the 64% of transfers that the detector missed and the 2 days before the alarm, subtracting the detected MW gains nothing and masking gains 21%, so the next step is a detector that alarms sooner and a forecast that drops recent power while a transfer is suspected. | primary |
| 00141 switching-detector-fed-into-gam-k | Telling GAM-K's Kalman filter when the RO-NC detector has seen a load transfer lowers the forecast's quantile score on days after a transfer begins, because the filter's level then adapts within hours instead of over days. The same signal given as plain features (event age and signed step size) does the same, more weakly. | not supported | not-for-production | Telling the forecast's filter when a load transfer started and ended helped inside the transfer and hurt in the 72 hours after it, because the filter holds the transfer's step over the whole 7-day forecast. The switching detector's alarm arrives 48 hours after the step and did no better than a random placebo. The next test is a correction that decays over the typical transfer duration. | primary |
| 00116 differentiable-physics-nra-edge-flows-between-primaries | Flows on the electricity network's interconnection edges, fitted to a frozen convex model's residuals with a sparse-step penalty, recover injected conserved transfers between MW-metered Primaries; removing them gives a normal running arrangement (NRA) model trained without hand labels that forecasts no worse outside events; and fed causally into the Kalman filter as a known input, they improve the metered forecast during and after a transfer. | inconclusive | not-for-production | The method that looks for load moved between neighbouring substations did not work: with the settings we fixed in advance it found under a quarter of the load we added artificially, and it changed no forecast. Looser settings found more, but they also found load moving between substations that are not connected, probably the slow drifts of demand that our model does not describe. The gate failed for a structural reason: at the settings we fixed in advance, the fit cannot find any transfer shorter than 7 to 9 days, and the longest test transfer lasts under 5 days, so the failure is a limit of the method as specified, not a bug in the solver. The zero flow we first saw on the unconnected pairs was also an artefact of the order the fit visits its paths, so those pairs were never clean. Adding a drift term to the model, or accepting a flow only where the two substations are connected, is the next thing to test. | also relevant |
| 00100 drop1-random-lag-dropout-widens-intervals | An XGBoost quantile model for Primary substation demand that is trained with labelled switching half-hours removed from its targets and with its lagged power inputs set to missing at random intervals, and that has those lags set to missing when they lie inside a labelled switching event, widens its 5% to 95% band on the half-hours whose lags are missing, so that the band's coverage moves from 0.784 towards the nominal 0.90, without widening the band elsewhere. | inconclusive | not-for-production | Training the demand forecast with lags hidden at random did not make its band wider on half-hours where a lag is missing: on real rows the band covered 0.80 (hidden in 10% of training rows) to 0.84 (30%) of actuals, against a nominal 0.90. Hiding lags in 30% of rows did remove the accuracy loss that masking alone causes on injected transfers (-0.02 points, 95% interval -0.35 to +0.38 against the plain model). Hiding lags in 10% of rows left +0.26 (98.75% interval -0.15 to +0.85), not separable from zero or from the loss. An input that tells the XGBoost model a lag is missing, or a separate XGBoost model of the quantile spread, are untested routes to a wider band. | also relevant |
| 00096 sw2-23-synthetic-switches-injected-into-training | A forecast of the normal running arrangement (NRA, the network as it is when no switching transfer is on) recovers the power a Primary substation would have drawn had no transfer happened better when its XGBoost model was also trained on many synthetic transfers than when it was trained on the real logged transfers only. In a synthetic transfer, known MW is added to the recorded power of a donor and its receivers, the half-hours of the transfer are dropped as training targets, the lags are corrected by the transfer's MW, and the MW is given as a column. The gain holds when the MW at forecast time comes from the causal refit-offset neighbour-count detector (RO-NC) or from the largest detector step, and not only when the MW is the injected truth. | inconclusive | not-for-production | C1, synthetic-trained minus real-trained with the injected MW: +0.079 points of capacity (98.75% interval -0.031 to +0.153). | primary |
| 00091 sw2-26-calibrated-probability-of-abnormal-arrangement | The refit-offset neighbour-count detector (RO-NC), run causally (with only the data before a forecast origin), can give for each past half-hour in the lag window a probability that the half-hour lies inside an abnormal running arrangement (ARA). The probability is calibrated, and it separates better than the detector's yes-or-no flag. | inconclusive | not-for-production | A probability built from the causal switching detector was not shown to beat the detector's yes-or-no flag: on the planned rows its Brier score was 3.3% lower (97.5% interval -7.6% to +1.6%), against the 20% the plan required. Only in a post hoc subset of injected transfers large enough to see was the gain 11.2% (-20.3% to -2.0%). Against NGED's switching logs it has almost no skill (area under the curve 0.52 to 0.53), so a rule that widened the uncertainty bands when the probability is high would cover only about 8% of the half-hours that the logs mark as switching. | primary |
| 00090 sw2-28-lag-unusualness-inputs-all-rows-and-switching | For the 16 Primary substation demand series, telling the monthly-refitted (P-roll) XGBoost model how unusual each of its three lags is (the probability integral transform, PIT, of each lag against a probabilistic climatology of the series) lowers the quantile score on all half-hours outside switching, and widens the quantiles on the half-hours whose lag lies inside a logged switching event, when both arms train on normal-arrangement targets only. | refuted | not-for-production | C1, clim_pit minus plain on all rows outside switching: +0.355 points (97.5% interval +0.209 to +0.513). |
primary |
| 00089 sw2-22-longest-history-nra-forecast-with-ara-lags-masked | For the 16 Primary substations, an XGBoost demand model refitted monthly and trained on the longest history of ECMWF ensemble control-member forecasts (2021-03 onward), with every half-hour inside an abnormal running arrangement (ARA) removed from the training targets, scores below the same model on the standard 12-month window, and setting to missing every lagged power input that lies inside an ARA lowers the score on the half-hours whose lags lie inside an ARA. | refuted | not-for-production | C1, LM minus L on rows with a lag in an ARA: +0.409 points (98.33% interval +0.281 to +0.517). | primary |
| 00088 sw2-21-nra-forecast-recovers-normal-arrangement-during-and-after-injected-switches | When a Primary substation (a substation that steps down to the local distribution network) is inside a load transfer, or has just left one, an XGBoost demand model that was trained without the transfer's half-hours as targets, with the lags corrected by the transfer's signed MW and the MW as a column ("full NRA", NRA being the normal running arrangement), forecasts the power the substation would have drawn had no transfer happened better than the plain model does. This holds when the MW comes from the RO-NC detector of experiment 82 (a logistic score on the 24-hour residual step) run causally on data up to the forecast origin, but by less than when the MW comes from the injected truth. | inconclusive | not-for-production | C1, arm 4 with RO-NC MW as built minus plain: -0.149 points of capacity (98.75% interval -1.174 to +0.148), closing 16% (-17% to +67%) of the gap to the ceiling, negative in 2 of 4 months. | primary |
| 00086 sw2-12-detector-labels-for-nra-forecast | On the 292 daily ENS forecast runs of experiment 48, giving the monthly-refitted (P-roll) Primary demand model the lags minus the MW that switching moved, together with that MW as three columns, lowers the quantile score on the half-hours whose lag sits inside a switching event, when the events and MW come from the RO-NC load-transfer detector of experiment 82 instead of NGED's logs, and the gain is smaller than with the oracle labels of experiment 41 by the share of events RO-NC misses. | inconclusive | not-for-production | Lags corrected with the detector's labels did not lower the quantile score on rows with a lag inside a logged event. | primary |
| 00075 sw2-11-nra-forecast-with-labelled-switching-masks | On the 292 daily ENS forecast runs of experiment 48, a weekly-refitted Primary substation demand model whose training targets leave out the labelled switching half-hours forecasts the normal running arrangement (NRA) no worse than the plain weekly model, and giving the model the signed MW that switching moved at each lagged power input, or replacing each lag by the lag minus that MW, lowers the quantile score on half-hours outside switching. | inconclusive | not-for-production | C1, masked minus plain: +0.004 points (98.33% interval -0.026 to +0.025). | primary |
| 00044 sw2-2-switching-share-of-best-guess-score | Half-hours in labelled switching events, and half-hours whose lagged power falls inside one, carry a larger share of the best guess's quantile score and tail score than their share of half-hours, and a perfect switching detector could lower the Primary quantile score or tail score by 3% or more. | refuted | not-for-production | In the band of events the labeller finds most reliably, a perfect switching detector would lower the best guess's quantile score by 0.42% (95% interval -0.47% to 1.96%) and its tail score by nothing, against a threshold of 3%. Switching in that band touches 3 of the 16 Primary series and few half-hours, in one validation year. | primary |