Is the gap between a no-lag forecast and a forecast with lags a detector of LSC events?
This experiment asks whether the gap between two forecasts can find load shifts. A load shift is a switching operation that moves part of one Primary substation's demand (a substation that steps 33 kV down to 11 kV) onto neighbouring substations for hours or days. The network operator NGED has no access to the system that records these operations. The two forecasts it receives from us are K0 (a Kalman filter on recent power readings, added to the generalised additive model forecast) and GAM_ONLY (the same model with no recent power, which forecasts how the network is normally switched). We turned the gap between them, and averages of recent forecast error over 6, 24, and 72 hours (observed power minus GAM_ONLY), into alarms. We set each alarm threshold on odd calendar months, using no knowledge of where shifts happened, and scored the alarms on the other months and on all months. The scoring covered the 35 real load shifts lasting longer than 1 day, 411 artificial shifts added to 44 simulated copies of the power history (experiment 141), and false alarms in ordinary rows. A row is one Primary at one 6-hourly forecast origin, and it is ordinary when no logged switching touched the substation or a neighbour in the 72 hours before. An alarm "matches" a shift when it opens from 2 hours before the onset until the shift ends. The plain gap matched 11 of 35 real shifts, the learned score LR_b20 10, and RO-NC 20 (the refit-offset neighbour-count detector), when each gap threshold reproduces the benchmark's false alarm rate on odd months. The learned score (experiment 227) re-ranks RO-NC's candidate alarms. The 24-hour mean forecast error matched 20 of 35.
Matched real shifts (35 shifts lasting over 1 day)
"While on" means an alarm episode opened from 2 hours before the onset to the end of the shift. "Within 4 days" (post hoc) counts an alarm that opens up to 4 days after the shift ends, because the learned score opens its alarms about 4 days after the step. "Share of moved MW" weights each shift by its size in megawatts. "Alarms per match" is all false alarms plus matched alarms divided by matched shifts. "Lag" is the forecast origin minus the onset. The threshold of each gap row reproduces the false alarm rate named in its row on odd months (1.29 per Primary-year for the learned score, 4.11 for RO-NC, and 0.86 for the pair alarm); the last column is read on even months.
| Detector | Matched while on (95% interval) | Within 4 days | Share of moved MW | Alarms per match | Median lag (hours) | False alarms per Primary-year, odd then even months |
|---|---|---|---|---|---|---|
| LR_b20, learned score | 10 (5 to 15) | 27 | 25% | 4.0 | 96 | 1.29, 0.55 |
| Plain gap, leads 0 to 6 h, at the learned score's rate | 11 (6 to 17) | 16 | 47% | 4.5 | 41 | 1.17, 1.21 |
| Gap with 6-hour minimum, at the learned score's rate | 15 (9 to 21) | 18 | 54% | 3.6 | 62 | 1.29, 1.09 |
| Gap with neighbour check, at the learned score's rate | 11 (6 to 17) | 15 | 35% | 5.0 | 83 | 1.04, 1.64 |
| 24 h mean forecast error, at the learned score's rate | 20 (14 to 26) | 20 | 78% | 2.8 | 29 | 1.23, 0.91 |
| RO-NC (RO_b80) | 20 (14 to 26) | 27 | 68% | 7.3 | 48 | 4.11, 3.64 |
| Plain gap, leads 0 to 6 h, at RO-NC's rate | 21 (15 to 27) | 22 | 76% | 7.3 | 47 | 3.93, 4.19 |
| Gap with neighbour check, at RO-NC's rate | 15 (9 to 21) | 18 | 43% | 10.4 | 49 | 3.93, 4.68 |
| Pair alarm (PG_g70) | 4 (1 to 8) | 6 | 14% | 8.2 | 7 | 0.86, 0.91 |
| Plain gap, leads 0 to 6 h, at the pair alarm's rate | 11 (6 to 16) | 16 | 47% | 3.6 | 41 | 0.86, 0.91 |
Checks and caveats
- Prediction (point estimates): the plain gap matches 11 against the learned score's 10, and 21 against RO-NC's 20. The differences are +1 (-5 to 7) and +1 (-6 to 7) shifts, so the experiment cannot rank the detectors. The learned score's false alarm rate was 1.29 on odd months and 0.55 on even months, while the gap's was 1.17 and 1.21, so on even months the gap raised about twice the learned score's false alarms.
- Where the plain gap's false alarms fall (at the learned score's rate, 39 episodes in 32.8 Primary-years of ordinary rows): 5 episodes in D9 rows (2.7 per Primary-year, against 1.2 over all series), 0 in D8 rows, 3 on bank holidays or within a day of a clock change, 8 from November to March, and 23 elsewhere. Holidays, clock changes, D8, and D9 do not hold most of the false alarms. The page cannot say how many of the 23 unattributed episodes are weather error, so the prediction's weather-error clause is undetermined.
- Other shift sets at the learned score's rate: the plain gap matches 1 of D8's 29 multi-day drops and none of the 17 unlabelled single days with block-shaped demand (the 24-hour mean error on those days stays under 2 noise units, and no detector, carried or gap, matches any of them). A noise unit is 1.4826 times the median absolute value of a score on odd-month ordinary rows. The injected shifts add 1,063 events at the six substations, because one transfer moves power at the donor and at each receiver. The plain gap matches 182 of them (17%): 5% of those lasting up to 2 days (380 events) and 39% of those lasting over 5 days (83 events). The median injected step is 0.87 MW.
- Neighbour check: 22 of the 35 shifts name two Primaries that are both metered, so the check can apply to those and not to the rest. It confirms an alarm origin when a connected metered neighbour has an opposite-signed gap at least 30% the size and the two sum to at most half the node's gap. The check is applied at each origin, so it removes alarm origins more strictly than a rule that kept a whole episode when any origin was confirmed; the episode-level rule is untested. The threshold had to fall from 3.2 to 2.1 to hold the false alarm rate, which explains the extra false alarms. The check cannot apply to injected shifts, whose receivers are unmetered.
- Match rule changes the ranking: the learned score's mask reaches back to the step stamp after an alarm that opens about 4 days late. When an alarm may open up to 4 days after the shift ends, the learned score matches 27 and the plain gap 16.
- Deviations from the plan: a noise unit is the robust scale of each score, not of the half-hourly forecast error; and the lag is measured from the forecast origin, which is 4 hours after the last reading the filter used.
- Data: K0 has no correction before a series' first reading (series 13 and 7 start late), so the gap is not computed there and those rows count for neither alarms nor exposure.
- Audit rule: none of the seven triggers in the
ml-experimentskill fires on this result. Experiment 309 reads the gap, and the maintainer excluded an Opus audit of this work. The median minus mean forecast and quantile-score contrasts do not apply, because no quantile forecast is scored here. - Lab conditions: the records, the generalised additive model, and the filter settings come from experiments 130, 141, and 258, with estimates of past weather and not ECMWF ensemble (ENS) forecasts. The thresholds of every detector were fitted on the same real record.
What to try next
- Experiment 309 tests whether the gap, and the 24-hour mean forecast error, can choose between the two forecasts, scored against observed power and against the normal running arrangement.
- Post hoc: a detector that combines the 24-hour mean forecast error with the gap could be tried.