Skip to content

Can a fitted wind-turbine physics model show that a wind farm has lost 25% of its capacity within 60 days, without raising false alarms on a slow seasonal drift?

The planned monitor found a 25% loss within 120 days at only about half the start dates at W1 and W2, and at none at W3. The verdict is not supported: the pre-registered rule for stopping (a median delay above 60 days at 2 or more farms at a 25% loss) is met. W1, W2, and W3 are three metered wind farms, anonymised.

Question and answer

We asked whether the fitted wind-turbine model of experiment 230 (called "the core" below) can show that a farm has lost capacity. The monitor divides observed power by the power the core expects, adds up both over a trailing 60 days of strong-wind half-hours (those where the core expects at least 95% of the farm's fitted maximum output, its "scale"), and divides by the same ratio before July 2024, so that 1.0 means normal. An alarm is raised when that relative ratio falls below the lowest value seen in 2021-09 to 2024-06.

The planned monitor was too slow in a simulation. We cut observed power by 10%, 25%, and 50% from 30 random start dates and counted the days until an alarm. A 25% loss raised an alarm within 120 days at 50% of start dates at W1, 63% at W2, and 0% at W3 (29 start dates at W3, after one that already alarmed was excluded).

A variant added after the results were seen did better but is not trustworthy. It uses every half-hour at 20% of scale or more. It found a 25% loss within 120 days at 100%, 87%, and 100% of start dates (26 at W3 after exclusions). A smooth seasonal swing of 9% in observed power with no loss, the size seen in experiment 183's solar farms, raised extra alarm days at all three farms.

Figure 1: share of start dates detected within 120 days, per farm and loss size

Figure 2: relative ratio over time for the real series, a 9% seasonal swing, and a 25% loss

Figure 3: share of days with at least 4 qualifying half-hours, by calendar month

Figure 4: monthly observed over expected power on the real series

Figure 5: monthly 99th percentile of observed power, all three farms

Results

  • Strong-wind half-hours are rare. They are 2% of half-hours at W1, 4% at W2, and 0.6% at W3 over 2019-09 to 2026-06. The plan predicted 3% to 14%, from counts that its own text gave wrongly; its stated counts imply 1.6% to 9%.
  • The planned monitor has a reading on few days. A 60-day window holds enough strong-wind half-hours (at least 4 half-hours on each of at least 6 days) on 21% of days at W1, 45% at W2, and 5% at W3. The post hoc rows give 97%, 97%, and 93%. W3's planned result is mostly a coverage failure, not a measure of sensitivity.
  • Alarm thresholds (planned rows): 0.956 at W1, 0.832 at W2, and 0.926 at W3. The W1 threshold sits 4% below 1, so a slow drift of 4% or more alarms there.
  • A 25% loss, planned rows: median delay 121.5 days at W1 (bootstrap 95% interval 79 to not reached), 105.5 days at W2 (77.5 to 162), and never at W3. Shares within 120 days: 0.50 (0.22 to 0.76), 0.63 (0.37 to 0.88), and 0.00. The intervals resample 18 calendar months of start dates.
  • A 25% loss, post hoc rows: median delay 27 days at W1 (24.5 to 36), 51.5 at W2 (42 to 75), and 34 at W3 (27 to 43). A 50% loss: 12.5, 27, and 15 days. A 10% loss is found within 120 days at 0.27 (W1), 0.60 (W2), and 0.73 (W3) of start dates.
  • Constant expected power on the planned half-hours gave the same shares and median delays as the core at the 25% and 50% losses at every farm, and within 0.04 in share at the 10% loss. The fitted power curve adds almost nothing on the flat top.
  • 9% seasonal swing, extra alarm days over two years: planned rows 50 at W1 and 42 at W2, and 0 at W3 (where only 5% of days have a reading); post hoc rows 10, 28, and 62. Swings of 15% and 25% raise 152 and 340 extra alarm days at W1 on the post hoc rows.
  • Real series with no loss, alarm days after 2024-07-01: planned rows 0, 102, and 36 (W1, W2, W3); post hoc rows 13, 32, and 202. The monthly observed over expected power on the post hoc rows ranges from 0.71 to 1.20 (W1), 0.74 to 1.31 (W2), and 0.35 to 1.27 (W3). On the planned rows, W2 falls to 0.39 in one month of winter 2025 to 2026, and W3 to 0.51 in one month of 2024 to 2025. No cause is claimed.
  • W1's output shows no step. Its monthly 99th percentile is mostly 18 to 21 MVA and falls to 9 to 15 MVA in some summer months, so it follows the wind.
  • Starts inside and outside the core's fitting data (before and after 2025-06-17), post hoc rows, 25% loss: shares within 120 days of 1.00 and 1.00 at W1, 0.79 and 1.00 at W2, and 1.00 and 1.00 at W3. Starts before 2025-06-17 sit inside the data the core was fitted on, so the after-2025-06-17 figures are the honest ones.

Checks

  • No forecast is produced, so forecast-error and tail-calibration checks do not apply.
  • Audit triggers (listed in the ml-experiment skill): trigger 1 fired (the 9% control raises extra alarms at W1 and W2) and trigger 2 fired (the predicted detection within 60 days at W1 and W2 failed; the rarity of strong-wind half-hours explains it, shown in Figure 3 and by the post hoc rows). A fresh Sonnet auditor re-derived the shares, thresholds, median delays, and drift alarm days with its own code from the raw inputs and matched every one. It found no error in window indexing, loss timing, or missing-data handling. It noted that the calibration years overlap the core's fitting data, which tightens the thresholds, and that the loss-free run's alarm days are masked, which makes the delays slightly longer than a plain count.
  • Hold-out check of the threshold: calibrating on 2021-09 to 2023-06, 51 of 104 days with a reading in 2023-07 to 2024-06 alarm at W1 and 50 of 50 at W3, so the minimum-over-years threshold is not stable out of sample there; at W2 none of 159 alarm.

Caveats and what to try next

  • The post hoc rows are not pre-registered, and they fail the 9% drift test at all three farms.
  • Lab conditions: the core is frozen, the loss multiplies observed power, and an export cap would hide a loss at strong wind, so the delays are optimistic. Start dates that already alarmed without a loss are excluded, so the farms are compared on slightly different start sets.
  • Next: a monitor on all half-hours above cut-in with a baseline that depends on the month, which may absorb the 9% swing (untested), then a check of the W2 and W3 low months against NGED's outage records. A learned correction of the ratio by wind speed is a candidate neural component; its prediction that could fail is that the 9% swing then raises no extra alarm at W1.