Skip to content

Requirements

Phased Rollout

Version 1 (current focus): 32 time series in NGED's trial area — 16 primary substations, 6 solar PV farms (5 extra-high-voltage, 1 high-voltage), 3 wind farms, 2 grid supply points (GSPs), 2 bulk supply points (BSPs), 1 biofuel generator, 1 battery energy storage system (BESS), and 1 reciprocating gas generator. All implemented with a single XGBoost model family.

Version 2 (future): Scale to approximately 2,500 time series (all of NGED's primary substations, BSPs, GSPs, and most customer meters). See Roadmap.

Core Objectives

This is a research project, and our NGED partners treat it as a research project: the single hard requirement is that the project gives NGED new information about forecasting for their assets. Even a negative result carries value — if we try our hardest and cannot, say, detect switching events from power data alone, a well-evidenced negative result would inform future decisions about extracting switching records from operational systems. The objectives below therefore sit on a priority continuum, not a must-have/nice-to-have split.

Highest priority — probabilistic power forecasts under the normal running arrangement (NRA):

  • Probabilistic, half-hourly, 14-day horizon forecasts updated every 6 hours. Within that horizon, users mostly act on forecasts roughly 1 to 10 days ahead, so skill in that band matters most.
    • The day-ahead forecast must be available by 11:00 and must cover midnight to 23:59 on the next day.
  • Cover substations (primary, BSP, GSP), metered generators (solar PV, wind, BESS, etc.), and customer meters.

Everything else exists primarily to improve those forecasts. Switching-event handling, effective-capacity tracking, and faulty-meter detection are pursued first and foremost because they make the NRA forecast better, and it is acceptable for the models to handle these phenomena implicitly. That said, explicit estimates are genuinely wanted where we can produce them:

  • Track the effective capacity of metered generators over time (turbine failures, inverter faults, PV panel degradation), including detecting misbehaving generators. The "effective capacity" ignores Active Network Management (ANM) curtailment because a curtailed generator is being held down rather than broken.
  • Detect and compensate for switching events — where power is diverted from one substation to another due to maintenance, changing the local demand signature. (Whether this ships as a discrete event table or as continuous switching-state signals is an open question — see the decision point.)
  • Automatically detect and flag faulty metering (stuck values, physically impossible values, missing data).
  • An optional "prevailing conditions" forecast assembled from the delivered building blocks — explicitly lower priority than the NRA forecast. See forecast building blocks.

The shapes of the five delivery tables were specified in a formal report to NGED, so NGED's agreement is needed to change a shape, such as replacing the discrete substation_switching table with continuous signals.

The worst case matters most: forecasting threshold exceedance

Flexpectation aims to save money for end users in two specific ways. The first is flexibility procurement: deciding, days ahead, whether to pay flexible customers to reduce their demand when a substation risks running beyond its capability. The second is generator curtailment: holding embedded generation down when export risks running beyond a limit. That limit is usually not the substation's own: a generation constraint typically binds above the primary, driven by the aggregated flow across several substations rather than by any single meter in isolation. The forecasts that matter for curtailment are therefore the forecasts that net and sum correctly up the hierarchy — which is why curtailment scoring nets at one primary before summing up the substation hierarchy. Curtailment savings accrue to the whole system rather than to NGED's own spend. Curtailment is nonetheless in scope alongside flexibility procurement, so the forecast requirement is the same for both decisions. So the question users ask of a forecast is rarely "what is the most likely load?" and usually "how likely is net demand to cross this limit?" — a mock-up of the operator view plots demand as headroom below a constraint line.

The project's value therefore concentrates in both tails of each forecast distribution: A model fails at the job if it is excellent on typical half-hours but unreliable in the handful of near-limit hours. The near-limit hours sit at both ends. Flexibility procurement turns on the upper tail, where demand rises towards firm capacity, and bites in winter. Curtailment turns on the lower tail, where export rises towards whichever limit binds because embedded generation is high and demand is low, and bites in summer. The 13 DELIVERY_QUANTILES are deliberately tail-heavy at both ends and symmetric about the median — p1, p2, and p5 matching p95, p98, and p99. The delivery shape therefore already serves both decisions. This is why evaluation includes tail & exceedance metrics alongside average-error metrics. (One honest complication: a substation's real limit is not a single number — it varies with ambient temperature and with how long an overload lasts — so the evaluation metrics use documented static proxies; see the threshold-choice discussion.)

Stretch Goals

Further down the same continuum:

  • Model and forecast unmetered solar PV and wind power on each primary substation by disaggregating net power flow.
  • Disaggregate and forecast other distributed energy resources (DERs): EV chargers, heat pumps, price-sensitive batteries.

ML experimentation at scale

Nearly every objective above is an open research question — improving NRA forecast skill, detecting switching events, estimating effective capacity, flagging faulty meters, disaggregating DERs — and we hold far more ideas than we can try at once. That turns experimentation throughput into an infrastructure requirement in its own right: we need to run on the order of hundreds of ML experiments per month, and the workflow must make each experiment as frictionless as possible. That the throughput produces a better forecast is a bet this project is making rather than a result the literature has settled, for the reasons set out in Our approach to MLOps. H2, a hundred experiments per person in a peak month is the pre-registered version of that requirement, deliberately relaxed to a peak-month, per-person claim so that months spent hardening the service are not spurious falsifications. The promotion half below is H3, one-click promotion and one-click rollback.

Three properties matter as much as raw throughput:

  • Re-runnability. We will inevitably find and fix bugs that invalidate earlier results — in feature engineering, in evaluation, in the data itself. When that happens we must be able to re-run old experiments cheaply and confidently (same configuration, same folds), so that results reflect the fixed world rather than a mixture of before and after.
  • A standardised leaderboard. Every experiment's metrics land in one comparable place, computed the same way, so "is this idea better?" is a lookup, not an analysis project. See Metrics & Leaderboard.
  • A short, safe path from R&D to production. Conducting experiments is only half the loop: an experiment that wins on the leaderboard must move into the live service as easily and as safely as possible. This is why R&D and production share a single unified codebase — the exact feature-engineering and model code that won the experiment is what runs in production, and promotion is an audited configuration change, not a rewrite or a port between systems.

This requirement shapes the architecture: it is why the experiment layer is built around per-(experiment, fold) partitions that can be run — and re-run — individually (see ML Orchestration Design), why R&D and production live in one repository and one execution path rather than separate codebases, and it was decisive in choosing Dagster over Airflow (see Why Dagster, not Airflow?).

Operating model & handover

The working assumption is that, after the Network Innovation Allowance (NIA) project, NGED runs the service on its own AWS account. That working assumption sets a standing design requirement for everything we build:

  • The service must be operable day to day by NGED staff who did not develop the code, working from the runbooks — every routine action reduced to a dashboard check, a button in the Dagster UI, or a runbook step. See Handover to NGED for the engineering consequences and the handover workstreams.
  • Uptime requirements are deliberately lenient — recovery is "next business day, via runbook", never a 2am page. See Uptime: lenient by design below for exactly why an outage does so little damage.

The phasing:

  1. For the duration of the NIA project, OCF develops and runs Flexpectation on OCF's own AWS account (unchanged from the existing plan).
  2. We will not know whether the service is truly hand-over-able until OCF has run the full v2 service (~2,500 time series) for a few months. The v2 gate is part of the working assumption.
  3. In the last few months of the NIA project, NGED progressively takes control of the service, with OCF support. NGED then decides whether to run it themselves.
  4. Post-NIA, OCF is no longer on call; NGED handles day-to-day issues. OCF may continue developing the software and models.

A hybrid model may prove beneficial: NGED runs the production service while OCF continues to develop the code and ML models, and perhaps runs a second service instance of its own (e.g. adapted for other DNOs). A hybrid model makes account-portable infrastructure doubly valuable — see Handover to NGED.

Uptime: lenient by design

Flexpectation carries no hard availability target. We aim for a highly robust service, but the requirement when something breaks is recovery "next business day, via runbook" — never a 2am page, and no on-call rota. The lenient uptime requirement follows from how the forecasts are consumed, not from aspiration. Three factors bound the damage of an outage:

  1. Every forecast extends 14 days ahead, refreshed every 6 hours, and users mostly act on the forecast roughly 1 to 10 days ahead (see Core Objectives). If the service stops producing new forecasts for a few hours — or even a day — the most recent forecast remains useful. It merely ages, degrading skill gradually rather than cutting NGED off.
  2. Delivery is decoupled from compute. Forecasts are delivered as Delta tables on S3 (see Forecast Delivery below), so every previously published forecast stays readable even while all of OCF's compute is down — the read path never touches our infrastructure.
  3. NGED's existing forecasting tools remain available if Flexpectation is temporarily unavailable.

One case these three arguments do not cover. All three assume the outage is our compute stopping. An extended NWP outage is different in kind: compute keeps running, but the forecast either degrades silently or — as the code stands today — hard-fails. Meanwhile the last good forecast ages out, so the 14-day-horizon argument expires along with it. Leniency about our uptime is therefore not the same as leniency about input outages, and the second is handled by a separate design principle: see Inherent Stability, whose degradation ladder sets out what the service should produce at each stage of input loss.

This section is the defensive half of the argument — an outage does little damage. The corresponding positive claim, that interventions will be rare in the first place, is stated and tested as H1, a service that mostly runs itself.

Missed forecasts are also not lost for evaluation purposes: once the service is back, missed slots are backfilled in replay mode, reconstructing what would have been forecast at the time — see Operating the live service: Backfilling a missed slot.

The same properties give the service built-in maintenance windows. New forecasts are produced only once every 6 hours, and NGED reads published forecasts directly from S3 rather than from any OCF-run service. So the gap between one forecast run and the next is a regular, roughly 6-hour window in which OCF can stop, patch, upgrade, or even rebuild its compute (most notably the always-on control-plane VM) without interrupting the forecasts NGED reads. Routine maintenance therefore needs no separate downtime window. Even a maintenance overrun causes only a missed slot, recovered by the replay-mode backfill above.

This requirement shapes the architecture: it is why a single always-on control-plane VM is an acceptable single point of failure (see Production Deployment — Design), and why the primary production alert is a missed-check-in alarm feeding a runbook rather than any paging or failover machinery.

Forecast Delivery

OCF delivers forecasts as Delta Lake tables on AWS S3, updated every 6 hours. Delta Lake provides ACID transactions, meaning NGED never reads an incomplete forecast. (Why Delta Lake rather than a REST API? See Forecast Delivery.) The tables are designed as "building blocks" that NGED can combine to construct:

  • A Normal Operation Forecast (MW or MVA): the power_fcst column is in MW (active power) or MVA (apparent power), with the unit given per time series in TimeSeriesMetadata. Assumes the grid is in a normal running arrangement with all generators at full unconstrained capacity.
  • A Prevailing Conditions Forecast (MW or MVA): planned. It will combine the forecast with the most recently observed effective capacity, reflecting current switching state and any reduced generator capacity — see Forecast building blocks.