Skip to content

Live Service

How to operate the production forecasting service day to day. Unlike the roadmap (forward-looking design for work not yet built), this is the durable home for live-service operational docs — step-by-step recipes for running what's already built — once each piece ships. Design rationale (the why) lives in docs/architecture/ instead; see that folder's Production Deployment — Design page for this area's counterpart. The roadmap's Live Service page sends readers here as its sections land (so far: the live_forecasts and promoted_model assets, local 6-hourly automation, the container build/verify runbook, the AWS bring-up runbook, and Sentry telemetry with the missed-check-in alarm; still to come: production monitoring, and per-task failure-email alerting). Once the whole v0.1 epic ships, the roadmap page is deleted and this section is the sole home for how the live service works.

This is distinct from ML Experimentation: that area covers training and backtesting candidate models against historical data; this area covers picking one of those candidates as the running production model and keeping live forecasts flowing from it.

Audience note: today these runbooks are written for OCF (Python-literate researchers), but after the NIA project the day-to-day operator is expected to be a non-expert at NGED — NGED's stated preference (2026-07-14, pending their internal sign-off) is to run the service themselves, on their own AWS account. Every routine operator action must therefore eventually reduce to a dashboard check, a button in the Dagster UI, or a runbook a non-Python-expert can follow; before handover, these pages get an editing pass with that operator as the audience, plus a top-level "operator contract" page indexing them. See Handover to NGED.

Documents

The pages split by which environment you're bringing up and then how to drive it — the driving is identical in both environments, so it lives on one shared page:

  • Running the whole stack locally — bring the entire service up on a laptop: .env, a persistent DAGSTER_HOME, dg dev, and the optional MinIO rehearsal.
  • Setting up the live service on AWS — every step to stand the service up on AWS, in order: S3 buckets and IAM (AWS's storage and permissions services; every AWS acronym is spelled out on first use there), promote a champion and build/verify/push its image, the Fargate task, the always-on control-plane box, and connecting to the Dagster UI over Tailscale.
  • Connecting to the AWS control plane — get a laptop onto the OCF tailnet to reach an already-running deployment: install Tailscale, view the Dagster UI, and SSH into the always-on box.
  • Operating the live service — driving a running stack day to day: promote a champion model, let the 6-hourly live_forecasts schedule run (or materialise a slot by hand), inspect a forecast, and backfill a missed slot in replay mode.
  • Setting up Sentry telemetry — point error reporting and the missed-check-in alarm at a Sentry project: get a DSN, test it from your laptop, and turn it on in production.
  • Configuration reference — what the storage roots, the derive-from-root convention, and the credential settings mean, and which combination each environment uses.