Production Deployment — Design

How the live service is orchestrated, and how the champion model gets from an MLflow leaderboard into a running production container — and why. For the step-by-step recipe — promote a model, build the image, verify it, push it, and stand up the AWS deployment around it — see Setting up the live service on AWS.

Run the Dagster control plane continuously on one small VM

The Dagster control plane — daemon + webserver + code-location server — runs continuously on one small VM, alongside MLflow and Marimo, managed via Docker Compose. Its schedules are responsible for the entire recurring pipeline, not just the forecasts: pulling fresh telemetry from NGED's bucket (hourly), downloading the ECMWF ensemble NWP from Dynamical.org (daily), and issuing the forecasts themselves (6-hourly). The box does none of that pipeline compute of its own, though: it dispatches every run — those schedules and UI-launched backtests alike — to an ephemeral Fargate task via EcsRunLauncher. Even the light data-ingest runs go to Fargate; the reasoning is in Running the data-ingest runs on the control-plane VM. This is the roadmap's accepted architecture; see Live service: AWS architecture for the costed options, and AWS Running Costs for what the deployed service costs to run — as built at v1, and projected at v2 scale.

Because scheduling lives inside Dagster rather than in any cloud-specific service, the entire stack stays portable: the laptop deployment and the cloud deployment are the same artifact, started with docker compose up. What switches a deployment between local and Fargate execution is not code but Dagster instance configuration: the run launcher is declared in dagster.yaml, which lives outside the image. The laptop's dagster.yaml declares no run launcher, so runs execute as local subprocesses (Dagster's DefaultRunLauncher); the box's declares EcsRunLauncher (Step 14 of the runbook), so runs launch as Fargate tasks. The code and image are byte-identical either way. The always-on daemon also provides the cross-run coordination that only a daemon can: schedules, sensors, run monitoring, and declarative automation.

The design accepts two trade-offs:

• Single point of failure. The daemon on one VM has no managed scheduler watching it; a quiet box failure could silently miss slots. The blast radius is small, because the project's uptime requirements are lenient by design: previously published forecasts stay readable from S3 and extend 14 days ahead, so a missed slot degrades forecast freshness rather than cutting NGED off. Mitigation: the missed-check-in alarm — each successful 6-hourly run checks in with Sentry (external to the whole deployment), and an alert fires when an expected check-in fails to arrive. The alarm is the only component that lives outside the box, and the stack does not depend on it to function.

• No run-level auto-retry after a hard crash. Accepted; covered by the existing replay/backfill mode for missed slots plus the missed-check-in alarm.

"Always-on" is also less demanding than it sounds, because the box comes with built-in maintenance windows: forecasts are produced only every 6 hours, and NGED reads published forecasts directly from S3 (see Forecast Delivery), so the gap between one forecast run and the next is a regular window in which the VM can be stopped, patched, or rebuilt without NGED noticing — and even an overrun costs only a single slot, recoverable via replay-mode backfill. See Requirements → Uptime: lenient by design.

Several other orchestration shapes were considered and rejected — see Considered but rejected designs.

Warn on stale power data with a Dagster asset check

power_time_series_and_metadata_job runs hourly and succeeds even when NGED has published nothing new, so a job that merely ran tells the operator nothing about whether fresh data actually arrived. If NGED's feed stalls, the Delta table silently goes stale. The power_data_is_fresh asset check closes that gap: it reads the power_time_series Delta table's actual data recency — the maximum time per time_series_id — rather than the asset's materialisation timestamp, and warns when any series has no data within a 24-hour staleness threshold. A native materialisation-freshness policy would miss this exact failure, because the materialisation keeps succeeding.

The check is attached to power_time_series_and_metadata, so the existing hourly schedule runs it every hour with no extra wiring. It reports two kinds of lateness: a series that once reported but has now gone stale, and a roster series (present in the TimeSeriesMetadata parquet) that has never sent data. The count of each, plus a table of the offending time_series_ids, lands in the check's Dagster metadata — Dagster's Checks view becomes the operator's at-a-glance "is the power data healthy?" status surface, showing a green tick when every series is current and a yellow warning naming the late count when the feed has stalled. The severity is a warning rather than a failure: a stalled feed is expected to self-heal once NGED recovers (the pipeline back-fills the gap automatically), so it must not block downstream assets.

This in-Dagster check is complementary to — not a replacement for — the missed-check-in alarm. The alarm fires on total silence from outside the deployment, because a dead daemon cannot report itself; this check reports per-series staleness from inside Dagster while the daemon is alive, and forwards that same staleness to Sentry as a warning (see the freshness-warning bullet under Send telemetry to Sentry). The freshness evaluation is a pure function, so the Sentry warning reuses the same PowerFreshnessResult the check already computed rather than recomputing it; the same result will later also feed the forecast-warnings delivery table.

Send telemetry to Sentry, and alarm on absence

Three independent Sentry mechanisms live in nged_substation_forecast._sentry, all active in every Dagster process (daemon, webserver, and each Fargate run worker) only when a SENTRY_DSN is configured — so laptops and CI stay silent by default.

• Error telemetry. init_sentry initialises the SDK once per process at import of the Dagster definitions module, and a Dagster @failure_hook (sentry_capture_failure, attached to the scheduled asset jobs) reports a failed step's exception — traceback intact — from inside the run worker. The explicit hook is used rather than Sentry's log-to-event capture, which init_sentry deliberately disables (it passes a LoggingIntegration with event_level=None): Dagster logs a step failure without exc_info, so a purely log-based capture would yield a message-only event with no stack trace, and — left at its default event_level=ERROR — it would turn every ERROR log in the process into a Sentry event, including Dagster's own startup and ad-hoc-run logs. The hook is attached to the three scheduled asset jobs, so it covers the whole unattended production workload; because log capture is off, failures in a manual UI materialisation, a replay backfill, or an experiment job are watched by the operator at the Dagster UI, not routed to Sentry.

• The missed-check-in alarm — the primary production alert. After each successful live live_forecasts run, the asset sends one success check-in (a heartbeat) to a Sentry cron monitor; Sentry raises the alarm when no heartbeat lands within the margin (the 6-hourly schedule plus ~2 h grace), regardless of cause. Evaluation must live outside the deployment because a dead daemon cannot report itself — the reasoning for why this, not a Dagster sensor, is the mechanism is in Alert on absence. Only success heartbeats are ever sent (never an error check-in), so the alarm keys purely off absence: an in-band run error is the failure hook's job, and a manual replay backfill — which reprocesses the past rather than proving the service is live now — deliberately does not check in.

• Freshness warnings. When the power_data_is_fresh asset check finds late series, report_power_freshness forwards that per-series staleness to Sentry as a warning-level event, reusing the PowerFreshnessResult the check already computed. It is gated on the DSN (like error telemetry, not the heartbeat flag), so it fires from wherever a configured environment runs the hourly check, separated by the environment tag. The event is fingerprinted per environment (["nged-power-data-stale", <environment>]) — Sentry's environment is a filter facet, not a grouping key, so without the environment in the fingerprint every deployment would share one issue; with it, each deployment gets its own ongoing issue, and the hourly re-reports of a continuing stall collapse into that one issue rather than a fresh issue each hour. The message body lists the late series and how late each is (series 12: 48.5h late (last seen …), or never reported), and the full per-series detail is attached as structured event context. Both are capped — the message to a short leading slice with an …and N more line, the context to a larger slice — so a whole-feed stall at V2 scale can't attach thousands of rows; the true late count is always carried by the n_late tag, so a capped list never makes a large stall look small. Sending is best-effort: report_power_freshness never raises, so a Sentry hiccup can't fail the blocking=False check (which, inside the hooked hourly job, would otherwise trip the failure hook and fail the run).

Freshness is a two-way state (stale ↔ recovered) modelled with a one-way primitive: a warning event has no "resolved" counterpart. So recovery is signalled by the events stopping — the issue's "last seen" timestamp stops advancing — and the operator resolves the issue by hand. The production alert rule should fire on regressions (and the project's Sentry auto-resolve window kept short) so that a fresh stall after a recovery re-pages rather than silently appending to a stale-but-unresolved issue. This is a deliberate trade: the missed-check-in alarm above is the primary two-directional signal; this warning is the richer per-series breadcrumb layered on top.

Separating laptop telemetry from production

Sentry's environment tag (from Settings.sentry_environment) keeps the two apart. The always-on box sets environment=production; each developer overrides the local default with <name>-laptop (e.g. jacks-laptop, alexs-laptop), so error events filter cleanly by origin. The missed-check-in alert rule is scoped to environment:production, so an intermittently-run laptop never pages. When a developer wants to exercise the heartbeat path locally, they set SENTRY_MONITOR_FORECASTS=true and point it at a throwaway monitor slug (live-forecasts-test), never the production live-forecasts monitor — otherwise the shared production monitor would be left expecting a 6-hourly check-in that the laptop won't keep sending, and would flag it as missed. The AWS runbook lists the exact environment variables set on the box.

Separate "is this run usable?" from "is it perfect?"

Data ingest poses two different questions, and we answer them with two different mechanisms so that neither compromises the other. "Is this run usable?" is a fatal gate: the contract validate inside the ingest asset rejects data that is structurally broken, so a bad run fails loudly and writes nothing rather than poisoning the table. "Is it perfect?" is a separate, non-fatal Dagster asset check: it records known-but-tolerable imperfections as a warning without blocking the materialisation or anything downstream. Collapsing the two — failing on every imperfection — throws away otherwise-good data; ignoring the distinction the other way lets genuinely broken data land silently.

The ecmwf_ens asset is the worked example. Its nwp_has_no_unexpected_nulls check surfaces the scattered per-pixel nulls that ECMWF ENS is known to carry (a WARN), while Nwp.validate still hard-fails a wholesale structural gap. The reasoning behind exactly where that fatal/tolerated line sits is documented in Known ECMWF ENS data-quality issues. The power_data_is_fresh check above is the same shape of tool pointed at a different question — staleness rather than completeness — and is likewise a warning, never a failure.

Bake the model into the image at build time

Production forecasts run as ephemeral Fargate tasks, dispatched by the always-on Dagster control plane described above. An ephemeral container has no persistent disk and, for v0.1, no MLflow tracking server to reach — so production inference needs some way to get a model without depending on either.

The champion model is therefore baked into the image at build time and loaded via a plain save/load — no MLflow, run ID, or cache involved at runtime. The model directory is produced once, out of band (a researcher picks the champion fold from the MLflow leaderboard and downloads its artifacts to local disk), then COPY'd into the image at build time. Promotion becomes rebuild + redeploy, which is auditable (image tags) and keeps MLflow completely out of the production runtime. (The rejected alternative — fetching the model from MLflow at container startup — is covered in Considered but rejected designs.)

This design also serves the preferred post-NIA operating model, in which a non-expert at NGED operates the service day to day (see Requirements → Operating model & handover): there is no tracking server on the hot path to break, and the model simply freezes between OCF's scheduled expert interventions — under a vendor-develops / operator-runs split, that is a feature, not a limitation.

This is deliberately simpler than reusing BaseForecaster.load_from_mlflow's cache (the mechanism the CV pipeline already uses — see ML orchestration: model artifacts): v0.1 has no MLflow dependency to cache against in the first place.

Future work: once production wants to pick up a new champion without a rebuild + redeploy (e.g. after the XGBoost quick wins start landing regularly), switch to fetching the champion model from MLflow dynamically — at that point load_from_mlflow's local-disk cache becomes the production-resilience mechanism again (serving from disk on a cache hit so the live service survives an MLflow outage), exactly as it does for CV today.

Resolve repo-relative paths via a workspace marker, not directory depth

contracts.settings.PROJECT_ROOT anchors every repo-relative default in Settings — the CV fold definitions (conf/cv/default.yaml), the NWP metadata CSV (metadata/nwp_metadata.csv), the local data/ roots, and the .env location. It is resolved by walking up from the installed settings.py to the nearest ancestor directory holding uv.lock (the file that exists only at the uv workspace root), falling back to the current working directory when no ancestor qualifies.

We resolve via a marker rather than a hard-coded directory depth (the previous Path(__file__).parents[4]) because the depth only held for an editable install. Under the production image's uv sync --no-editable, the installed file sits at <venv>/lib/python3.14/site-packages/contracts/settings.py, where the same depth silently resolved to the venv root and every default broke with FileNotFoundError (#287). The marker walk handles both layouts: a dev checkout resolves to the repo root, and the production image resolves to /app because the Dockerfile copies uv.lock there alongside conf/ and metadata/ — so the image needs no per-path env-var overrides, and conf/model/ stays available for running training jobs in-container.

The fallback case — a wheel installed into a venv outside any workspace checkout — is a deployment shape we don't currently have. If one appears, it must either run with its working directory laid out like the repo root or set each path setting explicitly via its env var. (Packaging the resource files into a wheel via importlib.resources was considered for #287 and deliberately deferred until such a target actually exists.)

Run live inference in single-run mode, not bulk

The live_forecasts asset engineers features in single-run mode — an explicit power_fcst_init_time supplied by the partition, joined across all 51 NWP ensemble members — never bulk mode's one-power_fcst_init_time-per-NWP-run derivation (see ML orchestration: forecast cadence under-sampling for where bulk mode's per-NWP-run derivation matters instead: CV/backtesting, not live inference). A production run issues one forecast for one explicit init time every 6 hours; bulk mode's derivation has no meaning for a single live materialisation.

Resolve NWP availability asymmetrically: live vs replay

live_forecasts' availability_mode config resolves which NWP run to join against, and the two modes are deliberately asymmetric:

• "live" joins the freshest NWP run actually present in Delta, with no modelled publication delay — reality already constrains the table to genuinely published runs, so a faster provider is used automatically without a config change.
• "replay" joins the freshest run at least nwp_publication_delay_hours old, reconstructing what was genuinely available at that historical init time. Without the delay, a replay would leak NWP runs that only landed after the fact — a lookahead-bias bug, not just an inaccuracy.

The scheduled path always uses "live"; backfills of missed or historical partitions use "replay". The mode is an explicit, manually-set flag rather than an automatic live-iff-recent rule — the ambiguity of "recent" isn't worth resolving for a backfill path that's already manually triggered.

Serve only the trained population

live_forecasts forecasts exactly the production model's trained_time_series_ids (recorded in meta.json), never the current day's eligibility set. This is the train==predict population invariant: a time series the model never saw during training must never receive a live forecast, even if it would otherwise qualify today.

Promote the champion via a Dagster asset, not a script

The "researcher downloads artifacts" step above is a manually-triggered Dagster asset, promoted_model (config mlflow_run_id), rather than a bare script — promotion becomes a materialisation, giving an audit trail and lineage for free. The download logic itself (ml_core._production_helpers.fetch_model_artifacts) is a pure, asset-independent helper, so nothing about this decision couples it to Dagster.

The Docker build reuses this same asset (headlessly, via dagster asset materialize) — no separate fetch script was built, since a bare script would have duplicated the asset's audit trail for no benefit. The docker build step itself stays outside Dagster: it only ever runs on a laptop today, and image build/push becomes a CI-shaped concern once an MLflow tracking server and AWS infra exist — not something worth orchestrating through Dagster in the meantime.

Considered but rejected designs

Each design below was seriously considered for the live service and rejected. Every subsection follows the same structure: first it describes the rejected design, then it explains why we rejected it.

No control plane: EventBridge Scheduler launching ECS tasks

The design. AWS EventBridge Scheduler fires on the 6-hourly cadence and launches an ECS RunTask that executes dagster asset materialize (or dagster job execute) directly in the production image. Dagster acts purely as the in-process execution framework, EventBridge acts as the scheduler, and compute is paid per run with nothing always-on; run history could still be inspected by pointing an on-demand dagster-webserver at shared Postgres run storage.

Why we rejected it (July 2026). Four reasons:

1. Portability is a hard requirement. The entire stack must run on a local laptop (or any cloud) without AWS-specific services. Once the schedule lives in EventBridge and run-level retry lives in Step Functions, the stack no longer runs with docker compose up. The Dagster-on-a-VM design keeps the laptop and the cloud deployment the same artifact.

2. Handover to NGED is simpler without AWS coupling. A Dagster deployment NGED can run anywhere is an easier handover than an EventBridge + Step Functions + ECS arrangement they would have to recreate inside their own AWS account. (This reverses an earlier assumption that "a cron and a container" would be the easier handover — that only holds if the receiving organisation is committed to AWS.) See Handover to NGED for the full handover plan.

3. The cost saving is not real for this workload. EventBridge's pay-per-run economics matter when the alternative is an idle fleet, but our alternative is one small VM that must exist anyway to host MLflow and Marimo. Splitting the scheduler out of that box would save roughly $10–15/month while adding architectural surface area.

4. Run-level retry is no better under EventBridge. EventBridge Scheduler retries the invocation of a task (with backoff, a configurable max age/attempt count, and an SQS dead-letter queue for failed invocations), but once ECS accepts the task, EventBridge considers its job done — it does not notice a run that starts and then crashes. Relaunching a whole crashed run would have required wrapping the task in a Step Functions state machine using the .sync integration, while failures inside a run are already handled by Dagster's RetryPolicy on individual assets/ops, which works in-process with or without a daemon. So neither design gives run-level retry out of the box, and at four runs per day the existing replay/backfill mode plus alerting is a sufficient answer in both worlds.

EventBridge is also graph-blind: it fires a cron and passes a payload, so asset dependency ordering would have been handled entirely by Dagster within a single self-contained run of the full asset selection, and cross-run coordination — sensors, declarative automation, freshness-driven materialisation — would not have been available without the daemon.

Door left open. EventBridge remains usable later as a pure trigger — an external service that pokes an otherwise portable stack (e.g. S3 event → EventBridge rule → webhook or task launch) — provided the stack never depends on it to run. Event-driven behaviour (e.g. "run when NGED drops a new file") can alternatively be handled natively by Dagster sensors, which the always-on daemon supports.

A periodically-woken control plane

The design. Run the Dagster daemon only periodically (say, hourly) instead of continuously, to save cost; on each wake-up, the daemon's schedule catch-up (max_catchup_runs, default 5) would fire any 6-hourly ticks missed while it slept.

Why we rejected it. Sensors, run monitoring, and retries all assume an always-on daemon; the web UI would be unavailable most of the time; and something else would still need to reliably wake the daemon — which reintroduces the scheduling problem one level up.

An always-on Fargate service for the control plane

The design. Run the always-on control plane — daemon, webserver, code-location server, and Postgres — as a long-running ECS service on Fargate, instead of on an EC2 VM. The appeal is real: Fargate has no operating system for us to install or patch (AWS owns and maintains the hosts), so the one piece of self-maintained OS in the deployment would disappear.

Why we rejected it. Five reasons:

1. Fargate's premium pays off exactly where this workload can't use it. Fargate's per-second billing wins for compute that runs a few minutes per day — which is why the forecast worker runs there. For a box that runs 24/7, the same premium just compounds: at eu-west-2 on-demand rates, an always-on 2 vCPU / 4 GB Fargate service costs roughly 2.4× the t4g.medium VM it would replace (~$66/month vs ~$27/month, from the ARM Fargate and EC2 rates in the region price table).

2. The box is planned to host more than the control plane. The same VM is the intended home of the MLflow tracking server and the Marimo dashboard — which is why the EventBridge rejection counts the VM as a cost that exists anyway. A VM absorbs each additional long-running service as one more entry in the same docker-compose.yml, using headroom already paid for; as Fargate services, each would be its own always-on task paying the premium above over again — and MLflow brings its own persistence needs (its backend store and artifact root), running into the disk problem below a second time.

3. Postgres needs a disk that Fargate doesn't have. A Fargate task has no persistent local storage, so the Postgres container (run, event, and schedule history) could not come along. It would have to move to managed RDS — more cost, and one more AWS-specific service to recreate at handover — or onto EFS, a network filesystem that Postgres tolerates poorly. On the VM, Postgres's data is simply a Docker volume on the instance's own disk.

4. It trades a portable artifact for AWS-specific glue. The control plane's deployment description is its docker-compose.yml: the laptop and the cloud run the same artifact. As an ECS service, that description becomes task definitions, service configuration, and (per the previous point) RDS — the same AWS coupling that helped reject the EventBridge design, in both its portability and its handover form. The Tailscale-based access design adds a quieter version of the same friction: on the VM, joining the tailnet is one tailscale up, whereas a Fargate service needs a Tailscale sidecar container with its own state management.

5. The benefit is smaller than it looks. The OS burden Fargate would remove is one Ubuntu box that patches itself (unattended-upgrades), can be stopped for maintenance in any of the built-in 6-hourly maintenance windows (above), and is slated for a tested rebuild-from-scratch script (see Handover: de-pet the control-plane box) so that the answer to a sick box is destroy and recreate, never diagnose.

Running the data-ingest runs on the control-plane VM

The design. The recurring data-ingest jobs — the hourly NGED telemetry pull and the daily Dynamical.org NWP download — are deliberately much lighter than inference, so they could run directly on the always-on box instead of each being dispatched to its own Fargate task. That would save the per-run Fargate cost and the minute or so of task-provisioning latency each dispatch pays.

One sizing fact frames the whole question: Dagster launches runs, not assets, onto Fargate. A schedule tick launches one job run, which becomes one Fargate task, and every asset in that job's selection executes inside that single task — so a tiny asset only pays for a task of its own if it has a schedule of its own (the one-off h3_grid_weights, materialised by hand, never does). The question is therefore about the recurring scheduled runs, and only one of them is genuinely tiny: the hourly NGED telemetry poll — which, because NGED publishes new files only roughly every 5 hours, spends most of its ticks discovering there is nothing to do.

Why we rejected it. Five reasons:

1. Dagster has one run launcher per instance. The EcsRunLauncher configured in dagster.yaml applies to every run the daemon launches — there is no per-job "run this one locally" switch. "Ingest on the box, forecasts on Fargate" would therefore mean writing a custom hybrid launcher that reads a run tag and delegates to one of two launchers. That is only a few dozen lines, but it is bespoke glue on the critical path of every run — exactly what the handover constraint says to avoid.

2. Blast-radius isolation. The box's only job is to always be up, and it is deliberately tiny: a t4g.medium whose 4 GB is shared by the daemon, webserver, code-location server, and Postgres, with MLflow and Marimo planned on top. The ingest jobs are lighter than inference but not negligible — the NWP ingest decodes a full daily ENS run (~7 million rows) and is CPU-hungry enough that its download once starved itself through thread contention (#276). Several busy worker threads on a 2-vCPU box would starve daemon heartbeats and the UI, and a memory spike (upstream format drift, an unusually large file) risks the OOM killer taking out Postgres or the daemon. A bad data file killing an ephemeral task is a shrug; killing the control plane is the exact failure this architecture exists to avoid.

3. The box would have to grow. Hosting ingest means sizing the box for ingest peaks rather than for coordination — realistically a t4g.large at roughly double the t4g.medium's ~$27/month, which roughly cancels the Fargate saving.

4. One execution path. With everything on Fargate, every run has the same image, the same log destination (CloudWatch), the same IAM story (the task role), and the same debugging experience. Two execution environments means two sets of failure modes for an operator who is, post-NIA, a non-expert at NGED.

5. V2 scaling. At ~2,500 time series the ingest workload grows roughly 78×. On Fargate that is a task-size change; on the box it is another round of resizing the component that is hardest to touch.

What the accepted design costs. Every ingest run currently inherits the task definition's 4 vCPU / 16 GB — sized for inference's ~9 GB peak, and oversized for an hourly telemetry pull. At the eu-west-2 ARM rates in the region price table (~$0.21/hour for that task size), ~720 hourly runs of a few minutes each — most of them the no-ops described above — come to on the order of $5–10/month: the same magnitude of saving the EventBridge rejection dismissed as "not real for this workload". A second, quieter cost: ecmwf_ens's not-yet-published retry loop (RetryRequested, 30-minute waits) retries inside the same run, so on a late-publication day its 16 GB task sits idle for up to 4 hours, billed the whole time (still well under $1).

Door left open. If those figures ever start to matter, two fixes exist inside the accepted architecture, and neither moves work onto the box:

• Right-size the ingest tasks. EcsRunLauncher supports per-run ecs/cpu and ecs/memory tags, so the ingest jobs can declare a small task size while forecasts keep 4 vCPU / 16 GB — a config change, not an architecture change.
• Replace polling schedules with Dagster sensors (#324). Sensor evaluation functions run on the daemon — that is, on the always-on box — while only genuine runs go through the launcher. A sensor that cheaply lists NGED's bucket every few minutes and fires the ingest job only when new files actually exist puts the seconds-of-work detection on the box, where Dagster runs it natively, and launches Fargate only for the ~5 real ingests a day; the same pattern lets a sensor watch for NWP publication instead of holding a Fargate task idle through the retry loop. This is the hybrid that reason 1 said would need a custom launcher — except Dagster already ships it, because the tiny recurring work here is detection, and detection is what sensors are for.

Either way: shrink or skip the ingest tasks; don't move the work onto the box.

Fetching the champion model from MLflow at container startup

The design. Host the MLflow artifact root on S3 and have each production container fetch the champion model from it at startup, instead of baking the model into the image.

Why we rejected it. It adds runtime moving parts, needs tracking-store access from production, and slows cold starts — baking the model in has none of those costs. The rejection gets stronger under the preferred post-NIA operating model, in which NGED run this code themselves (see Requirements → Operating model & handover): with the model baked in, NGED never has to run — or depend on — an MLflow tracking server at all, and the model simply freezes until a new image arrives. (OCF may well continue developing the model post-NIA and release new container images for NGED to test, but that arrangement is TBD — and either way it only changes which image NGED runs, never whether their production runtime needs MLflow.)

Rejecting this design says nothing against MLflow itself — MLflow remains the backbone of ML experimentation: training runs log their models, configs, and metrics to it, and the champion is chosen from an MLflow leaderboard (see ML orchestration: model artifacts for the design, and ML experimentation for the day-to-day workflow). The boundary this rejection draws is between ML R&D and production: research uses MLflow constantly, while production's hot path never touches it. MLflow's job ends at the moment of promotion, when the chosen champion is copied out of the tracking store and baked into the image.

The idea may still return in a stronger form: the future work note at the end of Bake the model into the image at build time — the section describing the accepted design this one lost to — describes fetching the champion dynamically once redeploys become frequent, with load_from_mlflow's local-disk cache as the production-resilience mechanism.

See also

• Live service roadmap — the full v0.1 design, including the costed AWS architecture options behind the accepted architecture (small control-plane box + EcsRunLauncher) and its implementation workstreams.
• AWS Running Costs — what this architecture costs to run: the v1 estimate as deployed, and the projected estimate at v2 scale.
• Setting up the live service on AWS — the step-by-step runbook: promotion, image build/push, and the full AWS bring-up including the control-plane box.
• Setting up Sentry telemetry — the how-to for the error reporting and missed-check-in alarm designed above: get a DSN, test from a laptop, enable in production.
• Configuration reference — where data tables and local artifacts live, and how to point Settings at S3.
• ML Orchestration Design — why production inference doesn't reuse the CV pipeline's MLflow-artifact cache.
• Why Dagster, not Airflow? — why Dagster was chosen at design time, whether a migration to Airflow would be possible today, and what a port would take.