Skip to content

Getting started on your laptop

This is the single walkthrough for going from a fresh clone to a running Dagster instance that downloads real data and trains a model — entirely on your laptop, with no AWS account of your own to set up. Follow it top to bottom the first time; later pages go deeper on each part and are linked as you reach them.

Prerequisites

  • Python 3.14+ and uv — install uv following its official instructions. uv manages the Python version and the virtual environment for you, so you do not need to install Python separately.
  • Credentials for NGED's telemetry bucket — the URL, access key, and secret for the S3 bucket where NGED publishes its telemetry. Ask another team member if you do not have them. Nothing in this project works without them, because the raw power data lives there.

Step 1 — Clone and install the project

git clone https://github.com/openclimatefix/nged-substation-forecast.git
cd nged-substation-forecast
uv sync                    # create the virtualenv and install all workspace packages
uv run pre-commit install  # install the git hooks (lint, format, markdown, type-check on commit)

uv sync reads the workspace's lockfile and installs every package under packages/ plus the root Dagster application in one step.

Step 2 — Create your .env

Configuration is read from a .env file in the repository root (and from environment variables, which win over .env). Copy the committed template and fill in the three required NGED credentials:

cp .env.example .env

Then edit .env and set the three NGED_S3_BUCKET_* values from the prerequisites:

NGED_S3_BUCKET_URL=<nged source bucket url>
NGED_S3_BUCKET_ACCESS_KEY=<key>
NGED_S3_BUCKET_SECRET=<secret>

Those three are the only values you must set. Every other setting has a working default, so with nothing else in .env all data and artifacts live under <repo>/data as plain files on disk. The Configuration reference explains the full menu — the three storage roots, the derive-from-root convention, and the credentials you would add to move the data tables to S3.

.env is git-ignored; never commit real credentials.

Using a git worktree? Settings reads .env from the repository root, so a worktree needs its own. Symlink the main checkout's file rather than copying secrets around: ln -s ../<main-checkout>/.env .env.

Step 3 — Give Dagster a persistent home

Dagster works without any of this, but by default it forgets its run history and schedule state when you stop it. Point it at a persistent directory so that state survives a restart — this also matters later, because the live 6-hourly schedule only keeps time while the daemon runs continuously.

  1. Create the directory and a dagster.yaml inside it:

    mkdir -p ~/dagster_home

  2. Put the following into ~/dagster_home/dagster.yaml:

    storage:
  sqlite:
    base_dir: "dagster_history"

concurrency:
  pools:
    default_limit: 1  # Used to limit concurrency of the ecmwf_ens asset.

run_monitoring:
  # Without this, a crashed/killed run can leak its concurrency-pool slot (e.g. the pool
  # above) forever, since nothing else frees a slot held by a run that never reached a
  # normal finally-block exit. This lets the daemon self-heal: any run finished (in any
  # terminal status) for longer than the threshold has its slots freed automatically.
  enabled: true
  free_slots_after_run_end_seconds: 300

python_logs:
  managed_python_loggers:
    - nged_data
  python_log_level: DEBUG

  3. Tell Dagster where it is by adding export DAGSTER_HOME=~/dagster_home to your .bashrc (or equivalent) and restarting your terminal.

dagster.yaml is read only at process startup, so restart dg dev after editing it.

Step 4 — Start Dagster

uv run dg dev

Leave it running and open http://localhost:3000. Everything from here is driven from that UI.

Step 5 — Download data and train a model

In the Dagster UI, materialise the data assets in order — this is what pulls the real data onto your laptop. The quickest first run is the smoke_test fold (a ~1-month train / ~1-month validation window for a fast end-to-end check), so the concrete values below target it:

  1. power_time_series_and_metadata — pulls NGED telemetry from S3 into a local Delta table (unpartitioned).
  2. h3_grid_weights — computes the spatial weights the NWP download needs (unpartitioned, one-off).
  3. ecmwf_ens — downloads ECMWF ensemble weather, one daily partition at a time. Materialise the partitions covering the fold's train and validation window; for smoke_test that is 2025-01-01 through 2025-02-28. Do not "Materialise all" on a first run — the full archive back to 2024-04-01 is billions of rows.
  4. eligible_time_series — determines which series have enough coverage for a fold. This asset is fold-partitioned, so materialise the smoke_test partition (not mid_2025_to_mid_2026) to match the run below — otherwise training finds an empty population and raises.

Then register an experiment and train a model. The full recipe — the run config for each job, what smoke_test versus full_cv does, and how the trained model is tracked in MLflow — is Running an ML experiment end-to-end. Register the experiment with run_mode="smoke_test" to check the whole pipeline is wired up before committing to a long full_cv training run.

Where to go next