Delta Store API

`delta_store`

Physical storage policy for the project's Delta tables.

Why this package exists

contracts owns each table's logical shape and meaning. This package owns its physical layout: parquet writer properties (codec + per-column encodings), compression-friendly sort orders, and significand-precision rounding, plus the write helpers that apply them. Dagster assets stay thin by writing through this package rather than calling write_deltalake with ad-hoc settings — and it becomes impossible to land rows in a table without its storage format applied.

The flagship example is the internal power_forecasts table: ZSTD + DELTA_BINARY_PACKED timestamps + BYTE_STREAM_SPLIT floats + member-adjacent sorting + rounding power_fcst to a 13-bit significand shrank the 403.6M-row development table from 6.33 GB to 0.73 GB.

precision.round_to_significand_bits() — rounds a Float32 expression to a chosen number of significand bits in pure Polars arithmetic (Veltkamp splitting), zeroing the low mantissa bits so BYTE_STREAM_SPLIT + zstd can compress them away. The trick and its preconditions are rigorously documented on the function.
power_forecasts — the power_forecasts table's writer properties, sort order, precision policy, and write_power_forecasts().

`delta_store.precision`

Reduce the significand precision of Float32 columns so parquet compression can work.

Nearly every full-precision Float32 value in a large forecast or weather table is distinct, so the low significand bits are incompressible noise — they defeat every general-purpose codec. Rounding each value to a small number of significand bits zeroes those low bits, after which BYTE_STREAM_SPLIT + zstd compress the column dramatically (see delta_store.power_forecasts for measured numbers), at the cost of a strictly bounded relative error.

The rounding here is pure Polars arithmetic (no bit-twiddling, no numpy round-trip), which works because of a classical floating-point identity — Veltkamp splitting — documented in detail on :func:round_to_significand_bits.

Attributes

`FLOAT32_SIGNIFICAND_BITS = 24` `module-attribute`

IEEE 754 binary32 significand precision p: 23 explicit fraction bits + 1 implicit bit.

Functions:

`round_to_significand_bits(expr, *, keep_bits)`

Round a Float32 expression to keep_bits significand bits (round-to-nearest).

The result is exactly representable with a keep_bits-bit significand, so the low 24 - keep_bits explicit fraction bits of every finite output are zero — which is what lets BYTE_STREAM_SPLIT + zstd compress the column — and the relative error is bounded by the unit roundoff of a keep_bits-bit format:

|result - x| <= 2**-keep_bits * |x|

(e.g. keep_bits=13 -> max relative error 2^-13 ~= 1.2e-4). Rounding is to nearest, so unlike truncation it introduces no systematic bias toward zero.

How it works — Veltkamp splitting. With s = 24 - keep_bits and the constant C = 2**s + 1, the expression computes, entirely in Float32 round-to-nearest arithmetic (RN)::

c = RN(x * C)  # = RN(x*2^s + x)
result = RN(c - RN(c - x))

Veltkamp's theorem (Veltkamp 1968; Dekker 1971, "A floating-point technique for extending the available precision", Numerische Mathematik 18; Muller et al., Handbook of Floating-Point Arithmetic, 2nd ed., §4.4, Algorithm 4.9 "Split") states that for 2 <= s <= p - 2 and no overflow, both subtractions are exact and result is x rounded to nearest onto p - s = keep_bits significand bits. The intuition: x*C stacks a copy of x shifted s exponent positions above itself; rounding that sum to p bits (forming c) is precisely what discards — with correct rounding — the low s bits of the original x; the two exact subtractions then peel the shifted copy back off, leaving x with its low s significand bits rounded away.

Preconditions — each one is load-bearing:

expr must be Float32. The splitting constant is pinned to Float32, but Polars promotes mixed arithmetic: if a Float64 expression is passed, the whole computation runs at p = 53 and silently keeps 53 - s bits instead of keep_bits. (A .cast(pl.Float32) is deliberately not applied here — silently changing the dtype of a Float64 column would be its own trap; callers own their dtypes.)
Arithmetic must be evaluated operation-by-operation in IEEE round-to-nearest. Polars (Rust) guarantees this: floats are never reassociated and a*b + c is never contracted into an FMA, either of which would break the exactness of the subtractions.
Non-finite and overflow cases are guarded: for |x| > f32_max / C the product c overflows to inf and the naive result would be inf - inf = NaN; for x NaN or ±inf, c is likewise non-finite. Wherever c is non-finite the input value is passed through unchanged (a huge-but-finite value is stored at full precision rather than corrupted; NaN/±inf survive verbatim).
Subnormal x (|x| < 2**-126) degrades gracefully: the relative-error bound loosens because precision is already at the absolute floor of the format, but the result is still a faithful nearby value.

Parameters:

Name	Type	Description	Default
`expr`	`Expr`	A `Float32` expression (see the dtype precondition above).	required
`keep_bits`	`int`	Significand bits to keep, in `[2, 22]` (the theorem's `2 <= s <= p - 2`). Note this counts significand bits — the implicit leading 1 plus explicit fraction bits — so `keep_bits=13` keeps 12 explicit fraction bits.	required

Returns:

Type	Description
`Expr`	A `Float32` expression: `expr` rounded to `keep_bits` significand bits, with
`Expr`	non-finite and overflowing inputs passed through unchanged.

Source code in packages/delta_store/src/delta_store/precision.py

def round_to_significand_bits(expr: pl.Expr, *, keep_bits: int) -> pl.Expr:
    """Round a ``Float32`` expression to ``keep_bits`` significand bits (round-to-nearest).

    The result is exactly representable with a ``keep_bits``-bit significand, so the low
    ``24 - keep_bits`` explicit fraction bits of every finite output are zero — which is what
    lets ``BYTE_STREAM_SPLIT`` + zstd compress the column — and the relative error is bounded by
    the unit roundoff of a ``keep_bits``-bit format:

        |result - x| <= 2**-keep_bits * |x|

    (e.g. ``keep_bits=13`` -> max relative error 2^-13 ~= 1.2e-4). Rounding is to nearest, so
    unlike truncation it introduces no systematic bias toward zero.

    **How it works — Veltkamp splitting.** With ``s = 24 - keep_bits`` and the constant
    ``C = 2**s + 1``, the expression computes, entirely in ``Float32`` round-to-nearest
    arithmetic (``RN``)::

        c = RN(x * C)  # = RN(x*2^s + x)
        result = RN(c - RN(c - x))

    Veltkamp's theorem (Veltkamp 1968; Dekker 1971, "A floating-point technique for extending
    the available precision", *Numerische Mathematik* 18; Muller et al., *Handbook of
    Floating-Point Arithmetic*, 2nd ed., §4.4, Algorithm 4.9 "Split") states that for
    ``2 <= s <= p - 2`` and no overflow, both subtractions are **exact** and ``result`` is ``x``
    rounded to nearest onto ``p - s = keep_bits`` significand bits. The intuition: ``x*C``
    stacks a copy of ``x`` shifted ``s`` exponent positions above itself; rounding that sum to
    ``p`` bits (forming ``c``) is precisely what discards — with correct rounding — the low
    ``s`` bits of the original ``x``; the two exact subtractions then peel the shifted copy back
    off, leaving ``x`` with its low ``s`` significand bits rounded away.

    **Preconditions — each one is load-bearing:**

    - ``expr`` must be ``Float32``. The splitting constant is pinned to ``Float32``, but Polars
      promotes mixed arithmetic: if a ``Float64`` expression is passed, the whole computation
      runs at ``p = 53`` and silently keeps ``53 - s`` bits instead of ``keep_bits``. (A
      ``.cast(pl.Float32)`` is deliberately *not* applied here — silently changing the dtype of
      a ``Float64`` column would be its own trap; callers own their dtypes.)
    - Arithmetic must be evaluated operation-by-operation in IEEE round-to-nearest. Polars
      (Rust) guarantees this: floats are never reassociated and ``a*b + c`` is never contracted
      into an FMA, either of which would break the exactness of the subtractions.
    - Non-finite and overflow cases are guarded: for ``|x| > f32_max / C`` the product ``c``
      overflows to ``inf`` and the naive result would be ``inf - inf = NaN``; for ``x`` NaN or
      ``±inf``, ``c`` is likewise non-finite. Wherever ``c`` is non-finite the input value is
      passed through unchanged (a huge-but-finite value is stored at full precision rather than
      corrupted; ``NaN``/``±inf`` survive verbatim).
    - Subnormal ``x`` (``|x| < 2**-126``) degrades gracefully: the relative-error bound loosens
      because precision is already at the absolute floor of the format, but the result is still
      a faithful nearby value.

    Args:
        expr: A ``Float32`` expression (see the dtype precondition above).
        keep_bits: Significand bits to keep, in ``[2, 22]`` (the theorem's ``2 <= s <= p - 2``).
            Note this counts *significand* bits — the implicit leading 1 plus explicit fraction
            bits — so ``keep_bits=13`` keeps 12 explicit fraction bits.

    Returns:
        A ``Float32`` expression: ``expr`` rounded to ``keep_bits`` significand bits, with
        non-finite and overflowing inputs passed through unchanged.
    """
    shift = FLOAT32_SIGNIFICAND_BITS - keep_bits
    if not 2 <= shift <= FLOAT32_SIGNIFICAND_BITS - 2:
        raise ValueError(
            f"keep_bits must be in [2, {FLOAT32_SIGNIFICAND_BITS - 2}], got {keep_bits}"
        )
    splitter = pl.lit(float(2**shift + 1), dtype=pl.Float32)
    c = expr * splitter
    rounded = c - (c - expr)
    return pl.when(c.is_finite()).then(rounded).otherwise(expr)

`delta_store.power_forecasts`

Storage policy for the internal power_forecasts Delta table.

Owns everything about how PowerForecast rows are laid out on disk: the parquet writer properties (codec + per-column encodings), the compression-friendly row order, and the power_fcst precision reduction. Callers write through :func:write_power_forecasts so it is impossible to land rows in the table without this format applied.

Measured impact: rewriting the full 403.6M-row development table into this format shrank it from 6.33 GB (delta-rs defaults: SNAPPY, dictionary encoding, unsorted, full precision) to 0.73 GB. See the POWER_FORECASTS_WRITER_PROPERTIES docstring for the per-lever breakdown.

Attributes

`POWER_FCST_SIGNIFICAND_BITS = 13` `module-attribute`

Significand bits kept when storing power_fcst (1 implicit + 12 explicit fraction bits).

Rounding to nearest at 13 significand bits caps the relative error at 2⁻¹³ ≈ 1.2×10⁻⁴ — orders of magnitude below forecast error — and zeroes the 11 low fraction bits, which are otherwise pure entropy that defeats every compression codec (nearly every full-precision power_fcst value is distinct). Mirrors the significand-rounding scheme delta_store.nwp applies to NWP data — but with different writer properties; see that module's docstring for why the choice doesn't transfer between tables. See POWER_FORECASTS_WRITER_PROPERTIES for the measured size impact.

`POWER_FORECASTS_SORT_COLS = PowerForecast.PRIMARY_KEY` `module-attribute`

Within-file row order for power_forecasts writes.

Placing the ~51 ensemble members of one (series, init time, valid time) target on adjacent rows makes power_fcst locally smooth and the timestamp columns stepped sequences — exactly what the BYTE_STREAM_SPLIT and DELTA_BINARY_PACKED encodings in POWER_FORECASTS_WRITER_PROPERTIES need to compress well. Leading with time_series_id also lets parquet row-group statistics prune scans that filter on one series.

Defined as PowerForecast.PRIMARY_KEY rather than repeating its columns, because the set that identifies a row uniquely is exactly the set whose adjacency makes a row compress. The order is this module's own choice and is pinned by test_sort_cols_lead_with_time_series_id — a reordered primary key is still a correct key but a worse layout, so the tuple is not free to follow one.

`POWER_FORECASTS_WRITER_PROPERTIES = WriterProperties(compression='ZSTD', compression_level=3, column_properties={'valid_time': _TIMESTAMP_COLUMN_PROPERTIES, 'power_fcst_init_time': _TIMESTAMP_COLUMN_PROPERTIES, 'nwp_init_time': _TIMESTAMP_COLUMN_PROPERTIES, 'power_fcst': ColumnProperties(encoding='BYTE_STREAM_SPLIT', dictionary_enabled=False)})` `module-attribute`

Parquet writer settings for the power_forecasts Delta table.

The delta-rs defaults (SNAPPY + dictionary encoding everywhere) leave power_fcst — ~76% of the bytes — essentially uncompressed. Measured on the table's largest single file (18.3M rows, 105 MB): ZSTD-3 alone → 80% of the original size; adding these column encodings plus the POWER_FORECASTS_SORT_COLS row order → 50%; adding the POWER_FCST_SIGNIFICAND_BITS precision reduction → 29%. (That file is the table's least compressible slice — across the full table the format achieved 11.5%.)

Classes

Functions:

`write_power_forecasts(forecasts, table_uri, *, replace_partition=None, replace_predicate_extra=None, storage_options=None)`

Write PowerForecast rows to the power_forecasts Delta table in its storage format.

Applies the three storage levers before writing: rounds power_fcst to POWER_FCST_SIGNIFICAND_BITS significand bits, sorts rows by POWER_FORECASTS_SORT_COLS, and writes with POWER_FORECASTS_WRITER_PROPERTIES.

The table is partitioned by (experiment_name, fold_id). A multi-chunk materialisation passes replace_partition on its first chunk — overwriting that partition so prior rows are always replaced, even when the first chunk is empty — and None (append) on the rest.

Parameters:

Name	Type	Description	Default
`forecasts`	`DataFrame[PowerForecast]`	Validated forecast rows. `experiment_name` / `fold_id` are `String` (their `PowerForecast` dtype), which is exactly what delta-rs needs for Hive-style partition directories — no cast required.	required
`table_uri`	`str \| Path`	Path or URI of the `power_forecasts` Delta table.	required
`replace_partition`	`tuple[str, str] \| None`	`(experiment_name, fold_id)` to overwrite, or `None` to append. Passed explicitly rather than derived from `forecasts` so an empty first chunk still clears the partition.	`None`
`replace_predicate_extra`	`str \| None`	An additional `AND`-ed SQL predicate clause narrowing the overwrite below the `(experiment_name, fold_id)` partition — e.g. `"power_fcst_init_time = '2026-07-04T06:00:00+00:00'"` so `live_forecasts` can replace one 6-hourly slot's rows without wiping the rest of the `"live"` fold's partition. delta-rs' `replaceWhere` supports predicates on non-partition columns (confirmed empirically: a `datetime.isoformat()` literal round-trips correctly against a `Timestamp` column). Only meaningful alongside `replace_partition`.	`None`
`storage_options`	`ObjectStoreOptions \| None`	delta-rs object-store options (credentials/endpoint) for a remote `table_uri`; `None`/empty for a local path.	`None`

Source code in packages/delta_store/src/delta_store/power_forecasts.py

def write_power_forecasts(
    forecasts: pt.DataFrame[PowerForecast],
    table_uri: str | Path,
    *,
    replace_partition: tuple[str, str] | None = None,
    replace_predicate_extra: str | None = None,
    storage_options: ObjectStoreOptions | None = None,
) -> None:
    """Write ``PowerForecast`` rows to the ``power_forecasts`` Delta table in its storage format.

    Applies the three storage levers before writing: rounds ``power_fcst`` to
    ``POWER_FCST_SIGNIFICAND_BITS`` significand bits, sorts rows by
    ``POWER_FORECASTS_SORT_COLS``, and writes with ``POWER_FORECASTS_WRITER_PROPERTIES``.

    The table is partitioned by ``(experiment_name, fold_id)``. A multi-chunk materialisation
    passes ``replace_partition`` on its first chunk — overwriting that partition so prior rows
    are always replaced, even when the first chunk is empty — and ``None`` (append) on the rest.

    Args:
        forecasts: Validated forecast rows. ``experiment_name`` / ``fold_id`` are ``String``
            (their ``PowerForecast`` dtype), which is exactly what delta-rs needs for Hive-style
            partition directories — no cast required.
        table_uri: Path or URI of the ``power_forecasts`` Delta table.
        replace_partition: ``(experiment_name, fold_id)`` to overwrite, or ``None`` to append.
            Passed explicitly rather than derived from ``forecasts`` so an *empty* first chunk
            still clears the partition.
        replace_predicate_extra: An additional ``AND``-ed SQL predicate clause narrowing the
            overwrite below the ``(experiment_name, fold_id)`` partition — e.g.
            ``"power_fcst_init_time = '2026-07-04T06:00:00+00:00'"`` so ``live_forecasts`` can
            replace one 6-hourly slot's rows without wiping the rest of the ``"live"`` fold's
            partition. delta-rs' ``replaceWhere`` supports predicates on non-partition columns
            (confirmed empirically: a `datetime.isoformat()` literal round-trips correctly
            against a ``Timestamp`` column). Only meaningful alongside ``replace_partition``.
        storage_options: delta-rs object-store options (credentials/endpoint) for a remote
            ``table_uri``; ``None``/empty for a local path.
    """
    prepared = (
        forecasts.with_columns(
            power_fcst=round_to_significand_bits(
                pl.col("power_fcst"), keep_bits=POWER_FCST_SIGNIFICAND_BITS
            )
        )
        .sort(*POWER_FORECASTS_SORT_COLS)
        .to_arrow()
    )
    if replace_partition is not None:
        experiment_name, fold_id = replace_partition
        predicate = f"experiment_name = '{experiment_name}' AND fold_id = '{fold_id}'"
        if replace_predicate_extra is not None:
            predicate = f"{predicate} AND {replace_predicate_extra}"
        write_deltalake(
            table_or_uri=table_uri,
            data=prepared,
            mode="overwrite",
            predicate=predicate,
            partition_by=["experiment_name", "fold_id"],
            writer_properties=POWER_FORECASTS_WRITER_PROPERTIES,
            storage_options=typeddict_to_dict(storage_options),
        )
    else:
        write_deltalake(
            table_or_uri=table_uri,
            data=prepared,
            mode="append",
            partition_by=["experiment_name", "fold_id"],
            writer_properties=POWER_FORECASTS_WRITER_PROPERTIES,
            storage_options=typeddict_to_dict(storage_options),
        )

`delta_store.nwp`

Storage policy for the nwp Delta table.

Owns everything about how Nwp rows are laid out on disk: the parquet writer properties, the compression-friendly row order, and the significand-precision reduction of the continuous weather variables. Callers write through :func:write_nwp so it is impossible to land rows in the table without this format applied.

Stores plain Float32 + delta_store.precision.round_to_significand_bits — the technique used by delta_store.power_forecasts, but with different writer properties. Measured on real NWP data (9 partitions spread across two years of history): BYTE_STREAM_SPLIT made every continuous column larger, not smaller — the opposite of the power_forecasts result. Working hypothesis: significand rounding collapses NWP values into a small set of repeats (many H3 cells / ensemble members round to the same value), which Parquet's default dictionary+RLE encoding captures directly; BYTE_STREAM_SPLIT scatters that repetition across four separate byte planes and loses more than it gains. power_forecasts's target values have no such repetition (near-continuous ML output), so BYTE_STREAM_SPLIT wins there instead — the two tables need different writer properties. See https://openclimatefix.github.io/nged-substation-forecast/architecture/overview/ for the measured GB/yr and read-latency numbers.

Attributes

`NWP_SIGNIFICAND_BITS = 13` `module-attribute`

Significand bits kept for every continuous NWP variable (1 implicit + 12 explicit fraction bits) — the same budget as delta_store.power_forecasts.POWER_FCST_SIGNIFICAND_BITS. Caps the relative error at 2⁻¹³ ≈ 1.2×10⁻⁴. Measured max absolute error on real data: ≤0.004 °C for temperature, ≤8 Pa (0.08 hPa) for mean-sea-level pressure — both well inside tolerance (temperature ≤0.25 K, MSL pressure ≤1 hPa).

`NWP_SORT_COLS = ('init_time', 'ensemble_member', 'valid_time', 'h3_index')` `module-attribute`

Within-file row order for nwp writes — member before valid_time (the opposite priority from power_forecasts, which sorts member-adjacent for a different reason: there it's about compressing near-duplicate ensemble values; here it's about row-group pruning). Sorting ensemble_member early means each ~1M-row Parquet row group spans only a handful of member values instead of all ~51, so a single-member predicate (the control-member read every training run does) can skip most row groups via min/max stats instead of decoding the whole partition — provided that predicate reaches the Parquet scan unchanged, which requires Nwp.scan_delta's cast to be a no-op (see the Nwp.ensemble_member field). The speed and storage cost of this ordering versus a valid_time-first sort need re-measuring against real production data; the old figures predate a period where that cast was not a no-op and are not restored here.

`NWP_WRITER_PROPERTIES = WriterProperties(compression='ZSTD', compression_level=3)` `module-attribute`

Deliberately no per-column encoding overrides (no BYTE_STREAM_SPLIT, DELTA_BINARY_PACKED, or disabled dictionary encoding) — see this module's docstring for why that choice, which won for power_forecasts, measures worse here.

Classes

Functions:

`write_nwp(nwp, table_uri, storage_options=None)`

Write one NWP run into the nwp Delta table in its storage format.

Rounds every continuous weather variable to NWP_SIGNIFICAND_BITS significand bits, sorts rows by NWP_SORT_COLS, and writes with NWP_WRITER_PROPERTIES. The table is partitioned by (nwp_model_id, init_time), matching Nwp.scan_delta's partition-pruning assumptions; the first write creates the table.

The write replaces the (nwp_model_id, init_time) partition named by the frame's first row, so re-materialising an ecmwf_ens partition leaves one copy of the run. delta-rs checks every row against that predicate and rejects the whole write, table untouched, if any row falls outside it (confirmed empirically against deltalake 1.6.2, locally and on S3, on a partition column despite its percent-encoded Hive directory name). Two materialisations of the same partition at once contend and the loser raises CommitFailedError; disjoint partitions do not.

schema_mode="overwrite" is passed on every write, not just to migrate an old table. This safety claim holds only for a widening contract change, which is the only kind this function has been used for so far: confirmed empirically (deltalake 1.6.2) that widening updates only the table's logical schema (the _delta_log metadata) to match the incoming frame's dtypes, and leaves every other partition's physical Parquet bytes untouched — a later read at the new logical dtype is correct and lossless even for a partition still physically stored at an older, narrower dtype. Since nwp's input is always an already-validated pt.DataFrame[Nwp], carrying the full column set at the current contract's dtypes, the only way this can ever change the table's schema is a deliberate future Nwp dtype change like this one — it cannot silently drop a column.

A narrowing contract change is a different, worse failure mode, also confirmed empirically: the write that narrows a column succeeds silently — schema_mode="overwrite" accepts it at write time — and the table is left with a logical schema that its own, previously-written partitions can no longer safely satisfy. Nothing fails until a later read of the whole table (by anyone, not necessarily the writer that broke it), which then raises SchemaError: incoming dtype cannot safely cast to target dtype — a confusing error far removed from its cause, especially if further partitions land on the bad schema before anyone reads across all of them. Without schema_mode="overwrite", delta-rs instead auto-widens the incoming (narrower) data up to the table's existing (wider) schema at write time, and the table stays readable throughout. This is an accepted, disclosed risk of leaving the flag on permanently, not a defect: nothing in this contract's own dtype-widening history has ever narrowed a column, and there is no guard against a future change that does.

Parameters:

Name	Type	Description	Default
`nwp`	`DataFrame[Nwp]`	Validated, non-empty NWP rows for a single `(nwp_model_id, init_time)` partition.	required
`table_uri`	`str \| Path`	Path or URI of the `nwp` Delta table.	required
`storage_options`	`ObjectStoreOptions \| None`	delta-rs object-store options (credentials/endpoint) for a remote `table_uri`; `None`/empty for a local path.	`None`

Source code in packages/delta_store/src/delta_store/nwp.py

def write_nwp(
    nwp: pt.DataFrame[Nwp],
    table_uri: str | Path,
    storage_options: ObjectStoreOptions | None = None,
) -> None:
    """Write one NWP run into the ``nwp`` Delta table in its storage format.

    Rounds every continuous weather variable to ``NWP_SIGNIFICAND_BITS`` significand bits, sorts
    rows by ``NWP_SORT_COLS``, and writes with ``NWP_WRITER_PROPERTIES``. The table is
    partitioned by ``(nwp_model_id, init_time)``, matching ``Nwp.scan_delta``'s
    partition-pruning assumptions; the first write creates the table.

    The write **replaces** the ``(nwp_model_id, init_time)`` partition named by the frame's first
    row, so re-materialising an ``ecmwf_ens`` partition leaves one copy of the run. delta-rs checks
    every row against that predicate and rejects the whole write, table untouched, if any row falls
    outside it (confirmed empirically against ``deltalake`` 1.6.2, locally and on S3, on a
    partition column despite its percent-encoded Hive directory name). Two materialisations of the
    *same* partition at once contend and the loser raises ``CommitFailedError``; disjoint
    partitions do not.

    ``schema_mode="overwrite"`` is passed on every write, not just to migrate an old table. This
    safety claim holds only for a **widening** contract change, which is the only kind this
    function has been used for so far: confirmed empirically (``deltalake`` 1.6.2) that widening
    updates only the table's *logical* schema (the ``_delta_log`` metadata) to match the incoming
    frame's dtypes, and leaves every other partition's physical Parquet bytes untouched — a later
    read at the new logical dtype is correct and lossless even for a partition still physically
    stored at an older, narrower dtype. Since ``nwp``'s input is always an already-validated
    ``pt.DataFrame[Nwp]``, carrying the full column set at the *current* contract's dtypes, the
    only way this can ever change the table's schema is a deliberate future ``Nwp`` dtype change
    like this one — it cannot silently drop a column.

    A **narrowing** contract change is a different, worse failure mode, also confirmed
    empirically: the write that narrows a column succeeds silently — ``schema_mode="overwrite"``
    accepts it at write time — and the table is left with a logical schema that its own,
    previously-written partitions can no longer safely satisfy. Nothing fails until a *later* read
    of the whole table (by anyone, not necessarily the writer that broke it), which then raises
    ``SchemaError: incoming dtype cannot safely cast to target dtype`` — a confusing error far
    removed from its cause, especially if further partitions land on the bad schema before anyone
    reads across all of them. Without ``schema_mode="overwrite"``, delta-rs instead auto-widens
    the incoming (narrower) data up to the table's existing (wider) schema at write time, and the
    table stays readable throughout. This is an accepted, disclosed risk of leaving the flag on
    permanently, not a defect: nothing in this contract's own dtype-widening history has ever
    narrowed a column, and there is no guard against a future change that does.

    Args:
        nwp: Validated, non-empty NWP rows for a single ``(nwp_model_id, init_time)`` partition.
        table_uri: Path or URI of the ``nwp`` Delta table.
        storage_options: delta-rs object-store options (credentials/endpoint) for a remote
            ``table_uri``; ``None``/empty for a local path.
    """
    continuous_vars = sorted(Nwp.continuous_var_names())
    rounded = nwp.with_columns(
        **{
            var: round_to_significand_bits(pl.col(var), keep_bits=NWP_SIGNIFICAND_BITS)
            for var in continuous_vars
        }
    ).sort(*NWP_SORT_COLS)

    prepared = rounded.to_arrow()

    write_deltalake(
        table_or_uri=table_uri,
        data=prepared,
        mode="overwrite",
        schema_mode="overwrite",
        predicate=(
            f"nwp_model_id = '{nwp.item(0, 'nwp_model_id')}' "
            f"AND init_time = '{nwp.item(0, 'init_time').isoformat()}'"
        ),
        partition_by=["nwp_model_id", "init_time"],
        writer_properties=NWP_WRITER_PROPERTIES,
        storage_options=typeddict_to_dict(storage_options),
    )

Delta Store API

delta_store

Why this package exists

Contents

delta_store.precision

Attributes

FLOAT32_SIGNIFICAND_BITS = 24 module-attribute

Functions:

round_to_significand_bits(expr, *, keep_bits)

delta_store.power_forecasts

Attributes

POWER_FCST_SIGNIFICAND_BITS = 13 module-attribute

POWER_FORECASTS_SORT_COLS = PowerForecast.PRIMARY_KEY module-attribute

Classes

Functions:

write_power_forecasts(forecasts, table_uri, *, replace_partition=None, replace_predicate_extra=None, storage_options=None)

delta_store.nwp

Attributes

NWP_SIGNIFICAND_BITS = 13 module-attribute

NWP_SORT_COLS = ('init_time', 'ensemble_member', 'valid_time', 'h3_index') module-attribute

NWP_WRITER_PROPERTIES = WriterProperties(compression='ZSTD', compression_level=3) module-attribute

Classes

Functions:

write_nwp(nwp, table_uri, storage_options=None)

`delta_store`

`delta_store.precision`

`FLOAT32_SIGNIFICAND_BITS = 24` `module-attribute`

`round_to_significand_bits(expr, *, keep_bits)`

`delta_store.power_forecasts`

`POWER_FCST_SIGNIFICAND_BITS = 13` `module-attribute`

`POWER_FORECASTS_SORT_COLS = PowerForecast.PRIMARY_KEY` `module-attribute`

`write_power_forecasts(forecasts, table_uri, *, replace_partition=None, replace_predicate_extra=None, storage_options=None)`

`delta_store.nwp`

`NWP_SIGNIFICAND_BITS = 13` `module-attribute`

`NWP_SORT_COLS = ('init_time', 'ensemble_member', 'valid_time', 'h3_index')` `module-attribute`

`NWP_WRITER_PROPERTIES = WriterProperties(compression='ZSTD', compression_level=3)` `module-attribute`

`write_nwp(nwp, table_uri, storage_options=None)`