Our Approach to MLops

A short explainer about this project's approach to MLops tooling: what this project's experiment automation changes, and why it makes the path to production safer rather than riskier.

Modern MLops (as used in this project) changes two things:

1. Throughput. The grunt work of an experiment — assembling features, training, cross-validating, recording results — is automated, so a small team can run hundreds of experiments per month instead of one or two. Humans are still the ones deciding what to try; the infrastructure runs those experiments. (See Running an ML experiment end-to-end.)
2. No translation gap. The artifact we experimented on is the artifact we deploy. There is no "now rewrite the research code for production" step, because every experiment runs on the exact same code as the production pipeline from the start. The gap is closed by raising research to the production standard, not by lowering production to accept a research notebook: an idea can be explored anywhere, but it only becomes a runnable experiment once it lives in the pipeline's own code.

An analogy

Traditional ML R&D is a chef inventing dishes in their home kitchen: every winning recipe has to be laboriously re-created on the restaurant's equipment before it can go on the menu, and much is lost (or silently changed) in translation. We are building the restaurant where R&D happens on the service line itself: hundreds of tastings a month, every dish judged by the same tasting panel, and the winning dish on the menu the same night — because nothing about it needs translating.

Nothing gets rewritten on the way to production

The model that wins the evaluation is, bit for bit, the model we deploy — not a re-implementation of it. Promotion to production takes minutes, and that speed is a consequence of rigour, not a trade against it: by the time promotion is on the table, the candidate has already been trained, cross-validated (see Cross-validation folds), and evaluated on the same pipeline, under the same standardised protocol, as every model before it.

That is what makes a one-command promotion safe to press rather than merely quick. The largest risk in a conventional setup — that the artifact measured and the artifact deployed are two different pieces of code — does not exist here; the comparison that picked the winner was made against every other candidate on identical folds; and the way back to the previous champion is a single command too. A fast promotion route that nobody trusts enough to use is worth no more than a slow one.