https://modelmachines.io/ Essays on production machine learning infrastructure: evaluation design, inference latency and cost, model and data versioning, drift detection, retraining triggers, feature pipelines and batching. en https://modelmachines.io/why-offline-gains-vanish-online/ https://modelmachines.io/why-offline-gains-vanish-online/ Wed, 29 Jul 2026 00:00:00 +0000 Evaluation The metric improved, the launch did nothing. There are a small number of mechanisms that cause this, and each of them can be checked before you spend a release cycle finding out. https://modelmachines.io/feature-stores-solve-a-skew-problem/ https://modelmachines.io/feature-stores-solve-a-skew-problem/ Sat, 18 Jul 2026 00:00:00 +0000 Data The case for a feature store is not storage or reuse. It is that training and serving compute the same feature twice, in different code, at different moments — and the two answers diverge. https://modelmachines.io/drift-is-a-question-not-an-alarm/ https://modelmachines.io/drift-is-a-question-not-an-alarm/ Fri, 10 Jul 2026 00:00:00 +0000 Monitoring Input distributions move constantly and most of the movement is harmless. Wiring a retraining trigger to a drift statistic buys expensive churn; tying it to outcomes buys something worth having. https://modelmachines.io/versioning-the-model-means-versioning-the-data/ https://modelmachines.io/versioning-the-model-means-versioning-the-data/ Wed, 01 Jul 2026 00:00:00 +0000 Reproducibility A model checkpoint is one member of a tuple. Storing it without the data, code and configuration that produced it gives you an artefact you can serve and cannot explain. https://modelmachines.io/batching-is-a-scheduling-policy/ https://modelmachines.io/batching-is-a-scheduling-policy/ Mon, 22 Jun 2026 00:00:00 +0000 Serving Grouping requests is usually treated as a throughput trick. It is really a scheduling decision that determines who waits, for how long, and what happens when memory runs out. https://modelmachines.io/latency-and-cost-are-the-same-dial/ https://modelmachines.io/latency-and-cost-are-the-same-dial/ Sat, 13 Jun 2026 00:00:00 +0000 Serving Serving cheaply and serving fast are opposing settings of one control. Understanding why makes capacity planning tractable and stops teams chasing a target that only exists on an idle machine. https://modelmachines.io/evaluation-that-survives-production/ https://modelmachines.io/evaluation-that-survives-production/ Thu, 04 Jun 2026 00:00:00 +0000 Evaluation An evaluation suite earns its keep by failing when the product would fail. Most suites fail somewhere else entirely — on a frozen set, against an average that hides the cases anyone would complain about.