Category See all →

MLOps

Training pipelines, model registries, GPU infrastructure and model serving in production.

27 stories

Banner: training-serving skew from different preprocessing in training and production

AI & Machine Learning Bug Report

Model with 0.91 AUC offline performs near random online: training-serving skew

Ryan Pham · Nov 27, 2024 · 5 min read

Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.

Banner: MLflow model registry stages and aliases pointing to different model versions

MLOps Bug Report

MLflow stages vs aliases: the wrong model version stayed in production for three weeks

Ryan Pham · Oct 16, 2024 · 5 min read

Data scientists promoted a new churn model by setting the champion alias, as the newer MLflow docs recommend. Our deploy script still loaded models:/churn/Production, which pointed at the old version. Stages are deprecated; we moved the whole pipeline to aliases and added a promotion check.