Recommendations

MLOps Maturity Roadmap

Gradually increase automation.

Many enterprises start with a highly manual, inefficient deployment process. Enterprises starting from scratch need to iteratively automate more processes with fully automated pipelines as the end goal.

See recommendations for gradually achieving maturity.

Gradually increase automation.

Robust Pipelines

Build up resiliency to errors.

Even the most experienced team can’t anticipate every error, but they can design intelligent safeguards to mitigate their risk and cost. Follow some of these tips to deploy resilient systems:

Calibrate for cost of error: For user-facing errors, do not display the prediction if the value exceeds a certain threshold. A virtual assistant should say “I don’t know” instead of generating a random or low-confidence response.
Supervisory model: Train a model to predict which inputs will likely cause a model error. If an error is forecasted, the supervisory model should not make the prediction.
Shotgun approach⁸: Where prediction confidence is low, display multiple predictions for users to choose from. This is like Google search results presenting a group of pages predicted to be most relevant.
Human-in-the loop: If confidence is low, transition to a human agent. A chatbot unable to answer a query can transfer users to live customer support.
Contingency models: If a model starts behaving erratically, have a fallback model. This could mean reverting to a previous version, deploying a simpler model (e.g., replacing a random forest classifier with predictions from a logistic regression model), or utilizing programmatically implemented decision heuristics.

Log monitoring events to enhance diagnostics where pipeline fragility needs additional attention. Data exploration tools⁹ (either external or native to the monitoring tool) can slice, summarize, and visualize the data to uncover patterns in errors and spotlight the issue’s location.^xxvi

Deployment Strategies

Mitigate risk by diversifying deployment resilience.

Single model (re)deployment is simple since each pipeline works only with one model in production. However, this approach carries significant risk, as any bug will affect all users.

Consider one of the deployment strategies below when replacing a model:

Silent deployment: A new model deploys in parallel with an existing model. Users only see predictions or actions from the old model; new model predictions are recorded and assessed for bugs and errors. After enough validation time, the error free model is replaced. This option requires twice as many models to run and evaluating the new model could expose users to predictions depending on the use case.
Canary deployment: Only a subset of the user population receives the new models for evaluation. Minimizing exposure prevents bugs from affecting a large population and allows prediction and effect evaluation when presented. This approach doesn’t require the resources for double deployments. Two potential drawbacks: This deployment likely won’t detect rare bugs and errors and having to manage multiple models imposes additional complexity.
Multi-armed bandits: The MAB algorithm can select the best version of a model in the production environment. Founded in probability theory, each arm is a model and served to a user based on maximization of the expected value. One common implementation is UCB1 (Upper Confidence Bound). UCB1 starts with low confidence about model performance and may serve suboptimal models to increase confidence. As confidence increases based on the performance of each served model, UCB1 converges towards the best one and eventually serves only that model. MAB executes both evaluation and deployment steps in the pipeline.^xxvii