Considerations

MLOps Is Not DevOps

The difference is in the details.

Enterprises cannot blindly apply DevOps principles to production machine learning due to technical and contextual complexities inherent to the domain.

For example, technical debt in traditional software development creates problems, but the debt is less likely to go undetected and compound than with machine learning systems. Machine learning creates code in a dynamic non-linear way, which makes it a chaotic system.² In this environment, technical debt evades detection and creates exponential error cascades and feedback loops.^xvi

Machine Learning vs. Programming

Figure 4 The indirect nature of coding machine learning models exposes production models to uncertainty and risk.^xvii

The difference in code development contributes to the chaotic contrasts. A software engineer fully controls the code when programming traditional software. A data scientist trains a model by choosing an algorithm and an objective function to optimize. The algorithm generates the code with a mapping function that sets parameters that best fit the data to the chosen objective. Data scientists can’t control the code directly and opacity increases with model complexity.

Since the data dictates the code, and the code represents a real-life relationship, a training-serving skew will occur when (not if) the relationship being modeled changes. This data drift degrades model performance, as the function fit to the training data may no longer represent the real-life relationship well. This shift is a real problem resulting in model responses that don’t generalize to reality. The effect of COVID-19 mentioned previously created a data drift that degraded the performance of many models.^xviii

Figure 5 DevOps’ main concern is code, but code is only a small part of the ML system creating opportunities for complex problems.^xix

Idiosyncratic challenges of machine learning that go beyond the nature of the code include:

Automation Goals

How to enable continuous integration and delivery

Enterprises can improve model reliability through pipeline automation. Automation limits the potential for human error, reduces deployment time, and frees data scientists to work on model development. With it, enterprises realize the consistent creation of higher-quality and more robust models along with adaptation and system scaling.

Like DevOps, MLOps incorporates continuous integration (CI) and continuous development (CD) as major pipeline automation components. While some principles are common to both, machine learning systems have unique challenges:

CI: DevOps and MLOps both require source control unit testing and integration testing, but machine learning systems require data, data schemas, and model validation.
CD: DevOps typically focuses on a single software package, but CD in machine learning pipelines deploys an entire model prediction service.
Monitoring: Developers need to monitor inference data and model KPIs in addition to bugs and errors.
Continuous Training: In addition to CI/CD, machine learning requires continuous training to automate retraining and redeploying models.^xxi

Stages and characteristics of an automated machine learning pipeline include:

Development and experimentation:
- ML algorithms iterated.
- Experiment orchestrated.
- ML pipeline source code generated.
- Output is source code pushed to repository.

Pipeline continuous integration:
- Source code built.
- Testing started.
- Outputs are pipeline components generated (packages, executables, and artifacts) for later deployment.

Pipeline continuous delivery:
- Artifacts from CI phase deployed to target environment.
- Output is a pipeline with new model implementation deployment.

Automated triggering:
- Pipeline executed based on schedule or in response to environmental trigger.
- Output is a trained model pushed to model registry.

Model continuous delivery:
- Newly trained model served as prediction service.
- Output is a deployed prediction service.

Monitoring:
- Model performance statistics collected based on production data.
- Output is a pipeline execution trigger that iterates all or part of the model and pipeline development cycles.^xxiii

Exploratory data analysis and modeling precede new pipeline iterations. Both processes require extensive manual implementation.⁶ ^xxiv