Nature of the Problem

Companies Struggle with Machine Learning Deployment

Realizing true value requires scale.

Large enterprises need scalability and sustainability from their production initiatives before the initiatives reliably deliver value at the magnitude of the world’s premier corporations.

While companies today deploy more machine learning models, deployment requires more effort and more staff. Illustrating the correlation of models to staff size, 41% of companies with more than 25,000 employees have 100+ models in production. Meanwhile, 42% of companies with fewer than 25,000 employees only had 1 to 25 models deployed.ⁱⁱⁱ

Time to deployment remains an issue as well. Most companies (64%) require a month or longer to deploy a machine learning model into production, and this time has increased year-over-year. With model development also requiring over a month, companies may need to wait more than a year for deployment from the time they first identified a use case.^iv

Figure 1 Deploying models has become more time consuming.^v

As stated earlier, companies have increased their hiring of data scientists, but in almost 40% of organizations, data scientists spend half of their time deploying models. Data scientists working at companies with more models deployed spend more time dealing with model deployment than their counterparts at less advanced companies. This fact tells us that companies are manually scaling their deployments by hiring more data scientists.^vi

Figure 2 Data scientists spend more of their time on deployment as the enterprise develops more models.^vii

Increasing manual input to linearly scale deployment creates a major roadblock to the financial viability of enterprise machine learning. A narrow labor market means data scientists are expensive. Demand will only increase as machine learning and AI expand to more industries. Enterprises should use most of their data scientists’ time to experiment and innovate by applying models to use cases, not deploying, monitoring, and managing models.

Deployment Is Only the First Step

Models degrade and become difficult to maintain in a complex environment.

Getting to the deployment stage clears one hurdle, but it also brings about many more. Manasi Vartak, CEO of Verta.ai, said it best: “In our experience helping organizations deploy hundreds of ML models, models begin to degrade the moment they get deployed. This is particularly true for models built on time-varying data, but it also holds for models built on so-called static data, like natural images, because the deployed model is used on new and unseen data.”^viii

Many factors cause model performance to deteriorate, including differences between real-world and training data, fragility, bugs, bad actors, and changes in the real-world environment.^ix

While slow development cycles impose high direct costs, opportunity costs linked to delays create the bigger obstacle to value. Even if a model works, it typically has significant room for improvement. The glacial pace of development prevents rapid iteration and models from realizing their full potential. Deployments lose value as model relevance deteriorates and infrequent updates hinder performance.

Companies encounter performance degradation challenges across a variety of use cases. During the COVID-19 pandemic, changes in human behavior caused performance deteriorations in many models as the data they were trained on no longer generalized to reality. Deterioration can also cause harm to society. After operating for five years, Google Flu Trends underestimated the peak of flu-season by 140% in 2013 due to associations with search terms no longer staying relevant.¹