Contributors and References

Meet the Authors & Works Cited

JOE GRANICK

Lead Research Analyst – Data Intelligence  

Joe is the lead data intelligence research analyst at Stratascale. He helps clients optimize decision making by leveraging statistics, mathematics, and machine learning through his background in data analytics, economics, and financial planning and analysis. 

Follow Joe on LinkedIn

DEREK SHANK

Head of Research

Derek is head of research at Stratascale’s Innovation Labs. Throughout his career—from academia to personal finance to IT research—he has focused on uncovering insights and helping people put those insights to good use. 

Follow Derek on LinkedIn

CHRIS HUDSON

Lead Technical Advisor - Digital Experience

Chris has more than 20 years of experience in software development and IT operations. He brings thought leadership to Stratascale clients by creating innovative strategies focusing on digital experience, DevOps, and automation and aligning business outcomes with those IT initiatives.

Follow Chris on LinkedIn

References

"2021 enterprise trends in machine learning." 2021. 

Archuleta, Michelle. "Don’t Forget MLOps When Deploying AI/ML Models." RS21, 12 May 2021. https://medium.com/rs21/dont-forget-mlops-when-deploying-ai-ml-models-86589da2aa30.

Bloem, Jaap, et al. "Your Big Data Potential: The Art of the Possible (Report 4)." 2013. 

Bughin, Jacques, et al. "Notes from the AI frontier: Modeling the impact of AI on the world economy." 2018. 

Burkov, Andriy. Machine Learning Engineering. True Positive Ink, 2020. 

Colyer, Adrian. "150 successful machine learning models: 6 lessons learned at Booking.com." The Morning Paper, 7 Oct. 2019. https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models.

Cunningham, Wayne. "Science at Uber: Powering Machine Learning at Uber." Uber Engineering, 10 Sept. 2019. https://eng.uber.com/uber-science-machine-learning-platform.

Gadaleta, Francesco. " MLOps: what is and why it is important." Data Science at Home, Amethix. 11 May 2021.

Google. "MLOps: Continuous delivery and automation pipelines in machine learning." Google Cloud, 2021. https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning#devops_versus_mlops. 6 June 2021. 

Hering, Mirco. DevOps for the Modern Enterprise. IT Revolution Press, 2018. 

Kahneman, Daniel. Thinking Fast and Slow. Farrar, Straus and Giroux, 2011. 

Karayev, Sergey. "Lecture 11: Deployment & Monitoring." Full Stack Deep Learning. 2021. https://fullstackdeeplearning.com/spring2021/lecture-11/#1-why-model-degrades-post-deployment. 

Lorica, Ben and Paco Nathan. "Model Monitoring Enables Robust Machine Learning Applications." Gradient Flow, 26 May 2021. https://gradientflow.com/machine-learning-model-monitoring.

Morris, Charles. “From Idea to Value: A process for managing the data science lifecycle in the enterprise.” Microsoft, 23 March 2020. https://cloudblogs.microsoft.com/industry-blog/microsoft-in-business/ai/2020/03/23/from-idea-to-value-a-process-for-managing-the-data-science-lifecycle-in-the-enterprise.

Ng, Andrew. "The Batch Issue 91." Deeplearning.io, 12 May 2021. https://www.deeplearning.ai/the-batch/issue-91.

Null, Christopher. “10 Companies Killing It at DevOps.” TechBeacon, 19 June 2021, techbeacon.com/devops/10-companies-killing-it-devops. 

Paleyes, Andrei, Raoul-Gabriel Urma and Neil D. Lawrence. "Challenges in Deploying Machine Learning: A Survey of Case Studies." arXiv, 2021. https://arxiv.org/pdf/2011.09926.pdf.

Qiao, Lin. "PyTorch builds the future of AI and machine learning at Facebook." Facebook AI, 2 June 2021. https://ai.facebook.com/blog/pytorch-builds-the-future-of-ai-and-machine-learning-at-facebook. 

Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems 28. Conference on Neural Information Processing Systems, 2015. 

Stumpf, Kevin. "Why We Need DevOps for ML Data." Tecton, 28 April 2020. https://www.tecton.ai/blog/devops-ml-data.

ⁱ(2021 Enterprise Trends in Machine Learning)
ⁱⁱ(2021 Enterprise Trends in Machine Learning)
ⁱⁱⁱ(2021 Enterprise Trends in Machine Learning)
^iv(2021 Enterprise Trends in Machine Learning)
^v(2021 Enterprise Trends in Machine Learning)
^vi(2021 Enterprise Trends in Machine Learning)
^vii(2021 Enterprise Trends in Machine Learning)
^viii(Lorica and Nathan)
^ix(Lorica and Nathan)
^x(Cunningham)
^xi(Colyer)
^xii(Bloem, van Doorn and Duivestein)
^xiii(Stumpf)

^xiv(Google)
^xv(Google)
^xvi(Sculley, Holt and Davydov)
^xvii(Archuleta)
^xviii(Karayev)
^xix(Sculley, Holt and Davydov)
^xx(Google)
^xxi(Google)
^xxii(Google)
^xxiii(Google)
^xxiv(Google)
^xxv(Google)
^xxvi(Burkov)
^xxvii(Burkov)

¹The model found spurious correlations in search terms that were seasonally related to the flu, like searches for “high school basketball,” and assigned them predictive significance. When the correlations ceased to hold, prediction accuracy deteriorated.

²Some academics have framed machine learning systems as chaotic attractors, like a double pendulum or weather system. Partial Differential Equations (PDEs) are much more subject to the butterfly effect than Ordinary Differential Equations (ODEs).

³CRISP-DM is a widely adopted idealized analytics process model. Data scientists’ skill sets tend to be concentrated in the first four steps of the process. MLOps practices can naturally be placed in deployment but can also be incorporated into other phases of the model development process, like monitoring and improving data quality in the data preparation phase.

⁴ Degrees in math or statistics rather than computer science are common.

⁶Aspects of these processes may be automated depending on the use case context.

⁷ NaN stands for not a number. This occurs if the model executes an operation that produces an undefined value like dividing by zero.

⁸ “It’s got a good spread!” – Bill Burr

⁹ These can either be native to your monitoring tool or your preferred data munging and visualization tool.