Recommendations

Know your limits as X approaches infinity and plan accordingly.

Recommendations

Leaders should plan for the complexities of data. They should tailor the strategy to the idiosyncrasies unique to their enterprise. If they don’t plan for complications, they are more likely to fund unproductive science experiments and wind up with “orphaned analytics” projects.⁹

Plan around limits of historical decisions

Historical decisions put present constraints on data monetization options. Amazon invested in building its data assets for years before being profitable. The acquisition cost was selling books below cost to learn customer characteristic profiles and purchasing tastes. They now leverage this with recommendations to match customers with products they would be likely to purchase. A competitor attempting to copy Amazon will likely be unable to acquire the same volume detailed data, rendering their recommendations dramatically less relevant.¹⁰

Understand the limits of your data

Data collected in the past is not necessarily relevant to current use cases, so it may be necessary to invest further.¹¹ Data acquisition represents a significant expense, either directly, or more likely and consequentially, as an opportunity cost.

When Fairbanks and Morris devised their information-based strategy, the necessary datasets didn’t exist. Banks had credit scoring models to evaluate default risk, but the predictive value was irrelevant to the question “Which applicants will be profitable?” Fairbanks and Morris eventually acquired this data through an expensive experiment phase.¹²

More data typically allows for better performance, but more complexity can lead to diminishing returns. The curse of dimensionality limits the utility of complexity. Flexible algorithms fit datasets closely. When there is a complex set of variables, a larger sample of observations will be needed to filter the signal form the noise, because each additional variable increases the likelihood of finding a spurious correlation, leading to false positives, over or underestimations, and/or overconfident predictions.

⁹“One-off analytics developed to address s specific business need but never ‘operationalized’ or packaged for reuse across the organization” (Schmarzo 106)
¹⁰(Provost and Fawcett 319)
¹¹⁽Provost and Fawcett 29)
¹²(Clemons and Thatcher)

Recommendations Continued

Understand the limits of your data (continued from previous page)

Build uncertainty into your strategy

Organizations need to account for the uncertainty inherent to data-based predictions. To deal with uncertainty, leaders should:

Treat long-term forecasts with suspicion
Respect the margin of error

Treating long-term forecasts with suspicion

Business forecasts are like weather forecasts. Forecast accuracy depends on estimated initial conditions. Expanding the time horizon compounds inaccuracies in the estimate. Accurate short-term prediction will rapidly approach irrelevance if extended.

Calibrate for a range of outcomes and build flexibility to change strategic assumptions in response to a changing environment.

¹⁴(Silver 177-179)

Recommendations Continued

Respecting the margin of error

Organizations must build the margin of error of any forecast into their action plan. Failing to do so can have catastrophic consequences, as happened in April 1997, when the Red River overflowed and flooded Grand Forks, North Dakota, causing billions in cleanup costs, and damaging or destroying 75% of the city’s homes.

The National Weather Service (NWS) expected runoff from melting snow and forecasted that the river would rise to forty-nine feet. The forecast was accurate: the waters rose to fifty-four feet, well within the margin of error, but the levees were only fifty-one feet high.

The NWS’s prediction interval meant there was a 35% chance of flooding exceeding the levee’s capacity. This could have been planned for, but NWS did not communicate the uncertainty with the forecast, fearing the public would doubt its validity.¹⁴

Given the real risk of overflow, additional investments to reinforce the levee, or to re-route floodwater would have been far less expensive. Similarly, your strategy should consider forecast uncertainty, and build appropriate contingencies.

Business leaders know that decisions can be wrong. They must make decisions in the face of uncertainty. In adopting a strategy to use data, organizations should calibrate decisions with this uncertainty in mind, building flexibility for variable degrees of belief, and investing in safeguards for outcomes with wide confidence intervals, or catastrophic downside risk.

Recommendations Continued

Promote a culture of experimentation

To effectively execute a data strategy built on research and development you will need to foster and promote a corporate culture of nuance and experimentation, particularly in leadership positions.

Executives typically value direct answers. Most corporate culture incentivizes the “certainty” of leaders with confident worldviews. President Harry S. Truman lamented “Give me a one-handed Economist. All my economists say, 'on the one hand...', then 'but on the other...”

This corporate overconfidence is detrimental to the survival and success of machine learning, where uncertainty is a foundational principle, and nuance and experimentation drive value.

In many organizations, employees will often resist new data driven recommendations and decisions. It may be difficult for influencers to get buy-in from leadership, especially if there is a lack of data literacy through the organization.

Analytics is rooted in uncertainty and degrees of belief, so an executive culture insistent on certainty can doom an initiative from the start.

When facing such resistance, we should consider how leadership from the first 16 banks approached responded to Fairbank’s and Morris’s Big Idea.¹⁵

“It can’t be done!” — it’s too expensive, and the data that you need on individual consumers to make the strategy succeed is simply not available to the bank.”
“And we don’t need your help; we already do it!”¹⁶

¹⁵(Clemons and Thatcher)
¹⁶(Clemons and Thatcher)

Recommendations Continued

Take an R&D approach to analytics

Analytic solutions require iteration to generate value and should be treated more like research and development as opposed to a software engineering project. Organizations should take time to focus on problem decomposition and process integrity, rather than chasing after quick wins.

Analytics teams need to spend time iterating on experiments. The companies that realize a significant return commit to a long-term data strategy.¹⁷ Like interest on financial assets, value from small improvements in decision making and efficiency compounds over time.

Data scientists command high salaries. Even digitally savvy companies like IBM, Uber, and AirBnB laid off many data scientists during the COVID-19 pandemic.¹⁸ For companies with less mature practices, leadership may make rash cuts, seeking quick savings from projects that aren’t yet profitable.

Protect the asset in an environment designed for R&D. Otherwise the firm will not have enough time to filter successful models from inaccurate ones and will not develop the maturity needed for successful deployments.¹⁹

At Signet Bank, Richards and Fairbanks spent years running failed tests before they could turn a profit. The first year consisted of building databases, learning the business context for account management, and figuring out how to implement the strategy.

In August 1989, they began implementing their “test and learn strategy.” They ran marketing tests, determined which tests were profitable, then iterated with more campaigns like the successful ones. With many initial failures, the lag between testing and deployment imposed significant expense.

¹⁷(Huyen)
¹⁸(Sharwood), (Cao), (Liffreing)
¹⁹(Huyen)

Recommendations Continued

Take an R&D approach to analytics (continued from previous page)

Between then and December 1991, test solicitation charge offs drove a doubling of total charge offs as a percentage of the bank’s portfolio
Toward the end of 1991, the information-based strategy started to turn a profit
From 1992 to 1994 receivables grew 427% and accounts grew 302% while charge offs declined
Signet grew faster than 996 others on the Fortune 1000²⁰

Craving certainty, a bank executive could have prematurely seen an opportunity to cut “unproductive” expenses. The CEO and the Executive Vice President of the Bank were strong advocates and understood the losses were an acquisition cost for the investment. By 1994 leadership decided to spin Fairbanks and Morris’s experiment into their own IPO, Capital One, with a vision built on the information-based strategy. By 2013 Capital One was running 80,000 tests a year.²¹

Data’s tendrils cross the organization. Stakeholders across the enterprise require input, and degrees of ownership will transition across phases of a deployment. To maximize the value of a deployment, leaders must put in place governance that defines how to transfer ownership and enables the organization to make decisions that balance potentially competing success metrics.

Poorly defined ownership may cause issues due to improper transitions, and failure to gain buy-in increases the likelihood of derailment by a single overconfident executive with institutional influence.

Figure 1: CRISP-DM Model for data mining project development shows the iteration necessary due to complicated relationship of stakeholder concerns²²

²⁰(Clemons and Thatcher)
²¹(Bloem, van Doorn and Duivestein)
²²(Shearer)

Recommendations Continued

Invest in infrastructure and data engineering

Most enterprises need to invest significantly in infrastructure before extracting value from big data.

Greater diversity and volume of data improves predictive performance more than better algorithms. This complexity requires more storage capacity, and specialized transformation pipelines to store and manage the data.²³
The algorithms that take advantage of big data are computationally intensive and can exceed an enterprise’s processing capacity.

At Signet bank, Richards and Fairbanks were essentially starting from scratch. They ended up building the largest Oracle database at the time. The database needed to contain detailed varied customer data, support analytic methods like regression analysis, and store the results.²⁴

Invest in infrastructure and data engineering (continued from previous page)

At Capital One, they continued to invest heavily in their data infrastructure. They spent $49 million to break the contract with their data processor, to develop the internal control needed to have flexibility in their data ecosystem.

Data may be non-depletable, but loss of relevance, along with technical debt, can cause value of data assets to depreciate. Relative to other IT initiatives, technical debt poses a larger risk to analytics projects, as it is:

Faster to accumulate
More complex
Difficult to detect

The dependencies driven by technical debt make it impossible to know what effect changes will have system wide. This accumulated debt can grind predictive performance to a halt²⁵

To be effective and sustainable, data architectures must be engineered to be:

Reliable
Scalable.
Maintainable
Adaptable²⁶

If an enterprise plans to extract the value that big data can fuel, it will need to make the requisite investments in storage capabilities and processing power. To continue to deliver value, data engineering teams must design this infrastructure in a composable way, so it can be adapted to future business goals and data requirements.

²³(Halevy, Norvig and Pereira)
²⁴(Clemons and Thatcher)

²⁵(Sculley, Holt and Davydov)
²⁶(Huyen)

Recommendations Continued

Appoint and empower a Chief Data Officer (CDO)

Executive leadership should build a strategy that organizes the company to create sustained competitive advantage from data assets. Organizations should appoint a CDO as the executive with responsibility for developing and implementing the organizational strategy for successful data investments.

Organizations that appoint a CDO are much more likely to realize benefits such as:

An effective CDO should control or have input for all decisions that are dependencies for the data strategy, including:

Monetization strategies
Evaluation of costs and benefits of data investments related to the monetization strategy
Gap analysis of investments needed to leverage value from data
Defining technical and project management requirements for human capital
Governance and ownership decisions regarding the data pipeline and solution development
Evaluating limits on performance
Hedging downside risks of uncertainty

Functional Outcome	*Increased Likelihood*
Business-driven analytics governance	*1.9x*
Big data and analytics strategy	*1.7x*
Enterprise-wide analytics deployment	*1.8x*
Data driven sustained competitive advantage	*1.5x²⁷*

²⁷(IBM Institute for Business Value)