Background

Dismal Predictions

The economics of decision-making under conditions of uncertainty

Today, artificial intelligence (AI) is in the same place that the internet was in the mid-1990s.

The last restrictions on commercial internet traffic vanished on April 30, 1995, when the National Science Foundation ended their sponsorship of the NSFNET backbone service.¹ The market rapidly adopted the now-unchained network of networks, and after a few years the internet was ubiquitous.ⁱ

This internet moment inspired the creation of a new digital economy. The internet spawned new business models and upended existing ones. Companies with no revenue went public (often questionably) with astronomical valuations² solely due to the “dot-com” in their name.³

Economists (“The Dismal Science”) didn’t see a so-called new digital economy.⁴ Rather, they saw the same economy playing by the same rules: the internet was merely a disruptor because it reduced the price of existing operational inputs. Dynamics changed as digital transformation reduced the price of distribution, service, search, and communication.

¹Digital Revolution

²See the dot-com bubble

³Dot-com bubble

⁴Economics is often referred to as the Dismal Science.

The era’s competitive landscape changed dramatically, but the economics of supply and demand illuminate why. Companies like Amazon excelled because they designed their processes around economies of scale:

They used the internet to acquire a wide customer base without expensive store fronts.
Their acquisition strategy fit into the customer-centric focus.
That created a high switching cost, which reduced customer churn.ⁱⁱ

The current AI moment is analogous to the dotcom boom, and the economic perspective is similarly applicable. Cutting-edge AI applications inspire the imagination, but the underlying supply-side economics paint a comparatively banal picture. Organizations can extract the value from predictions in a broader context because cost-benefit dynamics increasingly favor it — if not compel it outright.

Just like it leveraged the economies of the internet, Amazon also leverages the economies of predictions:

Amazon uses data science and machine learning to predict customer behavior.
Amazon aligns the predictions with their sales strategy, presenting customers with items they're more likely to buy.
In logistics, Amazon places items in distribution centers based on their predictions for purchases nearby.
Orders arrive faster, bolstering loyalty and increasing supply-chain efficiency by reducing uncertainty.ⁱⁱⁱ

Surplus predictions change the dynamics of the decision-making process. It is imperative that companies determine how they can improve decisions with the available predictions. Within the nascent field of Decision Intelligence, effective executives can capture value by organizing their data predictions and decisions.⁵

⁵VRIO Framework: To be a source of sustained competitive advantage something has to be Valuable, Rare, difficult to Imitate, and Organized to capture value

"Numbers is hardly real..."⁶

Reading the numerical tea leaves alone won't lead to better decisions

The success of bombing missions in World War II often came down to armor placement. Insufficient armor left bombers vulnerable to enemy fighters and anti-aircraft guns. Too much armor decreased maneuverability and fuel efficiency, preventing planes from reaching their intended targets. The tradeoff represented a deadly serious optimization problem.^iv

The issue was perfectly suited for “the most extraordinary group of statisticians ever organized, taking into account both number and quality.” These were the earliest days for the Statistical Research Group (SRG), which used statistical analysis to support the war effort. Military officers brought the armor placement question and relevant data to Abraham Wald, arguably the smartest man in a very smart room.⁷ Observing that most bullet holes occurred in the plane’s fuselage, officers decided to reinforce the armor there and requested Wald’s analysis to determine the optimal amount.^v

Wald approached the problem with a more nuanced perspective. He determined the engines needed more armor.

The officers blindly followed the data to reach a naïve conclusion, but Wald considered the data in the context of a broader model. He reached an unintuitive but undoubtedly higher value conclusion because he realized that bullet patterns should be randomly distributed. Plenty of engines had bullet holes—they just never returned to the airfield!

Planes that take heavy fire to the engine go down. Downed planes don’t make it back for analysis. The engines in the dataset accumulated fewer holes because the dataset excluded engines with a lot of bullet holes.

⁶"Numbers is hardly real and they never have feelings. But you push too hard, even numbers got limits." -Mos Def

⁷A room that included Frederick Mosteller, Leonard Jimmie Savage, and Milton Friedman among others.

The officers committed a Type III error: they used the right data and mathematics to ask the wrong question. Wald found the right question to ask because he had a mental model of the plane that mapped the data to its environment.

Figure 1 Statistician Abraham Wald recognized survivorship bias of where bullet holes accumulated because he a had a well-designed mental model.⁸

Business leaders today need to think like Wald. Many are not. Google Chief Decision Scientist Dr. Cassie Kozyrkov described the problem: “Type III error plagues data science today. We’ve spent so much effort developing the math, developing the algorithms, getting the data sets, figuring out the code, and we’ve spent almost no effort figuring out how to ask the right questions and how to run an end-to-end applied process without messing up.”⁹

Many leaders fall into the trap of making “data-inspired decisions” like the well-intentioned military officers who asked the wrong question about armor placement. They try to manufacture rationality with math. Decision intelligence (DI) provides a system for leaders to augment their decision-making process with quantitative input from data science and qualitive inputs from the decision sciences — leading to consistently better decisions.^vi

⁸https://www.technewsiit.com/hidden-history-abraham-wald-and-survivorship-bias

⁹Making Friends with Machine Learning

The Effective Executive Decision

Today’s business leaders should mirror Wald’s approach in a different context. For this reason, decision intelligence has been rapidly emerging as a new discipline and a critical capability for organizational success.

Complexity in decision-making grew rapidly after Peter Drucker¹⁰ expounded the virtues and challenges of knowledge work and economy. In the second half of the 20th century, process efficiency gave way to the synthesis of a diverse set of inputs from highly specialized employees. AI and ML further complicated the contemporary environment. The fledgling field of Decision Intelligence (DI) studies how to optimize decisions in complex environments.^vii

Figure 2 Decision making environments are complex and can be greatly affected by unknown events. Source: xkcd

¹⁰Peter Drucker is regarded by many as the father of modern management science.

According to Dr. Lorien Pratt, a pioneer in ML, AI and co-founder of the DI company, Quantellia,

Decision intelligence, like artificial intelligence, is an entirely new field of computer science. DI integrates existing tools (like AI) and disciplines (like behavioral economics) and human expertise to answer questions of the form, ‘If I make a decision to take this action, then how will that impact my desired outcomes?’.”

DI provides a common language to understand decisions. It bridges the gap between traditional data science disciplines — like statistics and ML—and qualitatively-oriented decision sciences — like organizational psychology and management sciences:

People often think of a technology as a ‘silver bullet’ that, alone, will make a difference. This is rarely the case, as people, process, and technology must all work in a coordinated fashion to be effective”

Kozyrkov describes this academic cross-pollination as “Data Science++.” Decisions that synthesize knowledge from the data and social sciences outperform naïve application of various mathematical flavors from the data science sundae. DI combines quantitative and qualitative aspects of decision-making, giving the mathematics context to operate.^viii

And it is these perspectives that avoid the “type III” problems described above. DI gives us an understanding of the context and the purpose for which we’re building solutions, to ensure that we’re solving the right problems, using the right approaches.”