Recommendations

Align and Iterate

Composing decision

An effective decision framework synthesizes decision elements across domains, allowing functions with different technical and contextual expertise to interface through an aligned abstraction. Address stakeholder concerns and synthesize expertise across domains by using a workshop to align the framework. This workshop should include representatives for every function of the decision at hand.^xxxi

Use the workshop to decompose the decision:

Begin by brainstorming and classifying any relevant decision elements, then classify and organize them into a causal decision chain.
Use the initial framework to define relationships and map the chain of causality from decision to outcome. (Use simple assumptions to start.)
Refine the framework using a process like the Cross-industry standard process for data mining (CRISP-DM) to integrate more complex machine learning/data mining models, replacing the initial static assumptions and simple calculations in the causal chain.¹³
To orchestrate AI/ML supported decision-making, build MLOps¹⁴ practices around the elements of the decision framework. This makes the assumptions more adaptable and enhances the ability to monitor and adjust assumptions based on feedback from decision implementation.

Pratt described the contribution of a diverse sample of disciplines to the field of DI:

A discipline that might surprise you is Organizational Change Management, which can help technologists to understand how to introduce and message around their solutions to help them to be adopted”

¹³See data as a strategic asset

¹⁴See MLOps

Running the Workshop

Pratt recommends using between half a day to multiple weeks building an initial model for decision making that separates tasks requiring divergent thinking¹⁵ from tasks requiring convergent thinking¹⁶. This is the best way to utilize the two different systems of thinking. The project should have an executive sponsor as well as representative members from each stakeholder and function that can affect the decision or be affected by the decision.^xxxii

Reprinted with permission

Decision Modeling Steps:^xxxiv

1. Setup: Set up the room with a blank whiteboard and markers. The project’s executive sponsor should clearly state the meeting’s objective. Precede the meeting with introductions and team building depending on the familiarity and level of comfort amongst the team.^xxxv

2. Starting the Meeting: Start the meeting by clearly defining the decision being faced. Define the intractable constraints on the decision: elements that will not change even if it leads to a better outcome.^xxxvi

3. Brainstorm Outcomes (Divergent): Promote open brainstorming around goals and potential outcomes. Ideas should cover outcomes that can be tangibly measured as well as intangible goals like “maintaining employee morale” that the organization would like to influence through the decision. Post-it notes can be used to encourage sharing from less assertive team members.^xxxvii

¹⁵Divergent thinking is a thought process used to brainstorm new ideas in an associative way, what Daniel Kahneman would classify as System 1 thinking.

¹⁶Convergent thinking is the analytical thought process typically used to evaluate and organize the ideas generated via divergent thinking, what Daniel Kahneman would classify as System 2 thinking

4. Brainstorm Levers (Divergent): Similar to the process for brainstorming outcomes, you should develop a complete inventory of the set of decisions available. This is how outcomes will be influenced. A failure of imagination will limit what outcomes can be achieved.^xxxviii

5. Analyzing Levers (Convergent): Once everyone is tapped out of ideas, switch to a convergent style of thinking and assess the ideas. Consolidate categorically similar domains, like advertising and competitive marketing. Confirm each lever is controllable and not actually an external. Each lever should then be assessed as something that can be chained by analyzing if it is caused by more primitive elements. If another decision must be made that set the lever into motion, then reclassify the lever as an intermediate. Continue the process until a base how/lever is defined.^xxxix

6. Analyzing Outcomes and Goals (Convergent): Each outcome and/or goal should be evaluated based on its importance. If the outcome listed is “improve customer experience,” ask why. This may lead to a chain:

Figure 8 Why Chain for customer value source Quantellia LLC^xl

In this example, customer happiness is a proxy goal. It may be measurable or manageable but setting it as the target outcome to change could lead to misleading results as it is not a direct representation of the true goal. Every goal/outcome should be assessed on measurability. Can you determine success? Once the goal/outcome is confirmed, disambiguate it. Change “Improve profits” to “maximize revenues net of capitals costs.” Clarify further for when to measure profit. Next define “maximize” as “2% net profit on new product net of capex in 18 months of launch,” "ensure employee morale is above measured score of 80%,” and “keep environmental impact within allowable limits.” Separate the outcome (what you will measure for feedback) from the goal (the outcome value that defines success).^xli

7. Brainstorming Externals and Information Sources: List factors outside the organization’s control. These are assumptions made about the operating environment, including competitor pricing and the macroeconomic environment. These should be listed, then varies, to examine how changes may impact the model and the ideal decision. More sophisticated models can incorporate forecasts and likelihood, changing levers based on the likelihood of the assumptions and the magnitude of their impact when changed.^xlii

8. Wring Up the Model (Convergent): This is where team members link levers to outcomes. Intermediates that chain these together should be determined. This is where feedback loops and interaction effects can be considered.^xliii

9. Using the Model: The decision maker should now have a greater understanding of the factors affecting their decision and can now use AI/ML and data to adjust assumptions and forecasts where they will be most appropriate. From here the model can be refined and iterated on based on new data gathering, changing assumptions about the world. Predictions based on machine learning can now replace heuristic assumptions.^xliv

"Death just illustrates the ultimate absurdity of life"¹⁷

Use the appropriate mix and amount of data sciences

Decision makers should take advantage of the decision framework and identify the most applicable paradigm for the various factors and relationship in the model, then apply AI and ML where it is most effective.^xlv

Pratt explains that machine learning is useful when there are relevant, reliable data sets to train the models:

If I've got a historical data of the relationship between how many signs I put up to say ‘wear your mask’ and the number of people that actually wear masks then machine learning model is a good candidate. I can use that data to understand the form of that cause and effect link. This is just one part of a decision intelligence model / Causal Decision Diagram (CDD).”

Criteria to look for:

Patterns that can be learned
Objective Function as target to maximize/minimize
Patterns with enough complexity to make normal decision making sub-optimal
Data that's needed exists or can be acquired/created
Existing data has common patterns relevant to prediction target
Problems predictive in nature^xlvi

Goals/problems with the following characteristics are most conducive to analytic solutions:

The task is repetitive or needs to be repeated frequently
The problem or task is scalable
Patterns are constantly changing

¹⁷Influential philosopher AJ Soprano

Complex models require a lot of data. The less data that is available, the more domain expertise will be necessary to limit models to essential features. Expertise is also needed to inform which features will need to be synthesized with a data scientist’s creativity in feature engineering and dimensionality reduction.^xlvii

Beware: If it is a potentially damaging decision, hedging toward caution is advisable. Randomness accounts for much more than most appreciate. The expected value of an outcome depends on the benefit/cost of the outcome in addition to the probability of it happening. Economist Nassim Taleb described this idea through the probability in Russian roulette.

"One can illustrate the strange concept of alternative histories as follows. Imagine an eccentric (and bored) tycoon offering you $ 10 million to play Russian roulette, i.e., to put a revolver containing one bullet in the six available chambers to your head and pull the trigger. Each realization would count as one history, for a total of six possible histories of equal probabilities. Five out of these six histories would lead to enrichment; one would lead to a statistic, that is, an obituary with an embarrassing cause of death. The problem is that only one of the histories is observed in reality."^xlviii

Remember the decision maker is responsible for deciding the metric. They need to define success and the metrics to optimize.