Solutions

Create Cross-Functional Teams

Bringing business and data together

For better alignment, management should place their data team close to the business where they can develop products relevant to the business stakeholders’ use cases. Alignment begins with planning the data strategy, through which the enterprise defines how the data product will return value, if it will be internal or external facing, how it will augment or automate decision-making, and other desired outcomes.^xviii Successful data projects often involve diverse stakeholders, so their success depends on flexible management and sound governance.^xix

Figure 12 Source: Algorithmia 2021 Enterprise Machine Learning Trends

In developing the strategy, enterprises should ensure their working group includes people who know what data the company has on hand, what the company can realistically acquire, and what the data team can do with it from a technical perspective.

Management’s role does not end after the strategy development. Executives must continue to interface with data teams to guide them toward business objectives throughout the life of each project.^xx

To identify potential business use cases, management needs adequate technical understanding of what the data teams do. They need to set reasonable goals and should be comfortable with uncertainty. The data science workflow often resembles research, and disproving a hypothesis is a completely valid result.

Even if the organization does not have a Chief Data Officer (CDO), the data team should have representation in upper-level management. Data teams can add more value if the organization creates a safe culture for them to participate in discussions about strategy, including voicing their concerns or disagreements with executives’ decisions.^xxi

Figure 9 Building the AI Powered Organization

When business use cases require close attention and deep contextual understanding, enterprises may wish to embed the data science and possibly data engineering teams in the business unit. Embedded data science teams should have a dotted line relationship to the central data team. Such a relationship ensures the embedded teams receive adequate training and exposure to best practices. Meanwhile, hiring tasks will no longer fall to business managers who typically do not understand the skills and abilities required or have the technical knowledge to accurately assess a candidate’s competencies.

Embedding can evolve gradually with a hub-and-spoke approach where members spin off into business units as the central team grows. The business-level data teams build and maintain close relationships with the central team.^xxii

Synchronize Data Teams Across Three Types of Roles

Complementary skills and traits are key

As mentioned earlier, data scientists alone do not make a data team. Too often, leadership views all data positions as equal, especially when skills and technologies overlap. Not differentiating roles dooms data projects and teams to failure.^xxiii

An effective data team typically requires three distinct functions: Data Science, Data Engineering, and Operations. Each team has specialized knowledge, skills, and abilities in their own domain. The effectiveness of the data team depends on the effectiveness of each domain. Like the economic principle of trade, each team can spend more time and create higher output in their specialization and then “trade” the output to others who have demand but lack specialization.^xxiv

Data Science: The data science team produces analytics products for the enterprise, ranging from analytics using advanced statistics and higher-level math to ML models. Data science requires strong skills in mathematics and statistics. Data scientists should demonstrate an adequate level of programming skills, but not to the extent of a data engineer. Python and R are the most used programming languages due to the availability of libraries for machine learning, data wrangling, data visualization, and, particularly for R, statistics. However, the Julia data science community is growing rapidly.

If the data engineering team provides enough support, the data science team only needs limited distributed systems knowledge. Data scientists need domain knowledge to effectively decompose a business problem, frame it as an analytical problem, and identify the data necessary to inform the analytics. While they don’t need the same depth of knowledge as subject matter experts in the business, they need more than surface-level understanding to effectively capture business problems. Data scientists also need to translate complex results couched in statistics and mathematics into an understandable and actionable format for the business consumer.^xxv

StitchFix hires data scientists from a variety of backgrounds not typically associated with e-commerce, including former biologists and chemists. The company’s Chief Algorithms Officer Brad Klingenberg explained that the company looks for candidates capable of solving problems with technology and data in a formalized system. “Working on a hard problem and really being trained to approach problems scientifically is a pretty general skill set. We generally find that people with a background in solving scientific problems and using technology and data do pretty well.”^xxvi

Data Engineering: Google’s Chief Decision Scientist Dr. Cassie Kozyrkov writes, “If data science is the discipline of making data useful, then data engineering is the discipline of making data usable.”^xxvii

The data engineering team performs many essential tasks, including:

Transforming models from the data science teams into usable products for the business.
Building the architecture the data science team uses to build their models.
Building the architecture the operations team uses to deploy the products.
Building data pipelines that process and transform raw data into a shape appropriate for the use case per the data scientists or the organization.

The data engineering team needs a strong understanding of distributed systems, because moving data across computers and orchestrating their operations requires more complicated architecture and design patterns. Data engineers need to understand programming to code the interaction of 10 to 30 different distributed systems along with the data technology to build data pipelines. They should also know how to test code and handle the continuous deployment and integration of data products and pipelines.

Visual communication skills help tremendously, as data engineers often use visualization to present data products. Similarly, data engineers need strong verbal communication since they commonly function as intermediaries between the business and data science team. If they can’t communicate the availability of data products to the business, they will go unused and will not generate value. If they can’t communicate the availability of data, the data science team won’t know what models they can build or what data they need to acquire.

In cases where an organization develops a formal data engineering function within an existing data program, the data engineers must start by cleaning up technical debt created by the data science team and make their code more robust so they can deploy the data products to the business.^xxviii

¹Learn about the importance of supplying high quality data.

Figure 13 Data Science and Engineering Skills^xxix

Data Operations: The operations team deploys software into production and makes sure end users can access it reliably with minimal downtime. The team also troubleshoots the system when issues occur. To do so, they require understanding of:

Distributed systems for proper monitoring.
Hardware troubleshooting.
The underlying structure of the data.
How software interacts with the operating system in a distributed framework.

Team members need the capacity to perform complex troubleshooting due to the large number of components and configurations involved across the layers of the technology stack.

The operations team should implement all monitoring and instrumentation to facilitate rapid troubleshooting. They should also operationalize best practices to prevent incidents and work with the Infrastructure and Operations team to adapt disaster recovery capabilities to the idiosyncrasies of data science, machine learning, and other data-intensive applications and workflows.^xxx

Without data engineers, data scientists end up doing the data engineering—and doing so poorly. Data scientists burdened with extra tasks will have less time to focus on creating analytics and models, and their haphazard data engineering efforts will result in technical debt and data products that don’t scale.

Without a data science team, analytics and data products are limited to what data engineering, business intelligence, and/or data analytics can create. Employees in these roles typically lack the knowledge to do machine learning and advanced statistics and they may lack the programming knowledge as well.

Without an operations team, the data science and engineering teams will create unreliable data products and infrastructure. Without dependable data products and infrastructure, the business can’t use the output for decision making. To maximize effectiveness, data teams need all three functions staffed and coordinated.^xxxi