Extending the Principles of Data Mesh to Maximize the Value of Mission Critical Data

Apiphani this week publicly launched our new Data Analytics Practice and Services. We have taken and integrated our decades of industry experience with Deep Automation™ tools to architect a data mesh solution that drives extreme efficiency and reliability with data pipelines for our client’s most important business intelligence, machine learning, artificial intelligence, and digital products.

Data is the lifeblood of today’s organizations. From the boardroom to the production floor, through engineering, finance, and project management, reliable, easily accessible role-specific data is essential for data-driven decision making, BI, ML, and AI.

Yet most organizations spend 80% of their time gathering, cleansing, and preparing their data and only 20% in analysis. And they do it again and again, week after week, to produce the same reports with little time for analysis, insight, and innovation.

Apiphani’s Data Analytics services can automate the process of managing data quality and data complexity, creating data pipelines designed to provide reliable, streamlined, self-service data. We take a business-first approach to continuous data delivery, freeing data out of technology silos – trapped in whatever application created it – and organizing it and prioritizing it within data domains aligned with your business needs.

I invite you to read our case study on apiphani’s data and analytics work with Power Systems Mfg, LLC (PSM).

A key part of what we are doing at PSM and with other clients is built around a modern data architecture that embodies the principles of data mesh. Zhamak Dehghani, in her 2021 book Data Mesh, outlined the four principles of Data Mesh: Data domain orientation, data as a product, self-service platform, and federated governance.

Maximizing the value of mission-critical data, however, requires extending beyond these four data mesh principles to put an emphasis on:

1. Opportunity-driven data domain priorities and expectations
2. Evolving the modern data platform beyond self-service
3. Delivering commercial-grade data products, and
4. A strategic approach to data governance.

Let’s look at each of these in a little more detail.

Opportunity-driven Data Domain Priorities and Expectations

“To make it out ahead and justify data teams work, providing data isn’t enough. It also needs to be reliable, and purpose built.  Like any type of change, bringing our work closer to the business will take time, ruthless prioritization, and the ability to understand where we can drive the most value.” Barr Moses CEO Monte Carlo.

There is a temptation to do too much all at once. Better is to pick the handful of opportunities that will have the greatest impact in a timely basis relative to the business. Early success with those high-impact opportunities will drive faster adoption.

Relentless opportunity-driven priorities and value expectations are first principles for all new technology-enabled value. Leaders need familiar methodologies that determine and value opportunities and priorities within and across data domains. Both strategic planning and ongoing adjustment to priorities require product life cycle methods: Product concept, business plan, development, and market launch with domain and product owners from across the organization.

Every organization has a product life cycle methodology. A typical product life cycle can be adapted to data mesh efforts in two ways, primarily focused on product concept and business plan: 1) include the specific characteristics of a data product definition, e.g., data sources and 2) sustaining operating model roles and responsibilities for data domains and center of excellence, e.g. data product roadmap and governance management.

Evolving the Modern Data Platform Beyond Self Service

“The growth of the data infrastructure industry has continued unabated since we published a set of reference architectures in late 2020. Nearly all key industry metrics hit record highs during the past year, and new product categories appeared faster than most data teams could reasonably keep track”. Andreessen Horowitz – Emerging Architectures for Modern Data Infrastructure.

The array of data infrastructure tools available is simply astounding. This includes tools from industry-leading companies like AWS and Microsoft to boutiques like Fivetran, databricks, dbt, Monte Carlo, Atlan, Nextdata and many others, all of which have collectively laid a path for data and infrastructure architectures.

The challenge is sifting through the ocean of choices (see Matt Turck’s modern data stack menu – The Next Data Crisis ). Today’s data platform stack is not only a defined and integrated set of tools for ingestion, storage, streaming, processing, and consuming data with embedded security and privacy governance, it is an evolving architecture of tools and opportunities with the potential to bring advanced data value for business intelligence, machine learning, artificial intelligence, and data products.

There is a big focus on building a data environment that is self service, delivering the right data to the right people at the right time. We need to move beyond self service. As Matt points out, there is a symbiotic relationship between the foundational aspects of data infrastructure and the analytics/BI and ML/AI that consume the data, all coming together in the end applications they enable.

Delivering Commercial-grade Data Products

“Speeding time to market for data-analytics applications: Data products can react more responsively to data demand and provide business users with scalable access to high-quality data through the direct exchange between data producers and data consumers.” McKinsey

Even if the resulting data products are only for internal use, they should still be commercial-grade – that is, supported, maintained, and grown just as if they were being offered for sale to the public. Data products must not only follow a product line management methodology but also require a commercial-grade DevOps CICD delivery infrastructure and methodology. Products flow through an experimentation and discovery sandbox until the product concept and business plan are formed, then move through Dev and Prod and general availability using tools like Terraform for module structures and configuration.

Strategic Approach to Data Governance

“Take a strategic approach to D&A governance and position it as an essential business-centric model versus a tactical approach, where D&A teams operate governance reactively, focusing on just one asset – data-only governance. Instead, rescope governance to target tangible business outcomes, make it sensitive to opportunity and risk, and agile and scalable.” Gartner

Data domain-focused governance provides a way to prioritize based on opportunity and risk. A vivid example is safeguarding the reliability and security of power generation data through a set of standards defined by the federal government and the North American Electric Reliability Corporation’s ‘Critical Infrastructure protection.’ The degree of management in this case is paramount. It’s not about the data. It’s about the business.

The Apiphani Data Pipeline

At apiphani, we are committed to providing our clients with a data pipeline platform and the accompanying services to help them uncover the trapped value of the data from operational systems. As our work with our co-innovation partner demonstrates, laying a data foundation and delivering expertise in data engineering, DevOps and MLOps engineering, and visual design enables their organization to increase and more effectively use their business and engineering expertise in BI, ML, and AI initiatives, as well as to data products for both internal use and external customers.

We like to say that we provide the resources and tools to do the hard work to deliver organized, trusted, and advanced data value to you when and where needed …so you don’t have to!