API Platform and Data Mesh: Why Bring Them Together

Build a self-service platform based on domain-driven products

The first similarity between an API platform and data mesh is that both aspire to create a self-service platform based on domain-driven products.

A data mesh architecture embraces a decentralized and domain-aligned approach to data management. Similarly, an API platform promotes a decentralized, domain-aligned and microservice-based architecture.

Both API platform and data mesh aspire to decompose the systems and data into distributed services built around business domains. This approach advocates for cross-functional business and technology teams that operate with autonomy and manage long-lasting business domains.

By anchoring the strategy around long-lasting business domains, organizational design in IT has fundamentally changed. Teams autonomously own a business domain and productize it to be offered as a self-service asset to internal and external customers.

Domain-driven products

But what do we exactly mean by business domain? Let's see a concrete example based on a retail e-commerce platform. There are several business domains that we expect to see in that scenario.

Figure 1. Common business domains in a retail e-commerce environment.

A domain-driven approach sees each one of those domains as products that can evolve with a high degree of autonomy. Each domain is complex enough to merit a cross-functional team that works autonomously and at high speed. The objective is to expose their business logic and data via APIs, hence delivering a self-service platform.

While an API platform productizes the enterprise processes and business logic, a data mesh productizes the data. We can see them as two complementary pieces of the same block as illustrated in Figure 2.

Figure 2. Retail e-commerce business domains combine data and business logic.

The personalization domain product

To explore a concrete example, let's focus on the personalization domain to see how it relates to other domains when it comes to consuming business logic and data.

All retail e-commerce businesses want to personalize their offering across all channels by knowing who the customers are and what they like. Data analytics processes and machine learning algorithms in the personalization domain leverage customer data, product data, and order data.

The outcome is exposed both as a Personalization API and as new, enriched datasets that relate product, content, merchandising, and search data with individual and aggregated customer groups. For this scenario to be a reality, the personalization team depends on high-quality data and API services from other domains.

Figure 3. The personalization business domain consumes business logic and data from other domains via APIs.

In fact, there is a network of dependencies among domains. They interface with each other via APIs, either request-response, event-based, streaming, direct database connections, or other styles — hence creating a chain of interactions. The following diagram shows how those interactions take place in our retail e-commerce example:

Figure 4. Business domains behave both as consumers and providers in a network of API interactions.

We can see in Figure 4 how the Personalization domain relates to other domains:

• Product Information, which at the same time gets pricing information from another domain.
• Customer Profile, which typically comes from a customer data platform domain that enriches customer profiles based on their basic data, search history, web and mobile analytics, digital communications interactions, and loyalty data.
• Order Management, which records the interactions between customers and products that resulted in a sale.

As you can see in the illustration above, almost all domains play the roles of customer and provider. Managing domains as products means that each domain has to reach out to its internal and external customers, understand their expectations, and build the relevant interfaces to serve them.

Business domain reference architecture

How can a business domain prepare to become part of such a network of interactions? By providing an API layer that acts as the touch point interface to other domains — what sits behind the APIs should be a black box to them.

Figure 5 is a high-level reference architecture of a domain. Some API styles are best suited for data-centric interfaces like ODBC/JDBC or Object Storage APIs like Amazon S3, while other styles like Event Driven, GraphQL, or RESTful are able to cover both data and business logic scenarios.

The Kong API platform provides the mechanisms to set up federated governance in enterprise scenarios while simplifying the management of authentication, security, traffic control, observability, transformations, and logging for all API styles.

Figure 5. A business domain productizes data and business logic via APIs.

When delivered as separate workstreams, an API platform and a data mesh initiative may duplicate efforts, creating overlapping or competing products that confuse customers. (Here "customer" can refer to other business domains that consume the services.)

We believe it's better to approach customers as a single product. In the end, the product is the set of API contracts that each domain offers to other teams, whether the APIs are delivering curated data, machine learning model outcomes, or business logic.

Align to business objectives to drive priorities

The second similarity between an API platform and data mesh is that business alignment can be derived in the same way.

Once business domains are identified as in Figure 1, the next step is to develop a plan to build and mature them in a way that is aligned with business priorities. Here we offer a framework to determine what business domains to prioritize based on business objectives.

At the highest level, we can group enterprise priorities into three big buckets:

• Increase revenue
• Optimize margin
• Manage risks

Performance in each one of those buckets is tracked via KPIs and KPIs are influenced by the implementation of use cases. Continuing with our example, Figure 6 shows some of the most important KPIs (light blue) and use cases (dark blue) in the retail industry, organized around revenue, margin, and risks.

Figure 6. Business priorities are measured via KPIs. An agile company tests use cases to validate that the right KPIs are influenced. Use cases are quickly implemented when there is a self-service mesh of capabilities.

Once management identifies its strategic objectives and KPIs, lines of business work on deciding what use cases will be most effective at influencing the KPIs. We're then ready to select the business domains we need to focus on to support the use cases.

Let's assume in our retail example that management is focusing on increasing revenue as the company is not on-par with similar businesses. Conversion is now a focus KPI. The e-commerce line of business has decided that website personalization is the use case that most likely will impact conversion. To test the initial hypothesis, it's enough to focus on maturing the business domains of product information management, basic customer profile, and order management. In later iterations, more advanced personalization can be achieved by introducing a customer data platform and personalizing other areas like search or merchandising.

The point here is that the API platform and the data mesh need to evolve in synchrony as a single unit to deliver on the business outcomes.

Figure 7. Maturing business capabilities in an iterative and incremental fashion is driven by the use cases we want to test to influence the KPIs that drive business objectives.

Define a common capability model

The third similarity between an API platform and data mesh is that they share many capabilities. (Here we understand "capabilities" as the set of practices performed in the context of an API platform or data mesh initiative.)

There are a number of capabilities that you need to master if you want to excel at creating an API platform or a data mesh environment. The good news is that many capabilities are common to both scenarios. Working with our customers, we've identified more than 50 capabilities organized around 10 pillars:

1. Documentation — How you document your API and Data offering
2. Training — How to train and onboard your engineers
3. Support — What support channels do you offer to deal with questions and incidents
4. Development tools — How to support rapid development that focuses on business logic
5. Runtime platform — Where to deploy API and data solutions
6. Operations — How to observe and automate the platforms.
7. Strategy — How to decide what to work on and how it will be delivered
8. Design — Best practices for product design
9. Security — Embed security in the API/data lifecycle.
10. Governance – Set up the institutions and processes that provide guardrails

An API platform will benefit from capabilities with a pink background. A data platform will benefit from capabilities with a green background. But both initiatives will benefit from capabilities with a pink/green background.

Figure 8. Capability matrix organized around 10 pillars. An API platform and a data mesh share many capabilities.

Running API platform and data mesh initiatives as separate products risks duplicating efforts around capabilities.

Developing more than 50 capabilities is, of course, a significant effort. Our recommendation is to build the capability matrix following a layered approach that will benefit both an API platform and a data mesh. We propose three layers.

1. Runtime layer
2. Operating model layer
3. Enablement Llayer

Runtime Layer

Start with a few teams and focus on maturing the capabilities that create a self-service runtime platform. This includes:

• A hardened API gateway distributed via a golden image
• Automation of the data and API platform via infrastructure as code
• Automation in the data and API platform via APIOps, DataOps, and MLOps
• Integration with Identity Provider and Secrets / CA Management
• Observability and alerting setup
• Security procedures like penetration testing and security scanning
• High availability and disaster recovery strategy

The elements in the above list should be consolidated in a reference architecture to communicate and replicate consistently the setup.

Figure 9. Main capabilities to develop during the Runtime Layer phase.

Operating Model Layer

Once a base runtime layer is in place, mature your operating model towards a federated style and establish governance guardrails. This includes:

• Establish business metrics and KPIs.
• Set up a community of practice for API and data mesh to promote guardrails development and knowledge sharing.
• Train product owners on contract-first, hypothesis-driven development, and go-live strategies.
• Work on data mesh metadata, API documentation, and technical writing to create a data and API portal that API and data consumers can use for discovery and onboarding.
• Develop a federated operating model to ensure consistency across teams while still allowing for autonomy.

The federated operating model will be further developed as more business domains join the platform.

Figure 10. Main capabilities to develop during the Operating Model layer phase.

Enablement Layer

Now it's time to establish an enablement practice and full governance. This includes:

• Developing API guidelines and data guidelines as the foundation for the tools and services of enablement. These guidelines should cover design templates, sample code, SDK, mocking, testing frameworks, evolvability, and lifecycle management.
• Training is essential for ensuring that teams are equipped to follow the guidelines and use the enablement tools effectively.
• Set up support with a knowledge base, developer forum, and service desk to help teams troubleshoot issues and share knowledge.

Overall, a strong enablement practice and full governance are crucial for ensuring consistency and promoting the reuse of API and data assets in a federated operating model. By providing teams with the necessary guidelines, tools, and support, they can effectively collaborate and deliver high-quality solutions that meet business needs.

Figure 11. Main capabilities to develop during the Enablement Layer phase.

At this stage of maturity, the organization has evolved into a federation of business domains, each of which productizes business logic and data via an API layer. All of them are supported by a central enablement team that takes care of shared assets and enablement practices as illustrated in Figure 12.

Figure 12. Federation of business domains.

Conclusion

Data mesh and API platform initiatives aim to productize business domains as self-service digital assets.

By promoting product thinking and placing business domains at the heart of organizational design, both initiatives aspire to offer a self-service platform to foster autonomy and innovation.

We advocate bringing them together as a single workstream to leverage their strategic and tactical similarities and evolve them in synchrony to deliver on business outcomes. The end result is a federation of business domains exposing an API layer that supports a myriad of interaction styles.