Federated Connectivity: Unlocking Data Silos with API Gateways

"The whole is more than the sum of its parts." Aristotle is credited with this quote, and it's true in the world of data. Legacy systems typically approached their role in a limited manner.  Each system was intended to be used by a certain user set and handle well-defined processes and associated data. The result was a disintegrated environment with data being difficult to obtain, and frequently out of date. The parts couldn't easily cooperate to make a whole.  

This warranted the need for integration and messaging which introduced its own costs and complexities. Lately, with cloud native technologies and approaches to building applications, integration is arguably simpler through common protocols.  However, various cross-cutting concerns remain important. Cross-cutting concerns are also known as non-functional criteria, which are requirements that are not business logic, and nonetheless important. This includes security, reliability, logging, latency, and the like.

In this article, we'll go over how API gateways can help address these requirements.

The difficulty with data silos

For a good while, traditional applications typically tackled a few business processes and their associated data in a segregated manner. 

Let’s take an example of a benign application, perhaps to approve vacations and expense reports. And now let us extend this to various other core and supporting functions in organizations. The result is a proliferation of applications and their associated data silos along with a lack of integration, both at a departmental and business unit dimension.  

In fact, data silos often mimic, or overlay, organizational structures. This reduces cooperation and introduces manual steps, which, in turn, introduces higher chances for errors, and slows down cross-functional processes. A classic example will be to sign up for a service, only to still receive solicitation mail from the vendor the service was purchased from. This is a hint that the account management system and marketing system aren't synchronized.

Invariably, vendors such as SAP and Oracle made a good set of pre-integrated applications that are made available in modules. ERP implementations usually begin with HR and Finance, plus the modules necessary for core processes. Despite this, organizations discovered that it isn't possible to have that single system to manage all processes within the organizational boundaries, and 70 percent coverage is considered a good achievement.  

Enter ESBs, and the promise to unlock data silos, and integrate business processes across various units and cross-functional teams — as well as partners, suppliers, and customers, that are beyond the boundaries of the organization.

While ESBs continue to have a function, organizations discovered that they themselves can become the monolith, and become brittle over time. They become slow to change and hard to scale. This invariably results in data again being (again) difficult to share, or inaccessible outright, which indirectly affects productivity, process performance, and ultimately end-user or customer satisfaction. Is there a better way to break down data silos?

The allure of federated API architecture  

Decentralized architectures using APIs and microservices in cloud native environments are the tried and tested approach to breaking down data silos. Teams are able to tailor and connect their applications using a lingua franca of APIs, messaging, or streaming. Teams continue to implement transformational logic, or business rules, as they did in the ESB world. They do this using a cloud native approach, with all the benefits it provides of compute efficiency, elasticity, portability of containers, and on-demand capacity. Therefore with federated API architectures, we have:

System and data access across domains

As each team produces and maintains APIs, other teams may use them to access data or functionality necessary for their use cases.  Data is no longer missing or inaccessible.

Breaking down barriers

With data and systems being available for use, there are lowered barriers for building, rebuilding, scaling, integrating, and transforming applications.

Innovation incentives 

With both functions and data being accessible, teams can use this flexibility to launch new applications and innovations and more accurately adapt and react to the dynamics they face.  They are no longer limited by the systems they use in terms of access, or scale. In fact, they're rewarded for finding and capturing new opportunities, instead of reacting to them.

The fly in the ointment: Security concerns

This all sounds great, except by going to a federated API architecture, we now may have introduced a few concerns . . . 

Verifying identities

How can each API in an organization accurately identify that the consumer is known and to be trusted?

Managing access 

Assuming the identity of the consumer is known, how can each API endpoint determine what data an API consumer has access to?  And how can it stay up to date when this access is revoked?

Data protection needs

How can APIs ensure that the data being exchanged is not subject to eavesdropping or tampering during transit?  How can APIs help prevent data leakage?

Detecting threats

Assuming an API is made available to a variety of teams, how can we ensure that abuses, or potential abuses, are detected and diffused?

Clearly the move to a federated API architecture carries concerns as it does benefits. In fact, it arguably enlarges the attack surface compared to traditional legacy applications.

API gateways: Your trusted federated intermediary

A common approach is to ensure each API is designed with appropriate protections and mitigations for the aforementioned concerns. However, given that different teams build the APIs, we can also expect different implementations of those protections. It's unlikely that the implementations will be consistent, or synchronized.  This is a rather costly maintenance overhead as well.

Fortunately, API gateways can mitigate these concerns, and also be managed in a federated manner. By using an API gateway, APIs can be appropriately:

• Designed
• Tracked
• Documented
• Published
• Protected
• Observed
• Provisioned

As such, APIs can be relied upon to be both secure and available when needed.

Unlocking the API gateway magic

These are quite a few promises, but how can an API gateway accomplish this, and does it do so alone?  Let's examine the typical API gateway capabilities required to assist with these objectives.

Want to learn more about getting your microservices boundaries and API surface area right? Learn more about aligning microservices to business process models, providing the right information architecture for APIs, and more in the on-demand webinar Building APIs as a Product.

Protocols for access management  

In a federated API landscape, the identification of a consumer, and access management requires support for common Authentication and Authorization protocols, possibly including a combination of:

• API key
• Basic authentication credentials
• JSON web token
• HMAC
• LDAP
• SAML
• OAuth
• OpenID Connect

While some of these mechanisms may be controlled in the gateway, the ideal approach is to rely on an identity provider as the source of truth. A good gateway will have support for a variety of identity providers.

By standardizing the mechanism with which API consumers gain access to APIs, a significant concern for API security is alleviated.

Developer enablement

Developing APIs is one matter, making them consumable is another. For an API to be consumable it should be well documented, discoverable, and accessible in a streamlined manner. This is why an API gateway will either include a developer portal or offer integration with one so that developers can find APIs and gain access to services. 

A developer portal should, at minimum, allow hosting of API specification, examples, and other documentation. More ambitious portals will offer credential and access management, role-based access control, and analytics for showback and chargeback purposes.

Analytics for visibility

If we think of APIs as organisms that grow, thrive, and eventually are retired, then analytics help us determine their health. 

We use analytics to monitor the health of APIs, how often they fail, how fast or slow they are, how frequently they're being used, and in what context. API Gateways can capture and propagate such information. There are a variety of off-the-shelf and OSS solutions for monitoring, logging, and tracing. A good API gateway will integrate with these systems.

Policy enforcement

Finally, it's quite natural for non-functional criteria to cover other concerns, including:

• Traffic controls, such as canary, rate limiting, and caching
• Static security posture support, such as IP restriction, CORS support, and bot detection
• Transformation of headers, parameters, or bodies of requests 
• Support for third party, or serverless integrations

A level of extensibility is always good, as long as API developers do not turn the API gateway into an application framework. As a general rule of thumb, anything low to moderate complexity when it comes to extensibility and remains concerned with non-functional criteria is acceptable. Orchestration, aggregation, or composition are generally not activities that an API gateway is preferred for. 

Gateway-powered API federation examples

Many organizations took on this approach with predictable success. If you're curious to learn more, check out the following case studies: 

• Yahoo! Japan accelerates development and saves development efforts in building APIs
• A prominent hospital system in the USA leverages API with partners
• Mercedes-Benz optimizes digital interactions using an API gateway

Summary: API gateways connecting systems and organizations

Federated API architectures have proven their superiority over traditional legacy systems in supporting speed, scalability, and innovation.  Appropriately, a scalable, cloud-native API Gateway is a natural complement to accelerate the adoption of federated APIs.  With this approach, data silos are no longer an obstacle for organizations, and the inherent complexity of and risks of decentralization is mitigated among other requirements using an API Gateway.

If you're on your journey to adopt a federated architecture and are considering an API gateway for security purposes as discussed herein, check out How to Choose the Right API Gateway for Your Business.