We announced the Kong Premium Technology Partner Program at API Summit 2024, and Confluent was one of the first in the program. This initial development was all about ensuring that the relationship between Kong and Confluent — from a business and product perspective — fully represented our joint belief that the world of data streaming and the world of APIs are converging. Today, we’ve taken that partnership even further, with Kong joining the Connect with Confluent technology partner program.
To get started using this integration, head to https://docs.konghq.com/hub/kong-inc/confluent. Read on as we cover how to expose and control Apache Kafka® Data Streaming with Kong Konnect and Confluent Cloud.
Data streaming enables companies to build highly scalable and loosely coupled real-time applications. These applications can manage significant concurrency demands and simplify the development of services. Conversely, it's become critical to expand access to this infrastructure to a variety of new entities, including external applications that produce events for internal systems to process.
This exposure serves to foster collaboration among development teams, establishing a standardized, programming language–agnostic interface. However, merely exposing the infrastructure isn't enough; stringent control over exposure is imperative.
Here, the event gateway pattern emerges as a sophisticated solution, enhancing your data streaming platform with robust API management capabilities. It addresses both exposure and consumption control requirements, abstracting robust policies from the core data streaming infrastructure. Within this integration architecture, the API management platform assumes responsibility for critical tasks such as authentication/authorization, rate limiting, log processing, and real-time monitoring.
In this blog post, we'll delve into the pattern implementation, describing how Kong Konnect establishes an event gateway atop Confluent Cloud with advanced API management functionalities.
Built by the original creators of Kafka, Confluent Cloud is a fully managed, cloud-native, and complete data streaming platform available everywhere businesses need it—in the cloud, across clouds, on-premise, and hybrid environments. Confluent provides:
Overall, Confluent Cloud enables organizations to leverage the full power of Kafka data streaming without the overhead of open source infrastructure management, making it easier to build and operate real-time data pipelines.
Kong Gateway, the cornerstone of the Kong Konnect API management infrastructure, stands out for its platform-agnostic nature, scalability, and exceptional performance enabled by its plugin-based extensibility. Functionality like proxying, routing, load balancing, and health checking consolidate within Kong, serving as a central orchestrator for microservices or conventional API traffic.
A pivotal feature of Kong Gateway lies in its extensibility, facilitated through an extensive list of plugins that implement specific policies such as Authentication/Authorization, Rate Limiting, Proxy Caching, Requests and Responses Transformation, Traffic Control, Observability, and more.
For transformation policies, the Confluent plugin abstracts the Kafka infrastructure complexities, transforming REST requests into new messages within existing Event Processing infrastructure.
In summary, Kong Gateway's data plane exposes and manages service and application consumption through multiple policies and protocols. Acting as a gateway, Kong safeguards Kafka infrastructure and other backend systems, offering standardized access to consumers through REST, Websockets, gRPC, and other protocols.
The following diagram illustrates how the components work together:
Kong Gateway inherently supports hybrid deployments, allowing the exposure and protection of workloads across diverse platforms concurrently. The Konnect control plane supports the administrative tasks, while the data plane handles the API consumers' requests. In a hybrid deployment scenario, the control plane and data plane operate in distinct environments, ensuring elastic data plane support for varying throughput requirements.
The diagram considers that the Kafka Cluster deployed in Confluent Cloud is not being exposed to external applications or services. In this sense, the Gateway is responsible not just to expose your Kafka Platform but, more importantly, to control such exposure with typical policies like Authentication, Rate Limiting, etc.
In an architecture like this, Kong will translate a regular REST request into Kafka messages to be posted in existing topics.
The synergy between Confluent Cloud and Kong Konnect heralds a new era of connectivity in data streaming architectures. By combining the collective capabilities of these technologies, organizations can fortify their infrastructure with robust policies and streamlined traffic management, laying a solid foundation for resilient and scalable event streaming-based applications.
Ready to get started using the integration? Visit the docs to learn more.
Not yet a Confluent customer? Start your free trial of Confluent Cloud. New sign-ups receive $400 to spend during their first 30 days.
The challenges of an acquisition frequently appear in a number of critical areas, especially when dealing with a platform as important as Kafka: API Instability and Change : Merged entities frequently rationalize or re-architect their services, whic
Since our last Kong Konnect monthly update , we've been hard at work to deliver some more powerful features in Kong Konnect , including consumer groups, dynamic client registration using Curity and Auth0, plugin ordering, and streamlined Docker ru
InfoWorld’s annual awards recognize the most innovative software development, DevOps, cloud, data management, and AI/ML products on the information technology landscape. We are extremely proud to see Kong Konnect recognized for its role in unifying
Observability has become critical to ensuring the effective monitoring of application and system performance and health. It focuses on understanding a system’s internal state by analyzing the data it produces in the context of real-time events and a
Understanding and monitoring the performance and health of applications and systems is critical. This is where observability comes into play. Observability is about gaining a comprehensive understanding of a system's internal state by analyzing the
Kong Konnect and our code in production Kong Konnect is Kong’s infrastructural SaaS solution. We run the control planes and API management applications for the data planes (API gateway and mesh) that are run by our customers to power their APIs. So
APIs are essential to modern applications, but managing access and security policies can be complex. Traditional access control mechanisms can fall short when flexible, scalable, and fine-grained control over who can access specific resources is nee