Exposing and Controlling Apache Kafka® Data Streaming with Kong Konnect and Confluent Cloud

Data streaming enables companies to build highly scalable and loosely coupled real-time applications. These applications can manage significant concurrency demands and simplify the development of services. Conversely, it's become critical to expand access to this infrastructure to a variety of new entities, including external applications that produce events for internal systems to process.

This exposure serves to foster collaboration among development teams, establishing a standardized, programming language–agnostic interface. However, merely exposing the infrastructure isn't enough; stringent control over exposure is imperative.

Here, the Event Gateway pattern emerges as a sophisticated solution, enhancing your data streaming platform with robust API management capabilities. It addresses both exposure and consumption control requirements, abstracting robust policies from the core data streaming infrastructure. Within this integration architecture, the API management platform assumes responsibility for critical tasks such as authentication/authorization, rate limiting, log processing, and real-time monitoring.

In this blog post, we'll delve into the pattern implementation, describing how Kong Konnect establishes an Event Gateway atop Confluent Cloud with advanced API management functionalities.

Confluent Cloud 

Built by the original creators of Kafka, Confluent Cloud is a fully managed, cloud-native, and complete data streaming platform available everywhere businesses need it—in the cloud, across clouds, on-premise, and hybrid environments. Confluent provides:

• 10x faster performance than open source Kafka, GBps+ elastic scalability, a 99.99% uptime SLA, and infinite storage—all made possible by Kora, the Kafka engine built for the cloud
• 120+ pre-built connectors with enterprise-grade security, reliability, and support (80+ provided fully managed)
• Serverless Apache Flink® service, for stream processing fully integrated with Kafka
• Enterprise-grade security and governance controls
• And much more

Overall, Confluent Cloud enables organizations to leverage the full power of Kafka data streaming without the overhead of open source infrastructure management, making it easier to build and operate real-time data pipelines.

Kong Konnect

Kong Gateway, the cornerstone of the Kong Konnect API management infrastructure, stands out for its platform-agnostic nature, scalability, and exceptional performance enabled by its plugin-based extensibility. Functionality like proxying, routing, load balancing, and health checking consolidate within Kong, serving as a central orchestrator for microservices or conventional API traffic.

A pivotal feature of Kong Gateway lies in its extensibility, facilitated through an extensive list of plugins that implement specific policies such as Authentication/Authorization, Rate Limiting, Proxy Caching, Requests and Responses Transformation, Traffic Control, Observability, and more.

Kong Konnect and Confluent Cloud integration

For transformation policies, the Confluent plugin abstracts the Kafka infrastructure complexities, transforming REST requests into new messages within existing Event Processing infrastructure.

In summary, Kong Gateway's data plane exposes and manages service and application consumption through multiple policies and protocols. Acting as a gateway, Kong safeguards Kafka infrastructure and other backend systems, offering standardized access to consumers through REST, Websockets, gRPC, and other protocols.

The following diagram illustrates how the components work together:

Kong Gateway inherently supports hybrid deployments, allowing the exposure and protection of workloads across diverse platforms concurrently. The Konnect control plane supports the administrative tasks, while the data plane handles the API consumers' requests. In a hybrid deployment scenario, the control plane and data plane operate in distinct environments, ensuring elastic data plane support for varying throughput requirements.

The diagram considers that the Kafka Cluster deployed in Confluent Cloud is not being exposed to external applications or services. In this sense, the Gateway is responsible not just to expose your Kafka Platform but, more importantly, to control such exposure with typical policies like Authentication, Rate Limiting, etc.

In an architecture like this, Kong will translate a regular REST request into Kafka messages to be posted in existing topics.

Conclusion

The synergy between Confluent Cloud and Kong Konnect heralds a new era of connectivity in data streaming architectures. By combining the collective capabilities of these technologies, organizations can fortify their infrastructure with robust policies and streamlined traffic management, laying a solid foundation for resilient and scalable event streaming-based applications.

Contact sales@konghq.com and sales@confluent.com if you have questions or need support.

Not yet a Confluent customer? Start your free trial of Confluent Cloud. New signups receive $400 to spend during their first 30 days.