Running Apache Kafka on Amazon Managed Streaming for Apache Kafka (MSK) gives you a managed broker with no ZooKeeper to operate, automated patching, and multi-AZ replication handled by the service. What it doesn't give you is a safe, governed way to expose Kafka access beyond your VPC boundary.
That problem looks simple on the surface. It isn't. And how you solve it has significant implications for security posture, operational complexity, and monthly cost.
This article walks through the DMZ pattern for MSK using Kong Event Gateway: how the architecture works, how to configure it on AWS, and why it's the right design for any organization that needs to share Kafka access across team boundaries, VPCs, or organizational perimeters.
Amazon MSK brokers live in private subnets by default. That's the right default. Kafka's protocol wasn't designed for untrusted networks — it has no concept of rate limiting, no built-in field-level encryption, and its ACL model assumes you control who can reach the cluster at the network layer.
When you need to give external teams, partners, or services in other VPCs access to MSK, you have a few options — and most of them are worse than they look.
MSK Public Access (the obvious answer): MSK supports enabling public access on brokers. This works, but it's operationally messy and expensive. Public access requires brokers to use TLS, forces you to manage per-broker TLS certificates resolvable from the internet, and exposes your broker topology directly. Every broker gets a public DNS name. Clients bootstrap against those names and connect to individual brokers directly. There's no policy enforcement layer, no centralized auth, and no place to add access controls without modifying every application. Data transfer out through public broker endpoints adds to your AWS bill — on high-throughput topics, this accumulates fast.
VPC Peering / Transit Gateway (the internal answer): You can peer VPCs or use Transit Gateway to give services in other VPCs direct access to MSK. This works for trusted internal teams but doesn't extend to external partners, doesn't provide per-team access control, and doesn't solve the authentication or observability problem.
Neither approach gives you a single enforcement point. Neither gives you the ability to apply different policies for different consumers. Neither gives you field-level encryption or schema validation. Both expose your broker topology to anyone with network access.
In classical network architecture, a DMZ (demilitarized zone) is a network segment that sits between an untrusted external network and a trusted internal network. Services in the DMZ are reachable from outside but cannot themselves reach the internal network arbitrarily — traffic flows through controlled paths, and the internal network remains hidden from external parties.
Applied to Kafka on AWS:
The gateway is the only component that crosses the boundary. Kafka clients — wherever they live — connect to the gateway. The gateway authenticates them, applies policy, and forwards traffic to MSK over a private network path. MSK never has public access enabled. Broker IP addresses are never exposed outside the VPC.
One controlled crossing point, not a porous boundary.
Use three subnet tiers across two availability zones:
The NLB sits in public subnets and receives inbound TCP on ports 9092–9094 from clients. It forwards to the gateway in the private app subnets. The gateway connects to MSK on port 9096 in the private data subnets. MSK has no route to the public subnets.
Three security groups form the DMZ chain:
sg-nlb (Network Load Balancer):
Inbound: TCP 9092–9094 from 0.0.0.0/0 (or scoped CIDRs for partner access)
Outbound: TCP 9092–9094 to sg-keg
sg-keg (Kong Event Gateway):
Inbound: TCP 9092–9094 from sg-nlb
Outbound: TCP 9096 to sg-msk (SASL/SCRAM over TLS — gateway service account)
TCP 443 to 0.0.0.0/0 (Konnect control plane sync)
sg-msk (MSK Cluster):
Inbound: TCP 9096 from sg-keg (SASL/SCRAM over TLS — gateway service account only)
Outbound: (none required — broker replication stays within sg-msk)
The critical property: sg-msk has no inbound rule from the internet, the NLB, or any subnet outside the app tier. MSK is unreachable except through the gateway security group. Port 9096 is the MSK SASL/SCRAM over TLS port — distinct from the client-facing ports (9092–9094) used between the NLB and the gateway.
An NLB is required here — ALB is HTTP/HTTPS only and cannot proxy the Kafka binary protocol over TCP.
Kong Event Gateway uses a port-mapping strategy to handle multiple MSK brokers behind a single address. The gateway assigns one external port per MSK broker: clients always bootstrap on port 9092, and then the gateway returns per-broker metadata pointing clients to ports 9093, 9094, and so on. The NLB has one listener and one target group per port in the range.
This means the NLB needs one listener per port in the range, not a single listener on 9092. Add one port per broker when you scale MSK beyond two brokers.
For production multi-tenant deployments where different teams need separate bootstrap hostnames, the gateway also supports SNI-based routing: a TLS listener on the NLB with a wildcard ACM certificate (e.g. *.kafka.acme.com) lets the gateway read the TLS SNI hostname and route each connection to the matching Virtual Cluster. Teams connect to team-payments.kafka.acme.com:9092 and partner-logistics.kafka.acme.com:9092 — both resolving to the same NLB, routed at the gateway layer.
MSK 4.x uses KRaft mode — no ZooKeeper. Enable SASL/SCRAM on your MSK cluster and create a service account credential. MSK 4.x requires a kafka.m5.large instance type or larger, and SCRAM secret names must start with AmazonMSK_ and be encrypted with a customer-managed KMS key:
# Store the gateway's SCRAM credential in Secrets Manager
aws secretsmanager create-secret \
--name AmazonMSK_keg-service-account \
--kms-key-id alias/msk-scram-key \
--secret-string '{"username":"keg-service-account","password":"<strong-password>"}'
# Associate the secret with the MSK cluster
aws kafka batch-associate-scram-secret \
--cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/my-msk-cluster/... \
--secret-arn-list arn:aws:secretsmanager:us-east-1:...:secret:AmazonMSK_keg-service-account
This service account is the only identity that can connect to MSK directly. All client identities are managed at the gateway layer — MSK sees only the gateway's single service account regardless of how many teams or partners are connected.
Kong Event Gateway is managed through Konnect, Kong's cloud control plane. The gateway data plane runs on ECS Fargate in your VPC — you own the runtime — while Konnect holds the configuration, issues data plane certificates, and provides the management API. The gateway pulls its config from Konnect over an mTLS-authenticated connection on port 443.
The Backend Cluster tells the gateway how to reach MSK. Create it in the Konnect UI or via Terraform:
resource "konnect_event_gateway_backend_cluster" "msk" {
name = "msk"
gateway_id = konnect_event_gateway.keg.id
authentication = {
sasl_scram = {
algorithm = "sha512"
username = "keg-service-account"
password = "<scram-password>"
}
}
bootstrap_servers = [
"b-1.my-msk-cluster.abc123.c2.kafka.us-east-1.amazonaws.com:9096",
"b-2.my-msk-cluster.abc123.c2.kafka.us-east-1.amazonaws.com:9096"
]
tls = { enabled = true }
}
Each team or partner gets their own Virtual Cluster — an isolated logical view of MSK with dedicated credentials and scoped access policies. The gateway enforces ACLs at the Virtual Cluster layer; MSK sees only the gateway's single service account.
resource "konnect_event_gateway_virtual_cluster" "internal" {
name = "internal"
gateway_id = konnect_event_gateway.keg.id
destination = { id = konnect_event_gateway_backend_cluster.msk.id }
acl_mode = "enforce_on_gateway"
dns_label = "internal"
authentication = [{
oauth_bearer = {
mediation = "terminate"
jwks = {
endpoint = "${konnect_identity_auth_server.kafka.issuer}/.well-known/jwks"
}
}
}]
}
Rather than federating with an external IdP, Kong Event Gateway ships with Kong Identity — a built-in OAuth 2.0 authorization server that issues short-lived tokens directly. This removes the dependency on Okta, Azure AD, or any external IdP for service-to-service Kafka access, while remaining compatible with external IdPs when you need them.
Three resources wire it up:
# The issuer — exposes a JWKS endpoint the virtual cluster validates tokens against
resource "konnect_identity_auth_server" "kafka" {
name = "kafka-auth-server"
audience = "https://kafka.example.com"
}
# The scope clients must request
resource "konnect_identity_auth_server_scope" "kafka" {
auth_server_id = konnect_identity_auth_server.kafka.id
name = "kafka"
}
# Machine-to-machine client (client_credentials flow)
resource "konnect_identity_auth_server_client" "kafka_client" {
auth_server_id = konnect_identity_auth_server.kafka.id
grant_types = ["client_credentials"]
allow_scopes = [konnect_identity_auth_server_scope.kafka.id]
access_token_duration = 3600
}
Kong Event Gateway ACL policies evaluate [CEL (Common Expression Language)](https://cel.dev/)CEL (Common Expression Language) expressions, which lets you write identity-aware access rules that adapt to the token presented at runtime — not just static principal lists.
The gateway ships with a read-only policy that applies to every authenticated user with no condition:
resource "konnect_event_gateway_cluster_policy_acls" "readonly_all" {
name = "readonly-all"
gateway_id = konnect_event_gateway.keg.id
virtual_cluster_id = konnect_event_gateway_virtual_cluster.internal.id
# no condition — applies to all authenticated users
config = {
rules = [
{ action = "allow", operations = [{name="describe"},{name="read"}],
resource_type = "topic", resource_names = [{match="*"}] },
{ action = "allow", operations = [{name="describe"},{name="read"}],
resource_type = "group", resource_names = [{match="*"}] }
]
}
}
To grant elevated access to specific clients, add a custom JWT claim that encodes which topics a client is allowed to write to. Kong Identity supports static claim injection at token issuance time:
resource "konnect_identity_auth_server_claim" "topics" {
auth_server_id = konnect_identity_auth_server.kafka.id
name = "topics"
value = jsonencode(["orders", "payments"]) # embedded in every token
include_in_token = true
include_in_scopes = [konnect_identity_auth_server_scope.kafka.id]
}
A token issued to this client will carry "topics": ["orders", "payments"]. The ACL policy can then read that claim at request time using a CEL expression:
condition: "'topics' in context.auth.token.claims"
resource_names: context.auth.token.claims.topics # resolved to ["orders","payments"] at runtime
operations: [describe, describe_configs, read, write, create, delete, alter, alter_configs]
Broker topology is completely hidden. External clients connect to the NLB address. They never see b-1.my-msk-cluster.abc123.kafka.us-east-1.amazonaws.com. If you replace the MSK cluster, move regions, or change broker counts, the client configuration doesn't change.
One enforcement point for all access. Every produce, every consume, every connection flows through the gateway. ACL changes take effect immediately. Revoking a partner's access is one policy change — not a Kafka ACL update plus credential rotation plus TLS certificate revocation.
Audit trail without custom instrumentation. Every connection, authentication event, and authorization decision is emitted as OpenTelemetry traces and metrics. When a compliance audit asks "who consumed from the payments topic between March 1 and March 31," the answer is a query, not an investigation.
Add capabilities without touching MSK or clients. Schema validation, field-level encryption, record filtering, header modification — all gateway policies. Adding PII encryption to a topic doesn't require changes to MSK or to any existing producer.
The DMZ pattern only delivers its full value when the gateway at the crossing point is itself easy to operate. A stateful gateway — one that holds connection state, caches credentials locally, or embeds configuration in its runtime — defeats the purpose. If the gateway becomes a fragile, hand-managed component, you've just moved operational complexity from MSK to the gateway.
Kong Event Gateway on ECS Fargate sidesteps this entirely: the gateway is stateless and ephemeral by design.
All configuration lives in Konnect, not on the gateway. Virtual Clusters, ACL policies, authentication settings, listener configuration — none of it is stored on the running container. An ECS task is a pure data plane: it connects to Konnect on startup, pulls its full configuration over an mTLS-authenticated channel, and starts serving. Terminate the task and start a new one — it's fully operational in seconds, with no warm-up and no state migration. Scale from one task to five — all five have identical configurations immediately. This means:
The enterprise Kafka backbone stays untouched. Internal teams — data engineers, application teams, ML pipelines — connect to MSK directly or through their own internal paths. Their credentials, ACLs, consumer group configurations, and broker-side policies are theirs. External access is a perimeter concern. Adding it doesn't require touching any of that.
When you need to onboard a new external partner, the sequence is:
When a partner relationship ends or a credential is compromised:
The blast radius of external access is bounded. Because the gateway's SCRAM service account is the only identity MSK ever sees from the external direction, a security incident on the external perimeter — a leaked client credential or an over-permissioned Virtual Cluster — cannot propagate into MSK's ACL space. The gateway service account holds exactly the permissions the gateway needs to proxy traffic; it doesn't inherit the permissions of any external client.
Audit and compliance are concentrated at the perimeter. Every external connection enters through one place. OpenTelemetry traces carry the client identity, the Virtual Cluster, the topic accessed, the operation, and the outcome — all emitted by the gateway before the request ever touches a broker. Compliance teams audit one log stream, not a mix of MSK CloudWatch metrics, broker-side ACL logs, and per-team credential audit trails scattered across accounts.
The MSK cluster grows in stability as external access grows in complexity. That's the core property this architecture is designed to deliver. Onboarding a tenth external partner is as simple as the first: one Virtual Cluster, one set of credentials, and one policy. The enterprise Kafka backbone never knows it happened.
The full Terraform implementation of this architecture — VPC, security groups, MSK cluster, ECS Fargate service, NLB, and all Konnect resources — is available in the companion repository at [github.com/hguerrero/kong-event-gw-aws](https://github.com/hguerrero/kong-event-gw-aws)github.com/hguerrero/kong-event-gw-aws. Deploy the complete stack against a real MSK cluster in two terraform apply steps.
The Kong Konnect Terraform provider covers all Event Gateway resources — Backend Clusters, Virtual Clusters, Listeners, Kong Identity, and ACL policies — making this architecture fully GitOps-compatible from day one.
Kong Konnect has a free trial at [cloud.konghq.com](https://cloud.konghq.com)cloud.konghq.com. Full Event Gateway reference documentation is at [https://developer.konghq.com/event-gateway/](https://developer.konghq.com/event-gateway/)https://developer.konghq.com/event-gateway/.
Kong Event Gateway deploys as a standard container workload on ECS Fargate and connects to AWS MSK via SASL/SCRAM over TLS on port 9096.
Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services. Ultimately, we may need to grant access to such infrastructure to
We announced the Kong Premium Technology Partner Program at API Summit 2024, and Confluent was one of the first in the program. This initial development was all about ensuring that the relationship between Kong and Confluent — from a business an
The architecture is built around two data planes, both managed by Kong. The first is the sync data plane — Kong AI Gateway — which handles all synchronous traffic between your agents and the outside world. Every inbound client request, every outbo
Before making the Kafka argument, it's worth being precise about what the memory log actually needs to do. A durable commit log for agentic AI isn't just a message bus. It's the substrate that makes replay, auditability, and governance possible. Tha
The widespread adoption of Kafka and event streaming platforms is evident across several enterprises, where they serve as the backbone of critical operations, ranging from financial transactions to AI inference pipelines. However, in the domains of
The Problem with Traditional Kafka ACLs Kafka ACLs are powerful, but they come with significant tradeoffs: Static Definition: They are defined at the broker level and lack context awareness (e.g., who the caller is, their role, or current environm
The challenges of an acquisition frequently appear in a number of critical areas, especially when dealing with a platform as important as Kafka: API Instability and Change : Merged entities frequently rationalize or re-architect their services, whic