The Problem with Traditional Kafka ACLs

Kafka ACLs are powerful, but they come with significant tradeoffs:

• Static Definition: They are defined at the broker level and lack context awareness (e.g., who the caller is, their role, or current environment).
• Central Bottlenecks: They require central coordination for every single topic change.
• Scaling Friction: They don’t scale well across multi-team or multi-tenant environments.

In practice, this leads to over-permissive access policies (the dreaded "just give them topic-*"), operational bottlenecks, and configuration drift across environments.

The Shift: Identity-Aware Policies

Instead of defining access in Kafka, you define it in your identity provider.

Using Kong Identity, you can embed authorization data directly into a token and let Kong Event Gateway enforce it. The token becomes the single source of truth for what a Kafka client is allowed to access, including specific topic names, topic patterns, and contextual scopes (team, environment, application).

At a high level, the architecture looks like this:

1. A Kafka client authenticates using OAuth (Bearer token).
2. The token is issued by Kong Identity (OIDC) and contains a custom claim (e.g., topics).
3. Kong Event Gateway intercepts the request, validates the token via JWKS, extracts the claim, and applies it dynamically to ACL policies.
4. The client only sees and accesses the allowed topics—zero Kafka ACL updates required.

Step 1: Configuring Token Claims in Kong Identity

The first step is setting up the identity provider to attach specific claims to user scopes.

For this implementation, we configure a custom JSON array claim called topics. This array contains the exact names or expressions of the topics the client is allowed to access (e.g., ["clicks", "transactions"]). This claim is attached to a specific scope (like "Team A" within a "payments" context).

When the Kafka client requests a token, Kong Identity generates a JWT with this array embedded directly in the payload:

{
 "sub": "client-id-123",
 "scope": "payments-team-a",
 "topics": ["clicks", "transactions"]
}

Step 2: Securing the Virtual Cluster

Next, we configure our Kafka virtual cluster within Kong Event Gateway to enforce authentication and read these claims.

We enable OAuth Bearer authentication, point the configuration to the Kong Identity JWKS endpoint, and enable token claim extraction so the gateway can access the payload data at request time:

auth:
 type: oauthbearer
 jwks_uri: https://<kong-identity>/.well-known/jwks.json

Step 3: Crafting the Dynamic Identity-Aware ACL Policy

With the token validated and claims extracted, we apply an ACL policy to the virtual cluster. Instead of hardcoding topics, we define an expression-based ACL policy that reads directly from the token:

acl:
 topics: expression(auth.claims.topics)

Using Kong's expression language, auth.claims.topics dynamically resolves into a list of allowed topics per request. You can still mix static lists, expressions, and patterns, but the real power comes from letting identity drive the authorization layer.

Seeing It in Action

Let's look at how this behaves from the client's perspective using a tool like kafkactl.

• Direct Backend Access: If an admin queries the raw backend Kafka cluster, they see a vast list of internal topics (e.g., payments.clicks, payments.transactions, payments.user_actions, plus system topics).
• Gateway Access (Without Identity ACL): Querying the virtual cluster with standard routing might strip prefixes and return all localized topics (clicks, transactions, user_actions).
• Identity-Aware Access: When the client connects using their generated JWT, Kong Event Gateway intercepts the request. Because their token specifically contains the claim ["clicks", "transactions"], running kafkactl context payments-user list topics will only return clicks and transactions.

The Power of Dynamic ACLs

The true power of this architecture shines when permissions need to change.

If a team suddenly needs access to user_actions, you do not touch the Kafka cluster or update the gateway's ACL policies. You simply update the scope claims in Kong Identity to ["clicks", "user_actions"].

The next time the Kafka client fetches a token, the access is immediately updated. The system adapts instantly when identity changes.

Advanced Patterns

Once you have this foundation, you can extend it further using Kong's expression language:

Topic Patterns

"topics": ["payments-*", "clicks"]

Environment Isolation

"topics": ["dev-*"]

Role-Based Access

"roles": ["producer"]
"topics": ["transactions"]

Expression:

expression(auth.claims.roles.contains("producer") ? auth.claims.topics : [])

Conclusion

By shifting Kafka access control to Kong Event Gateway and utilizing JWT token claims, you move from static, infrastructure-defined ACLs to dynamic, identity-driven policies.

This approach reduces operational overhead, eliminates the risks of misconfigured static ACLs, and centralizes your access management. It is a foundational shift in how event systems are secured and operated.

Instead of asking, "What topics should this client access?" you start asking, "Who is this client, and what should they be allowed to do right now?" And the answer lives entirely in the token.

Ready to move beyond static Kafka ACLs and simplify your access management? Start building a secure, identity-driven streaming infrastructure today. [Try Kong Event Gateway](https://konghq.com/products/event-gateway)Try Kong Event Gateway and see how easy it is to take control of your event streams.