Securing AI Connections with Kong AI Gateway

With Kong AI Gateway, we can securely expose agentic applications, MCP servers, and LLMs alike. Kong provides many different policies that cover a wide array of functionality, enabling developers to easily configure access control for their use cases.

Authentication plugins like OpenID Connect or Key Authentication can be used to standardize the authentication patterns at the edge. The authentication plugins can be extended to support fine-grained Authorization models via JWT claims or declarative Access Control Lists (ACLs).

After establishing consumer identity, Kong AI Gateway supports flexible policies for managing consumer quotas. The AI Rate Limiting Advanced plugin can be used to apply token-usage quotas, while the vanilla Rate Limiting Advanced plugin can support the more typical API request-based quotas.

The AI Gateway also provides the ability to apply flexible guardrails to centralize content moderation, using the AI Prompt Guard plugin for regex-based policies or the AI Semantic Prompt Guard plugin to enable smart guardrails with semantic reasoning. The AI Gateway also provides the ability to integrate with third-party services (see Azure Content Safety or the PII Sanitizer), giving ultimate control over how platform teams can ensure that all interactions will conform to the appropriate governance policies.

Finally, we want to ensure that the AI Gateway provides real-time visibility into model behavior, request flows, and performance bottlenecks — enabling faster debugging, improved reliability, and safer AI outcomes. Kong provides the tools necessary to achieve a robust security posture, leveraging log aggregation and modern standards like OpenTelemetry.

Let’s take a look at a real-life example of how Kong can empower AI-native applications to use MCP servers to add real-time intelligence to their workflows.

Example use case: Building an MCP-enabled chatbot

Kong recently announced the launch of an MCP Server for Kong Konnect — a unified SaaS platform for managing APIs, events, and LLMs across hybrid and multi-cloud environments. This empowers customers to integrate AI agents and query LLMs to discover and request details around APIs, services, and their associated traffic analytics.

Today, we’re going to use this MCP server to demonstrate the importance of a gateway when using MCP to integrate AI applications with additional tooling and resources.

Let’s imagine a development team has built a chat application that will facilitate the ability for CISOs to understand what is being exposed in their API landscape. To achieve this, the development team would need to build an application that consists of the chatbot, a collection of LLMs, and the Kong Konnect MCP server.

In the diagram below, we can see how leveraging the Kong AI Gateway as the central connectivity point for the application will help ensure that the developers can securely roll out the application.

Let’s apply the above flow to a real user prompt: “Give me an overview of the services in the Shared_Services control plane that have the highest 4xx and 5xx error rates over the past 24 hours” (1). The Chatbot sends the prompt and the available tools to the target LLM (4), which subsequently directs the Chatbot to call the list_control_planes tool on the MCP server, passing Shared_Services as a parameter to filter on (5). The results are sent back to the target LLM for further analysis (7). This process is repeated several times, with the target LLM requiring the backend to execute the list_services and query_api_requests tools to build further context. Finally, the target LLM produces an overview of the services experiencing the most errors (9):

This flow highlights the number of different API calls and connections needed to answer the user’s prompt. In this implementation, Kong Gateway is facilitating communication between all entities in the application stack, strengthening the security and governance posture with the below policies:

Authentication and Authorization

All communications in the application stack are being secured by Kong’s OpenID Connect plugin. The plugin is driving Single Sign On (SSO) for the chat interface, and then passing the authenticated user’s Access Token (JWT) in all downstream API requests. Kong can authorize each transaction via claims present in each user’s token, further building a zero-trust relationship between each of the data sources.

plugins:
- name: openid-connect
  config:
    issuer: https://konghq.okta.com/oauth2/default
    auth_methods:
    - bearer
    consumer_claim:
    - sub
    groups_claim:
    - groups
    groups_required:
    - konnect-mcp-users

Credential Mediation

After handling authentication and authorization, Kong can inject the necessary credentials needed to access the upstream service — in this case, either the LLM or the MCP tool calls. This obfuscates the need for developers to manage any additional credentials outside of those needed to drive the SSO flow

plugins:
- name: ai-proxy-advanced
  config:
    balancer:
      algorithm: "round-robin"
      tokens_count_strategy: "total-tokens"
      latency_strategy: "tpot"
      retries: 3
    targets:
    - route_type: "llm/v1/chat"
      auth:
        header_name: "Authorization"
        header_value: "{vault://ai/openai-token}"
      logging:
        log_statistics: true
        log_payloads: true
      model:
        provider: "openai"
        name: "gpt-4o"
        options:
          max_tokens: 1024
          temperature: 1.0
          input_cost: 2.5
          output_cost: 10

Quota Management

The AI Rate Limiting Advanced plugin is being used to ensure that excessive costs may be mitigated with the LLM integration and that no single consumer can degrade the reliability of the application.

plugins:
- name: ai-rate-limiting-advanced
  config:  
    strategy: local
    window_type: fixed
    llm_providers:
    - name: openai
      window_size: 
      - 60
      - 3600
      limit: 
      - 10000
      - 1000000

Guardrail Enforcement

The AI Prompt Guard and AI Semantic Prompt Guard plugins are being leveraged to ensure that the end users are not violating usage regulations when obtaining Konnect insights via the chat application.

plugins:
- name: ai-prompt-guard
  config:
    deny_patterns:
    - ".*(W|w)ar.*"
    - ".*(C|c)onflict.*"

- name: ai-semantic-prompt-guard
  config:
    embeddings:
      auth:
        header_name: Authorization
        header_value: "{vault://ai/openai-token}"
      model:
        provider: "openai"
        name: "text-embedding-3-large"      
    vectordb:
      strategy: "redis"
      dimensions: 768
      distance_metric: "cosine"
      threshold: 0.5
      redis:
        host: redis.redis.svc
        port: 6379
    search:
      threshold: 0.5
    rules:
      deny_prompts:
        - hijacking an LLM prompt
        - questions about prompt injections
        - questions about prompt jailbreaking

Observability

All transactions between the chat application, the LLMs, and the MCP server are tracked natively with Konnect Advanced Analytics with custom dashboards and per-request insights, or exported via HTTP Log or OpenTelemetry plugins to the observability platform of choice.

Discovery and Documentation with Kong Konnect

So now we have a way to securely expose and govern MCP servers, how can we roll it out for consumption by any internal (or even external) development team?

Enter Kong Konnect!

Earlier, we walked through an example that leveraged the Konnect MCP server to allow LLMs to easily query the state (e.g., API analytics) of a particular Konnect organization. Inside the Konnect platform, we have the ability to ensure that any API or service can be rolled out in an efficient manner to promote easy discovery and onboarding for development teams. Let’s take a look at how we can expose MCP servers to internal and external audiences.

Konnect’s Service Catalog offers a comprehensive catalog of all services running in your organization. By integrating with Konnect-internal applications (like Konnect Analytics) and external applications like Github and PagerDuty, Service Catalog provides a 360 overview into each MCP server the organization is making available.

Below we can see the Konnect MCP server that we integrated our chatbot with being able to publish developer-specific documentation streamlines how distributed teams within an organization can discover and consume these services in their applications. 

Konnect also provides the ability to publish Services to external-facing Dev Portals, enabling developers to locate, access, and consume API services. Having the ability to expose MCP Servers to a Dev Portal not only accelerates developer onboarding, but also provides a platform on which organizations can start establishing a go-to-market plan for generative AI solutions and potentially unlock additional revenue streams. 

Conclusion

While we have gotten into the weeds on how to securely expose a single chatbot application, we can see that many moving parts need to be accounted for, secured, and monitored, in the development stack for AI applications. A failure to accommodate development teams or a compromise in security can negatively impact innovation and time to market for teams looking to leverage generative AI solutions. It’s a macro problem at scale — and many of the top organizations are attacking these issues by rolling out an AI-centric developer platform that promotes easy discovery, onboarding, and innovation.

Kong AI Gateway is not just a proxy: it’s a trust layer for all generative AI applications. Combined with the Kong Konnect platform, organizations will have the tools to build a developer-centric AI platform that will allow the best and brightest to operate efficiently.

AI is powerful, and with great power comes great responsibility. Build boldly, build smart, and if you need a guide along the way, we're here to help.