The explosion of AI-native applications is upon us. With each new week, massive innovations are being made in how AI-centric applications are being built. There are a variety of tools developers need to consider, be it supplying live contextual data via the [Model Context Protocol (MCP)](https://konghq.com/blog/learning-center/what-is-mcp)Model Context Protocol (MCP) or leveraging the new Agent2Agent Protocol (A2A) to standardize how their agentic applications will communicate.

The modern AI application can include communication between many different entities, including:

• - **Applications**: User-facing interfaces, enterprise tools, or services triggering AI workflows.
• - **Agents**: Autonomous or semi-autonomous orchestrators making tool-based decisions.
• - **MCP servers**: A service exposing context, such as tools and resources, for LLMs in a standardized format
• - **LLMs**: Foundation models (e.g., OpenAI, Claude, Mistral, Bedrock) executing natural language reasoning.

In [a recent blog](https://konghq.com/blog/learning-center/what-is-mcp)a recent blog, we explored the many challenges of integrating MCP servers into AI-native applications — and MCP is just one piece of the puzzle. With all of the components being leveraged in AI application development, robust infrastructure is needed to manage authentication, enforce rate limits, and provide detailed observability across all interactions. At the same time, they must guard against misuse and remain flexible as models and providers evolve. 

These are the exact challenges the [Kong AI Gateway](https://konghq.com/products/kong-ai-gateway)Kong AI Gateway was built to solve. Kong’s AI Gateway is an enterprise-grade solution to help secure and govern connectivity between AI-native applications.

## Securing AI Connections with Kong AI Gateway

With Kong AI Gateway, we can securely expose agentic applications, MCP servers, and LLMs alike. Kong provides many different policies that cover a wide array of functionality, enabling developers to easily configure access control for their use cases.

Authentication plugins like [OpenID Connect](https://docs.konghq.com/hub/kong-inc/openid-connect/)OpenID Connect or [Key Authentication](https://docs.konghq.com/hub/kong-inc/key-auth/)Key Authentication can be used to standardize the authentication patterns at the edge. The authentication plugins can be extended to support fine-grained Authorization models via JWT claims or declarative [Access Control Lists](https://docs.konghq.com/hub/kong-inc/acl/)Access Control Lists (ACLs).

After establishing consumer identity, Kong AI Gateway supports flexible policies for managing consumer quotas. The [AI Rate Limiting Advanced](https://docs.konghq.com/hub/kong-inc/ai-rate-limiting-advanced/)AI Rate Limiting Advanced plugin can be used to apply token-usage quotas, while the vanilla [Rate Limiting Advanced](https://docs.konghq.com/hub/kong-inc/rate-limiting-advanced/)Rate Limiting Advanced plugin can support the more typical API request-based quotas.

The AI Gateway also provides the ability to apply flexible guardrails to centralize content moderation, using the [AI Prompt Guard](https://docs.konghq.com/hub/kong-inc/ai-prompt-guard/)AI Prompt Guard plugin for regex-based policies or the [AI Semantic Prompt Guard](https://docs.konghq.com/hub/kong-inc/ai-semantic-prompt-guard/)AI Semantic Prompt Guard plugin to enable smart guardrails with semantic reasoning. The AI Gateway also provides the ability to integrate with third-party services (see [Azure Content Safety](https://docs.konghq.com/hub/kong-inc/ai-azure-content-safety/)Azure Content Safety or the [PII Sanitizer](https://docs.konghq.com/hub/kong-inc/ai-sanitizer/)PII Sanitizer), giving ultimate control over how platform teams can ensure that all interactions will conform to the appropriate governance policies.

Finally, we want to ensure that the AI Gateway provides real-time visibility into model behavior, request flows, and performance bottlenecks — enabling faster debugging, improved reliability, and safer AI outcomes. Kong provides the tools necessary to achieve a robust security posture, leveraging [log aggregation](https://docs.konghq.com/hub/?category=logging)log aggregation and modern standards like [OpenTelemetry](https://docs.konghq.com/hub/kong-inc/opentelemetry/)OpenTelemetry.

Let’s take a look at a real-life example of how Kong can empower AI-native applications to use MCP servers to add real-time intelligence to their workflows.

## Example use case: Building an MCP-enabled chatbot

Kong recently announced the launch of an [MCP Server](https://konghq.com/blog/product-releases/mcp-server)MCP Server for Kong Konnect — a unified SaaS platform for managing APIs, events, and LLMs across hybrid and multi-cloud environments. This empowers customers to integrate AI agents and query LLMs to discover and request details around APIs, services, and their associated traffic analytics.

Today, we’re going to use this MCP server to demonstrate the importance of a gateway when using MCP to integrate AI applications with additional tooling and resources.

Let’s imagine a development team has built a chat application that will facilitate the ability for CISOs to understand what is being exposed in their API landscape. To achieve this, the development team would need to build an application that consists of the chatbot, a collection of LLMs, and the Kong Konnect MCP server.

In the diagram below, we can see how leveraging the Kong AI Gateway as the central connectivity point for the application will help ensure that the developers can securely roll out the application.

Let’s apply the above flow to a real user prompt: “Give me an overview of the services in the `Shared_Services` control plane that have the highest 4xx and 5xx error rates over the past 24 hours” (1). The Chatbot sends the prompt and the available tools to the target LLM (4), which subsequently directs the Chatbot to call the `list_control_planes` tool on the MCP server, passing `Shared_Services` as a parameter to filter on (5). The results are sent back to the target LLM for further analysis (7). This process is repeated several times, with the target LLM requiring the backend to execute the `list_services` and `query_api_requests` tools to build further context. Finally, the target LLM produces an overview of the services experiencing the most errors (9):

This flow highlights the number of different API calls and connections needed to answer the user’s prompt. In this implementation, Kong Gateway is facilitating communication between all entities in the application stack, strengthening the security and governance posture with the below policies:

### Authentication and Authorization

All communications in the application stack are being secured by Kong’s OpenID Connect plugin. The plugin is driving Single Sign On (SSO) for the chat interface, and then passing the authenticated user’s Access Token (JWT) in all downstream API requests. Kong can authorize each transaction via claims present in each user’s token, further building a zero-trust relationship between each of the data sources.

plugins:
- name: openid-connect
  config:
    issuer: https://konghq.okta.com/oauth2/default
    auth_methods:
    - bearer
    consumer_claim:
    - sub
    groups_claim:
    - groups
    groups_required:
    - konnect-mcp-users

### Credential Mediation

After handling authentication and authorization, Kong can inject the necessary credentials needed to access the upstream service — in this case, either the LLM or the MCP tool calls. This obfuscates the need for developers to manage any additional credentials outside of those needed to drive the SSO flow

plugins:
- name: ai-proxy-advanced
  config:
    balancer:
      algorithm: "round-robin"
      tokens_count_strategy: "total-tokens"
      latency_strategy: "tpot"
      retries: 3
    targets:
    - route_type: "llm/v1/chat"
      auth:
        header_name: "Authorization"
        header_value: "{vault://ai/openai-token}"
      logging:
        log_statistics: true
        log_payloads: true
      model:
        provider: "openai"
        name: "gpt-4o"
        options:
          max_tokens: 1024
          temperature: 1.0
          input_cost: 2.5
          output_cost: 10

### Quota Management

The AI Rate Limiting Advanced plugin is being used to ensure that excessive costs may be mitigated with the LLM integration and that no single consumer can degrade the reliability of the application.

plugins:
- name: ai-rate-limiting-advanced
  config:  
    strategy: local
    window_type: fixed
    llm_providers:
    - name: openai
      window_size: 
      - 60
      - 3600
      limit: 
      - 10000
      - 1000000

### Guardrail Enforcement

The AI Prompt Guard and AI Semantic Prompt Guard plugins are being leveraged to ensure that the end users are not violating usage regulations when obtaining Konnect insights via the chat application.

plugins:
- name: ai-prompt-guard
  config:
    deny_patterns:
    - ".*(W|w)ar.*"
    - ".*(C|c)onflict.*"

- name: ai-semantic-prompt-guard
  config:
    embeddings:
      auth:
        header_name: Authorization
        header_value: "{vault://ai/openai-token}"
      model:
        provider: "openai"
        name: "text-embedding-3-large"      
    vectordb:
      strategy: "redis"
      dimensions: 768
      distance_metric: "cosine"
      threshold: 0.5
      redis:
        host: redis.redis.svc
        port: 6379
    search:
      threshold: 0.5
    rules:
      deny_prompts:
        - hijacking an LLM prompt
        - questions about prompt injections
        - questions about prompt jailbreaking

### Observability

All transactions between the chat application, the LLMs, and the MCP server are tracked natively with Konnect Advanced Analytics with custom dashboards and per-request insights, or exported via HTTP Log or OpenTelemetry plugins to the observability platform of choice.

## Discovery and Documentation with Kong Konnect

So now we have a way to securely expose and govern MCP servers, how can we roll it out for consumption by any internal (or even external) development team?

Enter Kong Konnect!

Earlier, we walked through an example that leveraged the Konnect MCP server to allow LLMs to easily query the state (e.g., API analytics) of a particular Konnect organization. Inside the Konnect platform, we have the ability to ensure that any API or service can be rolled out in an efficient manner to promote easy discovery and onboarding for development teams. Let’s take a look at how we can expose MCP servers to internal and external audiences.

Konnect’s Service Catalog offers a comprehensive catalog of all services running in your organization. By integrating with Konnect-internal applications (like Konnect Analytics) and external applications like Github and PagerDuty, Service Catalog provides a 360 overview into each MCP server the organization is making available.

Below we can see the Konnect MCP server that we integrated our chatbot with being able to publish developer-specific documentation streamlines how distributed teams within an organization can discover and consume these services in their applications. 

Konnect also provides the ability to publish Services to external-facing Dev Portals, enabling developers to locate, access, and consume API services. Having the ability to expose MCP Servers to a Dev Portal not only accelerates developer onboarding, but also provides a platform on which organizations can start establishing a go-to-market plan for generative AI solutions and potentially unlock additional revenue streams. 

## Conclusion

While we have gotten into the weeds on how to securely expose a single chatbot application, we can see that many moving parts need to be accounted for, secured, and monitored, in the development stack for AI applications. A failure to accommodate development teams or a compromise in security can negatively impact innovation and time to market for teams looking to leverage generative AI solutions. It’s a macro problem at scale — and many of the top organizations are attacking these issues by rolling out an AI-centric developer platform that promotes easy discovery, onboarding, and innovation.

Kong AI Gateway is not just a proxy: it’s a trust layer for all generative AI applications. Combined with the Kong Konnect platform, organizations will have the tools to build a developer-centric AI platform that will allow the best and brightest to operate efficiently.

AI is powerful, and with great power comes great responsibility. Build boldly, build smart, and if you need a guide along the way, we're here to help.