Securing, Observing, and Governing MCP Servers with Kong AI Gateway

Let’s imagine a development team has built a chat application that will facilitate the ability for CISOs to understand what is being exposed in their API landscape. To achieve this, the development team would need to build an application that consists of the chatbot, a collection of LLMs, and the Kong Konnect MCP server.

In the diagram below, we can see how leveraging the Kong AI Gateway as the central connectivity point for the application will help ensure that the developers can securely roll out the application.

Let’s apply the above flow to a real user prompt: “Give me an overview of the services in the Shared_Services control plane that have the highest 4xx and 5xx error rates over the past 24 hours” (1). The Chatbot sends the prompt and the available tools to the target LLM (4), which subsequently directs the Chatbot to call the list_control_planes tool on the MCP server, passing Shared_Services as a parameter to filter on (5). The results are sent back to the target LLM for further analysis (7). This process is repeated several times, with the target LLM requiring the backend to execute the list_services and query_api_requests tools to build further context. Finally, the target LLM produces an overview of the services experiencing the most errors (9):

This flow highlights the number of different API calls and connections needed to answer the user’s prompt. In this implementation, Kong Gateway is facilitating communication between all entities in the application stack, strengthening the security and governance posture with the below policies:

Authentication and Authorization

All communications in the application stack are being secured by Kong’s OpenID Connect plugin. The plugin is driving Single Sign On (SSO) for the chat interface, and then passing the authenticated user’s Access Token (JWT) in all downstream API requests. Kong can authorize each transaction via claims present in each user’s token, further building a zero-trust relationship between each of the data sources.

Credential Mediation

After handling authentication and authorization, Kong can inject the necessary credentials needed to access the upstream service — in this case, either the LLM or the MCP tool calls. This obfuscates the need for developers to manage any additional credentials outside of those needed to drive the SSO flow

Quota Management

The AI Rate Limiting Advanced plugin is being used to ensure that excessive costs may be mitigated with the LLM integration and that no single consumer can degrade the reliability of the application.

Guardrail Enforcement

The AI Prompt Guard and AI Semantic Prompt Guard plugins are being leveraged to ensure that the end users are not violating usage regulations when obtaining Konnect insights via the chat application.

Observability

All transactions between the chat application, the LLMs, and the MCP server are tracked natively with Konnect Advanced Analytics with custom dashboards and per-request insights, or exported via HTTP Log or OpenTelemetry plugins to the observability platform of choice.

Discovery and Documentation with Kong Konnect

So now we have a way to securely expose and govern MCP servers, how can we roll it out for consumption by any internal (or even external) development team?

Enter Kong Konnect!

Earlier, we walked through an example that leveraged the Konnect MCP server to allow LLMs to easily query the state (e.g., API analytics) of a particular Konnect organization. Inside the Konnect platform, we have the ability to ensure that any API or service can be rolled out in an efficient manner to promote easy discovery and onboarding for development teams. Let’s take a look at how we can expose MCP servers to internal and external audiences.

Konnect’s Service Catalog offers a comprehensive catalog of all services running in your organization. By integrating with Konnect-internal applications (like Konnect Analytics) and external applications like Github and PagerDuty, Service Catalog provides a 360 overview into each MCP server the organization is making available.

Below we can see the Konnect MCP server that we integrated our chatbot with being able to publish developer-specific documentation streamlines how distributed teams within an organization can discover and consume these services in their applications. 

Konnect also provides the ability to publish Services to external-facing Dev Portals, enabling developers to locate, access, and consume API services. Having the ability to expose MCP Servers to a Dev Portal not only accelerates developer onboarding, but also provides a platform on which organizations can start establishing a go-to-market plan for generative AI solutions and potentially unlock additional revenue streams. 

Conclusion

While we have gotten into the weeds on how to securely expose a single chatbot application, we can see that many moving parts need to be accounted for, secured, and monitored, in the development stack for AI applications. A failure to accommodate development teams or a compromise in security can negatively impact innovation and time to market for teams looking to leverage generative AI solutions. It’s a macro problem at scale — and many of the top organizations are attacking these issues by rolling out an AI-centric developer platform that promotes easy discovery, onboarding, and innovation.

Kong AI Gateway is not just a proxy: it’s a trust layer for all generative AI applications. Combined with the Kong Konnect platform, organizations will have the tools to build a developer-centric AI platform that will allow the best and brightest to operate efficiently.

AI is powerful, and with great power comes great responsibility. Build boldly, build smart, and if you need a guide along the way, we're here to help.