***UI + MCP***
When an application calls a model provider directly, the service needs the provider's API key, and any control over that traffic has to be built into the service itself. For one service calling one model, that is the right amount of machinery.
It holds up less well as the application grows. Every new service needs the key too, limits and logging are reimplemented in each one.
An AI gateway moves those concerns onto the path the requests already take. This guide explains what one is and does, then sets one up on [_Kong Konnect_](https://konghq.com/products/kong-konnect)_Kong Konnect_ and sends a request through it. Extending that, you can also [set up a rate limit](#add-token-rate-limit)set up a rate limit on that traffic, counted in tokens rather than requests.
An AI gateway is a reverse proxy built for model traffic. Your application sends an OpenAI-formatted request to the gateway instead of to a provider, and the gateway handles the rest of the trip.
For each request it:
In your application, the base URL changes first:
1# calling the provider directly
2client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
3
4# calling through the gateway
5client = OpenAI(
6 base_url="http://localhost:8000/v1",
7 api_key=os.environ["GATEWAY_KEY"],
8)The `model `field in your request body changes too. It carries a name you define on the gateway rather than the provider's model id:
1response = client.chat.completions.create(
2 model="chat-default", # a name you configure, not "gpt-4o"
3 messages=[{"role": "user", "content": "Say this is a test"}],
4)That indirection is what the rest depends on. chat-default can resolve to gpt-4o today and to a Claude model tomorrow, while the calling code stays the same. The same name is what policies attach to, what usage is recorded against, and what limits are scoped by.
1Direct
2 your app ─────────────────────────► OpenAI
3 provider SDK, provider key, per-app limits,
4 per-app logging, per-app retry logic
5
6Through a gateway
7 your app ──► AI gateway ──► OpenAI
8 │ Anthropic
9 │ self-hosted model
10 │
11 identify caller · apply policy · select target
12 convert format · record tokens, cost, latencyThe gateway in this guide is [_Kong AI Gateway_](https://developer.konghq.com/ai-gateway/)_Kong AI Gateway_, which runs on Konnect and speaks the OpenAI request format in front of OpenAI, Anthropic, Azure, Bedrock, Gemini, Mistral, and others.
Which of these matter depends on what you are building. The ones that come up most often:
**Credential control.** Every service that calls a model needs the provider key, and any one of them that leaks it exposes the whole provider account. Rotation means a coordinated change across all of them. A gateway holds the provider key itself and issues each service its own credential, which you can revoke one at a time.
**Per-caller limits and attribution.** Provider billing sees one API key. It cannot tell you which service, team, or customer spent what, and it cannot stop one of them consuming the whole quota. A gateway sees every request with a caller identity attached, so limits and cost reporting can be scoped per consumer.
**Provider independence.** The provider SDK, request format, and model name are built into every service that calls them. Moving to a cheaper model, or pointing one service at a different provider, means editing and redeploying each one. Behind a gateway, the name your code sends stays fixed while the upstream behind it changes.
Beyond these, a gateway is also where caching, guardrails, data redaction, and tool access control can be applied once rather than per service. Those are linked in [where to go next](#wheretogonext)where to go next.[](https://docs.google.com/document/d/1ekkh64ql0M1vKaHNa8nnXy646R0rXPCA/edit#heading=h.lw5yrq2l3n1w)
Two outcomes, in order:
Three pieces of setup come first. They are done once in the Konnect UI.
After that, the remaining set up configuration can be built two ways and the guide covers both in full:
Pick one and follow it through. [_Kongctl_](https://developer.konghq.com/kongctl/)_Kongctl_, [_AI Gateway API_](https://developer.konghq.com/api/konnect/ai-gateway/)_AI Gateway API_ and Terraform do the same job and are linked at the end. For a self-hosted deployment rather than Konnect, see [_configuring AI Gateway on-prem_](https://developer.konghq.com/ai-gateway/configure-on-prem/)_configuring AI Gateway on-prem_.
You can use the **Kong Konnect UI** for setting up the prerequisites. The three steps in this section are prerequisites for both routes(UI and MCP approach)
Accounts and keys:
Provider calls are billed by the provider. The Konnect trial does not cover model usage.
Two terms, used throughout. The **control plane** is the Konnect-hosted service holding your providers, models, and policies. It is never in the request path. The **data plane node** is the proxy that is.
In [_Konnect_](https://cloud.konghq.com/ai-manager/v2/gateways)_Konnect_ UI, open **AI Gateway** in the left navigation and select **New gateway**.
Creating a gateway is a two-part wizard: the control plane first, then the data plane. The control plane step asks for a name and an optional description. This is what holds every provider, model, and policy you configure later, so name it something you will recognize. You will need this name again in the agent prompts, so note it. Select **Next**.
The second part of the wizard asks how you want to run the data plane. Choose **self-managed**, then pick a platform. **Docker** is the quickest for a local setup.
Leave the remaining options at their defaults and continue. Konnect generates a **certificate and a run script** containing it. Copy the script and run it on a machine with Docker running.
While the node starts, the wizard shows it as **awaiting connection**. Once it connects, that changes and you can select **View gateway**.
**Verify two things.** In Konnect, the gateway overview should now show the control plane and data plane as done, with models still outstanding. Locally, confirm the proxy is listening:
1curl -i http://localhost:80001{"message":"no Route matched with those values","request_id":"b176a014b5261a495b12f01f1c816101"}A `404` with that body, and a `Server: kong/<version>-ai-gateway` header, means the proxy is up and nothing is routed yet. That is the expected result here.
If the status never leaves awaiting connection, the node has not reached Konnect. Check that Docker is running, that the script finished without error, and that the machine can reach `*.cp.konghq.com`.
The gateway and its node are running, but there is nothing yet for them to forward requests to. The first thing to put in place is the provider key, because the provider you create next reads it rather than holding it.
Kong AI Gateway reads credentials from a vault entity that supports several providers. This guide uses the **Konnect** Vault, which keeps secrets inside Konnect and needs no external secret manager. The other backends, including AWS Secrets Manager, Google Secret Manager, Azure Key Vault, HashiCorp Vault and CyberArk Conjur, are covered in the [_AI Vault reference_](https://developer.konghq.com/ai-gateway/entities/ai-vault/)_AI Vault reference_.
Storing the key here means it is entered once, in Konnect, and referenced everywhere else. It is never typed into a provider field and never passed to an agent.
Open your gateway and go to the **Vaults** tab.
`model-keys`. A description is optional.Open the vault you just created and store the key.
`openai-key``Bearer <YOUR_OPENAI_KEY>`The `Bearer` prefix is part of the stored value, not part of the provider configuration. A vault reference replaces an entire field, so whatever you store here becomes the complete `Authorization` header. A secret holding only the key produces a header without `Bearer`, which the provider rejects.
Secret values cannot be read back once stored. To change one, open the secret and rotate it.
**Verify:** the vault appears under **Vaults**, and opening it lists `openai-key`.
The prerequisite setup is complete, you can continue the complete set up from
With the gateway, the data plane and the vault in place, two pieces of configuration remain. A provider and a model, so requests reach OpenAI. A rate limiting policy, so one caller cannot consume the whole provider quota.
With the key stored, you can add the provider. A provider is the gateway's connection to one upstream LLM service: which service it is, and how to authenticate to it. It is created before the model, because the model points at it.
Go to **Providers** and select **New provider**.
`OpenAI Main`, and a **Name**, `openai-main`. The name is how the model references this provider.`Authorization` header and the gateway passes it upstream. Use this when callers already hold their own provider credentials.`Authorization``model-keys` vault, then the **Secret ID** `openai-key`. **Secret key** is optional and can be left empty.The header value now holds a reference to the Konnect vault rather than the key itself.
Kong supports a wide range of providers, including Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Mistral, Cohere, DeepSeek, Databricks, and vLLM. The [_provider reference_](https://developer.konghq.com/ai-gateway/entities/ai-model-provider/)_provider reference_ documents the authentication shape for each.
**Verify:** the provider is listed. Reading it back returns the credential redacted, which indicates it is stored rather than that the write failed. The only conclusive test is a completion in [the first request](#step6-first-request)the first request.
The provider knows how to reach OpenAI, but nothing yet gives your application a URL to call. That is the [_AI Model_](https://developer.konghq.com/ai-gateway/entities/ai-model/)_AI Model_: the name clients address, the path it is served on, and the upstream target the gateway forwards to.
Go to **Models** and select **New model**.
`chat-default`. Keep the name provider-neutral, since the model is the stable address your application calls while the upstream behind it can change.`openai-main`, then the upstream model `gpt-4o.``/responses.``/v1`.`model `field. The model is already addressable by its **Name**, so an alias is only needed when callers must keep using a different string, for example a provider model id they already hardcoded. Leave it empty.**Base path** is only the prefix. The capability supplies the rest of the route, so Chat completions turns `/v1 `into `/v1/chat/completions,` and a request to `/v1 `on its own returns 404.
The **Name** is the gateway-side model name described in [What is an AI gateway.](#what-is-ai-gateway)What is an AI gateway. It is what clients send in the` model `field, and it is not the upstream model id.
**Verify:** the model shows as enabled, with base path `/v1 `and one target.
The route is live as soon as the model is created, so you can send something through it. The model's page in Konnect gives you a ready-made `curl` command. Copy it and make two changes:
`http://localhost:8000`, since your node is running locally.`Authorization `header. The gateway attaches the provider credential itself, so a client-side one is redundant here.What you end up with:
1curl -i -X POST http://localhost:8000/v1/chat/completions \
2 --json '{"model":"chat-default","messages":[{"role":"user","content":"Say this is a test"}]}'A working setup returns `200`:
1{
2 "id": "chatcmpl-EWsvXwtr7ceHSzfc3ZLbbNhY0bCMj",
3 "object": "chat.completion",
4 "model": "gpt-4o-2024-08-06",
5 "choices": [
6 {"index": 0, "message": {"role": "assistant", "content": "This is a test. How can I assist you today?"}, "finish_reason": "stop"}
7 ],
8 "usage": {"prompt_tokens": 12, "completion_tokens": 12, "total_tokens": 24}
9}Three things to check:
`200`.`choices[0].message.content`.`usage` is present. Those token counts feed the cost calculation, the [rate limiting policy](#add-token-rate-limit)rate limiting policy, and Konnect analytics. A response without `usage` is counted nowhere.The `model` field carries OpenAI's own resolved version of gpt-4o, not the chat-default name you sent. To see which target actually served a request, read the response header instead:
1X-Kong-LLM-Model: openai/gpt-4oThe value is `provider/target`, using the names you gave them.
In your application, this endpoint is the `base_url `shown in [What is an AI gateway](#what-is-ai-gateway)What is an AI gateway, and `chat-default` is the model name alongside it.
**Shortcut to be aware of.** This endpoint has no authentication. Anything that can reach port 8000 can spend your OpenAI credit. Add authentication before this node is reachable from anything other than your own machine.
Traffic is now reaching OpenAI through the gateway, which is the point at which most teams want a limit on it. Rate limits are normally counted in requests, but a request is a weak unit for model traffic: one can use fifty tokens and the next fifty thousand. The [_AI Rate Limiting Advanced_](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/)_AI Rate Limiting Advanced_ policy counts the tokens the provider reports in each response, which is the number that tracks spend.
Open your gateway, go to the **Policies** tab, and select **New policy**. Find **AI Rate Limiting Advanced** in the policy list. The form has three numbered sections.
**1. General information**
`chat-default-token-limit`.**2. Scope**
Choose **Scoped**, which applies the policy to specific models, MCP servers and agents.. **Global** would apply it to every model on the gateway, which is not what you want while one model is under test.
**3. Policy configuration**
**Identifier** sets what the counter is keyed by. Accepted values are `ip`, `credential`, `consumer`, `service,` `header`, `path` and `consumer-group.` It defaults to `consumer,` which gives each authenticated caller its own budget. You have not configured consumers yet, so for now the limit applies across all traffic to this model. It starts behaving per caller once you add authentication . Leave it at `consumer`.
**Policies** hold the rules. Select **Add policies** to create one. A rule has four parts.
`sliding`, `fixed` or `calendar. sliding `is the default and the right choice here, because it measures the trailing window continuously rather than resetting on a clock boundary.Inside the rule, select **Add limits**:
`60`, which is low enough to trigger while you are watching.` sliding `and `fixed`. Enter `60`.`week` or `month`, used only when **Window type** is `calendar`, as an alternative to **Window size**. Leave it empty.`monday` through `sunday`, used only with a week period. Leave it empty.`month`, ignored otherwise. Leave it empty.`usage` object is counted. `total_tokens` counts prompt plus completion, and is the default. The alternatives are `prompt_tokens, completion_tokens` and `cost`. Leave it at `total_tokens`.**Limit** and **Window size** are the only two values this form actually needs. Everything else above has a working default.
If you later want one rule per team rather than one for the whole model, select **Add match** inside the rule:
`consumer`, `consumer_group`, `credential`, `ip`, `header`, `path`, `model`, `provider` and `service`.`header `match on `X-Team`, the key is `X-Team` and the values are the team names.Below the rules:
`local` keeps them in the node's memory and is correct for the single node you are running. `redis` is required once you run more than one node, because `local` counts per node and the effective limit becomes your number multiplied by the node count. `cluster `is the third option.`local `strategy writes counters to. Leave it at `kong_rate_limiting_counters`.Leave **Show additional settings** closed. Create the policy, then attach it to `chat-default `when the wizard asks which entities it applies to.
When you set a real limit later, account for agent traffic. A tool loop can fire five or six calls to satisfy one user request, so a number that looks generous per user can cut a conversation off partway through.
**To verify:**
1X-AI-RateLimit-Limit-minute-policy-1: 60
2X-AI-RateLimit-Remaining-minute-policy-1: 412. Within a few requests you should get a `429` whose body is `{"message":"AI token rate limit exceeded for provider(s): "}`, carrying both the per-policy headers and unsuffixed ones:
1X-AI-RateLimit-Remaining-minute-policy-1: 0
2X-AI-RateLimit-Reset: 42
3X-AI-RateLimit-Retry-After: 423. Set a realistic limit before moving on. A sliding window means the counter drains gradually, so the endpoint stays limited for up to a full window after you change it.
Kong publishes a [_Konnect MCP server_](https://developer.konghq.com/konnect-platform/konnect-mcp/)_Konnect MCP server_, which gives an AI coding agent access to the Konnect configuration API. This section reaches the same end state as the UI walkthrough, and is self-contained: you do not need to read the UI steps to follow it.
It is organised in two parts. By the end of the first, an OpenAI request is flowing through your gateway and you have a` curl `command that proves it. The second rate-limits that traffic by token count.
**Complete **[**Required setup**](#prerequisites-ui-mcp)**Required setup**** first.** The agent configures Konnect, but it cannot create the parts that run on your machine, and it should not be given a provider key. You need:
`model-keys` holding a secret with the key openai-key, from [creating the vault](#create-vault-konnect)creating the vault.The prompts use the names `model-keys`, `openai-key`, `openai-main` and c`hat-default`. If you followed the setup as written, the only thing to change is your gateway name. If you chose different names, change them in the prompts to match.
This guide uses Claude Code:
1claude mcp add --transport http kong-konnect https://global.mcp.konghq.comAuthorization uses OAuth. A browser window opens so you can sign in to Konnect and grant access, and no token is needed. If the browser does not open, invoke the server once by asking the agent to use a `kong-konnect` tool; the authorization flow starts at that point.
If you would rather not use OAuth, pass a Konnect personal access token instead. Generate one from the [_Konnect tokens page_](https://cloud.konghq.com/global/account/tokens)_Konnect tokens page_:
1claude mcp add --transport http kong-konnect https://global.mcp.konghq.com/ \
2 --header "Authorization: Bearer $KONNECT_PAT"Confirm the connection with `claude mcp list`. Regional endpoints are available if you prefer one: `us.`, `eu.`, `au.` and `in.mcp.konghq.com`.
For **VS Code, Cursor, GitHub Copilot, Windsurf, Eclipse and other clients**, the per-client JSON config is in Kong's [_installation guide_](https://developer.konghq.com/konnect-platform/konnect-mcp/installation/)_installation guide_.
Optionally add the [_Kong skills_](https://developer.konghq.com/skills/install/)_Kong skills_, which give the agent Kong product knowledge alongside API access:
1/plugin marketplace add kong/ai-marketplace
2/plugin install kong-konnect@ai-marketplaceFor agents without plugin support, `npx skills add kong/ai-marketplace` installs the same set.
With the gateway, the node and the vault in place, the two entities that carry traffic are the provider and the model. The provider holds the connection to OpenAI and reads its credential from your vault; the model is the name your application addresses and the route it is served on. One prompt creates both, and the agent finishes by handing you a `curl `command to test with.
Requires the three items in [Required setup](#prerequisites-ui-mcp)Required setup. The equivalent in the UI is [adding a provider](#add-provider)adding a provider and [creating a model](#create-a-model)creating a model.
1Using the kong-konnect MCP server, configure the AI Gateway named
2"<YOUR GATEWAY NAME>".
3
4Context you can rely on:
5- A Konnect vault named model-keys already exists on this gateway.
6- It holds a secret with the key openai-key, whose value is a complete
7 Authorization header value in the form "Bearer <key>".
8- Do not create, modify or read any secret. Reference the existing one.
9
10Create the following:
11
121. An AI model provider:
13 - name: openai-main
14 - display name: OpenAI Main
15 - type: openai
16 - upstream authentication: a single header named Authorization whose value
17 is the vault reference {vault://model-keys/openai-key}
18
192. An AI model:
20 - name: chat-default
21 - display name: chat-default
22 - format: openai
23 - route base path: /v1
24 - capability: chat completions
25 - one target: upstream model gpt-4o on provider openai-main
26
27When you are done:
28- Read the provider and the model back and show me their configuration.
29- Give me a curl command that sends a chat completion to chat-default through
30 my data plane at http://localhost:8000. It must not include an Authorization
31 header, because the gateway supplies the provider credential.Run the `curl `command the agent returns. A working setup answers 200, with a `usage `object carrying the token counts and an `X-Kong-LLM-Model: openai/gpt-4o `response header naming the target that served it. [Sending your first request](#step6-first-request)Sending your first request describes the full response and what each part is for.
Your OpenAI traffic is now proxied through Kong AI Gateway. From here, [set up the rate limit](#setting-rate-limit)set up the rate limit to cap token consumption on this model.
With traffic flowing, the next thing to add is a limit. Rate limits are normally counted in requests, but a request is a weak unit for model traffic, since one can use fifty tokens and the next fifty thousand. This prompt creates an [_AI Rate Limiting Advanced_](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/)_AI Rate Limiting Advanced_ policy that counts the tokens the provider reports, and attaches it to your model alone rather than to the whole gateway.
Requires the previous section. The equivalent in the UI is [adding a token rate limit](#add-token-rate-limit)adding a token rate limit.
1Using the kong-konnect MCP server, on the AI Gateway named
2"<YOUR GATEWAY NAME>":
3
4Create an AI Rate Limiting Advanced policy:
5- name: tok-limit
6- display name: tok-limit
7- limit: 60 total tokens
8- window size: 60 seconds
9- leave tokens_count_strategy at total_tokens, window_type at sliding, and
10 strategy at local
11
12Attach the policy to the model named chat-default only. Do not enable it
13globally and do not attach it to any other model.
14
15When you are done, read the policy back and confirm that chat-default lists it.Send the same request four or five times. Successful responses carry `X-AI-RateLimit-Remaining-minute-policy-1`, counting down with each call, and then a 429 whose body is `{"message":"AI token rate limit exceeded for provider(s): "}`.
Sixty tokens per minute is deliberately low so the limit is easy to observe. Raise it to something realistic before you use the gateway for anything else. A sliding window drains gradually, so the endpoint stays limited for up to a full window after you change the number.
Review the entities the agent created before sending production traffic. It is writing configuration to a running gateway.
To create the gateway and the data plane node without clicking through [creating the gateway](#create-ai-gateway)creating the gateway and [connecting a data plane](#connect-data-plane-node)connecting a data plane, use the quickstart script. Generate a personal access token from the [_Konnect tokens page_](https://cloud.konghq.com/global/account/tokens)_Konnect tokens page_ first:
1export KONNECT_TOKEN='your-konnect-pat'
2curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKENThat creates the AI Gateway control plane and runs a data plane node locally, which covers the first two setup steps. You still [create the vault](#create-vault-konnect)create the vault, then the [provider](#add-provider)provider and the [model](#create-a-model)model.
Q: What is an AI gateway?
A: An AI gateway is a specialized reverse proxy designed specifically for model traffic. It sits between your applications and LLM providers (such as OpenAI or Anthropic) to handle request routing, provider authentication, format conversions, policy enforcement, rate limiting, and analytics.
Q: How does rate limiting work with AI Gateway?
A: Unlike traditional rate limiting that measures traffic by request volume, Kong AI Gateway offers AI Rate Limiting Advanced, which tracks token usage (prompt, completion, total tokens, or cost) reported by model providers. This ensures limits align directly with actual usage and compute spending.
Q: How does the AI Gateway secure API keys?
A: Provider keys are stored centrally in a secure vault (such as Konnect Vault, AWS Secrets Manager, or HashiCorp Vault) rather than hardcoded into application source code or passed via agent prompts. The gateway injects the credential dynamically into upstream requests.
Q: Can I set up Kong AI Gateway using AI agents?
A: Yes. Kong provides a Konnect MCP (Model Context Protocol) server that enables AI coding agents (like Claude Code, Cursor, or Windsurf) to configure control planes, providers, models, and policies directly using natural language prompts.
Q: Does setting up an AI gateway alter application request formats?
A: No. Applications can continue using standard OpenAI SDK formats. You only need to update the base URL to point to the data plane endpoint and send custom gateway model names instead of raw provider model identifiers.
Multi-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s
Multi-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:
Most teams start the same way. A developer creates an API key, calls OpenAI or Anthropic, and ships a prototype. The problems surface when that prototype becomes five production services calling three providers. Hardcoded provider dependencies are
This post is based on Kong's webinar on token cost management. Watch it below, or keep reading for the breakdown. Token cost management is how a business tracks, controls, and governs what it spends on AI model usage, the same way it already track
Built on top of Kong API Gateway, the Kong AI Gateway is designed to address key challenges in enterprise AI adoption. Modern AI applications rarely rely on a single model; instead, they orchestrate multiple GenAI providers, agent frameworks, Age
Together, the following components represent the three layers of the new AI platform: AI Gateway: Kong AI Gateway (including MCP support) controls both GenAI and MCP flow and orchestrates the existing services like Vector Databases, Event Streaming,
