# Spend Intentionally: Cutting LLM Costs Without Cutting Quality with Kong AI Gateway

LLM costs are difficult to forecast, difficult to attribute and generally only visible once the monthly invoice arrives. By that point the spend has already occurred and most organizations cannot determine which team or application was responsible for it.

At the same time, most requests are sent to the most capable model available, because no one has measured whether a less expensive model would have been sufficient. That assumption is frequently the largest single line item on the bill.

This session examines the cost controls available in Kong AI Gateway, which operates between your applications and your LLM providers. Because these controls are applied at the gateway rather than in the application, they can be introduced across every team and service without changes to application code.

Over 45 minutes, we will configure and demonstrate the following capabilities in Kong AI Gateway:

• - **Semantic caching** (`ai-semantic-cache`) — Kong AI Gateway matches incoming prompts against previous requests by meaning rather than exact text, and returns matching responses from cache. Repeated and closely related questions are answered without a call to the provider.
• - **Cost-aware and semantic routing** (`ai-proxy-advanced`)— Kong AI Gateway selects a target model based on measured cost per token, or on the semantic content of the prompt. When provider pricing changes, traffic distribution adjusts automatically and without redeployment.
• - **Prompt compression** (`ai-prompt-compressor`)— Kong AI Gateway reduces prompt length before the request is forwarded upstream.
• - **Token and cost-based rate limiting** (`ai-rate-limiting-advanced`) — Kong AI Gateway enforces token or currency budgets per team, consumer, provider, or model. Limits are isolated, so a team reaching its ceiling has no effect on the others.
• - **Quality measurement with LLM as a judge** (`ai-llm-as-judge`) — Kong AI Gateway scores a sample of responses using a separate, inexpensive model, establishing where a significantly cheaper model performs acceptably and where it does not. This capability adds cost rather than removing it. Its value lies in determining how far the other controls can safely be applied.
• - **Spend dashboards and cost attribution** (Konnect Analytics) — Kong AI Gateway calculates the cost of every request from the provider's token counts. Kong Konnect shows it as a single dashboard: total spend, spend over time, and a breakdown by team, model, and provider.

All configuration examples shown during the session, including the decK YAML for each Kong AI Gateway plugin, will be available to download afterwards so that you can reproduce the setup in your own environment.

Intended for platform engineering teams, API and AI platform owners and those accountable for large language model expenditure.