Every team running production LLMs has had the same idea: not every request needs the frontier model. Intelligent model routing (or LLM routing)— choosing a model per request on criteria such as task complexity, cost, latency, or quality — enables more efficient model usage.
The idea is easy. Shipping it is not. The moment routing logic goes into the request path, it becomes infrastructure — and now it's holding your provider credentials, needs to comply with security requirements (guardrails, auditing, etc.), and is standing between your applications and every model they depend on, which requires reliable, highly available, and redundant infrastructure. Teams can stall at this point because the application or ML team defining the routing strategy should not also have to own the infrastructure and security boundary. Meanwhile, the platform team responsible for that boundary may not have the application context needed to continually tune routing policies and algorithms.
The solution is to separate model selection from traffic management. Intelligent model routing is a combined system: NVIDIA NeMo Switchyard provides the model selection, and Kong AI Gateway provides the connectivity, governance, and traffic management. Switchyard is a customizable selection library with multiple algorithms that teams can configure according to their own requirements. Kong AI Gateway routes the traffic — dispatching each request to the selected model while managing credentials, guardrails, PII masking, rate limits, and auditing through a scalable, reliable gateway.
As AI adoption scales, applications evolve into complex systems of agents, orchestration layers, and context servers. Infrastructure lags behind — struggling with authentication, cost control, and data security across a provider list that changes every quarter.
Kong AI Gateway is the runtime for LLM traffic management those systems already run through. Its Universal API standardizes interfaces across providers, decoupling applications from provider-specific SDKs and centralizing credential management as part of a broader AI governance strategy. On top of that, the gateway enforces what production actually requires:
That last pair matters more than it looks. Your agents don't only call models — they call REST APIs, MCP servers, and event streams. Governing the LLM hop alone leaves most of the attack surface ungoverned. Further, an enterprise network is complex, with workloads running on-prem, across clouds, etc.
Per-request model selection can account for task complexity and other application-specific requirements. NeMo Switchyard is an open-source Apache 2.0 model selection library from NVIDIA. It makes several selection approaches available through an open-source library that teams can configure and extend for their own use cases.
In this integration, teams can deploy it as a model selection service: Kong provides the information needed to evaluate the configured selection policy, Switchyard returns a target, and Kong maintains control of the request path. Its selection algorithms - including complexity-based approaches such as `stage_router` give teams configurable building blocks for evaluating requests and selecting among model targets. Teams choose the algorithm, candidate models, thresholds, and other criteria appropriate to their applications.
Two configurable capabilities can help teams operate this pattern reliably:
For detailed setup and advanced routing configurations, refer to the NeMo Switchyard documentation.
Because model selection is one decision, and routing production AI traffic is many more.
A selection library helps teams automate model choice according to their configured policies. It is not intended to replace gateway capabilities such as credential management, per-team token limits, PII masking, cost attribution, or audit logging. And it doesn't govern the other 90% of your traffic — the APIs, MCP servers, and event streams your agents depend on.
Put the router in the data path instead, and you've added a second hop that sees prompts, holds keys, and carries its own CVE surface — with no SLA behind it. Use it as a configurable decision service, and teams can apply their chosen Switchyard selection strategy while Kong keeps routing the traffic and the gateway boundary intact.
*Kong AI Gateway routes every request; NVIDIA NeMo Switchyard supplies the model selection*
The data path begins when an application sends a request to Kong AI Gateway. Kong invokes the Switchyard decision service, which evaluates the request using the selection algorithm and criteria configured by the user and returns a model target. Kong then routes the request to that target, applies the organization’s configured policies, and returns the response to the application. If the decision service is unavailable, Kong can use a customer-configured fallback target, helping preserve request-path availability.
This pairing succeeds because it respects organizational boundaries. Model selection quality is an ML problem. Routing, security, compliance, infrastructure, and even cost control are platform problems. This architecture lets each team excel without overstepping.
Routing requirements and algorithms can evolve independently from gateway infrastructure. With a clear contract, teams configure how model targets are selected through Switchyard, while Kong routes, secures, meters, and logs the resulting traffic. This separation allows routing strategies to change without moving the gateway’s security and governance boundary.[](https://konghq.com/contact-sales)
To validate this architecture, Kong tested the integration using [_OpenThoughts-TBLite_](https://github.com/open-thoughts/OpenThoughts-TBLite)_OpenThoughts-TBLite_, a benchmark featuring 20 difficulty-calibrated agent tasks across different categories of easy, medium, hard, and expert.
The results demonstrate how precisely tuning a single `stage_router` parameter can optimize the cost-to-performance ratio. By increasing the confidence threshold from 0.3 to 0.5, escalations to the frontier model dropped from 85% to just 17%. This shift yielded a **43.7% reduction in cost per completed task** while maintaining consistent task completion rates.
In this configuration, the efficient tier utilized GLM-5.2, while the frontier tier ran Claude Opus 4.8. While these preliminary results highlight the potential of intelligent routing, Kong is continuing full evaluations against Terminal-Bench 2 for final verification.
*Learn more: *[_*Kong AI Gateway*_](https://developer.konghq.com/ai-gateway/)_*Kong AI Gateway*_* · *[_*NVIDIA NeMo Switchyard*_](https://github.com/NVIDIA-NeMo/Switchyard)_*NVIDIA NeMo Switchyard*_* · *[_*AI Proxy Advanced*_](https://developer.konghq.com/plugins/ai-proxy-advanced/)_*AI Proxy Advanced*_
*Get started with the API & AI platform: *[_*Book a demo*_](https://konghq.com/contact-sales)_*Book a demo*_
**NVIDIA NeMo Switchyard gives teams configurable algorithms for automating model selection according to their own criteria. Kong AI Gateway routes, secures, meters, and logs. Intelligence advances at ML speed; the security boundary never moves.**
Traditional APIs are, in a word, predictable. You know what you're getting: Compute costs that don't surprise you Traffic patterns that behave themselves Clean, well-defined request and response cycles AI APIs, especially anything that runs on LLMs
Why AI guardrails matter It's natural to consider the necessity of guardrails for your sophisticated AI implementations. The truth is, much like any powerful technology, AI requires a set of protective measures to ensure its reliability and integrit
When an organization deploys AI agents at scale, high uptime and low latency are an important baseline. However, Platform owners and business stakeholders could be flying blind on several fronts: The Insights Gap: Non-technical stakeholders have li
Built on top of Kong API Gateway, the Kong AI Gateway is designed to address key challenges in enterprise AI adoption. Modern AI applications rarely rely on a single model; instead, they orchestrate multiple GenAI providers, agent frameworks, Age
Agent-to-agent communication is the next frontier of AI infrastructure. As teams decompose monolithic AI workflows into specialized agents — a research agent, a booking agent, a summarization agent — the calls between those agents become as importa
Bring Financial Accountability to Enterprise LLM Usage with Konnect Metering and Billing Showback and chargeback are not the same thing. Most organizations conflate these two concepts, and that conflation delays action. Understanding the LLM showb
