# Intelligent Model Routing: NVIDIA Selects the Model, Kong Routes the Traffic
Every team running production LLMs has had the same idea: not every request needs the frontier model. Intelligent model routing (or LLM routing)— choosing a model per request on criteria such as task complexity, cost, latency, or quality — enables more efficient model usage.
The idea is easy. Shipping it is not. The moment routing logic goes into the request path, it becomes infrastructure — and now it's holding your provider credentials, needs to comply with security requirements (guardrails, auditing, etc.), and is standing between your applications and every model they depend on, which requires reliable, highly available, and redundant infrastructure. Teams can stall at this point because the application or ML team defining the routing strategy should not also have to own the infrastructure and security boundary. Meanwhile, the platform team responsible for that boundary may not have the application context needed to continually tune routing policies and algorithms.
The solution is to separate model selection from traffic management. Intelligent model routing is a combined system: NVIDIA NeMo Switchyard provides the model selection, and Kong AI Gateway provides the connectivity, governance, and traffic management. Switchyard is a customizable selection library with multiple algorithms that teams can configure according to their own requirements. Kong AI Gateway routes the traffic — dispatching each request to the selected model while managing credentials, guardrails, PII masking, rate limits, and auditing through a scalable, reliable gateway.
## Kong AI Gateway: Connectivity and Governance
As AI adoption scales, applications evolve into complex systems of agents, orchestration layers, and context servers. Infrastructure lags behind — struggling with authentication, cost control, and data security across a provider list that changes every quarter.
Kong AI Gateway is the runtime for LLM traffic management those systems already run through. Its Universal API standardizes interfaces across providers, decoupling applications from provider-specific SDKs and centralizing credential management as part of a broader AI governance strategy. On top of that, the gateway enforces what production actually requires:
- - **Token-based rate limiting and metering** per team, application, and model - the control that caps AI spend, not just optimizes it
- - **Credentials in a vault, never in application code**, rotated centrally across every provider
- - **PII sanitization and prompt guardrails** applied before a request ever leaves your network
- - **Semantic caching** to eliminate redundant inference entirely
- - **Model and provider routing, with multi-provider failover, retries, and load balancing** — the traffic layer that survives a provider outage
- - **One control plane for APIs, AI, MCP, and events**, with RBAC, audit, and analytics across all of it
- - **Highly scalable, performant dataplanes that support hybrid, self-hosted, and air-gapped environments** — providing teams architectural freedom
That last pair matters more than it looks. Your agents don't only call models — they call REST APIs, MCP servers, and event streams. Governing the LLM hop alone leaves most of the attack surface ungoverned. Further, an enterprise network is complex, with workloads running on-prem, across clouds, etc.
## NVIDIA NeMo Switchyard: Configurable Model Selection
Per-request model selection can account for task complexity and other application-specific requirements. NeMo Switchyard is an open-source Apache 2.0 model selection library from NVIDIA. It makes several selection approaches available through an open-source library that teams can configure and extend for their own use cases.
In this integration, teams can deploy it as a model selection service: Kong provides the information needed to evaluate the configured selection policy, Switchyard returns a target, and Kong maintains control of the request path. Its selection algorithms - including complexity-based approaches such as `stage_router` give teams configurable building blocks for evaluating requests and selecting among model targets. Teams choose the algorithm, candidate models, thresholds, and other criteria appropriate to their applications.
Two configurable capabilities can help teams operate this pattern reliably:
- - **Configurable fallback behavior: **With classifiers that support a strong_target, teams can configure ambiguous requests to use a designated higher-capability model rather than forcing a lower-confidence routing decision.
- - **Session persistence:** For multi-turn conversations, teams can retain an initial routing decision across a session when appropriate, reducing redundant routing evaluations and their associated latency.
For detailed setup and advanced routing configurations, refer to the NeMo Switchyard documentation.
## Why Intelligent Model Routing Needs Both
Because model selection is one decision, and routing production AI traffic is many more.
A selection library helps teams automate model choice according to their configured policies. It is not intended to replace gateway capabilities such as credential management, per-team token limits, PII masking, cost attribution, or audit logging. And it doesn't govern the other 90% of your traffic — the APIs, MCP servers, and event streams your agents depend on.
Put the router in the data path instead, and you've added a second hop that sees prompts, holds keys, and carries its own CVE surface — with no SLA behind it. Use it as a configurable decision service, and teams can apply their chosen Switchyard selection strategy while Kong keeps routing the traffic and the gateway boundary intact.
## Architecture: Configurable Selection, Centralized Routing and Governance
*Kong AI Gateway routes every request; NVIDIA NeMo Switchyard supplies the model selection*
The data path begins when an application sends a request to Kong AI Gateway. Kong invokes the Switchyard decision service, which evaluates the request using the selection algorithm and criteria configured by the user and returns a model target. Kong then routes the request to that target, applies the organization’s configured policies, and returns the response to the application. If the decision service is unavailable, Kong can use a customer-configured fallback target, helping preserve request-path availability.
## Scale Your AI Strategy
This pairing succeeds because it respects organizational boundaries. Model selection quality is an ML problem. Routing, security, compliance, infrastructure, and even cost control are platform problems. This architecture lets each team excel without overstepping.
- - **For the ML team:** Choose and tune selection algorithms, candidate models, thresholds, and policies without embedding that logic in the gateway configuration.
- - **For the platform team:** Support automated model selection while keeping credentials, guardrails, PII masking, auditing, and other traffic controls centralized in the gateway.
- - **For the business:** Establish an intelligent model routing strategy that balances cost, latency, and quality. Teams can direct different request types to appropriate models and use gateway telemetry to measure usage and cost by request, application, or team.
Routing requirements and algorithms can evolve independently from gateway infrastructure. With a clear contract, teams configure how model targets are selected through Switchyard, while Kong routes, secures, meters, and logs the resulting traffic. This separation allows routing strategies to change without moving the gateway’s security and governance boundary.[](https://konghq.com/contact-sales)
## Performance and Efficiency Benchmarks
To validate this architecture, Kong tested the integration using [_OpenThoughts-TBLite_](https://github.com/open-thoughts/OpenThoughts-TBLite)_OpenThoughts-TBLite_, a benchmark featuring 20 difficulty-calibrated agent tasks across different categories of easy, medium, hard, and expert.
The results demonstrate how precisely tuning a single `stage_router` parameter can optimize the cost-to-performance ratio. By increasing the confidence threshold from 0.3 to 0.5, escalations to the frontier model dropped from 85% to just 17%. This shift yielded a **43.7% reduction in cost per completed task** while maintaining consistent task completion rates.
In this configuration, the efficient tier utilized GLM-5.2, while the frontier tier ran Claude Opus 4.8. While these preliminary results highlight the potential of intelligent routing, Kong is continuing full evaluations against Terminal-Bench 2 for final verification.
*Learn more: *[_*Kong AI Gateway*_](https://developer.konghq.com/ai-gateway/)_*Kong AI Gateway*_* · *[_*NVIDIA NeMo Switchyard*_](https://github.com/NVIDIA-NeMo/Switchyard)_*NVIDIA NeMo Switchyard*_* · *[_*AI Proxy Advanced*_](https://developer.konghq.com/plugins/ai-proxy-advanced/)_*AI Proxy Advanced*_
*Get started with the API & AI platform: *[_*Book a demo*_](https://konghq.com/contact-sales)_*Book a demo*_
**NVIDIA NeMo Switchyard gives teams configurable algorithms for automating model selection according to their own criteria. Kong AI Gateway routes, secures, meters, and logs. Intelligence advances at ML speed; the security boundary never moves.**