[llm-d](https://llm-d.ai)llm-d is an open-source, Kubernetes-native framework for serving large language models at scale. It doesn't run models itself. It sits in front of inference engines such as [vLLM](https://vllm.ai/)vLLM, another open-source, high-throughput serving engine, and decides how requests are spread across them.

Teams taking a self-hosted LLM to production usually start with a managed model API. They call a hosted model from OpenAI, Anthropic or a cloud provider over HTTP and pay per token. When evaluating self-hosted LLM vs managed OpenAI cost trade-offs, the managed route means there are no GPUs to provision and no servers to run. With low or unpredictable traffic, it is usually the cheaper option, because you never pay for idle hardware.

The alternative is to self-host an open-weight model on your own GPUs, in your data center or your cloud account. Open-weight models are now a real option for production work: Z.ai's GLM, Moonshot AI's Kimi and NVIDIA's Nemotron families publish their weights and are built for coding, reasoning and agentic tasks. Compared with a managed API, this is the trade:

• What you gain: control over the whole stack. Prompts and data stay in your environment, there are no provider rate limits, and you decide how inference runs. You can route requests to the GPU that already holds their cache, split prefill and decode into separate pools, or serve many fine-tuned LoRA adapters on one base model. Furthermore, when comparing vLLM vs OpenAI API latency, self-hosting eliminates unpredictable network hops and provider-side queueing, offering consistent performance dictated entirely by your own hardware's KV-cache reuse.
• What you give up: simplicity, and the protections a provider gives you. You operate the GPUs, the serving engine and the scheduling. The GPUs cost the same whether traffic arrives or not, so how busy you keep them decides your cost per token. And the endpoint comes with no API keys, rate limits or usage reporting.

llm-d and Kong AI Gateway each address one part of what you give up. When the GPUs are yours, every request that lands on the wrong one wastes money. llm-d makes self-hosted inference efficient by routing each request based on what every vLLM instance is doing. Kong AI Gateway adds back the API keys, rate limits and usage reporting a managed API would have provided.

llm-d's main use cases are about getting the most out of those GPUs:

• Prefix-cache-aware routing: each vLLM replica keeps a KV cache, the keys and values it computed for the prompts it has seen, and reuses it when a new request starts with the same text. A plain Kubernetes Service sends requests to any replica, so a repeated system prompt or conversation history is computed again on each one. llm-d's router estimates which replica already holds a request's prefix and sends it there, falling back to the least loaded replica when that one is busy. Part 2 runs this as llm-d's optimized baseline.
• Prefill/decode disaggregation: reading the prompt (prefill) is limited by compute, while generating tokens one at a time (decode) is limited by memory bandwidth. llm-d can run the two phases in separate pods and move the KV cache from one to the other, so each pool is sized for its own work and long prompts don't stall token generation. Part 2 runs it on the four GPUs of one node.
• LoRA adapters: a LoRA adapter is a small set of extra weights that specializes a base model for one task. vLLM serves adapters on top of a single copy of the base model, and llm-d's router prefers the replicas where the requested adapter is already loaded. Part 2 serves one adapter, exposed through Kong as its own model alias.

llm-d covers more than this series does. For example, wide expert parallelism spreads a large Mixture-of-Experts model, too big for one server, across the GPUs of several nodes.

This blog post is the first of a two-part series:

• Part 1 (this post) describes how llm-d, vLLM and Kong AI Gateway work together. It covers the llm-d architecture and the role each layer plays, then builds the stack on Amazon EKS: vLLM serving Llama 3.2 3B, an llm-d router in front of it, and Kong AI Gateway 2.3 exposing the model with per-application API keys and a per-consumer token rate limit.
• Part 2 uses the same deployment to run three llm-d use cases: the optimized baseline, compared with a plain Kubernetes Service; P/D disaggregation, which splits prefill and decode across separate GPUs; and LoRA adapters, served next to the base model behind their own Kong model alias.

llm-d Architecture

The [llm-d documentation portal](https://llm-d.ai/docs/architecture)llm-d documentation portal has an architecture diagram of the project:

Its Architecture landing page defines the main components:

• llm-d Router: the intelligent entry point for inference requests. It provides LLM-aware load balancing, request queuing, and policy enforcement. It is composed of two functional parts:

• Proxy: a high-performance L7 proxy conformant with the Gateway API Inference Extension (GAIE) that accepts user requests and consults the EPP via the ext-proc protocol to determine the optimal destination.
• Endpoint Picker (EPP): the routing engine that scores and selects model server pods based on real-time metrics, KV-cache affinity, and configured policies.

• InferencePool: the API that groups Model Server pods serving the same base model via a label selector. Conceptualized as an "LLM-optimized Service", it serves as the discovery target for the Router.
• Model Server: the inference engine (such as vLLM or SGLang) that executes the model on hardware accelerators (GPUs, TPUs, HPUs).

llm-d, vLLM and Kong AI Gateway

Each layer has one job:

• vLLM serves Llama 3.2 3B, one GPU per pod.
• llm-d runs a router in standalone mode, reachable only inside the cluster. For every request it picks the vLLM replica that serves it. Part 2 changes how it picks.
• Kong AI Gateway is the only public entry point. It authenticates callers, maps a model alias to an llm-d router, limits and records token usage.

| A note on "replica". In this post, a replica always means one pod of the vllm-server Deployment: one vLLM process serving its own copy of Llama 3.2 3B on one GPU, with its own KV cache. Kubernetes creates the replicas, through the Deployment's replicas field. llm-d doesn't create or own any; its router only chooses which replica serves each request. The pods in the P/D steps are called prefill and decode pods, not replicas. |

The cluster uses two node groups: a small CPU node for the gateways and controllers, and one g6.12xlarge with four NVIDIA L4 GPUs for the models. The lab serves Llama 3.2 3B because it fits on one L4; larger models such as GLM, Kimi or Nemotron run on the same stack with bigger GPUs. Part 1 uses one GPU; Part 2 takes turns on all four.

Prerequisites: aws, eksctl, kubectl, helm, jq, [kongctl](https://developer.konghq.com/kongctl/)kongctl, a Konnect personal access token exported as PAT, a Hugging Face token exported as HF_TOKEN whose account has accepted the Llama 3.2 license, and the AWSLoadBalancerControllerIAMPolicy IAM policy in your AWS account. If the policy doesn't exist yet, create it from the policy JSON in the AWS Load Balancer Controller's install docs.

Create the EKS cluster with a single node

Start with one CPU node. It will run Kong and the llm-d routers, none of which need a GPU.

Add the GPU node group

Node groups are added to the running cluster with eksctl create nodegroup. This one holds the models. Its minimum size is zero, so it can be parked when idle.

For GPU instance types, eksctl picks the EKS-optimized NVIDIA AMI. By default it also installs the NVIDIA device plugin as a DaemonSet that targets every node in the cluster, including the CPU node. --install-nvidia-plugin=false turns that off. Install the plugin with Helm instead. Its chart only schedules on nodes labeled nvidia.com/gpu.present=true, which is the second label on this node group, so it runs on the g6.12xlarge only:

Check the plugin:

The DaemonSet should show 1 desired and 1 ready, its pod should be on the GPU node, and the last command should print 4, the number of GPUs the node offers. Any later node group is added the same way, with a different --name, --node-type and label. When you are done for the day, park the GPUs:

Install the cluster add-ons

Three add-ons are needed. EKS Pod Identity gives pods IAM roles. The AWS Load Balancer Controller creates the NLB in front of Kong. The EBS CSI driver backs the volume that caches model weights.

Deploy vLLM and send a prompt

Everything model-related lives in the llm-d namespace. Create a gp3 storage class, the volume for the Hugging Face cache, and the token secret:

Then deploy vLLM with Llama 3.2 3B on one GPU:

The first start downloads the weights. Follow the logs until vLLM reports it is listening on port 8000:

Now test it with a simple prompt, straight to the pod's Service:

The model answers, but with no authentication, no usage tracking, and one replica.

Deploy a basic llm-d router

Start with llm-d in front of the single vLLM replica from the previous section. The router is installed in standalone mode: the router pod carries its own Envoy proxy next to the endpoint picker, so no Kubernetes Gateway is needed. Kong will be the gateway.

Three pieces work together inside the router:

• Envoy proxy is the data path. It receives each request, asks the endpoint picker which pod should serve it, and forwards the request there. It makes no routing decisions itself.
• Endpoint picker (EPP) is the scheduler. Envoy calls it for every request through its external processing hook, and the EPP scores the candidate pods with a set of plugins and returns the chosen one. Which plugins run is set by its configuration: here the chart's default, replaced with llm-d's optimized baseline in Part 2.
• InferencePool is the EPP's list of candidates. It holds the label selector and port of the vLLM pods, so the EPP picks up new vLLM replicas as soon as they match.

Before llm-d, requests reach vLLM through a plain Kubernetes Service, which spreads connections across pods without looking at the request. The EPP is what adds request-aware scheduling.

The chart includes an InferencePool resource, and Kubernetes doesn’t know that kind without the CRDs. When Helm builds the release, it checks every resource kind against the cluster’s API, so the install stops with an error.

Install the InferencePool CRDs the router uses:

Install the router from llm-d's chart. A basic install needs a single value: the label that selects the vLLM pods deployed above. Everything else comes from the chart's defaults, so the endpoint picker runs the llm-d Helm chart's default scheduling configuration.

The router is reachable through a Service named after the Helm release, llm-d-epp. Its Envoy proxy listens on port 8081. Send the same simple prompt as before, this time through llm-d:

The answer is the same as before. The difference is the path: the router's Envoy proxy received the request, asked the endpoint picker which vLLM replica should serve it, and forwarded it there. This router runs the chart's default scheduler. Next, Kong AI Gateway goes in front of it; Part 2 then switches the scheduler to llm-d's optimized baseline.

Install Kong AI Gateway 2.3

Kong AI Gateway 2.3 is managed from Konnect and deployed on Kubernetes through Kong Operator. Two environment flags turn on the controllers it needs.

The operator talks to Konnect with your personal access token, stored in a Secret:

KonnectAIGateway creates the AI Gateway control plane in Konnect:

AIGatewayDataPlane runs the AI Gateway 2.3 data plane on the CPU node and exposes it through an internet-facing NLB:

Once the NLB is provisioned, capture its address and port:

Configure the AI endpoint for your self-hosted LLM

kongctl or Kong Operator. The AI Gateway configuration can be managed with more than one mechanism: declaratively with kongctl, through Kong Operator CRDs, or with a mix of both. As an exercise, Part 1 configures the gateway with kongctl. Part 2 configures it with Kong Operator CRDs.
The gateway configuration is declared with kongctl. Five entities build the endpoint:

• AI Model Provider of type vllm. llm-d speaks the same OpenAI-compatible API as vLLM, so the provider type is the same. The auth block follows Kong's vLLM provider example; llm-d itself needs no credentials, since only Kong can reach it inside the cluster.
• AI Auth Strategy of type key-auth, reading the apikey header.
• AI Consumer with its own key, one per application.
• AI Model llama-3b on the path /llm-d, targeting the llm-d router Service.
• AI Policy llama-3b-token-limit of type ai-rate-limiting-advanced, attached to the AI Model, capping the tokens each consumer can use per minute. It is covered after the first tests.

Export the values the file reads from the environment:

This is how one request is handled. Kong checks the API key, matches the llama-3b alias in the model field, rewrites it to the real model name, and forwards the request to the llm-d router. llm-d then picks the replica, for now the only one. The client never sees the upstream model name, the llm-d address, or which pod answered.

| Heads-up. The KonnectAIGateway was created by Kong Operator, not by kongctl. If kongctl sync refuses to attach entities to it, run kongctl adopt ai-gateway kong-llmd-vllm --namespace kong-llmd-vllm --pat $PAT first. |

Test the endpoint

Send a prompt through Kong with the application key. The X-Kong-LLM-Model response header shows the upstream model the alias resolved to. The AI Model targets llm-d's /v1/chat/completions endpoint, so requests through Kong use the chat format; the /v1/completions calls above only work against vLLM and llm-d directly. The data plane serves Kong's default self-signed certificate, so curl runs with -k; put a real certificate on it before production.

Now try without a key:

The request is rejected at the Kong data plane and never reaches llm-d or a GPU. In Konnect, open the kong-llmd-vllm AI Gateway to see requests, tokens and latency broken down by AI Model and AI Consumer.

Limit tokens per consumer

An API key decides who can call the model. It doesn't limit how much they use, and on self-hosted GPUs one busy application can take the capacity every other application depends on. Kong's [AI Rate Limiting Advanced](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/)AI Rate Limiting Advanced policy caps the tokens each AI Consumer can spend in a time window.

That is the llama-3b-token-limit AI Policy declared in kong-aigw2-llm-d.yaml and listed under the llama-3b model's policies:

• identifier: consumer keeps a separate counter for each AI Consumer, so consumer1 using up its budget doesn't affect any other application.
• tokens_count_strategy: total_tokens counts prompt and completion tokens together.
• The limit is 300 tokens per sliding 60-second window. It's set low so the test below hits it quickly; a real budget would be much higher.
• strategy: local keeps the counters in the data plane's memory, which is enough for one data plane replica. With several replicas, use the redis strategy so they share one counter.

Send a few requests in a row with the same key, printing the status code and the rate-limit headers Kong adds:

Each request uses around 100 tokens. The first few return 200 with the remaining budget going down, then Kong starts returning 429 with API rate limit exceeded. Kong counts tokens from the model's response, so the request that crosses the limit still succeeds, and the block applies from the next one. A 429 is returned by the data plane, so the blocked requests never reach llm-d or a GPU. After a minute, the window frees up and requests go through again.

In Konnect, the same AI Gateway view shows the token usage that counts against each consumer's limit.

Konnect Observability

The data plane reports every request it handles to Konnect, so the traffic from the tests above is already there. Under Observability, Konnect provides ready-made dashboards for AI Gateway traffic. They can be used as they are or edited tile by tile. This one covers 15 minutes of the lab:

Each group of tiles answers a different question:

• Usage: GenAI model usage count and GenAI provider usage count show how many requests went to each AI Model and provider, and Total AI gateway requests gives the overall count. Here, 119 requests reached Llama 3.2 3B (llm-d) through the vLLM provider.
• Tokens: Token usage by provider plots total tokens over time, and Consumer token usage breaks them down by AI Consumer, the same count the rate-limit policy enforces for consumer1.
• Cost: Cost per consumer and Total cost turn token counts into spend, per application and overall.
• Health: LLM latency (avg) tracks how long the model takes to answer, and AI status codes and Errors by model surface failing requests. Both of those were empty in this window.

The time range, filters and tiles can all be changed from the top of the dashboard, and Explorer, in the same menu, builds ad hoc queries over the same data. The lab didn't configure anything for this: it comes with the data plane's connection to Konnect.

Konnect Tokenomics

Konnect Tokenomics provides a structured, transparent way to control, monitor, and allocate AI spend across complex enterprise workflows. Because Kong operates directly in the execution path of AI traffic, Konnect meters token consumption and cost as requests are processed, consistently across every supported model provider. Organizations can attribute that spend to users, teams, projects, and agents, follow multi-agent sessions as a single unit of work, and establish daily, weekly, or monthly budgets that alert stakeholders before limits are reached. Combined with AI Gateway capabilities such as intelligent routing, semantic caching, and prompt compression, and with Kong Metering and Billing for chargeback, Konnect gives finance and platform teams a consistent, auditable view of AI expenditure across every team, department, and AI tool.

What's next

The stack works end to end: Kong authenticates every call, maps the llama-3b alias, caps and records token usage per consumer, and llm-d routes each request to the vLLM replica. The router, though, still runs the llm-d Helm chart's default scheduling configuration, and there is only one replica to choose from.
Part 2 keeps this cluster and evolves the deployment through three llm-d use cases:

• Optimized baseline: four replicas, with the router sending each request to the replica that already holds its prompt prefix in cache.
• P/D disaggregation: prefill and decode split across separate pods, with the KV cache moving between GPUs.
• LoRA adapters: a fine-tuned adapter served next to the base model, exposed through Kong as its own model alias.

Cleanup

If you are continuing with Part 2, keep the cluster running. Otherwise, remove everything:

Delete the data plane before the control plane, so the operator can deregister the certificate cleanly.

References

• [Kong AI Gateway with Kong Operator](https://developer.konghq.com/operator/konnect/ai-gateway/)Kong AI Gateway with Kong Operator
• [vLLM provider for Kong AI Gateway](https://developer.konghq.com/ai-gateway/ai-providers/vllm/)vLLM provider for Kong AI Gateway
• [Kong AI Gateway with llm-d](https://llm-d.ai/docs/operations/serve-external-apis/kong)Kong AI Gateway with llm-d
• [llm-d optimized baseline](https://llm-d.ai/docs/well-lit-paths/foundations/optimized-baseline)llm-d optimized baseline
• [llm-d P/D disaggregation guide](https://github.com/llm-d/llm-d/tree/main/guides/pd-disaggregation)llm-d P/D disaggregation guide
• [llm-d router](https://github.com/llm-d/llm-d-router)llm-d router
• [vLLM on Kubernetes](https://docs.vllm.ai/en/latest/deployment/k8s/)vLLM on Kubernetes

Frequently Asked Questions (FAQ)

1. What is the difference between a standard API Gateway and an LLM API Gateway like Kong?
   A standard API Gateway routes HTTP traffic based on headers and paths but lacks awareness of the payload's context. An LLM API Gateway natively understands AI protocols (like the OpenAI API format), counts tokens in real-time, supports model aliasing, and enforces AI-specific rate limits. This allows you to cap usage by tokens rather than just HTTP requests.
2. Why is my vLLM Kubernetes GPU utilization low, and how does llm-d fix it?
   Low GPU utilization in vLLM on Kubernetes often occurs because standard Kubernetes Services use round-robin load balancing. This means requests are sent to random pods, causing cache misses and redundant computations. The llm-d router fixes this by using prefix-cache-aware routing, sending requests to the specific vLLM replica that already holds the prompt's KV cache, significantly improving hardware utilization and throughput.
3. How does self-hosted vLLM latency compare to OpenAI API latency?
   When comparing vLLM vs OpenAI API latency, self-hosting offers more predictable performance. While managed APIs are subject to internet routing, provider-side queueing, and multi-tenant throttling, a self-hosted vLLM deployment gives you exclusive access to your GPUs. With optimizations like llm-d's prefix caching, you can achieve consistently lower latency for your specific workloads.
4. Why are repeated prompts recomputed across my vLLM replicas?
   If you are exposing vLLM directly via a standard Kubernetes Service, it distributes traffic evenly across all pods without inspecting the request. If Pod A processes a system prompt, and the next request with the same prompt goes to Pod B, Pod B must compute it from scratch. Using an intelligent router like llm-d ensures requests with matching prefixes are sent to the pod that already has them cached.
5. How do I fix the "Helm install fails: unknown InferencePool kind" error?
   This error occurs when installing the llm-d Helm chart before applying the required Custom Resource Definitions (CRDs). To fix it, you must install the Gateway API Inference Extension (GAIE) CRDs by running: kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.6.1/v1-manifests.yaml before running helm install.
6. What does the "Exceeded token budget error x-ai-ratelimit-remaining: 0" mean?
   This error is returned by Kong AI Gateway when an AI Consumer has spent their allotted token limit within a specific time window (e.g., 300 tokens per minute). Because Kong counts tokens after the model responds, the request that crosses the threshold succeeds, but subsequent requests are blocked with an HTTP 429 status and the x-ai-ratelimit-remaining: 0 header until the sliding time window resets.
7. How do I resolve AWS Load Balancer Controller not targeting GPU nodes?
   When mixing CPU and GPU node groups in EKS, the AWS Load Balancer Controller might target the wrong nodes if labels are not strictly applied. Ensure you use node selectors (e.g., workload=kong-llmd for your CPU nodes running Kong, and workload=vllm for your GPU nodes) and set the Kong Operator deployment to specifically target the CPU node group.