Traditional observability shows whether infrastructure and services are available, responsive, and error-free. AI observability retains those signals, then adds context ordinary APM often lacks. This includes models, tokens, retrieval results, evaluations, policy decisions, and tool actions.

This distinction matters because an HTTP 200 response can still contain an unsupported answer, trigger an unauthorized action, or consume excessive tokens. A practical operating model pairs conventional telemetry with three additional forms of evidence:

• - **Evaluation:** Did the response meet task, groundedness, and user-experience criteria?
• - **Economics:** Which request, feature, team, tenant, model, retry, or cache result drove usage?
• - **Governance:** Which identity, policy, authorization, and safety decisions shaped the outcome?

Observability supplies evidence. It does not guarantee quality, safety, or compliance.

To add comprehensive AI request logging and token cost analysis, combine three instrumentation patterns. Route shared traffic through an AI gateway or proxy. Use OpenTelemetry GenAI conventions, then add application instrumentation for workload-specific context.

A practical architecture is:

**application or agent → AI gateway → OTel Collector → existing observability backend**

Correlate gateway telemetry and application spans with a common trace ID. This lets teams keep their current backend while [unifying logs, metrics, and traces with OpenTelemetry](https://konghq.com/blog/engineering/tracing-logging-metrics-unifying-observability-with-opentelemetry)unifying logs, metrics, and traces with OpenTelemetry.

Use the three patterns together:

1. - **AI gateway or proxy:** Capture request metadata, model routing, tokens, latency, errors, retries, cache outcomes, identity, and policy results. Apply this schema at a consistent traffic layer.
2. - **OpenTelemetry GenAI telemetry:** Represent model calls, retrieval, and agent operations with a shared vocabulary, then export through an OTel Collector. The GenAI semantic conventions remain in **Development** status, so pin instrumentation to supported versions and verify field names before deployment.[1][2][3]
3. - **Application or framework instrumentation:** Add business context such as feature, workflow, tenant, experiment, evaluation score, retrieval purpose, and task result. Instrument tool calls and handoffs that do not cross the gateway.

At minimum, record operation, requested and returned model, provider, input and output tokens, TTFT, end-to-end latency, errors, retries, cache result, and team or tenant. Add retrieval spans, tool spans, authorization outcomes, and safety decisions when those components are present.

For cost analysis, roll usage up by request, application, feature, team, tenant, model, provider, retry, and cache outcome. Avoid embedding static provider prices in dashboards; connect token measurements to a maintained pricing catalog or billing source. [Gateway-level AI token cost management](https://konghq.com/blog/enterprise/ai-token-cost-management)Gateway-level AI token cost management can centralize attribution and controls before requests reach providers.

Collect metrics only when they lead to a diagnosis, evaluation, or control. Set percentile SLOs from your workload baseline rather than adopting universal thresholds; benchmark metric terminology is not standardized across tools.[4]

• - **Performance:** Track TTFT, inter-token latency or delivery rate, queue time, end-to-end latency, throughput, concurrency, errors, retries, and rate-limit responses.
• - **Cost:** Track tokens, request or workflow cost, retries, cache effectiveness, budgets, and spend by team, tenant, feature, model, or provider.
• - **Quality:** Track task completion, answer relevance, groundedness, citation correctness, user feedback, regression-test outcomes, and evaluator disagreement.
• - **Safety:** Track policy decisions, suspicious instructions, prompt injection attempts, blocked actions, authorization denials, false positives, and downstream effects. [Monitoring OWASP risks in LLM applications](https://konghq.com/blog/engineering/owasp-top-10-ai-and-llm-guide)Monitoring OWASP risks in LLM applications supports layered defenses because current controls do not fully mitigate prompt injection.[5]
• - **RAG:** Track retrieval latency, result count, filters, source or collection, relevance and coverage, document scores, reranking, context use, and answer support.
• - **Agents:** Trace the parent agent invocation, model calls, tool actions, handoffs, retries, errors, loops, and authorization decisions. OTel’s agent model connects these child operations into one investigation path.[2]

For agentic systems, token cost at the model call is only part of the bill. Retrieval, repeated planning, tool execution, and failed loops can compound cost, which makes [full-path cost visibility for agentic AI](https://konghq.com/blog/enterprise/ai-cost-management-stopping-margin-erosion)full-path cost visibility for agentic AI essential.

Build dashboards around decisions rather than collecting every available field. An on-call view should connect service health, request traces, model latency, token growth, and policy failures. A product-quality view should compare evaluations, user feedback, retrieval performance, and release versions.

Use dimensions carefully to control telemetry volume. Keep high-cardinality identifiers in traces or logs, then aggregate stable dimensions such as application, environment, team, route, model, and outcome. Preserve enough context to reproduce an incident without storing every prompt or response.

A RAG trace should separate retrieval from generation. Follow the request through query preparation, retrieval, reranking, context assembly, model inference, citations, and answer evaluation; [how retrieval-augmented generation pipelines work](https://konghq.com/blog/learning-center/what-is-rag-retrieval-augmented-generation)how retrieval-augmented generation pipelines work provides the architectural foundation.

Measure whether retrieval found the right evidence, then measure whether the answer used that evidence correctly. Retrieved context does not guarantee a supported response. RAGTruth research found that RAG outputs can still contain unsupported or contradictory claims.[6]

Capture raw queries and documents only when necessary. OTel notes that retrieval queries may contain sensitive information.[3] Prefer identifiers, scores, filters, redacted text, or sampled payloads when they answer the operational question

Start with the user-visible symptom, locate an affected trace, and compare it with a known-good baseline.

• - **Slow first response:** Split queue time, gateway time, retrieval time, provider time, and streaming time. Check prompt size, concurrency, routing changes, retries, and cache misses.
• - **Unexpected cost growth:** Group tokens by feature, tenant, model, retry, and cache result. Look for longer prompts, unbounded outputs, agent loops, retry storms, or routing changes.
• - **Quality regression:** Compare model, prompt, retrieval source, context, and evaluator results against the previous baseline. Re-run a versioned test set before changing production policy.
• - **Unsafe or unauthorized action:** Review the instruction chain, identity, tool request, policy result, authorization decision, and downstream effect. Monitoring complements least privilege and [governance controls for autonomous AI agents](https://konghq.com/blog/enterprise/agentic-ai-governance)governance controls for autonomous AI agents; it does not replace them.

Telemetry payloads can contain sensitive information, including retrieval queries.[3] Apply data minimization: define what is captured, redact sensitive fields, sample payloads, restrict access, encrypt data, and set retention by purpose and risk.[7]

Separate telemetry, evaluation, and enforcement. Assign accountable owners for each boundary. Engineers can own instrumentation, evaluation teams can define quality criteria, and security or governance teams can set policy and retention requirements.

NIST’s voluntary GenAI profile supports post-deployment monitoring, evaluation, documentation, and incident response as risk-management practices.[7] Article 12 of the EU AI Act includes event-logging requirements for systems within its high-risk scope. Applicability requires legal review.[8]

• - **Establish the request record:** Instrument model calls, propagate trace IDs, count tokens, capture errors and latency, and create baseline dashboards. Add redaction and access controls before collecting payloads.
• - **Connect behavior to outcomes:** Add RAG and agent spans, quality evaluations, user feedback, cost attribution, safety events, and workload-specific SLOs. Create runbooks for latency, cost, quality, and security incidents.
• - **Turn evidence into controls:** Use tested signals to drive routing, caching, quotas, rate limits, alerts, regression gates, and policy reviews. Automate cautiously, with rollback paths and accountable owners.

[Kong AI Gateway](https://konghq.com/products/kong-ai-gateway)Kong AI Gateway provides a control layer between applications and the LLMs, agents, and MCP servers they use. It centralizes routing, token usage controls, quotas, access control, prompt guardrails, and traffic telemetry.

This gateway view complements application instrumentation. The gateway records consistent cross-provider traffic signals, while application spans explain business intent, evaluations, retrieval, and tool behavior.

**What is AI observability?**

AI observability is the practice of connecting logs, metrics, and traces with evidence about model calls, retrieval, agents, quality, safety, and cost. It helps teams explain an AI system’s behavior and troubleshoot the request path that produced an outcome.

**How does AI observability differ from traditional observability or APM?**

Traditional observability and APM focus on infrastructure, services, code paths, latency, and errors. AI observability adds model, token, evaluation, retrieval, tool-use, policy, and cost context needed to assess non-deterministic AI outcomes.

**What metrics should teams track for production AI and LLM applications?**

Track performance, token use and cost, quality, safety, retrieval, and agent execution. Choose workload-specific baselines for TTFT, end-to-end latency, errors, retries, task success, groundedness, policy outcomes, tool actions, and cost attribution.

**How do you monitor a RAG pipeline?**

Trace query preparation, retrieval, reranking, context assembly, generation, citations, and answer evaluation as separate operations. Measure retrieval relevance and coverage independently from whether the final answer is supported by the retrieved evidence.

**What is the difference between AI observability and LLM observability?**

LLM observability focuses on model calls, prompts, responses, tokens, latency, and evaluations. AI observability is broader because it also covers retrieval systems, agents, tools, policies, identities, and the surrounding application and infrastructure.

[Request a demo to explore Kong's API platform capabilities.](https://konghq.com/contact-sales)Request a demo to explore Kong's API platform capabilities.

[1] OpenTelemetry Semantic Conventions GenAI SIG. Client inference. Official OpenTelemetry repository. Accessed November 4, 2026. [https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md)https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md.

[2] OpenTelemetry Semantic Conventions GenAI SIG. Agent spans. Official OpenTelemetry repository. Accessed November 4, 2026. [https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md)https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md.

[3] OpenTelemetry Semantic Conventions GenAI SIG. Retrieval spans. Official OpenTelemetry repository. Accessed November 4, 2026. [https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/retrieval.md](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/retrieval.md)https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/retrieval.md.

[4] vLLM Project. Benchmark CLI. vLLM Documentation. Accessed November 4, 2026. [https://docs.vllm.ai/en/stable/benchmarking/cli/](https://docs.vllm.ai/en/stable/benchmarking/cli/)https://docs.vllm.ai/en/stable/benchmarking/cli/.

[5] OWASP GenAI Security Project. LLM01:2025 Prompt Injection. OWASP. 2025. [https://genai.owasp.org/llmrisk/llm01-prompt-injection/](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)https://genai.owasp.org/llmrisk/llm01-prompt-injection/.

[6] Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). August 2024. [https://aclanthology.org/2024.acl-long.585/](https://aclanthology.org/2024.acl-long.585/)https://aclanthology.org/2024.acl-long.585/.

[7] National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. July 2024. [https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf.

[8] European Commission. AI Act. Shaping Europe’s digital future. Accessed November 4, 2026. [https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai)https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai.

[9] NIST Computer Security Resource Center. Minimization. Glossary entry sourced from CNSSI 4009-2022 / OMB Circular A-130. Accessed November 4, 2026. [https://csrc.nist.gov/glossary/term/minimization](https://csrc.nist.gov/glossary/term/minimization)https://csrc.nist.gov/glossary/term/minimization.