{"id":32245,"date":"2026-09-29T13:43:59","date_gmt":"2026-09-29T08:13:59","guid":{"rendered":"https:\/\/opstree.com\/blog\/?p=32245"},"modified":"2026-09-29T13:43:59","modified_gmt":"2026-09-29T08:13:59","slug":"ai-agent-observability-200-ok-wrong-answer","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/ai-agent-observability-200-ok-wrong-answer\/","title":{"rendered":"AI Agent Observability: Why 200 OK Doesn’t Mean Your AI Agent Got It Right"},"content":{"rendered":"

A practical guide to AI agent observability: how tracing, evaluation and structured debugging help teams <\/span>monitor<\/span> quality, investigate failures, manage cost, and <\/span>operate<\/span> agentic systems more reliably.<\/span><\/span>\u00a0<\/span><\/p>\n

Quick answer:<\/b> AI agent observability is the practice of capturing an agent\u2019s execution path, model calls, retrieval context, tool inputs and outputs, evaluations, latency, token usage and cost so teams can determine whether the agent completed the user\u2019s task correctly, not merely whether the API returned successfully. OpenTelemetry<\/a> defines traces as the path of a request through an application, with spans representing individual units of work; agent observability applies that tracing idea to LLM calls, tools, retrieval and quality signals. OpenTelemetry traces document the path of a request and model the hierarchy through spans. Langfuse describes tracing as the starting point for ingesting and inspecting LLM and agent execution traces.<\/p>\n

Research Basis, Validation Scope and Source Quality<\/span><\/i>\u00a0<\/span><\/p>\n

This article should be treated as a research-backed technical guide, not a benchmark report. The conceptual foundation comes from OpenTelemetry\u2019s trace-and-span model, official Langfuse documentation for tracing, masking, datasets, experiments, OpenTelemetry ingestion and self-hosting, plus operational practices from production observability. Any SDK syntax, region availability, licensing detail, hosting dependency or performance number must be checked against the exact Langfuse version and deployment model used at publication time. OpenTelemetry defines traces and spans as the core structure for understanding request execution paths.Langfuse documents OpenTelemetry ingestion through its OTLP endpoint and SDK guidance.Langfuse documents datasets as test collections for evaluating applications.<\/p>\n

The Failure Your Dashboard Never Saw<\/span><\/span><\/h2>\n

Consider a support agent in production. Its latency and HTTP error-rate dashboards look healthy. Then a customer reports that the agent claimed a refund had been processed when it had not. The agent called search_orders, received an empty result, retried and eventually produced a confident but unsupported response. The exact numbers and sequence in this scenario are illustrative, the underlying failure pattern is a useful example of a successful request that still delivers an incorrect outcome.<\/span>\u00a0<\/span><\/p>\n

Your APM recorded that as a successful HTTP request.<\/span>\u00a0<\/span><\/p>\n

This is the gap. Traditional application performance monitoring (APM) helps answer whether a service responded and how it performed. Agent observability extends that view, did the agent complete the task correctly, use appropriate tools, rely on relevant evidence and stay within acceptable latency and cost limits? Both operational health and task quality matter.<\/span><\/p>\n

Here’s<\/span> the same request, seen through both lenses:<\/span><\/span>\u00a0<\/span><\/p>\n

\n\n\n\n\n\n\n\n\n\n\n
What you’re monitoring<\/th>\nTraditional APM sees<\/th>\nAgent observability sees<\/th>\n<\/tr>\n<\/thead>\n
The request<\/td>\nPOST \/chat \u2192 200, 4.2s<\/td>\nTrace with 14 nested steps<\/td>\n<\/tr>\n
The work<\/td>\nOne service call<\/td>\n6 LLM calls, 5 tool calls, 3 retrievals<\/td>\n<\/tr>\n
The failure<\/td>\nNothing – no exception thrown<\/td>\nsearch_orders returned [] on step 4<\/td>\n<\/tr>\n
The recovery<\/td>\nInvisible<\/td>\nAgent looped twice, then hallucinated<\/td>\n<\/tr>\n
The cost<\/td>\nNot tracked<\/td>\n\u20b918.40 for one conversation<\/td>\n<\/tr>\n
The quality<\/td>\nNot a concept<\/td>\nGroundedness score: 0.2<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Nothing crashed. <\/span>That’s<\/span> exactly the problem.<\/span><\/span>\u00a0<\/span><\/p>\n

\n

Also Read: Enterprise Data Discovery: Strategy, Tool Selection and DPDP Readiness<\/a><\/p>\n<\/div>\n

Why Agent Failures Are Structurally Different<\/span><\/span>\u00a0<\/span><\/h2>\n

In conventional services, many failures surface as explicit errors. Agent failures can also be semantic, a tool may return a valid response while the agent misinterprets it, skips a required step or produces an unsupported answer. The final response can sound plausible even when an earlier step was incorrect.<\/span>\u00a0<\/span><\/p>\n

Three properties make this hard:<\/span>\u00a0<\/span><\/p>\n

Non-determinism. The same input may lead to different outputs or execution paths, depending on model behavior, context, tool state, and system configuration. Capturing the original trace makes investigation more reliable, although replaying a request can still help reproduce some failures.<\/span>\u00a0<\/span><\/p>\n

Depth, A single user turn fans out into planning, tool selection, retrieval, tool execution, reflection and synthesis. Modern agents add sub-agents and handoffs on top of that. The <\/span>token overhead of loading tools alone<\/span><\/a> can dominate a request before any reasoning happens.<\/span>\u00a0<\/span><\/p>\n

Deferred symptoms, The bad tool argument on step 3 doesn’t surface until the summary on step 11. Without causal structure, you’re reading a wall of logs trying to work out which line poisoned the well.<\/span>\u00a0<\/span><\/p>\n

The failure taxonomy that actually shows up in production:<\/span>\u00a0<\/span><\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n
Failure mode<\/th>\nWhat it looks like<\/th>\nWhere you catch it<\/th>\n<\/tr>\n<\/thead>\n
Wrong tool selected<\/td>\nAgent calls send_channel_message instead of send_user_message<\/td>\nTool call span, input args<\/td>\n<\/tr>\n
Bad tool arguments<\/td>\nCorrect tool, malformed or hallucinated parameters<\/td>\nTool call span, input args<\/td>\n<\/tr>\n
Silent tool failure<\/td>\nTool returns empty\/error, agent proceeds anyway<\/td>\nTool output span + downstream generation<\/td>\n<\/tr>\n
Infinite or near-infinite loop<\/td>\nSame step repeats 8 times before hitting a cap<\/td>\nAgent graph view, step counts<\/td>\n<\/tr>\n
Retrieval miss<\/td>\nRight question, wrong documents retrieved<\/td>\nRetriever span, scored for relevance<\/td>\n<\/tr>\n
Context overflow<\/td>\nLong session, early instructions fall out of context<\/td>\nSession view, token counts per turn<\/td>\n<\/tr>\n
Groundedness failure<\/td>\nOutput not supported by retrieved context<\/td>\nLLM-as-a-judge score on the trace<\/td>\n<\/tr>\n
Cost blowout<\/td>\nOne user, one session, 40 model calls<\/td>\nTrace cost aggregation<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Notice that only two of these throw an exception. The rest need you to look at <\/span>content<\/span><\/i>, not status codes.<\/span>\u00a0<\/span><\/p>\n

Traditional APM vs LLM Observability vs Agent Observability<\/span><\/h3>\n
\n\n\n\n\n\n\n\n\n
Dimension<\/th>\nTraditional APM<\/th>\nLLM observability<\/th>\nAgent observability<\/th>\n<\/tr>\n<\/thead>\n
Primary question<\/td>\nDid the service respond reliably?<\/td>\nWhat did the model receive and generate?<\/td>\nDid the agent complete the task correctly across planning, tools, retrieval, and synthesis?<\/td>\n<\/tr>\n
Core data<\/td>\nLatency, errors, throughput, infrastructure metrics<\/td>\nPrompts, completions, tokens, model metadata, cost<\/td>\nNested traces, tool calls, retrieval context, evaluations, user feedback, cost attribution<\/td>\n<\/tr>\n
Failure visibility<\/td>\nStrong for exceptions and service degradation<\/td>\nStrong for model input\/output inspection<\/td>\nStrong for semantic failures, wrong tool use, retrieval misses, loops, and unsupported answers<\/td>\n<\/tr>\n
Best use<\/td>\nOperating distributed services<\/td>\nImproving LLM prompts and outputs<\/td>\nOperating production AI agents with multiple steps and dependencies<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

The Three Layers That Make Agents Operable\u00a0<\/strong><\/h2>\n

The rest of this guide follows the same operating loop: first capture the agent\u2019s execution path, then attach quality signals to that path, and finally use the combined evidence to debug, test, and govern future changes.<\/span>\u00a0<\/span><\/p>\n

Agent observability is not one thing. It’s three layers that feed each other in a loop and skipping any one of them breaks the other two.<\/em><\/p>\n

\"\"<\/p>\n

Tracing gives you the structure<\/strong> – a causal record of every step. Without it, you have nothing to evaluate and nothing to debug.<\/span>\u00a0<\/span><\/p>\n

Evaluation gives you judgement<\/strong> – a score attached to that structure, so “is it good?” becomes a number you can chart, alert on and gate deploys with.<\/span>\u00a0<\/span><\/p>\n

Debugging closes the loop<\/strong> – you find the broken step, turn that trace into a test case, fix it, and prove the fix with an experiment.<\/span>\u00a0<\/span><\/p>\n

The examples use Langfuse as an illustrative observability platform. Product capabilities, SDK APIs, licensing, hosting options, and integrations can change, so verify them against the version you deploy. The underlying practices, trace structure, evaluation, and evidence-led debugging also apply to other observability platforms<\/a> and OpenTelemetry-based architectures, though instrumentation details differ.<\/span>\u00a0<\/span><\/p>\n

Layer 1: Tracing – Capture What Actually Happened<\/span><\/span>\u00a0<\/span><\/strong><\/h3>\n

The data model you need to <\/span>internalise<\/span><\/span>\u00a0<\/span><\/h4>\n

Everything else depends on getting this right.<\/span><\/span>\u00a0<\/span><\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n
Concept<\/th>\nWhat it represents<\/th>\nExample<\/th>\n<\/tr>\n<\/thead>\n
Trace<\/td>\nOne end-to-end request<\/td>\nA user message and the agent’s full response<\/td>\n<\/tr>\n
Observation<\/td>\nOne step inside a trace<\/td>\nA single LLM call, tool call or retrieval<\/td>\n<\/tr>\n
Span<\/td>\nA unit of work with duration<\/td>\nplan_next_action, execute_tool<\/td>\n<\/tr>\n
Generation<\/td>\nA model call specifically<\/td>\nCaptures prompt, completion, model, tokens, cost<\/td>\n<\/tr>\n
Session<\/td>\nTraces grouped into a conversation<\/td>\nA 12-turn support chat<\/td>\n<\/tr>\n
User<\/td>\nTraces attributed to a person<\/td>\nEverything user u_8812 did this week<\/td>\n<\/tr>\n
Score<\/td>\nA quality judgement<\/td>\ngroundedness: 0.91 on a trace or a single step<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Observations nest, That nesting is the whole point, <\/span>it’s<\/span> what lets you see that the bad summary was <\/span><\/span>caused by<\/span><\/span> the empty tool result, rather than just noticing both happened.<\/span><\/span>\u00a0<\/span><\/p>\n

Why <\/span>OpenTelemetry<\/span> matters here<\/span><\/span>\u00a0<\/span><\/h4>\n

OpenTelemetry matters because it gives teams a common instrumentation path. Langfuse can receive OpenTelemetry traces through its OTLP endpoint, and its documentation recommends using the Langfuse SDK for Python or JavaScript\/TypeScript when available because the SDK handles Langfuse-specific attributes, propagation, media, filtering, and export. This makes it possible to route agent telemetry through familiar observability infrastructure while still using an AI-focused platform for trace inspection, scoring and debugging.<\/span>\u00a0<\/span><\/p>\n

Architecture flow:<\/span><\/b> user request \u2192 agent runtime \u2192 LLM, tool, retrieval, and guardrail spans \u2192 OpenTelemetry SDK or Langfuse SDK \u2192 optional OpenTelemetry collector \u2192 Langfuse for AI-specific trace analysis and evaluation \u2192 existing APM for infrastructure correlation \u2192 dataset and CI regression workflow for future changes. <\/span>Langfuse documents OTLP ingestion for OpenTelemetry traces.Langfuse datasets support test cases built from inputs and expected outputs.The Langfuse experiment GitHub Action can run experiments in CI and optionally fail a job when regressions are detected.<\/span>\u00a0<\/span><\/p>\n

This is the difference between observability that your DevOps team<\/a> adopts and observability that lives on one engineer’s laptop.<\/span><\/span>\u00a0<\/span><\/p>\n

Instrumenting: three levels of effort<\/span><\/span><\/h4>\n
\n

Level 1 \u2014 drop-in wrapper.<\/strong> Change one import, get traces:<\/p>\n

# Before\r\n\r\nfrom openai import OpenAI\r\n\r\n# After\r\n\r\nfrom langfuse.openai import openai\r\n\r\ncompletion = openai.chat.completions.create(\r\n    name=\"intent-classification\",\r\n    model=\"gpt-4o\",\r\n    messages=[{\"role\": \"user\", \"content\": user_input}],\r\n    metadata={\"tenant\": \"acme-corp\"},\r\n)<\/code><\/pre>\n
Level 2 – framework callback.<\/strong> For LangChain, LangGraph, CrewAI and similar, attach the handler and the framework’s internal structure becomes your trace structure:<\/p>\n
from langfuse.langchain import CallbackHandler\r\n\r\nlangfuse_handler = CallbackHandler()\r\n\r\nresponse = agent.invoke(\r\n    {\"messages\": [{\"role\": \"user\", \"content\": \"Where is my refund?\"}]},\r\n    config={\"callbacks\": [langfuse_handler]},\r\n)<\/code><\/pre>\n
Level 3 – manual instrumentation.<\/strong> This is where you earn your money. Custom agent loops, business logic, non-LLM steps that still matter:<\/p>\n
from langfuse import get_client\r\n\r\nlangfuse = get_client()\r\n\r\nwith langfuse.start_as_current_observation(\r\n    as_type=\"span\", name=\"refund-agent-run\"\r\n) as root:\r\n    root.update(\r\n        input={\"query\": user_query},\r\n        metadata={\"agent_version\": \"2.4.1\", \"tenant\": tenant_id},\r\n    )\r\n\r\n    # Planning step\r\n    with langfuse.start_as_current_observation(\r\n        as_type=\"generation\", name=\"plan\", model=\"gpt-4o\"\r\n    ) as gen:\r\n        plan = call_model(planning_prompt)\r\n        gen.update(output=plan)\r\n\r\n    # Tool execution \u2014 trace the tool, not just the model\r\n    for step in plan.steps:\r\n        with langfuse.start_as_current_observation(\r\n            as_type=\"tool\", name=f\"tool:{step.tool_name}\"\r\n        ) as tool_span:\r\n            tool_span.update(input=step.arguments)\r\n            result = execute_tool(step)\r\n            tool_span.update(\r\n                output=result,\r\n                metadata={\"empty_result\": len(result) == 0},\r\n            )\r\n\r\n    root.update(output=final_answer)\r\n\r\n\r\n# Required in short-lived processes (scripts, serverless, CI)\r\nlangfuse.flush()<\/code><\/pre>\n<\/div>\n
That <\/span><\/span>empty_result<\/span><\/span> flag is a thirty-second addition that turns an invisible failure into a filterable one. You can now search every trace where a tool came back <\/span>empty<\/span> and the agent answered anyway.<\/span><\/span><\/p>\n
What a good trace looks like<\/span><\/span><\/h4>\n
Bad traces are worse than no traces, because they create the illusion of visibility. Five rules:<\/span>\u00a0<\/span><\/p>\n
    \n
  1. Name spans semantically, not structurally. <\/span>validate_refund_eligibility<\/span>, not <\/span>step_3<\/span>. Six months from now you’ll be grateful.<\/span>\u00a0<\/span><\/li>\n
  2. Capture inputs and outputs at every step, not just at the boundary. The whole value is in the intermediate state.<\/span>\u00a0<\/span><\/li>\n
  3. Set <\/span>session_id<\/span> and <\/span>user_id<\/span> from day one. Retrofitting session grouping onto a live system is miserable.<\/span>\u00a0<\/span><\/li>\n
  4. Put business context in metadata \u2014 tenant, environment, agent version, feature flag, prompt version. These become your filter dimensions when you’re triaging.<\/span>\u00a0<\/span><\/li>\n
  5. Don’t trace everything. HTTP client spans and database queries from unrelated libraries will drown your agent trace in noise. Instrument deliberately.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n
    The agent graph<\/span><\/span><\/h4>\n
    When a trace contains typed observations beyond plain spans, some AI observability platforms can visualize the execution as an agent graph: nodes for steps and edges for control flow. Verify the exact graph capabilities, supported frameworks, and mode names against the platform version you deploy before making product-specific claims in the published blog.Langfuse states that agents can be represented as graphs and that traces can include LLM and non-LLM calls such as retrieval and API calls.<\/span>\u00a0<\/span><\/p>\n