AI Observability Sessions, Traces, and Spans in LLM Systems

An LLM system doesn't fail the way a normal app fails. It doesn't throw a 500 error when it gets something wrong, it just answers confidently and moves on. That's the whole problem observability for these systems is trying to solve, and sessions, traces, and spans are the three layers of recording that make it solvable. Each one captures a different slice of what happened, at a different zoom level, and together they're what let an engineering team go from "the output was wrong" to "here's exactly which step broke."

Old-school application monitoring was built on a bet: same input, same code path, same output. Errors showed up as HTTP status codes. A database call either succeeded or threw an exception. That whole model assumes the system is deterministic, and it mostly was.

LLMs break every part of that bet. A wrong answer still comes back as a clean 200. Nothing in standard telemetry looks at the actual content of the response, so a hallucinated order status and a correct one look identical to a monitoring dashboard. Send the same prompt twice and the model might call a different tool the second time, or skip a step it took before, so you can't even reproduce the failure by re-sending the request. And a single user request usually isn't one operation anyway. The model reads context, decides to call a tool, waits on the result, rewrites its answer. That's four or five separate operations stacked inside what looks, from the outside, like one API call.

Charity Majors put this plainly back in 2023: LLMs are black boxes that produce nondeterministic output and can't be debugged with the tools built for deterministic code. The only real option left is to record what the model actually did in production, and go study that recording after the fact. That's not a metaphor, it's the design principle behind the whole tracing stack described below.

And the failure modes themselves are new, too. A model can write something fluent, grammatically perfect, and completely wrong. Inference latency can spike not because a CPU is overloaded but because the KV cache got evicted. GPU memory fragmentation can cause requests to silently drop while every standard health check stays green, a pattern documented in a 2026 observability paper by Red Hat and Boston University (arXiv:2604.26152). None of this shows up in a log line that just says "request completed in 340ms." Logs and metrics still matter, but they're not enough on their own. Structured tracing is what makes individual LLM operations actually inspectable.

What sessions, traces, spans, and their siblings actually capture

The hierarchy is a tree. Not a list, not a flat log stream, a tree, where each level nests inside the one above it. That nesting is what makes root-cause analysis possible in the first place, because it preserves the relationship between a symptom at the top and a cause several layers down.

Session is the widest layer. It captures a full multi-turn interaction, like an entire chatbot conversation rather than a single question-and-answer pair. It holds context that carries across multiple user turns and, often, across multiple traces. If you're asking "did this user have a good experience across ten messages" or "how much did this conversation cost end to end," the session is the right unit to look at.

Trace sits one level down. A trace represents everything that happened to produce one single response, from the initial query to the final output. If a user got a bad answer, the entire explanation for that one bad answer lives inside one trace. This idea isn't new. It's borrowed straight from distributed tracing in backend engineering, same tree-of-spans structure. What's different is what fills the tree.

Span is the actual unit of work. One operation, timestamped, with its inputs and outputs recorded. A span carries input tokens, output tokens, latency, the model name, a status code, an error type if there was one. Parent-child links between spans preserve the full execution path across agents, tools, and retrievers, so you can walk backward from an outcome to its cause.

Underneath "span" sit a few subtypes that matter specifically for LLM systems:

• Generation span: logs one LLM call, the input messages, model parameters, the output, and the cost.
• Retrieval span: tracks a RAG query pulling context out of a knowledge base, the query itself, what entries came back, relevance scores, and how fresh that data was.
• Tool call span: records an external API call or tool execution, the tool's name, its arguments, the raw output, how long it took, retry count, and any error state.
• Event: not a span on its own, but a timestamped marker attached to one, flagging a significant moment or state change inside it.

None of this is reinventing distributed tracing. The tree structure is standard. What's new is that the nodes in the tree are model calls and tool calls instead of HTTP handlers and database queries.

Take a concrete example, a customer support agent handling "What is the status of my order #4521?" (as walked through in a futureagi.com guide on agent tracing). The root span is invoke_agent triage_agent, running 4.2 seconds total. Inside it, a chat gpt-5 span at 600ms decides to route the request to an order-lookup agent. That spawns a child span, invoke_agent order_lookup_agent, 2.8 seconds, which itself contains a tool span (execute_tool order_api, 1.9 seconds, a GET request to /orders/4521) and a second LLM span (900ms) that formats the returned order data. Then a final child, invoke_agent response_agent, 800ms, with one more LLM span composing the reply the user actually sees.

If that final reply is wrong, the trace tree tells you exactly where to look. Did the response agent misread the formatted data? Did the order API return something stale? Did triage route to the wrong agent in the first place? Without the tree, you just have a bad answer and no way to trace it backward.

How the four span types map to the four ways agents fail in production

Tool-call errors are probably the most common failure in production agent systems. The agent decides to call a function, but the parameters it generates are malformed, or the tool returns an error the agent doesn't know how to handle. Sometimes it retries with the same bad parameters. Sometimes it just ignores the failure and hallucinates an answer instead. A tool span needs to record the tool name, arguments, raw output, duration, retry count, and error state, because without that detail, malformed arguments look exactly like normal traffic in aggregate logs.

Silent failures are the hardest category, and arguably the scariest. Agent A hands Agent B incomplete or irrelevant context. Agent B produces a confident, well-written, wrong response. No exception fires anywhere. The monitoring dashboard stays green the entire time. What actually surfaces this kind of failure is a state transition span, one that records the working memory of the system across steps, so context loss and summarization drift become visible instead of invisible.

Hallucination compounding through multi-step chains is its own category. A fabrication that shows up in step two of a five-step chain corrupts everything downstream of it, and standard logging only captures the final output, not the moment the fabrication was introduced. Reasoning spans, the intermediate chain-of-thought and plan-act-observe transitions, are what surface plan drift and wrong-branch selection that a single LLM span can't show on its own. Scoring each LLM span for faithfulness against the retriever span that fed it is a workable pattern for catching this.

Latency compounding is more mechanical but no less damaging. Every agent added to a chain adds its own delay, and one slow step anywhere in the chain can push total response time past what a user will tolerate. Span-level timing, exact start and end timestamps per span, is the only way to find where the bottleneck actually lives. Aggregate request latency just tells you the total was slow, not why.

Memory operation spans round out the picture as a fourth pillar: reads and writes to long-term storage, capturing details such as the query, what was retrieved, and the relevance and freshness of that data. These are what expose stale reads, retrieval of the wrong entity, and memory leaking between users, none of which shows up in an aggregate metric.

The structural point underneath all four: each span type is built for one failure mode. The place where the failure becomes visible is also the place where you go to fix it.

Why multi-agent systems make each of these failure modes dramatically harder to catch

In a multi-agent system, a wrong final answer arrives with no built-in indication of which agent failed, which tool call returned garbage, or where the reasoning chain came apart. Agents hand tasks off to each other, share state, call outside APIs, and each makes its own decisions along the way. When one of them hallucinates or a tool call times out, that error doesn't announce itself, it just quietly propagates through everything downstream.

Traditional logging hands you fragments. A log line here, a log line there, none of them connected. What actually gives the full picture is a connected span tree that preserves parent-child links across every agent handoff.

The industry data backs up how widespread this problem already is. A late-2025 survey of 1,340 teams by LangChain found 89% running some form of agent observability, and 71% with detailed tracing already in place, yet quality was still named the top barrier to shipping AI agents. That's worth sitting with: most teams already have tracing, and quality is still the thing holding them back. Tracing alone doesn't solve the problem, it just makes the problem visible enough to attach an evaluation layer to.

The span hierarchy for a multi-agent system typically breaks down into a root span for the full workflow execution, an agent span for each individual agent's processing, LLM spans for individual model calls, tool spans for external API calls, retriever spans for knowledge base queries, and embedding spans for embedding generation. When Agent A hands off to Agent B, the child span links back to its parent, and that link is the entire mechanism that makes cross-agent root-cause analysis possible.

This is also why standardization matters. The OpenTelemetry GenAI working group, formed in April 2024, defines specific span operation types for agent work, create_agent, invoke_agent, invoke_workflow, execute_tool, alongside others like chat, retrieval, and plan. Having a shared vocabulary for these operations is what makes it possible to stitch traces together across different tools built by different vendors.

The five-layer observability stack and where session/trace/span instrumentation sits within it

The Red Hat 2026 paper (arXiv:2604.26152) lays out a five-layer taxonomy for AI observability, running from the deepest model internals down to raw infrastructure telemetry. Worth walking through, because it places tracing in context rather than treating it as the whole solution.

Layer 1 is model internals: watching activations, attention patterns, latent representations directly. This is the highest-fidelity signal available for what a model actually "believes" when it generates an answer, but it requires white-box access to the model, which most teams calling an API simply don't have.

Layer 2 is confidence calibration. Models trained with standard reinforcement learning tend to come out overconfident, stating wrong answers with the same certainty as right ones. Work out of MIT CSAIL on a method called RLCR augments training with a Brier score term specifically to penalize miscalibrated confidence. Without something like this, an operator has no way to distinguish a confident correct answer from a confident hallucination just by looking at the model's own stated certainty.

Layer 3 is behavioral monitoring, covering external observation of chain-of-thought, action sequences, and output properties — the layer where session, trace, and span instrumentation operates. It's external observation: chain-of-thought, action sequences like tool calls, properties of the output. No internal model access needed. Layer 4, sitting just above it, covers operational and request-level telemetry such as latency, error rates, and token counts. Layer 5 is infrastructure tracing proper, GPU kernel timings, memory allocation patterns, cross-node communication in distributed inference setups, the kind of non-intrusive inference-level tracing described in the TRUFFLD approach.

Layer 3 is the most actionable layer for most engineering teams, precisely because it doesn't need white-box access and it produces the structured data that debugging, cost attribution, and evaluation all depend on. But the signals from these five layers currently live in separate systems that don't talk to each other. Connecting a model's internal confidence signal to an infrastructure-level anomaly, and turning both into one coherent picture, is what the Red Hat and Boston University paper identifies as a central open problem going into 2026. Impressive depth at each individual layer. Very little integration across them.

How OpenTelemetry GenAI conventions are standardizing what spans carry and where they go

The OpenTelemetry GenAI working group started in April 2024 with a narrow scope: tracing LLM client calls. As of mid-2026, its conventions are still marked "Development," meaning experimental, and no attribute in the spec has reached Stable status yet. Still, the shape of what it defines is already doing real work.

The spec lays out a gen_ai.* attribute namespace covering model name, provider, input and output token counts, and finish reason. It defines the four span operation types for agents mentioned earlier, plus events for streaming responses, which matter because they let latency get measured at the level of individual tokens rather than only at full request completion.

Why any of this matters practically: any backend that speaks OTLP, Jaeger, Tempo, Datadog, Honeycomb, can receive these spans without custom translation work. Framework adapters for the OpenAI Agents SDK, LangGraph, Mastra, Pydantic AI, and custom orchestrators can normalize their spans into the same schema without anyone rewriting application code. For frameworks nobody's built an adapter for yet, the OpenTelemetry SDK still provides a fallback instrumentation path.

Datadog added native support for the OTel GenAI conventions starting with version 1.37, announced December 1, 2025, automatically mapping gen_ai.* attributes into its own LLM Observability schema. A team using standard OTel instrumentation gets that integration for free, no custom mapping code required. The practical upshot: instrumenting to the OTel GenAI spec today means building on a foundation multiple backends already read. Portability here isn't a future promise, it's already working.

Gaps remain, though. There's no stable, standardized schema yet for memory operation spans or state transition spans across frameworks. The spec defines the concept, but every implementation of it remains framework-specific as of mid-2026. That instrumentation still has to be hand-built per system.

What instrumentation needs to capture for traces to be useful in practice

A trace with only timestamps and status codes isn't useful. What makes a trace actually usable is semantic richness, enough metadata attached to each span that an engineer can filter, search, and correlate across thousands of them.

Unique identifiers have to exist at every level, session ID, trace ID, span ID, generation ID. Without consistent IDs threaded through the whole tree, correlated failures across multiple turns of a conversation can't be grouped together at all, they just look like isolated incidents.

Tags matter just as much: environment, user ID, experiment ID, model version, deployment parameters. These are the dimensions a team actually needs to answer questions like "did this regression start after we upgraded the model" or "which experiment cohort is driving the cost spike."

Full request and response capture is non-negotiable. The user's query, the model's response, any error messages, all model parameters and configuration in effect at the time of the call. If a parameter change quietly degrades output quality and that parameter isn't recorded per span, there's no way to catch it after the fact. Tool call arguments and results both need capturing, not just a boolean flag saying a tool was invoked, but what it actually returned. Intermediate states and errors need to be logged too, not only the terminal state at the end of the trace.

At minimum, per a Braintrust instrumentation guide, each span type needs to record specific fields. A tool span needs tool name, arguments, raw output, duration, retry count, and error state. A reasoning span captures the intermediate chain-of-thought and plan-act-observe transitions that a single LLM span cannot show on its own. A state transition span needs working memory captured both before and after each step, plus the handoff payload passed to the next agent. A memory span needs the query sent, the entries returned, relevance scores, and how fresh that data was at read time.

Get this level of detail into every span, and a trace stops being a record of what happened and starts being a tool for figuring out why.