part = 2 / 5

The price of speaking OTel

OpenTelemetry is a wire protocol, a data model, and an ecosystem of instrumentation — in roughly that order of rigor. Phoenix's decision to be a compliant OTLP collector is the most consequential architectural choice in the product. Here's the whole trade, both sides, priced honestly.

What OTel actually specifies

Strip the ecosystem away and OTel tracing specifies remarkably little, which is its genius and its limitation. A span is a protobuf message with: a 16-byte trace_id, an 8-byte span_id, an optional parent_span_id, a name, start and end timestamps, a status, a list of typed key–value attributes (strings, numbers, booleans, and arrays thereof — no nesting), a list of timestamped events, and a list of links to other spans. Spans are exported after they end, in batches, over OTLP. Alongside the data model sits context propagation (the W3C traceparent header) — the mechanism by which a trace id survives hops across processes, queues, and services.

Note what's absent: there is no trace message. No session. No user. No conversation. No "request outcome." The protocol ships leaves and lets everyone reconstruct trees. Anything richer lives in semantic conventions — agreed-upon attribute key names — which is where OpenInference (Phoenix's convention set) and OTel's own emerging gen_ai.* conventions operate. Conventions are soft law: universally violable, no compiler, no referee.

What compliance buys

Instrumentation for free, forever. Anyone's OTel instrumentation — OpenInference's, OpenLLMetry's, a framework's built-in tracer, or six lines of manual SDK code — can point at Phoenix by changing an endpoint URL. No proprietary SDK, no client library treadmill, no "supported frameworks" list that has to chase the ecosystem.
The collector ecosystem. Sampling, batching, redaction, routing, and fan-out (send the same spans to Phoenix and Datadog) are solved upstream by the OTel Collector. Phoenix never has to build a data-plane.
Cross-service traces by default. If the surrounding microservices speak traceparent, an LLM call deep in service C lands in the same trace as the HTTP request that entered service A. Proprietary AI-observability SDKs mostly cannot see outside their own process.
A credible no-lock-in story. The user's instrumentation investment is portable away from Phoenix — which, in practice, is a reason to choose Phoenix. This matters double for an open-source product; "our SDK, our format" is a much harder sell.

What compliance guarantees about our data shape

Because every producer speaks the same protocol, Phoenix gets to assume — structurally, not by convention — that every unit of telemetry it will ever ingest:

has a globally unique identity and a trace membership (span_id, trace_id);
has at most one parent, so every trace is a forest, and (if nothing is lost) a tree;
has a bounded time interval — spans always end;
carries only flat, typed attributes — every payload is queryable with one indexing strategy;
is immutable — once received, a span never changes, so ingestion is append-only, caching is trivial, and an annotation store can safely hang off the side.

These are real engineering guarantees. Append-only immutable rows with tree structure is about the most storage-friendly, cache-friendly shape telemetry can take, and Phoenix's schema (Part 1) leans on every one of these properties.

What compliance costs

The sacrifices are the mirror image of the guarantees. Five of them matter.

1. Nothing exists until it's over

Spans export on end. A 40-second agent run is invisible for 40 seconds, then materializes all at once — leaves first, root last. While it runs, Phoenix sees a partial trace with orphans; there is no moment at which any trace is knowably "complete," because a straggler span can always arrive. Scrub the timeline:

The observer's timeline Dashed = happening; solid = visible to Phoenix (span has ended and exported)

This is why "why is my trace empty/weird?" is a support FAQ for every OTel-based tool: mid-flight, the truthful rendering is an orphan forest. It's also why streaming LLM output can't be watched live through spans alone — the span carrying the completion doesn't exist until the last token does. (OTel's answer, real-time-ish span events or logs, is bolted on, not native.)

2. There is no place to put trace-level facts

"This request came from user X, cost $0.04, and failed" is a sentence about a trace. The protocol offers nowhere to say it. Every OTel-based product independently reinvents the same workaround — denormalize onto the root span and treat it as the trace's spokesperson — and inherits the same failure modes: the root arrives last, might not arrive at all, and in multi-entry-point systems (Part 5) might not be unique. Part 1 showed Phoenix's version of this: turns, trace lists, and the sessions page all wear the root span as a mask.

3. Sessions are somebody else's problem

Conversation identity requires the application to thread a session.id through every process that emits spans, stamp it reliably, and never typo it. OTel's context mechanism (baggage) can carry it, but nothing enforces or validates it. First-writer-wins (Part 1) is Phoenix coping with the fact that the protocol gives it no authoritative source.

4. Trees only — no joins, no reruns, no continuations

One parent_span_id means the shape of computation is a tree. Real agent systems produce DAGs: two parallel subagents whose results are merged; a workflow resumed from a checkpoint; a message that triggers work in another agent's session. OTel's official escape hatch, span links, is honest but second-class: links carry no semantics, most instrumentation never sets them, and most UIs (Phoenix's included) don't render them. So the model quietly forbids the most interesting structures modern agents produce. This is the load-bearing sacrifice, and Part 5 is entirely about it.

5. Flat attributes make chat payloads a second language

The natural unit of LLM observability is a structured message list. OTel attributes are flat. So OpenInference encodes structure into key paths:

What your code has vs. what the wire carries

Workable — Phoenix stores the bag as JSON and re-nests dotted keys — but it means size-capped attribute values silently truncate payloads, there's no schema to validate against, and every consumer re-implements the re-nesting. The message list, the single most important payload in the domain, travels steerage.

So is a Phoenix "trace" a conceptual container?

A real container has four things: identity, contents, a boundary, and a lifecycle. An OTel trace has identity only — a 16-byte number that spans claim membership in. Nothing is ever "in" it on the wire; it has no boundary (spans with that id may arrive forever) and no lifecycle (it is never opened, closed, or completed). Phoenix materializes a traces row, which adds a second thing — a time range — but as Part 1 showed, still no contents.

So the precise answer is: a trace is a label that the interface dresses up as a container. Users, reasonably, expect container semantics — "show me the request: its input, its output, whether it succeeded" — because that's what the word "trace" gestures at and what competitor products literally provide as an object. The gap between label and container is where most of the interface clumsiness this series catalogs actually lives: empty-faced traces when roots are missing, annotation aimed at spokesspans, completeness that can't be promised.

the trade, in one sentence OTel compliance buys Phoenix the entire instrumentation ecosystem and an append-only, guaranteed-shape storage model, and the price is that the two objects users most want to reason about — the request and the conversation — exist only as derived projections that the interface must fake convincingly at read time.

That price is payable. The wire format does not have to be the mental model — a point the competitors make for us, next.