part = 5 / 5
When one agent becomes forty
Everything so far assumed the classic shape: a user asks, a mostly-sequential pipeline answers, repeat. Multi-agent systems break the shape in four independent ways — and it's worth being precise about which layer each break lands on, because two of them stress OTel, and all four stress the waterfall.
The four breaks
- Fan-out: an orchestrator spawns N parallel subagents. Width replaces depth.
- Fan-in: their results get merged, judged, deduplicated. The computation is a DAG, not a tree.
- Long horizons: tasks run for hours or days, pause for humans, resume from checkpoints. "The request" stops being a natural unit.
- Handoffs: work crosses queues, schedulers, and agent-to-agent protocols, where trace context goes to die.
Does this stress the OTel model?
Less than you'd guess, in an instructive pattern: OTel handles the shapes it was born for — concurrency and distribution — and strains exactly where agents stop resembling RPC.
Fan-out: no stress at all. Twelve parallel children of one parent is a tree; OTel was designed by people whose services fanned out to hundreds of backends. The wire format, context propagation, and Phoenix's schema absorb it without comment. (One local caveat: Phoenix's ancestor-propagating cumulative token counts, Part 1, make each inserted span cost writes proportional to its depth — fine for wide-shallow swarms, worth watching for deep chains.)
Fan-in: real stress. One parent_span_id means the merge
node can name only one of its twelve inputs as parent; the other eleven relationships
need span links, which instrumentation rarely emits and UIs — ours
included — don't render. The lineage isn't unrepresentable, it's just uninstrumented
and invisible, which for a debugging tool is the same thing. This is the sharpest gap:
the question a multi-agent debugger asks constantly — which subagent's output
poisoned the final answer? — is precisely a fan-in lineage question.
Long horizons: stress, but on a different joint than people think. Nothing in OTel caps a trace's duration; what breaks is the experience, because spans surface only on completion (Part 2's scrubber, now hours long) and because a days-long trace makes "trace = the thing you look at" absurd. Practical systems chunk the work into many traces and group them — which quietly promotes the session from "nice for chatbots" to the load-bearing container. Phoenix's session is ready for this structurally (it's just a grouping key) and unready semantically: it means "conversation," carries a chat-shaped UI, and can't be annotated from that UI.
Handoffs: stress OTel shares with everyone. When work crosses a queue or an A2A boundary, either the context propagates (same ever-growing trace), or it doesn't (orphaned trace with no visible relation). The correct OTel answer is "new trace, linked to the producer span" — links again. No vendor's proprietary model solves this either; they just fail with different error messages. The difference is that application-first products can at least attach both traces to a server-side task object; we need the grouping-key equivalent.
Does it stress the interface? Watch it happen
The waterfall's implicit contract: reading top-to-bottom approximates reading time-order, and height is proportional to work. Both clauses fail under fan-out — parallel bars overlap in time while stacking in space, so the page grows linearly in spans while the information grows not at all. Drag the slider:
And the waterfall is only the most visible casualty. The sessions page renders each trace as a chat turn via its root span's I/O — a subagent's trace has neither a user message nor a chat-shaped root, so agent swarms produce sessions full of mute "turns." The root-span-as-face pattern breaks when continuation and handoff traces have entry points that aren't requests. Annotation targeting gets genuinely ambiguous: the thing you want to judge — "did the research task succeed?" — may span four traces and two sessions, an object no current level represents.
The practices, sorted
- Immutable spans, append-only ingest. Concurrency-proof by construction; swarms make this bet look better, not worse.
- Grouping keys as containers. The session mechanism (group by attribute) is exactly how task-, job-, and agent-level containers should work too. The mechanism generalizes; only the chat semantics don't.
- Annotations as side tables. Target-agnostic by design — already pointed at four levels, trivially pointable at new ones.
- OTel context propagation. The only credible story for traces that cross real process boundaries, which multi-agent systems do constantly.
- Waterfall as the default projection. Keep it for sequential traces; detect parallelism and offer lanes/groups.
- Root span as the trace's face. Resolve once, orphan-aware, with an honest fallback when there is no face.
- Session = conversation. Generalize to "thread of related traces," of which chat is one flavor.
- Depth-proportional cumulative rollups. Fine today; profile before agents get deeper.
- Span links, end to end. Emit them in OpenInference instrumentation, store them queryably, render them. Fan-in lineage is the killer feature nobody ships well yet.
- Agent identity as a dimension. Filter, color, and group by which agent — aligned with OTel's emerging gen_ai.* agent conventions rather than against them.
- Aggregate projections. "12 × research_subtask, 1 failed" as a first-class rendering, not a UX afterthought.
- Live partial traces. The orphan forest, labeled as in-progress, instead of pretending nothing is happening for 40 minutes.
- Turn = trace. True for chatbots, false for everything arriving next; every feature built on it inherits the falsehood.
- Annotate-the-root-as-proxy. Already wrong at session level (Part 1); indefensible when "the task" spans traces.
- Trace as the unit of user attention. The trace is becoming an implementation detail between span and thread. Design attention around the path, not the middle level.
- Waiting for completeness. No trace was ever knowably complete (Part 2); long-horizon agents just make the fiction expensive.