part = 5 / 5

When one agent becomes forty

Everything so far assumed the classic shape: a user asks, a mostly-sequential pipeline answers, repeat. Multi-agent systems break the shape in four independent ways — and it's worth being precise about which layer each break lands on, because two of them stress OTel, and all four stress the waterfall.

The four breaks

Does this stress the OTel model?

Less than you'd guess, in an instructive pattern: OTel handles the shapes it was born for — concurrency and distribution — and strains exactly where agents stop resembling RPC.

Fan-out: no stress at all. Twelve parallel children of one parent is a tree; OTel was designed by people whose services fanned out to hundreds of backends. The wire format, context propagation, and Phoenix's schema absorb it without comment. (One local caveat: Phoenix's ancestor-propagating cumulative token counts, Part 1, make each inserted span cost writes proportional to its depth — fine for wide-shallow swarms, worth watching for deep chains.)

Fan-in: real stress. One parent_span_id means the merge node can name only one of its twelve inputs as parent; the other eleven relationships need span links, which instrumentation rarely emits and UIs — ours included — don't render. The lineage isn't unrepresentable, it's just uninstrumented and invisible, which for a debugging tool is the same thing. This is the sharpest gap: the question a multi-agent debugger asks constantly — which subagent's output poisoned the final answer? — is precisely a fan-in lineage question.

Long horizons: stress, but on a different joint than people think. Nothing in OTel caps a trace's duration; what breaks is the experience, because spans surface only on completion (Part 2's scrubber, now hours long) and because a days-long trace makes "trace = the thing you look at" absurd. Practical systems chunk the work into many traces and group them — which quietly promotes the session from "nice for chatbots" to the load-bearing container. Phoenix's session is ready for this structurally (it's just a grouping key) and unready semantically: it means "conversation," carries a chat-shaped UI, and can't be annotated from that UI.

Handoffs: stress OTel shares with everyone. When work crosses a queue or an A2A boundary, either the context propagates (same ever-growing trace), or it doesn't (orphaned trace with no visible relation). The correct OTel answer is "new trace, linked to the producer span" — links again. No vendor's proprietary model solves this either; they just fail with different error messages. The difference is that application-first products can at least attach both traces to a server-side task object; we need the grouping-key equivalent.

Does it stress the interface? Watch it happen

The waterfall's implicit contract: reading top-to-bottom approximates reading time-order, and height is proportional to work. Both clauses fail under fan-out — parallel bars overlap in time while stacking in space, so the page grows linearly in spans while the information grows not at all. Drag the slider:

Fan-out stress test
Same trace, three projections — the series' theme one last time. The waterfall stays correct and becomes unreadable; lanes keep time legible by spending a row per agent; collapsed groups trade the timeline for an answer to "what happened," surfacing the one failed branch instead of hiding it at scroll position 61. A real trace view needs all three, plus rendered span links for the fan-in edges.

And the waterfall is only the most visible casualty. The sessions page renders each trace as a chat turn via its root span's I/O — a subagent's trace has neither a user message nor a chat-shaped root, so agent swarms produce sessions full of mute "turns." The root-span-as-face pattern breaks when continuation and handoff traces have entry points that aren't requests. Annotation targeting gets genuinely ambiguous: the thing you want to judge — "did the research task succeed?" — may span four traces and two sessions, an object no current level represents.

The practices, sorted

✓ scales — keep leaning on it
  • Immutable spans, append-only ingest. Concurrency-proof by construction; swarms make this bet look better, not worse.
  • Grouping keys as containers. The session mechanism (group by attribute) is exactly how task-, job-, and agent-level containers should work too. The mechanism generalizes; only the chat semantics don't.
  • Annotations as side tables. Target-agnostic by design — already pointed at four levels, trivially pointable at new ones.
  • OTel context propagation. The only credible story for traces that cross real process boundaries, which multi-agent systems do constantly.
△ strains — fix before it calcifies
  • Waterfall as the default projection. Keep it for sequential traces; detect parallelism and offer lanes/groups.
  • Root span as the trace's face. Resolve once, orphan-aware, with an honest fallback when there is no face.
  • Session = conversation. Generalize to "thread of related traces," of which chat is one flavor.
  • Depth-proportional cumulative rollups. Fine today; profile before agents get deeper.
+ new — build for this world
  • Span links, end to end. Emit them in OpenInference instrumentation, store them queryably, render them. Fan-in lineage is the killer feature nobody ships well yet.
  • Agent identity as a dimension. Filter, color, and group by which agent — aligned with OTel's emerging gen_ai.* agent conventions rather than against them.
  • Aggregate projections. "12 × research_subtask, 1 failed" as a first-class rendering, not a UX afterthought.
  • Live partial traces. The orphan forest, labeled as in-progress, instead of pretending nothing is happening for 40 minutes.
✗ abandon — quietly, now
  • Turn = trace. True for chatbots, false for everything arriving next; every feature built on it inherits the falsehood.
  • Annotate-the-root-as-proxy. Already wrong at session level (Part 1); indefensible when "the task" spans traces.
  • Trace as the unit of user attention. The trace is becoming an implementation detail between span and thread. Design attention around the path, not the middle level.
  • Waiting for completeness. No trace was ever knowably complete (Part 2); long-horizon agents just make the fiction expensive.
the series in one paragraph Phoenix stores exactly one honest object — the immutable span — and derives everything users care about. That architecture is the right side of the fork to be on: it bought the whole OTel ecosystem, and Part 3 showed the rich mental models competitors sell can mostly be materialized at read time on top of it. What can't be derived is interface honesty: a UI that names its levels, degrades visibly at the seams (missing roots, partial traces, swallowed conflicts), annotates what it says it annotates, and projects parallel work as parallel. The data model is further ahead than the interface — the schema already believes in sessions, traces, and spans equally. The screen should catch up before the swarms arrive.