part = 5 / 5

What may the judge read, and what may it mark?

The v1 spec gives each project evaluator one target type, and that single knob plays three roles at once: it decides when the evaluator fires, what it reads, and where its verdict lands. Design wants to split the reading from the marking. Backend warns that the split invites every evaluator to drag whole trees through the system. Both are right, and the interesting work is in the rules that let them both stay right.

Three roles, one knob

Give an evaluator e three scopes: T(e), the trigger — which event enqueues work; R(e), the read set — what data the judge sees; and W(e), the write set — which artifacts receive annotations. The spec'd attachment has one field, target ∈ {span, trace, session}, and sets T = R = W = target. For the founding use case — judge a sampled final-answer LLM span for hallucination — the collapse is harmless: the span triggers, the span is read, the span is marked.

It stops being harmless the moment an evaluation is relational — when the property being judged belongs to one artifact but is only visible in the company of others. The design team's motivating example is the cleanest one in the domain, so let's build it.

The redundant tool fire

An agent handles "Where is my order?": it plans, calls lookup_order for A-1234, gets a good result — and then calls lookup_order again with identical arguments before responding. A wasted call: latency and money spent re-learning a fact. Now try to catch it with an evaluator, and watch the scopes pull apart:

Judged alone, the second call is flawless. Well-formed arguments, sensible tool choice, successful result. Redundancy is not a property of the span; it's a property of the span's relationship to a sibling. Any span-in-isolation judge answers "fine" — confidently, repeatably, wrongly. So R(e) must include at least the sibling tool calls, and honestly the whole trace.
The useful verdicts live at two altitudes. For monitoring, you want contains_redundant_tool_call on the trace — filterable in trace tables, chartable over time, alertable. For debugging, you want redundant_call on the offending span — so the flag is sitting there when someone opens the waterfall. One is "how often does this happen"; the other is "which call, exactly." A verdict filed at only one altitude answers only one of those questions, and users ask both. So W(e) = {trace, span}.

One read, two writes: R = trace, W = {trace, span} — and no assignment of the single target knob expresses it. Set target=span and the judge is blind. Set target=trace and the verdict can't point. Run it yourself:

The scope machine Pick what the judge reads, when it fires, where it writes — then run

read scope R

trigger T

write scopes W span trace

configure the evaluator and run it — the tree shows what gets read (outlined) and what gets marked (badges)

Three things to try. (1) Read scope "target span only": every verdict is a confident pass — the defect is invisible by construction. (2) "+ sibling tools" with per-arrival triggering: the judge fires at the first tool call before the second exists, files a pass, then flags at the second — leaving a stale pass on span 9f03 and double the reads. (3) Whole trace, trace-idle, both write scopes: one evaluation, full context, verdicts at both altitudes — and the fewest spans read of any configuration that works.

This example is not exotic

It's tempting to file the refire under "advanced multi-agent stuff" and ship the single-scope model. But look at the spec's own canonical examples with R/W eyes. "Did the LLM call the right tool" is listed as a span-level eval — yet right is defined by the conversation the tool call serves; a tool-choice judge reading only the tool span is grading penmanship, not correctness. The hallucination judge wants the sibling retriever span's documents, which is why the document-annotation table exists. Session evals ("did the agent stay coherent") are relational across traces by definition. The pattern generalizes: evaluation scope ≥ annotation scope, usually strictly — because judgment needs context, and context is exactly the thing a lone span doesn't carry (companion series, Part 1: the span is the only thing with content; every larger unit is an assembly). The single-knob model doesn't remove the need; it forces a choice between blind judges and misfiled verdicts. Design's two-scope proposal is just the honest shape of what nontrivial evals already are.

Now backend's turn: the clobbering problem, priced

The objection is not aesthetic. If the attachment row lets any evaluator declare "read the enclosing trace," every evaluator author will — context helps every judge a little, and the cost lands on someone else's infrastructure. The generous option becomes the default, and small targets start roping their surrounding trees into every evaluation. Price the naive version: a trace of n spans, k span-targeted evaluators with trace-sized reads, triggered per arrival, is k·n evaluations reading ~n spans each — O(k·n²) span-reads per trace, where the honest configuration above cost k·n. For a 50-span agent trace and five such evaluators, that's 12,500 span-reads instead of 250 — and if the judges are LLMs, "reads" are prompt tokens, so the amplification is a bill, not just I/O. Add the churn: each early evaluation was judged on a partial tree, so later arrivals invalidate earlier verdicts, which the identity key silently overwrites — write amplification on top of read amplification, with a side of stale-verdict windows. And there's a second, quieter clobbering: if each run's audit record snapshots the context it read (and Part 4 argued run records are v1-blocking), storing trees per-verdict clobbers the reviewer — the run history becomes bigger than the telemetry it judges. Backend's worry, stated precisely: reads scale with R, and R is the one scope users don't see when they file a verdict.

Rules that keep both teams right

The resolution isn't to pick a winner — it's to notice that the expensive thing and the expressive thing are different axes. Five rules, each mechanical enough to encode:

1 · The trigger follows the widest read. If R(e) is trace-sized, T(e) is trace-idle — never per-arrival. This single rule converts O(k·n²) into O(k·n), guarantees the judge sees the sibling that hasn't arrived yet (the widget's case-2 failure), and makes the work queue's dedup natural: one unit per (artifact, evaluator, fingerprint). Per-arrival triggering is reserved for R = span. Readiness is already in the spec; this just binds it to R instead of W.
2 · Reads are declared extractions, not grabs. "The trace" is not a read scope; it's a fetch. The read scope should be a named, bounded extraction — "sibling tool calls: name, args, status," "root I/O plus tool summaries," "transcript, 4k-token budget, truncation marked" — the extraction-vs-mapping split Part 3 found already forming in the branch spec. Declared extractions make cost visible at authoring time (the builder can literally price them against a recent artifact), make run records small (store the extraction, not the tree), and make the whole-tree read what it should be: the explicit, expensive option, not the default the templates hand out.
3 · One read funds many writes. Multi-scope annotation is the mitigation, not the indulgence: the alternative to one trace-reading evaluator that writes trace + span verdicts is two evaluators each reading the trace. Letting a single evaluation fan its verdicts out amortizes the expensive scope over every cheap one. This does put a new demand on the output contract — a span-scope verdict must name its span, which means structured judge output with artifact references, resolution of those references against the read set, and a policy for the judge citing a span it never saw (refuse the write, log the defect). That's real work, but it's schema work, not physics.
4 · Sample and batch by the read cohort. Sample on the artifact that gets read, not the artifacts that get written — hash the trace id, and Part 4's shared-cohort property does the rest: every trace-reading evaluator co-samples the same traces, so the fetch happens once and feeds k judges. Fetch amplification drops from k to ~1 for the co-sampled set.
5 · Verdicts carry their evidence basis. A trace verdict is a judgment about a snapshot — the companion series established there is no final snapshot. So stamp it: evaluated-at-n-spans (or a frontier watermark) in the annotation's metadata, latest-wins for the visible row (Part 1's key, as designed), full history in the run store. Then a "changed verdict" is a legible event — new evidence, new judgment — rather than silent clobbering of the other kind.

Under these rules, design's scenario runs whole: the refire evaluator declares R = "sibling tool calls" (a bounded extraction), T follows R (trace-idle), W = {trace, span} with a structured output naming the offender, sampling hashes the trace, and the run record stores the extraction it judged. Backend's meter reads k·n, once, per quiet trace. Nobody had to lose.

the series in one paragraph Phoenix's second write path is one shape with one identity key, and that economy is the whole story: humans, scripts, experiments, and (soon) online evaluators all write the same row, distinguished by stamps and disciplined by an upsert. The evaluator grew from a notebook function into a stored, versioned, prompt-hub-backed machine bound to its data through a small mapping layer — and the online frontier is not a new evaluator but a new binding, whose genuinely open question is scope: judges need more context than their targets contain, verdicts need more altitudes than one, and reads are the only resource in the system that scales quadratically if nobody prices them. Split the read scope from the write scopes, bind the trigger to the read, make context a declared extraction, and the annotation tables — which have been ready for machines since the day the identifier column shipped — will hold whatever the judges decide.