series = a field guide to Phoenix's judgment layer

The second write path

Phoenix ingests telemetry it is forbidden to edit. Everything Phoenix itself has to say about your system — every human verdict, every LLM-judge score, every experiment result — lands in a second, mutable write path: the annotation. This series is about that path, the machines being built to write into it, and the design questions now open about what those machines may read and what they may mark.

The companion series ended on a claim: Phoenix stores exactly one honest object, the immutable span, and derives everything else. That was a story about facts — records of what happened, produced by someone else's process, arriving over a protocol that guarantees Phoenix will never change them. But an observability tool that only restates facts is a very expensive SELECT *. The product's actual job is to help you decide whether what happened was any good — and "good" is not an attribute on any span. It has to be written down somewhere, by someone, after the fact.

So the database has a second half, with opposite physics. Compare the two write paths:

The remarkable thing — and the architectural decision this series keeps returning to — is that every kind of judgment in Phoenix converges on one row shape: a name, an optional label, an optional score, an optional explanation, a metadata bag, and a set of provenance stamps saying who wrote it and how. A human clicking thumbs-down, a pandas dataframe of eval results, an experiment's LLM judge, and the proposed online evaluators all produce this row. They differ only in their stamps — and in one quietly consequential field called identifier. Try the writers:

An evaluator, in this vocabulary, is a machine that turns path-1 rows into path-2 rows: it reads telemetry (or dataset examples) and writes judgments, at a scale and regularity no human review process reaches. Phoenix's evaluator has been migrating for a year — from a Python function running in the user's notebook, to a first-class server-side object with a stored prompt, a typed output schema, and a binding layer that maps data into it (Parts 2–3). Today that machine runs only against data at rest: datasets and experiments. The current design work — "project evaluators," online evals, backfill — is about pointing it at the live span firehose (Part 4), which is where the open questions get interesting: what should an evaluator be allowed to read, what should it be allowed to mark, and what happens to cost and legibility when those two scopes stop being the same thing (Part 5).

One more loop worth noticing before the parts begin: the judges are themselves traced. Every server-side evaluator execution produces spans in a project of its own; every experiment eval row records the trace id of the judge's own reasoning; those traces can be annotated in turn, and the spec explicitly imagines correcting a judge's verdicts and training its successor on the corrections. The second write path feeds the first. It's spans all the way down here, too.

The series