part = 1 / 5

Judgment as a row

The annotation is the only data in Phoenix that Phoenix itself is allowed to change. Its entire social contract — who may disagree with whom, what a re-run replaces, why notes pile up while evals overwrite — is encoded in one three-column unique key. This part reads that key closely, because everything the later parts build stands on it.

span_annotationsuniform

Verdicts on a span. The reference implementation of the shape; everything below varies it (src/phoenix/db/models.py:1141).

trace_annotationsuniform

Same shape, keyed on trace_rowid (models.py:1181). The one way to say something about a trace, which itself contains nothing.

project_session_annotationsuniform

Same shape, keyed on the session (models.py:1260). Fully wired in schema and API; still without a create surface in the UI.

document_annotationsuniform +1

Same shape plus document_position (models.py:1218,1224) — a verdict on the n-th retrieved chunk inside a span.

experiment_run_annotationsoutlier

Evaluator results on experiment runs (models.py:1678). Drops three provenance columns, adds two the others lack. Part 2's subject.

The shape

Four of the five tables share an identical column set: a name (which metric this is), the verdict payload — nullable label, nullable score, nullable explanation, a JSON metadata bag — and three provenance stamps:

• annotator_kind ∈ {'LLM', 'CODE', 'HUMAN'}, enforced by a check constraint (models.py:1152–1154). Who kind of thing judged: a model, a program, a person.
• source ∈ {'API', 'APP'} (models.py:1164–1166). Which door the verdict came through: the REST API always stamps API; the UI's GraphQL mutations stamp APP. When the columns were introduced, existing rows were backfilled by the rule HUMAN → APP, everything else → API (migration 2f9d1a65945f) — a tidy statement of the assumed world: people click, machines POST.
• user_id, nullable, SET NULL on user deletion (models.py:1167) — verdicts outlive their authors.

And then there is the odd one out, the column that looks like bookkeeping and is actually the whole design: identifier, a string, NOT NULL, defaulting to '' (models.py:1159–1163).

The identity key is the conflict policy

Every uniform table carries the same unique constraint (models.py:1172–1178):

UniqueConstraint(
    "name",
    "span_rowid",      # or trace_rowid / project_session_id (+ document_position)
    "identifier",
)

and every write path upserts against it. The REST endpoints and the async ingestion queue both compile to INSERT … ON CONFLICT DO UPDATE on exactly this key (src/phoenix/db/insertion/helpers.py:34–77), with the update set defaulting to every column except the primary key and created_at (helpers.py:80–89). The GraphQL creates do the equivalent select-then-update by hand (span_annotations_mutations.py:101–124). The consequences are worth spelling out, because they are the API's real semantics and no endpoint name says so:

• Writes are idempotent replacements. Re-sending a verdict with the same (name, target, identifier) doesn't duplicate it — it silently replaces label, score, explanation, metadata, and provenance. An API write can overwrite a human's row if it collides on the key.
• "How many opinions may exist" is not schema — it's identifier policy. Distinct identifiers coexist; equal identifiers collide. The table itself has no opinion about which you want.
• The async path adds last-write-wins by timestamp: a queued annotation older than the stored row is discarded, not applied (insertion/span_annotation.py:94–105). And annotations can legally arrive before their span does — the inserter parks them and retries, then gives up (:115–123). Path 2 inherits path 1's out-of-order world.

Phoenix ships three identifier conventions, which produce three completely different social behaviors from one mechanism:

Annotation configs: a grammar that isn't a schema

Alongside the annotations sits their nominal vocabulary: annotation_configs, globally unique by name, attached to projects through a many-to-many link table (models.py:2556–2571). A config comes in three types (src/phoenix/db/types/annotation_configs.py): categorical (a list of labels, each with an optional score), continuous (optional bounds, lower strictly below upper), and freeform — each with an optimization_direction of MAXIMIZE, MINIMIZE, or NONE.

The load-bearing observation: configs are consulted by exactly zero write paths. None of the annotation mutations or REST endpoints reference AnnotationConfig at all — any label, any score, any name is accepted, whether or not a config exists, whether or not the value is in its list or inside its bounds. The single write-time rule in the whole system is that the name "note" is reserved (routers/v1/annotation_configs.py:745–756; rejected again at each annotation endpoint). Even deletion admits the relationship: the settings UI warns, accurately, that "Annotations made with this config will not be impacted."

Is that a defect? Mostly no — it's the same posture Phoenix takes toward OpenInference conventions on the first write path: believe what arrives, standardize how it's displayed. A validating annotation store would have to reject late eval results whose config changed in the meantime, which is a worse failure mode than rendering an off-list label plainly. But notice what the posture costs: a config rename or a choice-list edit silently forks the vocabulary (old rows keep old labels), and nothing in the system can tell you whether an annotation named correctness means the same thing this month as last. The config system records intentions; the annotation table records history; nothing reconciles them. Part 3 shows the one place the two are actually coupled — inside the evaluator builder, where the config is generated from the machine that will honor it.

Where the levels stand today

The companion series scored the annotation model "built for all levels, surfaced for one and a half" — humans can create span (and document) annotations in the UI, while trace and session annotations arrive only through the API. It's worth restating here with this series' emphasis: the write paths that exist are exactly the ones evaluators need. Machines don't click. A trace-level LLM judge writing through POST /v1/trace_annotations is fully served today; it is the humans who are waiting on frontend work. The consequence for Part 4 is quietly funny: when project evaluators ship, the session inspector's annotation rail will fill with machine verdicts that no human, standing in the same UI, can yet reply to.