part = 2 / 5

The evaluator becomes a row

For most of Phoenix's life, the evaluator was a Python function in your notebook, and Phoenix stored only its verdicts. Over 2025–26 the function itself moved into the database — prompt, output schema, sandbox, version history, binding — and the server grew a daemon to run it. This part traces that migration and reads the new tables, because they are the raw material for everything online evals will be built from.

Era one: the evaluator is your problem

The classic flow still works and still matters: run_experiment(dataset, task, evaluators=[…]) in the phoenix-client package executes everything — task and judges — in your process, and POSTs results to a server that is a passive store. An evaluator here is any callable; the client inspects its signature and binds parameters by name from a fixed menu — input, output, expected, reference, metadata, example — and coerces whatever it returns: a bool becomes score-plus-label, a float a score, a string a label, a tuple all three (packages/phoenix-client/…/experiments/evaluators.py:86–149). That name-binding convention is worth flagging now, because Part 3 shows the server reinventing it deliberately: an evaluator's parameter names are its input schema.

Two deletions mark the era's end. In March–April 2026 Phoenix removed the /v1/evaluations endpoint and its protobuf ingestion path entirely (#12382, #12538). Until then, "an eval" was a distinct thing you could ship to Phoenix, and the server stamped every arriving row annotator_kind='LLM', identifier='' for you. After it, evals stopped being an ingestion type: an eval result is just an annotation (Part 1), written through the same endpoints as any other, with the caller stating its own annotator_kind. One shape, as promised — achieved not by adding a concept but by deleting one.

Era two: the evaluator is a row

Meanwhile the evaluator itself became data. The migration (02463bd83119_add_evaluators.py, October 2025) created a polymorphic base table and three kind tables:

evaluatorsbase

kind ∈ {'LLM','CODE','BUILTIN'}, a globally unique name, description, owner (models.py:2712–2741). The registry entry; joined-table polymorphism hangs the specifics below.

llm_evaluatorsLLM

Owns no prompt text. It points at the prompt hub — prompt_id with ondelete="RESTRICT" (you cannot delete a prompt an evaluator depends on) plus an optional version tag pin (models.py:2744–2757). Output schema: a list of categorical configs only — labels with optional scores (:2761–2763).

code_evaluatorsCODE

Python or TypeScript source, executed in a pluggable sandbox (WASM, E2B, Daytona, Vercel, Deno, Modal — models.py:175), with source versioned append-only in a separate table (:2893–2921) and any output config type, numeric included (:2858).

builtin_evaluatorsBUILTIN

A database reflection of an in-memory registry (exact match, contains, regex, Levenshtein, JSON distance), re-synced at startup (models.py:2924–2967). Rows that exist so bindings have something to point at.

Three details here are architecture, not bookkeeping. First, the LLM evaluator's delegation to the prompt hub means judge prompts get the full prompt-management treatment — versions, tags, playground testability — and the internal spec is explicit about the intended workflow: iterate on the judge's prompt, benchmark it with an experiment, then promote it by tag, and the evaluator picks it up. The judge is itself an LLM task under test. Second, the categorical-only restriction on LLM outputs is a quiet statement of eval philosophy: LLM judges classify; numbers come from code. Third, code evaluators are append-only — every edit is a new version row — which will matter in Part 4, where "which version produced this verdict?" becomes a queue-deduplication key.

The binding: a global machine, locally adapted

An evaluator row is deliberately abstract — a function with named inputs. To run, it must be bound to a data source, and that's a separate row: dataset_evaluators (models.py:2970–3010) joins one evaluator to one dataset and carries everything local to the pairing:

• a per-binding name, unique within the dataset — the name the resulting annotations will carry;
• an optional output-config override — same judge, stricter rubric here;
• an input mapping, NOT NULL — literals plus a restricted-JSONPath path_mapping from the evaluator's parameter names into the data's shape (src/phoenix/db/types/evaluators.py:251–253; Part 3 is entirely about this object);
• and a project_id, NOT NULL, RESTRICT — the project that will receive the judge's own traces.

This is the "mapping layer" in the series subtitle, and it's the piece that makes "global evaluator" a coherent idea at all: the evaluator carries the judgment; the binding carries the adaptation. When project evaluators arrive (Part 4), the spec describes their attachment row as this table plus what a firehose needs — target type, filter, sampling rate, readiness — which is why it's worth internalizing now.

Execution: a daemon with table manners

Server-resident evaluators are run by the ExperimentRunner daemon (src/phoenix/server/daemons/experiment_runner.py) against experiment_jobs — a jobs table whose primary key is the experiment id, with a type (PROMPT runs a task then evals; EVAL_ONLY "no task, just run evaluators against existing runs"), a status machine, and multi-replica coordination columns: claimed_by, claimed_at, heartbeats, cooldowns (models.py:1731–1782). Replicas claim jobs by atomic update, heartbeat while working, and recover each other's orphans. Work dispatch prioritizes evals over retries over tasks, and every step is logged to structured per-run, per-evaluator log tables. Two properties deserve emphasis because Part 4 leans on them:

• The lease pattern is proven in-house. Claim-by-CAS, heartbeat, orphan recovery — the coordination shape online evals need already runs in production, one table over.
• The LLM plumbing is shared. A server-side judge speaks through the playground's client stack — same providers, same credentials. The evaluator didn't grow its own inference path; it borrowed the one users already trust.

The results table teaches the philosophy

Wherever an evaluator runs — your notebook, the daemon, the playground's streaming runs — its verdict lands in experiment_run_annotations. Compare it to the uniform annotation shape of Part 1:

And one non-column fact that surprises most people: experiment eval results never flow back onto spans. The eval endpoints write ExperimentRunAnnotation rows and nothing else; the experiment's task traces live in an auto-created project (Experiment-<hex>), joined by a string trace id that tolerates the trace never arriving. The experiment world and the tracing world are linked by reference, not by writes. Online evals (Part 4) invert exactly this: their entire point is to write into the span/trace/session annotation tables that experiments deliberately leave alone.

The loop: judges are traced

Every server-side evaluator execution is itself traced — into the binding's own project, inspectable span by span, with the eval row's trace_id pointing at it. The UI already leans on this: each evaluator detail page has a Spans tab; each experiment cell offers "view evaluation trace." The spec takes the loop further: annotate the judge's traces, collect corrected verdicts, and use them to train the next version of the judge. Path 1 feeds path 2 feeds path 1. The companion series worried about what happens when agents watch agents; here it's institutionalized — with the pleasant consequence that every skeptical question you can ask about your application ("what did it actually do?") can be asked of your evaluator, in the same UI.