part = 3 / 5

Building a judge in the browser

Phoenix now ships a full evaluator construction flow in the UI — and its most interesting decisions are all acts of reuse: the prompt editor is the playground, the output schema is an annotation config wearing a different label, and the input mapping panel is a small form sitting on top of the abstraction the entire online-evals future hangs on. This part reads the builder panel by panel, then takes the mapping layer seriously as architecture.

Where evaluators are born

There is a global /evaluators registry page, and it is deliberately read-only — a table of every evaluator, its kind, its prompt, its model, and which datasets use it (app/src/pages/evaluators/). Creation happens in context: the dataset's Evaluators tab and the playground both mount the same AddEvaluatorMenu — new LLM evaluator, LLM evaluator from a curated template, new code evaluator, or a built-in (app/src/components/evaluators/AddEvaluatorMenu.tsx). The flows are even deep-linkable (?createLlmEvaluator=true), which tells you the team expects other surfaces — docs, agents, empty states — to open them. Creation being dataset-scoped is era-2's shape (Part 2) showing through the chrome: an evaluator is born already bound, because an unbound evaluator can't be tested, and the builder is organized entirely around testing.

Three observations worth carrying forward

1. The output schema is an annotation config, and the UI says so out loud

The section where you define the judge's verdict — choices, each a label with an optional score, minimum two; an optimization direction; an "include explanation" switch — is titled "Evaluator Annotation" (EvaluatorCategoricalChoiceConfig.tsx). The name field is disabled, mirroring the evaluator's name; the store syncs them automatically. This is Part 1's config system, generated rather than authored — and it closes the loop Part 1 left open: configs validate nothing at the write path, but here the config is compiled into the writer. The judge's choices are constrained to the config's labels by construction (structured output), so machine-written annotations can't drift from their vocabulary the way human- and API-written ones can. The one writer that honors the config is the one whose config was generated from it.

2. The judge's prompt is a prompt, full stop

The prompt panel literally mounts the playground's template editor with save/tools disabled (EvaluatorChatTemplate.tsx wrapping PlaygroundTemplate) — same model picker, same mustache/f-string toggle, same variable derivation. Combined with the prompt-hub delegation from Part 2 (evaluator → prompt_id + version tag), the message is consistent: judge prompts get no special ontology. They are versioned, taggable, playground-testable prompts whose consumer happens to be a daemon. The intended promotion workflow — iterate, benchmark with an experiment, tag, and the evaluator follows the tag — treats the judge as an LLM task under test, which it is.

3. Convention first, mapping as the escape hatch

The mapping panel renders one row per template variable, derived live from the prompt, each switchable between a path into the data and a literal. Its caption is the design philosophy in one sentence: "You can leave these blank if your variable names match the field names." Name a variable {{input}} and it binds itself; name it {{question}} and you point it somewhere. Code evaluators get the same panel with variables read from the function signature instead — def evaluate(output, reference=None, input=None, metadata=None) — the client-era convention (Part 2) reborn as UI. And note the small vocabulary shift the test loop performs: when you pick a real example, the example's stored output is offered to the judge as reference, and "output" means the task's output. The builder quietly teaches the eval worldview: the dataset holds the question and the answer key; the thing being graded is always fresh.

The mapping layer, taken seriously

Under the panel is a two-field object that the whole future leans on (src/phoenix/db/types/evaluators.py:251–253):

class InputMapping(DBBaseModel):
    literal_mapping: dict[str, Any]
    path_mapping: dict[str, JSONPath]

The JSONPath dialect is a deliberately weakened RFC-9535 subset — dot and bracket access, indices, wildcards, slices; no recursive descent, no filters, no unions (types/evaluators.py:202–245). A mapping can point, but it cannot search. That restriction reads differently depending on the data source, and walking the three sources in order is the best preview of Part 5 available:

Dataset examples are flat little dicts — {input, output, metadata} plus the task's output. Pointing is trivial; the convention defaults usually suffice. This is the world the mapping layer was built in.
Spans are one entity with a deep attribute bag. The pointing still works, but the paths get long and conventional — attributes.llm.output_messages.0.message.content — inheriting every flattening quirk OpenInference imposes (companion series, Part 2). Nothing matches evaluator parameter names by accident anymore.
Traces break the model. A trace has no attribute bag, no input, no output — it's a label over spans (companion series, Part 1). There is nothing for a path to point at until something conjures a document from the tree: the root span's I/O, a filtered set of sub-spans, a rendered transcript. The internal spec work names this properly: extraction (hoist fields out of a tree into an evaluable payload) as a distinct step from mapping (rename payload fields into evaluator parameters). Today one dict does both jobs, because datasets never needed the distinction. Projects will.

Code evaluators: the same bench, sharper tools

The code path replaces the prompt panel with a CodeMirror editor (Python or TypeScript), starter templates for the classic heuristics, a sandbox picker, and — the detail that matters — versioning on every edit: saving changes creates a new code_evaluator_code_versions row rather than mutating the old one. Output configs may be numeric here, with bounds and a threshold ("the cutoff used to visually distinguish good from bad scores") — the LLM/code asymmetry from Part 2 surfaced as form fields. The test section is identical in spirit: run against a real example, see the annotation it would produce, via the same evaluatorPreviews mutation that powers the LLM preview — a dry run that executes the full pipeline (mapping included) and persists nothing.

That preview mutation deserves its sentence of respect: it means the builder's promise is behavioral, not cosmetic. What you saw in the preview is what the daemon will do, because both run the same evaluator, through the same mapping, on the same shapes. For a system whose Part-1 configs enforce nothing, the builder is where correctness actually gets manufactured — upstream, in the machine, rather than downstream, at the write.

part 3 in one paragraph The builder is reuse all the way down — playground editor for the prompt, annotation config for the output, convention-plus-mapping for the inputs, a real example and a dry-run mutation for the feedback loop — and the reuse is the argument: a judge is an ordinary LLM task whose output schema happens to be an annotation vocabulary. The mapping layer underneath is small and deliberately weak: it can point but not search, which is exactly enough for datasets and spans, and exactly not enough for traces — where there is nothing to point at until an extraction step conjures it. That missing step is where the next two parts live.