part = 4 / 5

The project is a firehose

Online evals change nothing about the evaluator and nothing about the annotation. What changes is delivery: the data source stops being a table you point at and becomes a stream that points at you. This part reads the current design work — a spec on main, a deeper rework and a queue spike on branches — as five distinct hard problems, plus the one the spec explicitly defers: backfill.

Status, honestly. On main today there is no hook from ingestion to evaluation — the span-insert event's only consumer invalidates dataloader caches (src/phoenix/server/dml_event_handler.py). What exists: a v1 spec (internal_docs/specs/online-evals.md, July 1), a substantially expanded rework on origin/online-evals-spec-update, a DB/queue spike on origin/dustin/queue-spike, and a working in-server precedent scoped to Phoenix's own agent (origin/ehutt/pxi-inference-evals-live-runner: a span-processor that buffers finished spans, samples traces, scores them, and writes span annotations). Branch material is marked as such below; it describes a conversation, not a commitment.

The attachment, one more time

The spec is emphatic that a project evaluator is not a new evaluator type — it's a new binding. Set Part 2's dataset_evaluators beside what the spec says a project attachment defines, and the delta is exactly the firehose tax:

Field	dataset binding (shipped)	project attachment (spec'd)
evaluator	✓ evaluator_id	✓ same registry
name, output override	✓	✓ (annotations need names here too)
input mapping	✓ NOT NULL	✓ — now doing extraction's job too (Part 3)
judge-trace project	✓ project_id	✓ presumably
target type	— (always: examples)	new: span \| trace \| session
filter	— (the dataset is the filter)	new: which artifacts are eligible
sampling rate	— (you run what you run)	new: what fraction to evaluate
readiness	— (rows are at rest)	new: when an artifact counts as evaluable
enabled	— (jobs are invoked)	new: a kill switch, because nobody invokes anything

Five new fields, five hard problems. In order:

1 · Eligibility: filters meet their limits

Span and trace filters get to reuse Phoenix's existing filter DSL — the same language that already supports annotations["name"].score < 0.5 (src/phoenix/trace/dsl/filter.py), which quietly enables a lovely pattern: evaluators triggered by other evaluators' verdicts, or by human thumbs-downs. But the DSL is span-scoped and has no topology predicates — you cannot say "spans whose sibling retried" or "traces containing a TOOL span" (a gap the branch rework calls out explicitly). And sessions have no filter vocabulary at all; the v1 spec's honest options are "define a small one or omit session filters." Filters also only face forward: changing one affects future artifacts, never the past — re-reading history is backfill's job, below.

2 · Sampling: a hash, not a coin

The spec requires an "intermediate value" of the sampling rate to select "a stable subset, so the same artifact is not randomly included or excluded across retries or restarts." The spike's implementation (branch) is the classic trick: md5(span_id) / 2¹²⁸ < ρ. No stored coin flips, no coordination — the artifact's identity is its lottery number. Two properties fall out that a random sampler doesn't have, and both are product features:

Deterministic sampling: cohorts, nested and shared Drag ρ — watch which spans enter and leave

evaluator A · ρ_A

evaluator B · ρ_B = ρ_A ÷ 2 (co-sampled)

Every span sits at a fixed position — its hash. Raising ρ only adds spans: yesterday's 10% cohort is inside today's 25% cohort, so longitudinal comparisons survive rate changes. And because both evaluators read the same hash, B's cohort is a subset of A's — their verdicts land on the same artifacts and can be joined. The branch rework calls this shared-cohort "density"; a per-evaluator random sample would scatter verdicts across disjoint subsets and quietly destroy every cross-evaluator query.

3 · Readiness: the completeness problem becomes a trigger

The companion series spent a part establishing that no trace is ever knowably complete — spans export on end, stragglers are always legal. That was a rendering nuisance. Here it graduates into semantics: an evaluator that reads a trace must decide when, and there is no event to wait for. The spec's answer is the only honest one available: idle time as a proxy for done. Spans are ready at storage; traces are ready after a quiet period; sessions after a user-configured idle window; and when a late span or a resumed session breaks the quiet, the artifact becomes eligible again — re-evaluated on the next silence, with "the visible annotation reflects the latest evaluation." Note which machinery that last clause is borrowing: it's Part 1's identity key. Latest-wins is not a new feature; it's what ON CONFLICT DO UPDATE on a derived identifier already does. The genuinely new obligation is the audit half — run history that preserves what earlier verdicts said, which the branch rework promotes to a v1-blocking requirement (a run/decision record store), since the annotation row, by design, forgets.

4 · Delivery: a queue, not a cron

The spike (branch origin/dustin/queue-spike) is small and shaped exactly like the lessons of Part 2's daemon. Two tables:

eval_work_cursors — a single-active-producer watermark per grain, with a lease: one elected producer walks the span arrival clock, matches attachments, samples, and fans out work. Producers are elected by compare-and-swap, exactly like experiment claims.
eval_work_units — one row per (span_rowid, evaluator_id, config_fingerprint), with status, per-item lease, attempts, cooldown, and a partial index over everything not DONE. Competing consumers claim, heartbeat, complete, or fail with cooldown — the EvalWorkCoordinator protocol.

Two of those columns are doing conceptual work worth pausing on. config_fingerprint makes "which evaluator?" a content-addressed question: edit the prompt, the tag, the mapping — the fingerprint changes, and the same span becomes new work. Version identity, which Part 2 noted the evaluator tables already keep append-only, becomes the queue's dedup key. And the claimed unit carries the derived annotation identifier — so a retried or duplicated execution upserts the same row instead of forking a second verdict. Idempotency all the way from producer to annotation table, riding on Part 1's key.

5 · Overload: skips you can see

The spec's hardest constraint is one sentence: project evaluators "must never delay or fail ingestion." Evaluation is strictly downstream, asynchronous, and sheddable — and the branch rework insists shedding be visible: an overload backstop that skips work and says so, plus a first-class lag signal (pending and running counts, frontier gap, oldest-pending age) already present in the coordinator protocol. The principle deserves its own sentence, because monitoring tools rot without it: a skipped eval is monitoring data; a silently skipped eval is a lie about your system. The same goes for sampling (an unevaluated span should be distinguishable from a failing one) and for judge errors (Part 2's tables made errors rows, not absences; the firehose must too).

The firehose, stepped by hand Three traces arrive interleaved; a trace-level judge waits for quiet

arrivals (the part evals must never slow)

eval_work_units → annotations

Per-span mode: eligible LLM spans (filter) inside the cohort (ρ = 0.5) become work units the moment they're stored. Trace-idle mode: nothing is enqueued until a trace goes quiet — then one unit per trace. Run the stream to the end and press "late span arrives": the idle-mode trace wakes, re-evaluates on its next silence, and its annotation absorbs a second write — Part 1's collider, playing itself.

Backfill: the same queue, facing backward

The v1 spec lists backfill as a non-goal, and the deferral is more interesting than it looks, because the queue design quietly makes backfill cheap to describe: backfill is a second producer. The online producer walks the arrival frontier forward; a backfill producer walks a historical range. Everything downstream — work units, fingerprints, consumers, derived identifiers — is unchanged. config_fingerprint is what makes "run the updated evaluator over last month" a well-defined request (new fingerprint, new units, no collision with the old run's records), and the derived identifier is what decides whether the new verdicts replace the old annotations or sit beside them — which is not a technical question but a product one, the same one the collider posed in Part 1: is this re-judgment the same opinion, revised, or a new opinion? Deferring backfill is defensible; what shouldn't be deferred is that choice of key, because the moment evaluator configs change (immediately) users will ask for their history re-scored, and the identifier scheme decided today is what makes that either an upsert or a mess.

part 4 in one paragraph Online evals reuse the evaluator, the binding idea, the annotation tables, the daemon's lease pattern, and the identity key's idempotency — the new work is a delivery system: forward-only filters (with real expressiveness gaps at trace and session level), hash-based sampling whose nested, shared cohorts keep verdicts joinable, idle-as-done readiness that turns the old completeness problem into re-evaluation semantics, a content-fingerprinted work queue, and overload behavior that skips loudly. Backfill falls out as a second producer on the same queue — provided the annotation identifier is designed for re-judgment from day one. What the delivery system does not yet answer is what an evaluator may read versus what it may mark — the scope question. That's Part 5.