Evals

An Eval dataset is a list of test cases (input → optional expected output → optional rubric) that exercises an Agent. Run the dataset; an LLM judge scores each case; metrics persist. Each run stamps the active workspace_sha so you can attribute a regression to a specific prompt change.

What makes a good eval case

Three flavors of case work well in practice:

Pinned outputs — case has expectedOutput; the judge compares actual vs expected by rubric. Use for "the answer here is unambiguous."
Rubric-only — case has rubric but no expectedOutput; the judge applies the rubric ("a good reply acknowledges the issue, asks at most one clarifying question, doesn't promise a refund"). Use for prose-shaped outputs where exact wording shouldn't matter.
Tags as slices — tags: [angry, refund] lets the dashboard report pass rates per slice. Useful when you want to know "we pass calm tickets but fail angry ones."

Authoring

One dataset per file at workspace/<org>/evals/<slug>.yaml:

slug: support_triage_baseline
name: "Support triage — baseline cases"
agentSlug: support-agent           # required — datasets are agent-scoped
description: "Eight representative tickets covering the range L1 should handle."
items:
  - input: "Order #SR-48201 hasn't shipped in 10 days. Where is it?"
    rubric: "Empathetic, acknowledges the wait, escalates to fulfillment without promising an SLA."
    tags: [shipping]
  - input: "I want a refund for the broken lamp I received."
    expectedOutput: "Acknowledges the defect, offers a return label, no auto-refund."
    tags: [refund]
  # …

Then npm run workspace:apply to register.

Running

From the dashboard at /dashboard/evals/<slug> click Run dataset. Each case becomes:

Agent generates output for the input.
An LLM judge applies the rubric (or expected-output comparison) and returns { verdict: pass | fail | error, score: 0-1, rationale }.
The case row persists with output + score + rationale + Langfuse trace id.

Once all cases run, metrics are computed: passRate, medianLatencyMs, failed count.

CI integration

npm run eval:run -- --dataset <slug> runs from CI. The convention: fail the build if passRate < 0.8. This catches the case where a "small prompt tweak" regresses several baseline scenarios — a clean way to keep prompt edits honest.

Dashboard surface

/dashboard/evals — every dataset, with last run's pass rate + run count
/dashboard/evals/<slug> — dataset detail: cases, recent runs, Run dataset button
/dashboard/evals/<slug>/runs/<runId> — per-case results: input, output, expected, judge rationale, score, Langfuse trace link