Evals

An eval dataset is a list of cases for one agent: a message to send, and what a good turn looks like.

# evals/lead-basics.yaml
slug: lead-basics
name: Lead basics
agentSlug: northwind-lead
passThreshold: 0.8               # default 0.8
items:
  - input: What's at risk at Acme this week?
    expectedOutput: Names the open Acme deals and the risk on each, with sources.
    checks:
      - toolCalled: search_knowledge
      - outputNotContains: I don't have access
      - latencyUnderMs: 60000
  - input: Draft a follow-up to Contoso.
    checks:
      - toolCalled: propose_action
      - toolNotCalled: run_code

Checks

Deterministic: toolCalled, toolNotCalled, toolCalledWith, toolReturned, toolCallCount, outputMatches, outputContains, outputNotContains, latencyUnderMs, turnsUnder. A case can also carry expectedOutput and rubric for the judge, plus expectedTrajectory, assertions and tags.

provider: vocion (default) grades in Vocion; provider: agentcore uses Amazon Bedrock AgentCore evaluations, with evaluators naming which.

Run

npm run eval:run --workspace @vocion/core -- --dataset lead-basics

Or from /dashboard/evals, or POST /api/v1/evals/{slug}/runs. Each run records the workspace version it ran on. npm run eval:upgrade and POST /api/v1/evals/{slug}/model-upgrade-test compare a candidate model against the current one on the same dataset.

Handoff-gate judges calibrate on an eval dataset named in judge.cases.