Quality control at the handoffs, without over-engineering

Status: shipped. vocion-core#672 — the grammar, the applier, the deterministic refusal and Returned to PM on Work — merged 2026-09-24 (a500d4cb). vocion-core#683 — the judge, the reason codes, escalation as an ask, sampling, the workspace dials — merged the same evening. Both live on the box since the 2026-09-25 pin bump. The deterministic half has refused writes in production; the first live judged crossing has not yet been observed. Docs: Handoff gates.

The problem, on one day

2026-09-24, running the software factory against real requests. A product manager put up a decision card that invented an iOS test harness for a web app. A contract was dispatched to a worker without anyone having read the repository. A bug report filed the wrong request. Chris: "the agent instructions alone will drift. The factory needs to know what good work looks like, check the evidence, and send weak work back to the seat that produced it."

Three seams. Three prompt edits would have covered them for a week.

What we rejected

The next seat as the judge. The obvious design: QA reviews the PM, Eng reviews QA. It is biased to proceed — the next seat wants work to do — it is the same model family grading itself, and "good enough" turns into silent repair: the next seat fixes the record instead of sending it back, and the seat that produced weak work never learns it did. The next seat keeps the right to refuse as the last line, not the first.

A review-queue product for gates. A second queue beside Review, with its own rows and its own page. The review queue already exists; anything a person must decide is an ask on it. A gate that needs a person files an ask of kind gate. One shape, used everywhere.

A send_back verb. A separate tool for returning work. It was queued, and then absorbed: a return is what the judge writes when it says return, and Reject-with-reason on the decision card writes the same two fields (returnedTo, returnReason). No new verb.

What we built

Two halves, one grammar, declared on the object type:

gates:
  - name: decision-ready
    when: {field: state, becomes: [in_scope]}
    producedBy: product-manager
    require:
      - {field: acceptance, minItems: 3, message: 'write at least three criteria a person can check'}
      - {field: expectedResult, present: true, message: say what will be different if this works}
    judge:
      rubric: rubric-product-manager
      cases: factory-reference
      escalateBelow: 0.6
      sampleRate: 1
      alwaysEscalate: {riskClass: [schema, billing, auth, infra]}

The deterministic half runs in the precheck of objects.update_meta — the action every agent write to a record already rides — against the record as it would be after the write. A transition that fails is refused: the record gets gate: {name, failed, at} and returnedTo, and the tool result names each missing thing in the seat's own words. No person is interrupted; the seat fixes the record and moves it again. The message ends A gate is not argued with in prose, because the first thing a model does with a refusal is explain why the refusal is wrong.

The judge runs after the deterministic half passes, asynchronously, best-effort. One model call at temperature 0: the seat's one-page rubric as the system prompt, the first four reference cases as calibration, the record as it stands, and one instruction — answer with one JSON object: pass, return, or escalate, a confidence, a reason code from a closed list, the one concrete thing that fails quoted from the record, and one sentence. Pass only when a product owner would make the same decision with this evidence.

• return undoes the transition and marks the record returned to the producing seat, with the reason code and the example. Closed list — unfaithful-to-asker, untestable-criteria, unsupported-recommendation, missing-evidence, wrong-record, missing-state, scope-invented, other — so returns can be counted by cause.
• escalate stamps the record and files an ask on Review naming the rubric line in doubt and the confidence. It is also what a verdict under escalateBelow becomes — pass or fail — and what any alwaysEscalate field value forces. A verdict the parser cannot read is an escalation carrying the raw text: the judge never guesses on a person's behalf.
• pass stamps gate.judged with the confidence.

Everything with a side effect is injected — the model call, the rubric mount, the dataset read, the stamp, the revert, the ask — so judgeHandoff is tested without a model or a database.

Where the numbers live

Core ships the grammar and the check and knows nothing about what a request is. The plugin declares the gates: which fields, on which transition, judged by whose rubric against which cases. The workspace turns the dials, for every gate at once:

defaults:
  gates: {sampleRate: 0.5, escalateBelow: 0.7}

And a workspace's own skills/rubric-product-manager/SKILL.md or evals/factory-reference.yaml replaces the plugin's by slug. Concrete → abstract → core: the meaning is a file a workspace can edit, the mechanism is TypeScript nobody has to.

What the reference runs measured

A gate is only as good as its rubric, and a rubric is only as good as the cases that calibrate it. So the same day, the factory reference set — fourteen cases, drafted for the PM and corrected by the person who owns the standard — was run on the box to get a number:

The bar is 80%. Five of the six early failures carried one tag: asked or offered options instead of deciding. That is one prompt change, and it is why the trend moved. The set is now the check every PM prompt, skill or model change is measured against, and its cases are what the decision-ready gate's judge reads on every crossing. Evals and reference sets has the baseline discipline.

What this does not do

It does not judge prose the model wrote in chat; it judges records at transitions. It does not run synchronously — the write lands, then the judge reads — so a returned record was briefly in its new state. It has not been observed returning a record in production yet; the deterministic half has refused several. Two older request-specific gates that live in core TypeScript are queued to become declared gates and be deleted, now that maxAgeDays and message are in the grammar. And sampling is a dial, not yet a ladder: the pass rate that should lower sampleRate for a seat that keeps passing is recorded, not yet acted on.

Where it lives: packages/core/src/libs/gates/handoffGate.ts, packages/core/src/services/gates/handoffJudge.ts, HandoffGateSchema in libs/workspace/schemas.ts, and the two declarations in templates/plugins/software-factory/objects/{request,engineering_task}/type.yaml.