What a week of Slate commits taught the factory

Status: shipped and measured. The first per-product reference set and its standards page landed in the instance that runs Vocion for Metacto (metacto-vocion-agents#197, 2026-09-25) and ran on the box the same day: eval run #5, slate-reference, 2 of 16 pass (12.5%). Docs: Evals and reference sets, The learning loop.

The question

The factory's first reference set — fourteen cases, drafted from our own internal tooling requests — had been run to a 60% baseline and was moving (Quality control at the handoffs). Then Chris, 2026-09-24: "these few-shot and eval samples are from our Vocion work. But we are building a software factory specifically for consumer SaaS products. Won't the feature types and user considerations be different?"

They are. A consumer app is judged on first-run and permissions, capture reliability, perceived speed and async state, trust and store compliance, monetisation posture, empty states — none of which a set built from internal tooling requests exercises. A PM graded 60% on our own requests could be graded 10% on a consumer product's, and we would not know.

"Categorise, and make synthetic data sets for evals and gold standards. Go trawl the updates pushed to the Slate repos in the last week for good examples. The solutions and the code are the gold standard."

The method

Source of truth is what shipped. One engineer's week on Slate — 2026-09-17 to 24, seven repositories, about 340 commits — was read and categorised. Not the commit messages as text to copy, but the decisions in them: what was asked, what was decided, why, and the shape of the change. Sixteen categories fell out of the week: first-run and permissions · capture reliability · recording feel · perceived speed and async state · editing IA · sharing and viewing · trust, safety and store compliance · security and pentest · monetisation posture · empty states and pitches · AI features · migration and import · notifications and inbox · appearance and theming · analytics and observability · back office.

The input is synthetic. For each case, the request is written fresh, the way it would actually arrive: a store review, a support mail, a pentest finding, an alarm, a message in chat. Fictional names, fictional addresses. This is the part the seat sees.

The gold is real. The case's expectedOutput is what the engineer actually decided and shipped — the decision, the reason, the code shape — linked by a gold:<repo>@<sha> tag to the commit. The judge measures the PM's decision against a decision a person who owns the product made, not against one the dataset author imagined a good PM would make.

Two artifacts, both core nouns. The dataset (evals/slate-reference.yaml, fifteen cases, tagged product:slate and category:<slug>) is measurement and never enters the model's context. The standards page (wiki/slate-standards.md: the product lens, then one block per category with up to three exemplars) is what the seat retrieves by search when a request lands. The rubric stays per seat; the lens changes per product. Putting the cases in the prompt would make the eval measure memorisation. Putting the standards in the prompt would fill the context with sixteen categories for a request that touches one.

Quota, not everything. About twelve cases per product per seat; one standards page per category with at most three exemplars; a case retires after five straight passes once a newer case covers its category. The mechanism to keep the set from growing without bound is designed in before the set exists.

The number

Run #5, on core 77b9edcc: 2 pass, 11 fail, 3 error. 12.5%, against an 80% bar.

The three errors were the eval infrastructure — empty responses while the tooling container logged seventeen model-call timeouts — not the seat. The eleven failures were one thing, in eleven outfits: the PM files the request and describes it well, and then does not decide to the standard the engineer set. It recommends the smaller fix instead of the one that changes the user's experience; it deflects the decision to the user; it delivers a filing where the gold delivers a decision. The judge's rationales read like one reviewer's notes on one habit.

Why 2 of 16 is the right starting number

A baseline is not a grade. It is the number that makes the next number mean something.

If the first run had come back at 70%, one of three things would be true: the cases are easy, the gold is soft, or the seat was already good at consumer product decisions without ever having been shown one — and the third is the least likely. A set that a seat passes on day one is not measuring the gap it was built to measure. A set that fails eleven of sixteen for the same reason has found a gap you can name in one sentence, which is the most useful thing a reference set can do on its first run.

It is also honest about what a standards page is for. The page exists now; the seat was not yet told to read it before deciding. The next change is exactly that — the eleven rationales become a few-shot pointer in the PM's skill to slate-standards, the seat reads the relevant block before it decides — and run #6 will say whether a page the seat reads moves a number the seat never sees. That is the experiment. 2 of 16 is its control.

What it costs, and what is next

A reference run is one model call per case plus the judge; the factory set ran at about fifty-one cents per passed case on the first clean run. Cheap enough to run after every change to the seat, which is the point.

Next: the PM reads the standards page before deciding, then run #6. The handoff gate's judge choosing its calibration cases by the request's product (casesByProduct), so a Slate request is graded against Slate cases. The eval page showing the gold link per case. And the same week-long trawl for the other products in the portfolio, each with its own page and its own dozen cases — the shape is now a file, not a project.

Nothing in the dataset or the page names a customer; the product is Metacto's own. The inputs are synthetic and the gold commits are referenced by hash. No commit text is reproduced.