VOCION
Get a demo
← All posts

The critic that ran out of room

The red team's first live run returned nothing parseable twice, and the agent read the document by hand instead and said so. The cause was a length budget, not a reasoning failure — and the fix was to stop asking a model for JSON in prose and declare a tool it has to call. vocion-core#457.

Vocion Teamvocion-v2.123.1
llmanthropicdocumentsproposalstoolsoperations

Status: shipped. vocion-core#457 (699df5ca) is merged and deployed. It fixes the tool that shipped four commits earlier in #455, written up in A proposal that reads itself as the buyer.

The Proposals plugin gained a red team: red_team_document reads a rendered document sheet by sheet as the sceptical operations lead who has to carry it into an internal business case, and returns numbered findings with a severity, the rubric rule each one breaks, and the edit that answers it. It is the check that a renderer cannot perform. A render-verify receipt tells you every sheet fits and the footer did not move; it says nothing about whether the buyer will believe a word of it.

The first time it ran on a real document, it returned nothing.

What the failure looked like

A seven-sheet client proposal. The agent called red_team_document, got back an unusable answer, called it again, got back an unusable answer again — and then read the document by hand, sheet by sheet, produced findings of its own, and told the person in the thread that the tool had not worked and that what followed was its own reading.

That last part is the only good news in the paragraph. A tool that fails and an agent that narrates the failure is a bad afternoon. A tool that fails and an agent that presents its improvisation as the tool's output is a trust problem you find out about months later.

But the findings were not in the document's record, nothing was stored against the version, and the export gate that depends on a review had nothing to read. As far as the system was concerned, the document had never been reviewed.

What actually went wrong

Not reasoning. Arithmetic.

The review was one model call that ended with an instruction to answer in JSON. The instruction was this, verbatim, as it stood before the fix:

Return STRICT JSON and nothing else: {"findings":[{"sheet":<n>,"severity":"block|fix|consider",
"rule":"<few words>","finding":"<one or two sentences>","fix":"<the edit>"}],
"keeps":"<one or two sentences on what to keep>"}. Fewer, sharper findings beat many small
ones; cap at the ones that matter.

And the call that carried it was capped at max_tokens: 3000.

Read those two together. A finding is one or two sentences. A fix is an edit. A rule is a few words. Multiply by a dozen findings — which is what a sceptical reader of a real proposal produces, because the rubric has ten separate things to be unhappy about — and the answer is several thousand tokens of output. The model wrote a good review, hit the ceiling partway through, and stopped mid-object.

The parser then did what a parser does with {"findings":[{...},{...},{"sheet":4,"severity":"fi — it returned null. Eleven complete, correct, useful findings were thrown away because a twelfth was cut in half.

And the reason given back to the agent was one generic sentence:

the review model returned something other than the findings JSON

Which is true, and tells you nothing. It reads identically whether the model refused, rambled, emitted prose around the JSON, or was simply cut off — four different problems with four different fixes, all wearing the same label.

The fix, in three parts

1. The shape comes from a schema, not from a sentence

packages/core/src/services/documents/redTeam.ts now declares the answer as a tool:

const FINDINGS_TOOL = {
  name: 'report_findings',
  description: 'Report the red-team findings on this document. Call this exactly once.',
  input_schema: {
    type: 'object' as const,
    properties: {
      findings: {
        type: 'array',
        maxItems: 20,
        items: {
          type: 'object',
          properties: {
            sheet: { type: 'integer', description: 'The sheet number the finding is on.' },
            severity: { type: 'string', enum: [...SEVERITIES] },
            rule: { type: 'string', description: 'The rubric rule it breaks, in a few words.' },
            finding: { type: 'string', description: 'What a buyer reads, in one or two sentences. Quote briefly where it helps.' },
            fix: { type: 'string', description: 'The edit that answers it.' },
          },
          required: ['sheet', 'severity', 'rule', 'finding', 'fix'],
        },
      },
      keeps: { type: 'string', description: 'One or two sentences on what to keep, so the fixes do not erase it.' },
    },
    required: ['findings'],
  },
};

and forces it:

tools: [FINDINGS_TOOL],
tool_choice: { type: 'tool', name: FINDINGS_TOOL.name },

The prompt line that used to carry a JSON grammar now carries only editorial direction, because the grammar has somewhere better to live:

Answer by calling report_findings exactly once, with nothing before it. Fewer, sharper findings
beat many small ones: at most fifteen, and keep each finding under sixty words so they all fit.

SEVERITIES is ['block', 'fix', 'consider'], re-exported from the card specs so the tool schema, the zod schema the stored findings validate against (documentRedTeamFindingSchema) and the receipt renderer all read the same three values from one place. A severity the tool cannot emit is a severity the storage layer never has to handle.

2. Room to finish

// A dozen findings on a twelve-sheet document is several thousand tokens
// of output; the first live run was cut off at 3000 and returned nothing
// usable. Budget for the whole answer rather than for the average one.
max_tokens: 8_000,

The comment is there because the next person to tune this will be looking at the average review, which fits comfortably, and will wonder why the budget is so generous. The budget is not for the average review. It is for the review of the document that most needs reviewing.

Input is bounded on the other side rather than by hope: MAX_SHEETS = 20 and MAX_CHARS_PER_SHEET = 6_000, the latter commented as "Enough of a 12-sheet proposal to read every sheet; a sheet is ~2–4k characters of text."

3. A partial answer keeps what validates

parseFindings no longer treats the answer as pass-or-fail. It tries the strict schema first; if that fails it re-parses with a loose schema that accepts an array of unknowns, validates each entry on its own, and keeps the ones that pass:

// One long finding must not lose the other eleven: keep the entries that
// validate, drop the ones that do not, and only give up when none survive.
const kept = loose.data.findings
  .map(f => FindingSchema.safeParse(f))
  .filter(r => r.success)
  .map(r => r.data);
if (kept.length === 0) {
  return null;
}

Eleven out of twelve is a review. Zero out of twelve is an outage. There are tests for both ends of that: one malformed entry (severity: 'shouting') is dropped while its valid sibling survives, and an array in which nothing validates still returns null.

And the receipt names the failure

When there is genuinely nothing to keep, the reason is no longer one sentence for four different problems:

const why = res.stop_reason === 'max_tokens'
  ? 'the review ran past its length budget before it finished — ask for fewer findings, or split the document'
  : `the review model answered in an unusable shape (stop_reason ${res.stop_reason ?? 'unknown'}${call ? ', tool call did not validate' : ', no tool call'})`;

Three distinguishable outcomes: out of room, called the tool but produced something the schema rejected, or did not call the tool at all. Each one has a different next move, and the person reading the receipt now gets told which one they are in — without opening a log.

The general rule, which this repo already believed

The build root's CLAUDE.md carries a standing rule that reads, in full:

Structural over prompting. A required model behaviour is enforced in code (typed events, deterministic post-processing, gated backstop), never by iterating prompt wording.

"Return STRICT JSON and nothing else" is prompting. It is a request, competing for attention with every other instruction in a long system prompt, and it has no mechanism behind it. A declared tool with a required schema and tool_choice pinned to it is structure: the provider constrains the output, the field names are not spelled out in prose where they can drift from the parser's expectations, and the failure modes collapse from "anything a language model might emit" to one — length.

That collapse is the actual win, and it is worth stating separately from the token bump. Before the change, an unparseable answer could have been any of a dozen things. After it, if the answer is unusable, the model ran out of room, and that is a number you can raise. A failure mode you can enumerate is a failure mode you can fix. A prompt instruction turns every failure into the same shrug.

The corollary, less comfortable: the original code was not sloppy. It asked clearly, it named the fields, it even gave an example object. It was a well-written version of the wrong mechanism, and no amount of further rewording would have found the bug, because the bug was a number on a different line.

Verified

The fix was run against a real client document with a real model, not a scripted one: 11 findings, 1 blocking, each carrying its sheet number, the rubric rule it broke and the edit that answers it. That is the run that would have produced nothing an hour earlier.

The receipt that reaches the thread has one fixed shape, and it is deliberately directive at the end. Here it is on an invented document, so that the format is legible without quoting anything from a real one:

Red team (claude-sonnet-4-6, 12 sheets): 1 blocking · 3 to fix · 2 to consider.
1. [BLOCK] sheet 9 · GROUNDING: the savings figure on the summary sheet is presented as a
   projection with no source in the room. → Mark it as a placeholder to be baselined, or cut it.
2. [FIX] sheet 4 · SCOPE IS UNAMBIGUOUS: the integrations list reads as included; two of the
   four are later phases. → Move them under "comes later" with the same heading weight.
3. [CONSIDER] sheet 12 · ASSUME THE YES: the closing line offers a way out before the ask.
   → Cut the hedge and end on the date.
Keep: the measurement sheet is the strongest thing here — the fixes must not flatten it.
A BLOCK is not sent. Fix each with edit_document by sheet, then run red_team_document again;
a finding that appears on a second document is a rule — add it as a learning so the next
proposal starts without it.

The open problem: the loop does not converge

Here is the part that is not solved, and it matters more than anything above.

The rubric is ten rules — grounding, no outcome promises, honest placeholders, the client's words as the spine, unambiguous scope, assume the yes, one well-framed decision, commercial clarity, no chatbot register, structure. A long document breaks several of them in several places. The review is one model call over the whole document, and it is asked for at most fifteen findings on a document that could honestly yield forty.

So two consecutive passes over the same unchanged document do not return the same set. They return two overlapping samples of a larger population. Fix everything the first pass named, run it again, and the second pass finds different things — not because the fixes failed, but because there was always more, and the cap plus the model's own sense of what matters most chose differently the second time.

Which means the obvious loop — review, fix, review again, ship when it comes back clean — does not terminate. There is no fixed point. An operator following the receipt's own instruction faithfully can rerun forever.

We are not going to solve that by asking for more findings; forty findings on a proposal is not a review, it is a refusal. The design question we are actually sitting with is whether the gate should block only on findings that recur — run the review twice, intersect the two sets, and treat agreement across independent passes as the signal that a problem is real rather than merely available. A finding both passes surface is a property of the document. A finding only one pass surfaces may be a property of the sampling.

That doubles the cost of every review and it has not been built. Stating it here because the alternative is shipping a "run it again until it is clean" instruction that we know cannot be satisfied, and letting operators discover the treadmill themselves.

One more thing worth saying plainly: the review is one call with no retry loop. A failed parse returns status: 'skipped' and stops. Retrying inside the tool would have papered over exactly the bug this post is about — two attempts, both truncated, both discarded — which is what the agent ended up doing by hand anyway.

Where this lives in the repo

vocion-core#457 (699df5ca), on top of #455 (91af4a40).

Not verified: the stop_reason === 'max_tokens' branch of the receipt has not fired against a live model since the budget moved to 8000 — it is exercised by a test with a stubbed response, not by a real truncation. Nobody has managed to truncate the new budget yet, which is the point, but it does mean that particular sentence has only ever been read in a test.

Related: A proposal that reads itself as the buyer, The prompt that produced the tells it forbade and The parameter Claude 5 will not take — three posts in a row about the same lesson arriving from three directions.