VOCION
Get a demo
← All posts

A sync that tells you what it did: honest caps, a real dedup key, and a crawl that reads the listing first

Eight fixes to the web connector and the candidate extractor, each found by running a real sync and reading the log. Listing-first crawl ordering, feed discovery scoped to the listing path, caps sized for one model call per document, a dry-run line that carries the dedup key, and a record that can no longer flag the card it is refreshing. Release notes 2026-09-16 — vocion-core#356, #362, #364, #365.

Vocion Teamvocion-v2.82.0
connectorssourcesextractionretrievaloperations

Status: shipped. Four pull requests — #356, #362, #364, #365 — merged to main and carried by v2.81.0 through v2.82.0. No migrations, no new environment variables. Part of the 2026-09-16 release notes.

Extraction left the agent loop a day earlier: a sync fetches documents and hands each one to a processor, and the candidate extractor turns a page into typed records for the review queue. This is what happened when it was pointed at real listings and the runs were read line by line — four passes, eight commits, one theme. A sync should be honest about what it did. Every number below came out of an actual run.

The crawl was reading the sidebar

A listing page publishes two kinds of link: the few that go to the things it lists, and the site chrome that goes everywhere. The crawler queued them in page order, which on most content-managed sites puts the navigation menu first. On one venue's calendar, a depth-1 crawl with a 60-page budget spent roughly 57 pages on menu destinations; three were calendar pages, and the single month page it reached was in the past.

#364 partitions a page's links before they reach the frontier, in libs/sources/web.ts:

(isUnderListingPath(next, start) ? ownPath : elsewhere).push({ url: next.toString(), depth: depth + 1 });
…
for (const batch of [ownPath, elsewhere]) {
  for (const entry of batch) {
    queue.push(entry);
  }
}

isUnderListingPath compares paths only — the query string is deliberately excluded, which is what makes …/calendar-of-events?month=10&year=2026 the listing's own page while …/services/notary is not. A trailing index, index.html or index.php is stripped first.

The property that matters is that this is a stable partition, not a filter. The include/exclude rules, the same-origin check and the visited set still decide membership; being under the listing path is not a pass. With enough budget every same-origin link is still fetched exactly once, in a better order. Two of the three new tests in web.crawl.test.ts assert precisely that.

A feed has to describe the listing it was found on

Feed discovery reads <link rel="alternate">. On WordPress, every page declares the blog's feed — so a calendar listing handed the connector a blog feed, the connector preferred it (a feed is cheaper and cleaner than a crawl), and the whole source became one document of about 1,460 characters of blog posts and zero events while the listing sat unread.

#362 adds describesTheListing(candidate, listingUrl) as a filter before the existing preference sort. An ics candidate passes unconditionally — a calendar feed is a calendar feed wherever it lives. A different origin is rejected. Otherwise the feed URL must be at or under the listing's path.

Rejection is not silent: it emits a run note — source: skipped rss feed outside the listing path <url> — on the same progress channel the rest of the sync reports on, so the row says a feed was found and why it was not used. With no eligible candidate the source falls back to listing-plus-crawl as before. The cost is stated rather than hidden: a legitimate cross-origin syndication feed now falls back, visibly, in the note.

A listed URL beside a crawl block is a seed, not a document

The sync read its config in the wrong order. If anything was listed in urls or resolved through urlsFrom, each entry was fetched as one document and the function returned — before feed discovery, before the crawl. A tenant resolving listings through a registry and configuring a crawl got one document per source: the raw listing HTML.

#356 reorders it. With no crawl block the legacy path is byte-for-byte unchanged. With one, a configured feedUrl wins and otherwise every listed URL becomes a listing seed; crawl.startUrl is the seed only when nothing was listed and no registry was configured, and an empty registry answer yields nothing rather than falling back to startUrl — that is how a disabled row spends nothing.

That creates a budget hazard worth naming. crawl() owned a local page counter, so one seed per listed URL would have made maxPages per seed — five seeds at 60 pages is 300 requests a run. The counter moved out into a PageBudget built once per sync and threaded through every seed, and a seed arriving with the budget spent is refused with a note rather than quietly starting. One of the six new tests is named spends ONE page budget across every seed, not one each.

Caps sized for what a document actually costs

SYNC_BUDGET_DEFAULTS was sized when a document meant a listing page. Under the processor pipeline a document is one feed entry or one detail page, so a healthy sync needs about one model call per document.

CapUnitBeforeAfter
maxModelCallsmodel calls per sync25150
maxInputTokensPerSyncinput tokens per sync120,000800,000
maxInputTokensPerCallinput tokens, one call10,00010,000
modelTimeoutMsms, one model call20,00060,000
maxProposalsPerSyncproposals per sync120120
maxWallClockMsms, processor side of a sync600,000600,000

Both raises were measured. The second pass over three sources ran 51 model invocations averaging ~3,300 input tokens — well under the 10,000 per-call ceiling — but one source crawled 59 detail pages, spent all 25 calls, hit a cap 34 times and skipped 44 documents. That set 150 calls and 400,000 tokens. The fourth pass found 400,000 still wrong for a first sweep of a large venue: 117 documents at ~4,500 tokens each meant tokens ran out after 88 of 150 calls, leaving 29 documents unread. 800,000 is 150 × the measured 4,500, and the reasoning is about ordering as much as size — it puts the token cap behind the call cap, where it belongs: the call cap is the one meant to stop a runaway source, and a sync should end because it ran out of pages. The figure is explicitly a first-sweep number; a steady-state sync re-reads only what changed.

The 800k is per sync run — not per document, not per workspace. createSyncBudget() is called once per run, and the docstring says why: up to MAX_CONCURRENT_INGESTS documents are in flight at once, so a per-document allowance would be eight times what it reads. Tokens are taken before the await, alongside the call count. A source manifest may author its own limits, and the merge is lower-only — caps[key] = Math.min(caps[key], authored) — so a manifest asking for more gets the default. A refusal is never an error: it bumps a counter and emits the sync's <cap> budget is spent, so the processor did less, once per distinct cap so the same sentence forty times cannot crowd out the run row.

One honest gap, left in the PR rather than papered over: maxPages and maxDetailHops are declared on the budget, lowerable and tested, but nothing spends them. What actually bounds a crawl today is the source row's own crawl.maxPages. Wiring the budget cap in, or deleting the two dead ones, is left as a separate change.

Timeouts a healthy call can meet, and a budget the processor owns

The model deadline was a hard-coded 20 seconds. The first pass measured 6 invocations, 18.2 s average, 19.8 s maximum, no errors — every one finishing just under a limit about to cut them off. The file now states the principle: a cap set below what a healthy call costs is not a cap, it is a guaranteed failure, and it costs the retry and then the whole document.

The per-document budget also changed owner. It used to be one constant in the sync runner; it is now declared by the processor as documentTimeoutMs on DocumentProcessor<TConfig>, because only the processor knows what its document costs. The candidate extractor declares 150,000 ms — two model attempts at the 60 s default, the one ticket hop a record may ask for, and writing the proposals — and it lives in the eager half of the module specifically so reading it does not drag LangChain into the Temporal worker bundle. Precedence in the runner is env override → the processor's declaration → a 25 s floor for a processor that never said. maxTokens went 2048 → 4096 for a related reason: maxRecordsPerDocument defaults to 25 and a record carries a description, so a full answer did not fit — and a truncated answer is invalid JSON, which costs the retry and then the document.

The dry-run line carries the key the real write would use

dryRun is the rollout gate: the extractor does everything except write. Its log line carried the title and identity fields — enough to compare keys across runs, not enough to diff what would have been written. It now carries fields, confidence, the document, and the dedup key itself.

The key is the interesting part. candidateDedupKey is a new export of the live action, and the action's own dedupKeyFor hook was reduced to a one-line call into it. The dry run does not reimplement the key; it calls the one the real write uses, because a second implementation drifts the day either one gains a rule. The shape is objects.propose_candidate:<objectType>|<value>|<value>…, each value NFD-normalised, de-accented, lowercased, collapsed to [a-z0-9-] and truncated to 80 characters — with an empty value rendered none rather than skipped, so a named field the extractor left blank still takes its slot and a missing venue cannot silently merge two candidates.

A record can no longer flag the card it is refreshing

This one has a user-visible symptom, and it is the theme in miniature.

The prompt's <known> block lists the venue's queued cards so the model can spot duplicates and series. But a sync's job is to re-read the pages behind those cards — so the card for the record being extracted is normally in that very list. The fourth pass had the model answer duplicateOf <that card> for 58 records deriving the same dedup key as the card they were refreshing. Live, that writes possible duplicate of <its own run id> into the flag field and attaches suggestedDecision: 'reject' — a recommendation that a reviewer turn down the card being refreshed.

#365 adds one comparison at the top of the record loop:

function isOwnCard(card: KnownCard | undefined, ownKey: string | undefined): boolean {
  return card !== undefined && ownKey !== undefined && card.dedupKey === ownKey;
}

Identity decides it, not the id — the record has no run id yet, so the only available comparison is the key the refresh would write against the key the card is stored under. Both sides come from candidateDedupKey, so a rule added to the key is added to this guard the same day. It covers duplicateOf and seriesOf alike; a self-match clears the field, pushes an issues note, and bumps a counter surfaced on the run row as extract.self_match. The record keeps its ordinary key, so the refresh still happens — and a different card with the same title on another day is still labelled.

The prompt is deliberately unchanged: the known block is rendered once per call, before any record exists, so nothing in it can name a card a record has not been extracted into yet, and a general warning would cost tokens on every call to catch what one comparison catches for free.

What changed for an operator

Nothing to configure. Two behaviours changed inside the new scope: a source with a URL list, a crawl block and a feedUrl now uses the feed where the list used to win; and a cross-origin syndication feed declared on a listing page is no longer taken. Both say so in the run row. Run rows generally get more useful — the skipped-feed note, the spent-page-budget note, the per-cap "the processor did less" note, and extract.self_match all report what a sync did instead of leaving it to be inferred from a document count.

Where this lives in the repo

Not verified end to end. Each pull request reports its suite green — the connector and processor suites reached 437 tests across 23 files — with types clean and negative checks recorded (neutering the partition fails exactly the two new ordering tests; removing the fields from the dry-run line fails exactly its test). But the proof that the right feed is read and a large venue finishes its sweep is the next full run after the deployment pin moves, and that had not happened when this was written. The measurements above come from the author's own instrumented passes, not a published benchmark.