VOCION
Get a demo
← All posts

Extraction leaves the agent loop: processors in the sync, a feed-aware web connector, a sources API

Release notes 2026-09-15 — vocion-core#319 (merged): a knowledge source may declare a processor that runs once per changed document inside the sync, with its own budget and failure scope; the web connector reads feeds and URL lists; sources and budgets get v1 routes.

Vocion Teamvocion-v2.69.0
sourcesconnectorsretrieval

Shipped: vocion-core#319, merged to main on 2026-09-15. Part of the 2026-09-15 release notes.

The outcome

Extracting structured candidates from documents — events from a venue's pages, say — used to happen inside an agent turn, which meant it happened when someone asked, cost an agent's context, and was hard to budget. Now it happens once per changed document, inside the sync that fetched it, under a cap, and the agent reads the result.

What it is

Four pieces, reviewable commit by commit in the PR.

Processors in the sync. A knowledge source may declare processor: in its manifest. The sync runner calls it once per created or updated document. The registry is lazy (slug → import()), so the model stage never enters the Temporal worker's static import graph (a test enforces this). Each sync carries a SyncBudget; processor failures land in their own failure scope with their own cap — they never reject the ingest, never hold the watermark, and never spare a document from tombstoning. Counts keep the six existing keys byte-identical and add extract.*, processorErrors and capHits. Off by default.

The candidate extractor. One bounded model call per changed document. The prompt carries the venue's known upcoming cards — pending, failed and done, next 60 days — so the model can say seriesOf or duplicateOf with context, and core accepts only ids it sent. Output is a proposal into the review queue, like any other agent action.

A feed-aware web connector. urlsFrom (remote URL lists, split into one document per item), crawl.include / exclude, feed discovery, structure on HTML extraction, and page metadata — parsed with cheerio, keeping JSON-LD, images, links and timestamps (packages/core/src/libs/sources/).

v1 routes. GET/POST /api/v1/sources, POST /api/v1/sources/:slug/sync, and GET /api/v1/budgets for agent budgets. The upsert is idempotent; if the row is written but Temporal refuses the schedule the route answers 503 SCHEDULE_UNAVAILABLE rather than pretending.

Tenant impact

With no processor declared: one if per document, no import, counts unchanged. Existing web sources see one metadataRefreshed spike on the next sync because page metadata is new. A source whose urls already point at a JSON array is split once into one document per item. A dashboard Save on a source written through the API drops the API-only keys it does not show (urlsFrom, feedUrl) until the next API upsert; reserved _-prefixed keys survive.

Docs: docs/entities/source.md.

Where this lives in the repo