Logging and Observability for a Team of AI Agents

Observability for a team of AI agents is not "add a tracer" — that part is solved. It is deciding what to tag on every call so cost, latency, and behavior can be sliced by org, by agent, by feature, and by the mission or workflow that triggered it. Vocion's answer: every LLM call — an agent turn, a skill run, an eval judge, a feedback classification — goes through one helper that stamps userId, metadata.orgId, and tags feature:<name>, org:<orgId>, slug:<agentOrOperationSlug> onto a self-hosted Langfuse trace. Separately, every tool call a workflow makes is a Postgres row carrying agent_slug, lead_agent_slug, mission_run_id, workspace_sha, and the langfuse_trace_id that row's cost lives under — so a tool call, a mission run, and a dollar figure are one join away from each other.

This is for the engineer who already knows Langfuse, or something like it, exists. You've read the SDK docs. What they don't answer is the harder question: once more than one agent is running, what do you actually put on every span so a bill or an incident traces back to one agent, one team, one mission — not just "some LLM call happened somewhere." Vendor docs for Langfuse, LangSmith, or Arize explain their own API surface. None of them tell you which dimensions matter once you have a workforce instead of a single chatbot.

What is the closed set of "feature" dimensions?

Vocion answers the tagging question with a closed enum, not free strings. It lives at packages/core/src/libs/Langfuse/features.ts, and the file's own header comment states the rule plainly:

/**
 * Closed enum of "feature" dimensions stamped on every Langfuse trace.
 *
 * Adding a new feature MUST mean editing this file, not passing a free
 * string at the call site. That's what keeps the Langfuse UI's
 * `tags = feature:<name>` filter useful for slicing cost / volume by
 * surface (chat vs. operation vs. eval).
 */

The enum itself declares twelve values: agent.chat, agent.dev, operation.run, eval.judge, workflow.step, feedback.classify, chat.chip-synthesis, source.oauth, retrieval.search, retrieval.embed, retrieval.ingest, retrieval.rerank.

Declaring a feature and wiring it are two different things, and it is worth checking which is which before you build a dashboard filter around one. Grepping packages/ for FEATURES.<NAME> call sites against the current tag shows eight of the twelve actually stamped on a trace today: AGENT_CHAT (services/AgentService.ts), EVAL_JUDGE (services/EvalService.ts), FEEDBACK_CLASSIFY (services/feedback/classifier.ts), CHIP_SYNTHESIS (services/chat/synthesis.ts), and the four retrieval features — RETRIEVAL_SEARCH (services/RetrievalService.ts), RETRIEVAL_EMBED (libs/retrieval/embedder.ts), RETRIEVAL_INGEST (services/IngestionService.ts), and RETRIEVAL_RERANK (libs/retrieval/reranker.ts). Four are declared with no call site in the codebase as of this version: AGENT_DEV, OPERATION_RUN, SOURCE_OAUTH, and — the one worth flagging explicitly since it's the one people ask about when they see "workflow" in a tagging doc — WORKFLOW_STEP. It is a reserved slot in the enum, not a live trace. No workflow step currently emits a workflow.step span. Treat any reference to per-step workflow tracing as forward-looking until a call site shows up.

What is the actual trace shape?

The tagging contract is one function, traceFor, documented in its own comment in packages/core/src/libs/Langfuse.ts:

/**
 * Create a Langfuse trace with the standard Vocion tagging shape:
 *
 *   name      = `${feature}:${slug}`
 *   userId    = caller-supplied (never undefined — use 'system' /
 *               'worker' / 'eval-runner' / 'mcp' for non-interactive)
 *   metadata  = { orgId, feature, slug, ...callerMetadata }
 *   tags      = [`feature:${feature}`, `org:${orgId}`, `slug:${slug}`]
 */

Every call site supplies a feature from the closed enum, a slug (agent, operation, or dataset identifier), an orgId, and a userId that is never left undefined — non-interactive callers pass a fixed string like 'system' or 'eval-runner' instead. That single rule is what makes an org filter, a user filter, and a feature filter all work against the same trace without special-casing background jobs.

How does a workflow tool call link back to cost?

Langfuse traces answer "what happened in this call." They don't answer "which mission run made this call, and what did it cost against that mission's tool budget." That join lives in Postgres, in the tool_call table (packages/core/src/models/Schema.ts, pgTable('tool_call', ...)). Relevant columns:

• agent_slug — the agent that made the call (the delegated specialist when nested, never the lead acting on its behalf).
• lead_agent_slug — the dispatching lead, when a delegated specialist made the call.
• mission_run_id — the mission run this call belongs to.
• provider — which harness executed the call: local, agentcore, or runtime.
• langfuse_trace_id — the trace this call's cost and latency live under.
• workspace_sha — the context version SHA active when the call executed.

Three indexes back the common queries: tool_call_org_created_idx (org, created_at), tool_call_org_agent_idx (org, agent_slug), and tool_call_org_tool_idx (org, tool). A tool call row names its mission run and its Langfuse trace in the same row, so you don't reconstruct that join from timestamps.

What does per-agent cost attribution look like day to day?

/dashboard/observability is a real page (packages/core/src/app/[locale]/(auth)/dashboard/observability/page.tsx), not a mockup. It's deliberately thin — the comment in the file calls it "a launch pad with three numbers that match the day-to-day questions... + saved-filter deep-links into Langfuse." Those three numbers are "Spend this period" (summed across listAgentBudgets), "Runs (last 24h)" (tool calls plus workflow runs, via ObservabilityService.countRunsLast24h, which counts rows in tool_call and workflow_run since now - 24h), and "Active agents" (agents with non-zero spend in the current period). Below that sits a "Top agents by spend" table, sorted by currentCents descending, each row linking out to Langfuse pre-filtered by slug:<agentSlug>. The page also renders two one-click buttons that build a Langfuse URL filtered by org:<orgId> and by name: 'agent.chat' or name: 'operation.run' respectively (FILTER_ENCODE in the same file) — useful shortcuts, with the caveat above that operation.run has no current call site, so that particular saved filter returns nothing until one exists.

How do budgets cap runaway spend before it happens?

BudgetService.preflightCheck (packages/core/src/services/BudgetService.ts) runs before an agent turn and checks a per-agent, per-period (daily or monthly) budget row for a hard token or cent ceiling. If no budget row exists for that org/agent/period, it returns { ok: true } — budgets are opt-in, not a default cap. chargeUsage runs after, from the Langfuse callback's on_chat_model_end hook, incrementing the period's token and cent counters; its own doc comment notes it is "no-op (returns silently) when no budget row exists yet." Period rollover is checked lazily on charge, comparing UTC year/month/day against periodStartedAt, so nothing needs a cron tick just to reset a counter.

How do you self-host the tracer?

One command. The root docker-compose.yml includes infra/docker-compose.platform.yml, so docker compose up -d brings a full Langfuse stack online next to the app: web, worker, its own Postgres, ClickHouse for trace storage, Redis for the ingestion queue, and MinIO for large payload blobs. A stack with an OpenTelemetry Collector container also ships in that same compose file (otel-collector, listening on 4317/4318); as of this version we did not find application code sending spans to it, so treat it as present in the stack rather than as a wired tracing path. First boot takes 60-90 seconds while migrations run. Once it's up, verify with:

npm run langfuse:smoke

This creates a trace, a generation, and a span, flushes them, then polls /api/public/traces/<id> until the row lands in ClickHouse, exiting non-zero on failure — safe to drop into a dev:up hook or CI. Reasonable sizing, from infra/README.md: Postgres 1GB RAM for small demo datasets, Langfuse 4GB (ClickHouse plus its own Postgres), Temporal 2GB, roughly 8GB+ recommended for the full stack. The complete boot sequence, including the auto-init project defaults and the one-time npm run langfuse:bootstrap step for pricing newer models, is in /docs/guides/observability — read that for setup, this page for what to tag and why.

Related

An adjacent question is not "what did this call cost" but "what version of the agent produced this output" — see AI Agent Versioning and Audit Trail, which covers the workspace_version rows and the same workspace_sha column that shows up on every tool_call row above. Both pages answer "trace this output back to what produced it," one by cost and call trace, the other by version.

If you're setting up a new agent workforce, decide your tagging dimensions before your first production run, not after the first unexplained bill.