Deploying an agent platform without losing the schema

Status: shipped. All five practices below are running as of 2026-09-15, the same day the rail train deployed with migrations 0094–0099 applied.

An agent platform is a normal web application with an unusual amount of state: conversations, runs, traces, review items, learnings, alignment decisions. Losing the schema is worse than losing the app, because the app redeploys in four minutes and the state does not come back.

A note on sourcing before anything else. Two of the things below are public and you can read them: the migration conventions and the concurrent-index rule live in vocion-core, under packages/core/migrations/CONVENTIONS.md, enforced by npm run check:migrations in .github/workflows/CI.yml. Everything else — the deploy scripts, the backup step, the pin-bump automation, the workspace CI job, the runbook — lives in a private deployment repository. Those are described here as practices and reasoning, with no file paths, because the reasoning is the transferable part and the paths are not yours to use anyway.

1. Dump before you migrate, and abort if the dump is empty

Before any migration runs, the deploy takes a pg_dump -Fc of the application database and of the workflow engine's database. Two databases, because the platform's durable state is split across them and restoring one without the other leaves you with a half-consistent system.

Two properties matter more than the dump itself.

The deploy aborts if the dump fails. Obvious, and frequently not implemented — a backup step whose exit code is discarded is a backup step that will one day be discovered to have been failing for a month.

The deploy also aborts if the dump succeeds and is empty. This is the less obvious half and it is the one that catches real incidents. pg_dump can exit zero having written almost nothing: pointed at the wrong database name, pointed at a fresh instance a connection string typo created, run with a role that can see the schema and none of the tables. A zero exit is not evidence of a backup. A non-trivial file size is weak evidence, and weak evidence is enormously better than none.

The last seven dumps are kept. Seven is not a compliance number; it is "long enough that a problem noticed on Monday can be restored from Friday".

The compressed custom format is deliberate: it is what pg_restore takes, and a selective restore of one table is the thing you actually want at three in the morning.

2. Log migrations one file at a time, then check what they claimed to build

Migrations are applied one file at a time, and the log names the file. This is not a hand-rolled applier for its own sake — the migration tooling is a development dependency that is trimmed out of the production image, so running the ORM's migrate command inside the production container fails with a module-not-found error. Rather than fatten the image, the applier talks to psql directly and keeps its own ledger table of applied filenames.

The failure message tells you the recovery, which is the point of naming the file:

a file applied inside a transaction rolled back completely, so re-running it is safe;
a file using CREATE INDEX CONCURRENTLY could not be wrapped in a transaction, so it may be half-applied and needs a person to look before re-running.

Those are different situations and a log that says only "migration failed" leaves you guessing which one you are in.

The newer half is a post-migration check. After the applier finishes, the deploy re-reads the schema and compares it against what the newest migrations declared — and names any table or column that was declared but does not exist. This exists because of a specific failure shape: a migration file full of CREATE TABLE IF NOT EXISTS and ADD COLUMN IF NOT EXISTS is idempotent, which is good, and silent when it does nothing, which is not. A file that was recorded as applied against the wrong database, or whose statements were skipped because an earlier object collided, exits zero and leaves the application to discover the missing column at request time, in production, as a 500.

The check turns "the migration step said OK" into "the columns are there", which is the claim you actually wanted.

Migrations run before the containers roll. That ordering does not remove the schema-skew window; it picks which side of it to take — the outgoing release serves against the new schema for the length of the roll. That is only the safe half if migrations stay backward-compatible with the release being replaced: expand now, contract in a later deploy. A DROP COLUMN or a rename shipped in one step breaks the running app the moment it is applied, and keeps it broken if the roll then fails.

3. Bump the pin by pull request, every day, automatically

The runtime is pinned as a submodule. A scheduled workflow runs daily, compares the pin against upstream main, and when they differ opens a pull request that bumps it — with the list of commits being pulled in written into the body.

The value is not saving the thirty seconds it takes to bump a submodule. It is that the diff between what is deployed and what is on main becomes a thing that exists, with a name, in a review queue, once a day. Without it, that diff is invisible until someone goes looking, and it grows until the bump is a hundred commits nobody can meaningfully review — at which point the bump gets approved on faith, which is the opposite of what pinning was for.

A daily PR that nobody merges is also a signal. Three of them stacked up means something is wrong upstream or nobody is minding the deployment, and either is worth knowing.

It opens a pull request rather than pushing. A robot that can move production on a schedule is a different risk category from a robot that can propose a change.

4. Dry-run every workspace in CI when the pin, the workspaces, or the applier change

A workspace is YAML — agents, teams, trust rules, automations, a mailbox — applied into the database. Which means a change to the applier, or to the runtime the applier lives in, can break a workspace that nobody edited.

So CI dry-run-applies every workspace on any pull request that touches the pin, the workspace definitions, or the apply script. Not on every pull request: the job needs a migrated database and takes real time, and a check that runs when it cannot possibly be relevant is a check people learn to ignore.

One detail worth knowing if you build the same thing: a dry run still needs a reachable, migrated database. It writes nothing, but computing the diff means reading the current rows. Migrate first, then check — a documented claim that the dry run is database-free was wrong and had to be corrected.

5. A rollback runbook, written down

Two steps, in this order: re-pin to the previous runtime commit, then pg_restore from the newest dump.

The order is the whole runbook. Restoring the database under the new application code gives you an old schema serving a release that expects a new one, which is a worse outage than the one you were rolling back from.

Written down because rollback is the procedure you execute least often and under the most pressure. A runbook that exists is worth more than a better runbook that lives in one person's memory.

What this does not give you is point-in-time recovery. A restore loses everything written since the last dump, and the platform also takes a daily crash-consistent volume snapshot, which recovers the way a database recovers from a power cut. Point-in-time would need write-ahead-log archiving, which is a separate decision nobody has made yet. Saying so is more useful than implying a recovery capability that is not there.

The activity view after a deploy. The question after a deploy is not "did the container start" but "is work still running". Runs, checks, syncs and tool calls land here, which is where a migration that half-applied shows up as behaviour rather than as a log line.

The public half

Two things in this post you can read for yourself, because they live in vocion-core:

The migration conventions — packages/core/migrations/CONVENTIONS.md — and the CI job that enforces them. npm run check:migrations rejects a migration that would take a write lock on a populated table. That is the rule the CREATE INDEX CONCURRENTLY handling above exists to serve: an index built concurrently does not lock writes, and cannot run inside a transaction, and those two facts together are why the applier has a special case at all.

The step is deliberately ordered before the linter in the workflow, with the reason in a comment:

      # Refuses a migration that would take a write lock on a populated table.
      #
      # Ahead of the linter deliberately. `npm run lint` currently exits 1 on
      # this repo's standing errors, so any step after it never runs at all —
      # a migration check placed later would be wired up and silently dead.
      - name: Migration conventions
        run: npm run check:migrations

That comment is the most transferable thing here. A CI step placed after a step that reliably fails is not a check. It is a check-shaped comment in a YAML file. If your pipeline has a known-failing stage, everything downstream of it is decoration until you either fix the stage or move the checks in front of it.

Where this lives in the repo

Public, and citable:

Migration conventions: packages/core/migrations/CONVENTIONS.md
The CI gate and its ordering: .github/workflows/CI.yml in vocion-core
The migrations themselves: packages/core/migrations/ — the 0094–0099 range this release applied is landing on main behind the release branch

Deployment-side, and deliberately uncited: the dump-and-abort step, the post-migration schema check, the daily pin-bump workflow, the workspace dry-run job, and the rollback runbook all live in a private deployment repository. They are described above as practices because that is the part worth copying.