The Migration Harness: How We Make AI CMS Migrations Verifiable

TL;DR #

• An agent can produce the first working transform in a day. Making its output trustworthy across a real corpus takes most of the project. That’s the part that turns content migration into an engineering problem, not a copy-paste exercise.

• A CMS migration is harder to grade by default. Code arrives with a compiler and a test suite that do most of the judging. Content arrives with neither, so the judge has to be built before anything moves — especially when the work includes a website migration content checklist, CMS migration content cleanup, and SEO content migration requirements.

• When we read Anthropic’s code-migration playbook, the strange part was how familiar it felt. We already had the parity gate, the component map and the phase gates. Their post also revealed three gaps and we closed them with the same site migration content strategy we use on large replatforms.

• The harness is running a migration of more than 20,000 stories right now, and it is portable. Ten weeks in: around 30 task pull requests merged, none reverted, editorial content rendering from the new CMS behind a flag on staging. The generated corpus and the cutover are still ahead.

Sixty-Four Agents Is the Least Useful Part of the Story #

Sixty-four agents, 11 days, and about $165,000 in model usage. Those are the numbers people remember from Bun’s Zig-to-Rust rewrite, which its creator wrote up in July: 535,496 lines across 1,448 files, all 60,624 tests kept. They are also the least useful part of that story for a CMS migration.

We do not run an agent for every story. In our current migration, 97% of more than 20,000 stories are generated from a small set of templates. An agent writes the transform once, and a boring script repeats it over everything. An agent per story would be paying for creativity exactly where you want determinism.

The hard part is proving that the script has not quietly removed an inline link, changed a canonical URL, or dropped a field an editor depends on. We call the system that provides that proof the migration harness, and it turned out to be very close to what Anthropic’s own post spends most of its length on.

Their post barely dwells on the agent count. They validated the judge first: the test harness had to pass the original code and fail deliberately broken code, otherwise a green result meant nothing. Then a rulebook of how idioms translate, a dependency map to order parallel work, and a gap inventory of what needed a redesign. Then a stress test on about three files, output thrown away, rules kept. Only then came the fan-out, from a mechanical, resumable queue, with one implementer agent and two or more adversarial reviewers per unit of work. One line of theirs carries the method: “Make review adversarial and verification mechanical.”

Why Content Is Harder to Judge Than Code #

A code migration starts with most of its judge already built. The compiler rejects anything it cannot parse or type-check, and the existing test suite gives a verdict on behavior. Anthropic still had to sort and rewrite tests so they ran against both versions, but the suite was there to extend.

Content has no compiler and no suite to extend.

Move a page from one CMS to another and nothing objects. A rich-text paragraph can lose every inline link and still import cleanly. A slug can pick up a trailing segment and publish without complaint, while the old address, the one search engines have indexed, quietly starts returning a not-found page unless someone notices and maps it. That’s why CMS migration content work needs an explicit content audit for migration before the first document moves.

So the judge for an AI CMS migration gets built on purpose, around what the business depends on.

Rendered-HTML parity comes first: take a real document from the old CMS, run it through the converter, load it into the new one, and diff the rendered page against the original. We compare rendered pages, not fields, because that’s what readers and crawlers actually receive.

The comparison runs on normalized output, and every intentional difference has to be documented as a decision before the gate accepts it.

Frozen URLs come second. Google is explicit that permanent redirects preserve PageRank, although any migration can still cause temporary ranking fluctuations. In practice, the fragile part is the mapping: one missing URL among twenty thousand is easy to ship and hard to notice.

A published story therefore keeps its URL. The parity check covers both the URL and the canonical tag emitted by the new platform, so there’s no redirect map to get wrong.

Metadata and structured data come third, as part of that same comparison, so they can block the launch too.

What a Migration Harness Is #

A migration harness is the part of an AI CMS migration that decides what counts as progress. It migrates nothing itself. It lives as plain files in the same repository as the migration code, which is where the project state lives too, with no chat window or ticket board to consult.

It opens with a constitution: a short list of MUST and MUST NOT rules, each with the reason it exists written next to it. Everyone reads it at the start of every session, humans included, so no rule gets quietly relaxed.

The unit of work is one markdown file with one executable verification command, ending in one pull request that a person reviews and merges. A task with no runnable verification command cannot be started. One task at a time. It sounds slow, but it is still faster than untangling three half-finished ones.

“Done” has three parts and all three must hold: the pull request is merged, the verification command was executed and passed with its output pasted into the task file, and the working tree is clean. A summary of what the code does is not evidence of any of that, and neither is a green badge. GitHub will report a pull request as mergeable and clean while the build that matters never ran, simply because it was not in the required checks. We read the executed command, not the badge.

Underneath sits one fail-closed gate, one command, and this is roughly what it prints on a good day:

$ bash migration/verify.sh
> BOARD.md regenerated · 37 task files valid
> checks/*                                   OK
> schema: 0 identifiers over the 63-char limit (self-test: FAIL, as expected)
> shared types: no drift from the live config
> tests: 2,167 passed across 4 migration packages
> GATE GREEN (exit 0)

Red blocks all new work. The self-test line is there because a check that cannot fail is worse than no check, so every checker gets fed a deliberate violation on every run to prove it still catches one. Postgres, for instance, truncates identifiers at 63 characters and does it silently, so that limit was spiked in the first phase and has been checked on every run since.

The last piece is a decision log: 232 entries so far, each with the alternatives we rejected and why. A fresh session inherits the reasoning instead of re-arguing it.

We write the tasks, argue about sequencing, review every pull request and approve the merge. The agent claims one task, implements it, puts the result through adversarial review agents, and pastes executed output into the task file as evidence. The harness constrains whatever coding agent runs inside it, so the rules outlive any particular model. A detailed engineering write-up follows this article.

The Same Playbook, Applied to Content #

Built independently, for different material, the two line up closely.

The three gaps their post showed us all went into the decision log in July 2026. We now validate the judge with a negative control: the parity gate has to go red on a deliberately broken transform, say a dropped SEO field or a mangled slug, before any green on the full corpus counts. We stopped hand-patching migrated output; if a document comes out wrong, the transform gets fixed and the batch regenerated, because a hand edit is exactly what the next re-sync will overwrite. And the import is resumable by construction, so a second run does nothing. That last one had a side effect we did not plan for and liked most: the final content re-sync before cutover, which we had scheduled as its own piece of work, stopped existing as a separate mechanism. It is the same import command, run one more time.

Anthropic also published an open-source starter kit for code migrations. Nothing equivalent exists for content. We would have used it.

What the Judge Caught That We Could Not See #

The judge earned its keep on something small. The old site has a habit: a paragraph made entirely of bold text renders as a heading. We had decided, deliberately, not to carry that quirk into the new CMS. The editorial import ran clean, the pages rendered on staging, and every check we had at the time was green.

Then the parity sweep over the editorial corpus came back with 423 differences, and 161 of them had that one cause. Neither CMS considered a single document broken. We reversed the decision on measurement rather than argument, wrote it into the log, and the same sweep then caught the sub-case: a bold paragraph that was also a link still rendered differently, because the converter had changed the shape of the node underneath it. That is what a validated judge is for. A human reading 500 pages would have signed off.

Two Migrations, One Method #

The method scales down as well as up.

The small shape is our Prismic to Payload walkthrough, published in April 2026. More than thirty custom types and 1,417 documents moved through four stages behind a single command: export, generate, transform, seed. No TypeScript was written by hand. At that size, one re-runnable command is most of the harness you need.

The large, gated shape is a migration from Storyblok into Payload for a large programmatic-SEO platform, and it’s running now.

Two phases are closed and tagged. The first focused on risk spikes: localization as a day-one property of every collection, because retrofitting it later is brutal; the identifier-length limit above; and a converter from Storyblok’s ProseMirror rich text to Payload’s Lexical, proven against real documents through rendered-HTML parity before anyone trusted it. That work sits inside the larger site migration content strategy, so every template, redirect and metadata rule is checked in the same place.

The second phase built the editorial content model from the consolidated component map.

This is not a postmortem. The generated corpus and the public cutover are still ahead of us. What we can report today is narrower: two phases closed, around 30 task pull requests merged with none reverted, 2,167 automated tests and 232 logged decisions guarding the work, and close to 500 editorial documents rendering from the new CMS behind a feature flag on staging.

Next, the thousands of generated pages go through the same gated pipeline, with the same parity requirement and frozen URLs. For that pass, the website migration content checklist is not a document on a desk; it is the gate itself.

At cutover, the flag switches only the public site’s render source. During the bake period—the window after the flip when the old system stays ready—editors keep writing in the old CMS. The same import carries their late edits across, because a re-run only imports what has changed. AI-assisted content migration is useful here only because the rules around it stay strict.

If we need to revert, we flip the same flag back. No edits are lost.

What You Actually Get #

Our own analysis found that 70% of website replatforming projects go over budget, and the usual cause is scope nobody could see at the start. The overrun sits in the tail, in rework and re-imports, because nobody defined what “done” meant until the end. A harness moves that definition to the front, so every task carries its own proof and the last phase is a cutover, not a scramble. Our breakdown of enterprise CMS migration costs shows where the money goes.

For a client, this comes down to five things. Your migration starts as a risk-sequenced task board with the killers at the top and a verification command against each one. Nothing merges without a human review. Your editors keep the fields they use and your published URLs do not move. Both are constraints in the repository, not checklist items. The cutover is a flag flip, and for the whole bake period traffic can go back to the old CMS without losing an edit. And a fresh clone of the repository answers “where are we, what is next, and why”, so the project survives handovers, including the one where you decide to finish it in-house. That is the practical shape of a content audit for migration before the first batch moves.

Planning a CMS Migration? #

If you are migrating off Storyblok, Prismic, Contentful, WordPress, or any other content system, and want the process managed with a harness like this, fill out the form at focusreactive and select “Migration” as the subject. Let us know what you are migrating from and where to. We will reply within a day, schedule an intro call, and deliver a scoped proposal within two to three days.

After an audit, the first artifact a client gets is a risk-sequenced task board for their own migration — the same kind of board we use ourselves. That is how we run all of our CMS migrations, and it is the exact place we start when executing a move off Storyblok.