How 33 Characters Froze Production Migrations for Weeks

For several weeks, every deploy of a production service reported success. For those same weeks, not a single database migration actually ran.

The gap between those two sentences produced scattered runtime failures that each looked like an application bug, a pile of unapplied schema changes, and eventually a database that had been manually told to lie about its own state.

The root cause was a revision ID that was 33 characters long.

The 33-character revision ID

Alembic — the standard migration tool for SQLAlchemy — records the currently applied migration in a table called alembic_version, in a column defined as VARCHAR(32).

Alembic's own autogenerated revision IDs are 12 hex characters. But nothing forces you to use them, and descriptive revision IDs are a popular convention — a readable slug tells you what the migration does. One of those slugs came out to 33 characters.

The migration itself executed fine. Then Alembic tried to record it as applied — an INSERT into a VARCHAR(32) column with a 33-character value — and the write failed. One character over.

From that moment, the version table could never advance. Every subsequent alembic upgrade head failed the same way, forever. The migration system wasn't broken loudly; it was wedged.

The failure that was told not to matter

alembic upgrade head || echo "WARNING: migration failed"

A wedged migration on every deploy should have set off alarms within the hour. It didn't, because the deploy entrypoint contained this:

That || echo means: if migrations fail, print a line into a log stream nobody reads, then keep deploying. The deploy stays green. The orchestrator is happy. The dashboard is happy.

So for weeks, new application code shipped on top of a schema that was frozen in the past. Code referencing new columns failed at runtime — but only on the specific paths that touched them, in scattered, intermittent ways. Each failure got debugged as an application bug, because the deploy pipeline said everything was fine.

Then it got worse

Eventually someone noticed the migration state was "out of sync" and reached for the standard-looking fix: alembic stamp head — force the version table to claim the database is fully migrated.

Except the database wasn't fully migrated. Columns were missing. Stamping head didn't fix the drift; it made the drift invisible to the one tool whose job is to track it. The version table now asserted a state the schema was not in, and every future migration decision would be made on top of that lie.

Why AI assistants specifically

Three AI signatures, one incident.

**Descriptive revision IDs.** Assistants name things verbosely — it's one of their genuinely good habits, right up until the name flows into a VARCHAR(32) column no one remembers exists.

**The || echo guard.** Ask an assistant to make a deploy script "more resilient" and it will happily wrap the risky step so failure doesn't block the deploy. That converts a hard failure into an ignored one. Resilience against the wrong thing.

**stamp head as a fix.** When migration state is out of sync, force-stamping is the top search-result-shaped answer, and assistants reproduce it confidently — without first verifying that the schema actually matches the revision being stamped. It's the exact difference between fixing the problem and silencing the check.

What the right version looks like

set -euo pipefail
alembic upgrade head   # fails the deploy if it fails — that's the point

The deploy step should be the loudest thing in the pipeline, not the quietest. Migrations either apply cleanly or the deploy stops — there is no useful third state:

And when migration state genuinely drifts, the fix starts with comparing the actual schema against what the migrations say it should be — not with stamping. Stamp head is only ever correct after you've verified the schema really is at head. Run alembic's autogenerate against production and read the diff; if it proposes changes, the database is not at head, and stamping it there just buries the evidence.

MIG001 and MIG002 exist because of this

This incident is where two shipped StableStack rules came from.

**MIG001 (alembic-revision-id-too-long)** flags any revision ID longer than 32 characters in your migration files — at lint time, before it ever meets the VARCHAR(32) column in production.

**MIG002 (migration-failure-masked)** flags deploy scripts and entrypoints where a migration command's exit code is swallowed — the || echo and || true patterns — along with force-stamp usage that can paper over real drift.

That's how every StableStack rule starts: a real incident, distilled into a check that fires in seconds instead of weeks.

pip install stablestack