Deploying to production
The Vercel build pipeline, the pre-build migration step, environment promotion, and the pre-deploy checklist.
ErmisAI deploys to Vercel as a single project in a single region. The deploy itself is one command with a migration step in front of it, so the interesting failure modes are not build failures — they are configuration that fails silently, and a schema that lands ahead of the code.
What vercel.json declares
Four things, and nothing else.
{
"regions": ["arn1"],
"buildCommand": "pnpm run vercel:build",
"functions": { "…": { "experimentalTriggers": [{ "type": "queue/v2beta", "topic": "…" }] } },
"crons": [ … ]
}Region is a single entry, arn1 (Stockholm). VERCEL_REGION is read at runtime and attached to every OTel span as deployment.region, falling back to local (src/instrumentation.ts:40-46).
Five crons, all authenticated by Authorization: Bearer $CRON_SECRET:
| Path | Schedule |
|---|---|
/api/admin/ai/metering/flush | */2 * * * * |
/api/admin/ai/ledger/retry/flush | */2 * * * * |
/api/admin/rss/refresh | */10 * * * * |
/api/admin/state/cleanup | 15 3 * * * (03:15 UTC) |
/api/admin/erasure/purge | 45 2 * * * (02:45 UTC) |
Four queue triggers of type queue/v2beta, binding topics rss.ingest, rss.enrich, rss.cluster and rss.synthesize to src/app/api/queues/rss/{ingest,enrich,cluster,synthesize}/route.ts.
The triggers are declared for all four topics, but only rss.ingest has an implementation, and it only does work when the rssQueuePipeline feature flag is on — which it is not by default. The other three consumers throw on purpose. Deploying does not turn any of this on. See Scheduled jobs and the queue subsystem.
The build command
pnpm run vercel:build
# = tsx scripts/deploy/prod-migrate.ts && NODE_ENV=production next buildNODE_ENV=production is pinned in both build and vercel:build (package.json:11-12). A shell-exported NODE_ENV=development leaking into next build otherwise breaks prerendering.
The build performs no type checking at all. next.config.ts:112-119 sets typescript.ignoreBuildErrors: true, because Next's build-time check goes through the TypeScript programmatic API that the native TS 7.0 compiler does not expose. Type safety comes from pnpm typecheck in the CI quality job, not from the deploy. If CI is red, the Vercel build will still go green. See Scripts, quality gates and CI.
The migration gate
scripts/deploy/prod-migrate.ts runs before next build and decides what to do from VERCEL_ENV alone:
VERCEL_ENV | Behaviour | Log line |
|---|---|---|
| unset | skip | [prod-migrate] Not a Vercel build (VERCEL_ENV unset) - skipping migration. |
anything ≠ production | skip | [prod-migrate] VERCEL_ENV=preview - skipping migration (production deploys only; preview builds must never migrate the production database). |
production | run pnpm exec drizzle-kit migrate | [prod-migrate] VERCEL_ENV=production - applying Drizzle migrations before build… then [prod-migrate] Migrations applied successfully. |
On a production build with neither SUPABASE_DATABASE_URL nor DATABASE_URL set, the script exits 1 and the deploy fails. It does not look at MIGRATIONS_DATABASE_URL (scripts/deploy/prod-migrate.ts:36-42). A non-zero exit from drizzle-kit migrate aborts the build too — nothing deploys.
That is deliberate. Production has no in-memory fallback for shared state, so a deployment serving traffic against an unmigrated platform_state throws on every state read.
Migrate-then-build means the schema can get ahead of the running code. If the migration succeeds and next build then fails, the migration is already applied and the old deployment keeps serving against the new schema. Every migration must be expand/contract for that reason — add nullable-or-defaulted columns first, drop only after no deployed code reads them. The full authoring rules are in Migrations and schema changes.
Production, preview and local
These behaviours are keyed on VERCEL_ENV:
| production | preview | local build | |
|---|---|---|---|
| Drizzle migrations run | yes | no | no |
ERMIS_FORCE_INMEMORY_STORE honoured | no | no | yes |
| Launch-config stderr block at boot | yes | yes | no |
Sentry environment | production | preview | from NODE_ENV |
isInMemoryStoreForced() returns true only when ERMIS_FORCE_INMEMORY_STORE === 'true' and VERCEL_ENV is unset (src/lib/platform/shared/persistence.ts:16-22). Any Vercel deployment, preview included, is hard-gated to durable storage. The launch-readiness warning is gated on NEXT_RUNTIME === 'nodejs' && process.env.VERCEL_ENV so it logs once per deployment rather than per edge region (src/instrumentation.ts:90-96).
A separate set of behaviours is keyed on NODE_ENV, not VERCEL_ENV, which matters because Next sets NODE_ENV=production for any built deployment:
- A direct (non-pooled) Postgres URL throws at client construction rather than falling through to the next candidate variable (
src/lib/db/client.ts:32-48, 97-104). - The CSP
script-src/connect-srcClerk origin ishttps://clerk.ermisai.comin production and the dev instance otherwise (src/proxy.ts:95-98). - Permissive CORS headers are development-only (
next.config.ts:50-61), and the SSRF guard only allows loopback targets outside production.
Preview deployments never migrate. A preview branch carrying a new migration runs its new code against the un-migrated production database. If it needs the schema, someone applies it manually first — and that application then lands in production ahead of the code, which is the other half of why expand/contract is mandatory.
Environment variables apply to the next deployment
Vercel applies environment changes only to new deployments (docs/ops/launch-checklist.md:44-45). Setting a variable does not fix a running deployment; set it, then redeploy. The full variable inventory is in the environment variable reference — .env.example in the app repo is more complete than README.md.
Launch-critical configuration
Ten keys fail silently when unset: the build, the typecheck and the tests all stay green while a capability is simply off. src/lib/platform/launch-readiness.ts:28-103 is the single source of truth for the list.
| Key | Category | What is silently off when unset |
|---|---|---|
BLOB_ARCHIVE_READ_WRITE_TOKEN | ops | Webhook payload archives (PII) fall back to inline DB storage; those rows are never pruned. Satisfied instead by SUPABASE_STORAGE_ARCHIVE_BUCKET + SUPABASE_SERVICE_ROLE_KEY when OBJECT_STORAGE_PROVIDER !== 'vercel-blob' |
CRON_SECRET | ops | Every cron 401s; AI overage never bills to Polar |
APP_URL (or NEXT_PUBLIC_APP_URL) | ops | WordPress OAuth callbacks and unsubscribe links resolve to the deployment origin |
SENTRY_DSN (or NEXT_PUBLIC_SENTRY_DSN) | ops | Server errors never reach Sentry |
RESEND_WEBHOOK_SECRET | The bounce/complaint webhook 503s; hard bounces are never suppressed | |
EMAIL_POSTAL_ADDRESS | Marketing sends hard-fail | |
EMAIL_UNSUBSCRIBE_SECRET | Marketing suppression throws in production | |
NEXT_PUBLIC_ERMIS_LEGAL_NAME | legal | Terms and privacy render "ErmisAI is operated by ErmisAI" |
NEXT_PUBLIC_ERMIS_PRIVACY_EMAIL | legal | Privacy and GDPR pages render a non-clickable circular fallback |
NEXT_PUBLIC_ERMIS_CONTACT_EMAIL | legal | The contact page has no clickable address |
They surface in two places. First, as a one-time stderr block at boot in any deployed environment, opening verbatim with:
[launch-readiness] WARNING: launch-critical environment variables are unset in this deployed
environment. These fail SILENTLY (build/typecheck/tests stay green while the capability is off):followed by one - <KEY> [<category>]: <impact> line per missing key, then Set them in the Vercel Production environment and redeploy. See docs/ops/launch-checklist.md.
Second, as the config block of GET /api/health, which is the authoritative check because it reads the live deployment rather than a gitignored local .env.production that Vercel never reads.
config.ready === false does not change the HTTP status. /api/health still returns 200. It is a go-public gate you have to read, not a probe failure.
/api/health semantics
Public, no-store, runtime = 'nodejs', dynamic = 'force-dynamic', every check capped at 3000 ms (src/app/api/health/route.ts). It is one of exactly two routes allowlisted as anonymous — in src/proxy.ts and in the auth-guard test's allowlist.
{
"status": "ok",
"checks": {
"database": { "status": "ok", "latencyMs": 0, "migrated": true },
"redis": { "status": "ok", "latencyMs": 0 }
},
"aiGuardrailsMode": "enforce",
"controls": { "aiGuardrailsMode": "…", "aiPaused": false, "signupsPaused": false,
"waitlistEnabled": false, "platformDailySpendCapCents": null,
"platformMonthlySpendCapCents": null },
"config": { "ready": true, "missing": [], "entries": [ … ] }
}- The database check runs
select 1and then probes theplatform_statetable. A failure there returnsmigrated: falseand the detailplatform_state unreachable (run db:migrate before serving traffic): <error>. - HTTP 200 when
checks.database.status === 'ok', 503 otherwise. That is the only input to the status code. - Redis is reported but never fails readiness;
not_configuredmeans no Upstash credentials are present. controlsexposes guardrail mode, the pause switches, the waitlist state and the platform spend caps anonymously. This is an intentional disclosure — key names and operator state, never values or secrets.
Post-deploy checks
Readiness
curl -s https://ermisai.com/api/health | jqstatus must be "ok" and checks.database.migrated must be true.
Launch configuration
curl -s https://ermisai.com/api/health | jq .configconfig.ready must be true. Any entry in missing means a capability is silently off; fix the variable in the Vercel Production environment and redeploy.
Cron authentication
curl -s -o /dev/null -w "%{http_code}" https://ermisai.com/api/admin/ai/metering/flush
# expect 401
curl -s -H "Authorization: Bearer $CRON_SECRET" https://ermisai.com/api/admin/ai/metering/flush
# expect 200isAuthorizedCronRequest() compares against Bearer ${CRON_SECRET} with timingSafeEqual and returns false when the secret is unset (src/lib/api/cron-auth.ts:46-65), so a missing variable can never leave a cron open — it closes it.
Triggers and crons in the Vercel dashboard
Confirm the four rss.* queue triggers are attached and that all five crons are live.
docs/ops/launch-checklist.md:84-87 says "all four crons" and lists four. vercel.json declares five — the GDPR erasure purge at 02:45 UTC is the one the checklist sentence omits.
AI spend protection
GET /api/health should report aiGuardrailsMode: "enforce". The database-persisted runtime config wins over the ERMIS_AI_GUARDRAILS_MODE env seed, so the live value is the only one that counts.
If it reads anything else, patch it as a super_admin:
curl -X PUT https://ermisai.com/api/admin/ai/config \
-H 'content-type: application/json' \
-d '{"aiGuardrailsMode":"enforce","aiMeteringEnabled":true}'The launch checklist says POST /api/admin/ai/config. The route exports only GET and PUT (src/app/api/admin/ai/config/route.ts:78, 89). Use PUT.
Walk the journey once
Sign up, complete onboarding, watch /app/feed populate, open a story, compose, submit for review, approve, then pull it through a delivery surface. Nothing else exercises the ingestion pipeline, the AI stages, the editorial state machine and delivery in one pass.
Observability wiring
The config is wrapped withSentryConfig(withBotId(withNextIntl(config))) (next.config.ts:441-472), org entro314, project ermisai.
- Source maps upload during the build when
SENTRY_AUTH_TOKENis present in the build environment. The CIbuildjob deliberately omits it so@sentry/nextjsjust builds (.github/workflows/ci.yml).pnpm sentry:sourcemapsis the manual path and needs a built.nextplus sentry-cli auth. - Cron monitoring rides on
_experimental.vercelCronsMonitoring: true. Thewebpackoption key —automaticVercelMonitors,autoInstrument*,treeshake— is a no-op under Turbopack and was removed;automaticVercelMonitorsalso never instrumented App Router route handlers. tunnelRouteis commented out (next.config.ts:458), so Sentry browser requests are not ad-blocker-proofed, even though/monitoring/sentryis already allowlisted insrc/proxy.tsand in the request-log ignore list. Uncommenting the line is the whole change.SENTRY_ENVIRONMENTfalls back toVERCEL_ENVthenNODE_ENV;SENTRY_RELEASEfalls back toVERCEL_GIT_COMMIT_SHA(src/instrumentation.ts:33-37).tracesSampleRateis 0.1 in production, 1 elsewhere.- Server OTel registers as
serviceName: 'ermisai'. Client-side Sentry starts only after the visitor accepts cookies, so browser errors from anonymous visitors who declined are not collected.
The marketing edge cache rotates per deployment
Cacheable marketing responses carry Vercel-CDN-Cache-Control and CDN-Cache-Control set to public, s-maxage=300, stale-while-revalidate=86400, and have Set-Cookie deleted, because Vercel will not cache a cookie-bearing response (src/proxy.ts:199-215).
Those pages therefore use a deployment-stable CSP nonce, base64("ermis-marketing-" + VERCEL_DEPLOYMENT_ID), falling back to local (src/proxy.ts:89-91). A per-request nonce would never match CDN-cached HTML and every script on those pages would be blocked. The value rotates on each deploy. Details in Public site, SEO and marketing routes.
What a deploy does not do
- Seed reference data.
billing_plansandsource_catalog_entriesare populated bypnpm db:seed-bootstrap, run by hand. Without it, onboarding and the source picker have nothing to show. - Sync the source catalog. Catalog entries reach Postgres only through the admin sync route or
db:seed-bootstrap— there is no cron and no deploy step. See Source catalog operations. - Change any flag or operator brake. Feature flags,
signupsPaused,waitlistEnabled,aiPausedand the platform spend caps all live inplatform_stateand survive deploys untouched. See Feature flags and runtime controls. - Migrate a preview deployment. Covered above; worth repeating because it is the most common surprise.
- Roll the schema back. Migrations are forward-only: each
drizzle/<timestamp>_<name>/directory holdsmigration.sqlandsnapshot.jsonand nothing else, and applied state lives indrizzle.__drizzle_migrationsin the database. Reverting to an earlier deployment reverts code only — the newer schema stays. That is the reason the expand/contract rule is not optional.
Migrations and schema changes
The drizzle-kit v1 layout and the rules that break migrations here.
Environment variable reference
Every variable the app reads and what breaks when it is unset.
Incident and recovery runbooks
Restore procedure, redrives, and per-dependency degradation.
Scheduled jobs and the queue subsystem
The five crons and the honest state of the queue pipeline.
Scripts, quality gates and CI
What must be green before merge, and what the build does not check.
Feature flags and runtime controls
Flags, operator brakes, and the ones with no UI at all.
