Incident and recovery runbooks

Everything here assumes you have the repo, the Vercel project, and a platform role in the operations console. Paths are relative to the app repo root.

First move: read the health probe

GET /api/health is auth-free and allowlisted in src/proxy.ts. It is the fastest way to tell an app fault from a dependency outage.

curl -s https://ermisai.com/api/health | jq

The response is { status, checks: { database, redis }, aiGuardrailsMode, controls, config } (src/app/api/health/route.ts:133-166). Read it like this:

Blast radius

Postgres is the dominant single point of failure and it is deliberately fail-closed. Production has no in-memory fallback for shared state: isLocalStateFallbackAllowed() returns false whenever VERCEL_ENV is set (src/lib/platform/shared/persistence.ts:24-34), so an unreachable or unmigrated database 503s every authenticated surface at once — feed, editorial, compose, billing, newsroom profile. A divergent in-memory store was judged worse than an outage (docs/ops/disaster-recovery.md:3-9).

The public marketing site is unaffected. Those routes are CDN-cached with Set-Cookie stripped, so they serve without touching Postgres or Clerk.

Two mitigations shorten the failure: the 10-second statement timeout means a saturated or locked database returns 503 fast instead of hanging a function to its 300-second ceiling, and /api/health distinguishes the two cases in one request.

Dependency degradation

The feed is empty or stale

Symptoms and what each one means

• Stories feed unavailable — HTTP 503, code stories_feed_unavailable. The route's internal timeout STORIES_ROUTE_TIMEOUT_MS = 95_000 fired (src/app/api/stories/route.ts:18-23). Almost always a cold synchronous whole-catalog build, not a hung database.
• No stories match your current filters. — the feed loaded and returned nothing. This string shows even with zero filters set.
• Stories present but hours old — the scheduled ingestion is not running.

Checks, in order

1. CRON_SECRET. isAuthorizedCronRequest compares Authorization: Bearer ${CRON_SECRET} with timingSafeEqual and returns false when the variable is unset (src/lib/api/cron-auth.ts:46-65), so a missing secret can never open the endpoint — it 401s every cron instead. With no cron there is no scheduled tenant ingestion, no alert sweep, and no metering flush; freshness silently degrades to whatever browser traffic triggers. /api/health → config lists it.
2. Source health. /admin/sources shows per-source ok / failed / empty parse / not polled, last polled time, 24-hour article count, and error rate. A source returning HTML instead of a feed reads empty parse.
3. Catalog reach. Only 13 of the 1,427 entries in news-sources/catalog.v1.json are both active and ingest-enabled, and those are the only sources the global build polls. If a sync deactivated rows, the feed narrows without any error.
4. Kill switches. ERMIS_DISABLE_AI_PIPELINE=true disables synthesis entirely; aiPaused blocks every AI invocation including the pipeline.
5. Pipeline diagnostics. /admin/ai → "News pipeline diagnostics" shows the last build's status, duration, article/cluster/story counts, and the last failure with its stage, model, and provider. It is an in-process, per-instance snapshot of the last global build; a cold instance renders "No diagnostics snapshot available yet." and a tenant-scope build never appears there at all.

Forcing a refresh

The tenant-facing control is the Refresh sources button on the feed, which calls GET /api/stories?forceRefresh=true and rebuilds only that tenant's scope.

Operator-side there is no button. Post to the cron route by hand:

curl -s -X POST https://ermisai.com/api/admin/rss/refresh \
  -H "Cookie: <your authenticated session>"

POST requires the sources admin surface and, with the queue flag off, runs listAggregatedStories({ forceRefresh: true }) inline — which refreshes the global catalog scope, not tenant scopes. Tenant ingestion happens only on the cron GET path, which runs ingestAllTenantScopedFeeds() serially under ERMIS_TENANT_INGESTION_TIME_BUDGET_MS (default 240000) with a rotating start offset, then sweeps alert rules (src/app/api/admin/rss/refresh/route.ts).

Deliveries fail

Webhook failures

Every dispatch writes a log entry whether or not a packet left the process (src/lib/platform/integrations/webhooks.ts:273-358). Read the code before assuming a network fault:

There is no retry queue, no backoff, and no dead-letter. A failed alert.triggered delivery is left out of the dedup set so the next alert evaluation re-attempts it; a failed webhook.test is never retried.

Two SSRF behaviours reject endpoints that look valid to the customer: validation is re-run at dispatch with DNS resolution, and the socket is pinned to the address that passed. Split-horizon DNS, a hostname resolving to any private or reserved address, and a load balancer that rotates addresses mid-flight all fail. Saving the configuration does not run the DNS check (requireResolvedPublicAddress: false, webhooks.ts:379-382), so a config can save cleanly and every delivery still fail with 400.

A fleet-wide delivery cap drop

If many tenants report Story is currently queued. (HTTP 402, code story_queued) and the amber queued for delivery (daily cap resets at midnight UTC) banner at once, suspect the billing database rather than the plans.

The comment at src/lib/platform/local-platform-data.ts:1909-1913 says delivery capacity fails open to unlimited, but the resolver it calls swallows every error and returns individual_free (src/lib/platform/newsroom-preferences.ts:512-513), which maps to 20 deliveries per day. A billing read failure therefore drops every tenant to the Free cap silently. Check checks.database and the tenant_subscriptions reads before touching plan configuration.

What the delivery surfaces will and will not tell you

/admin/deliveries reads a bounded 500-entry cross-tenant recent-activity mirror, described in code as "recent-N, not a ledger" (src/lib/platform/shared/delivery-log.ts:5-18). It silently drops older entries. Exact per-tenant volume lives in the per-UTC-day counters retained for 35 days, and only successful deliveries increment them.

There is no email delivery. ALERT_EMAIL_DELIVERY_ENABLED is a hard-coded false (src/lib/platform/alerts/index.ts:25), so an EMAIL tile reading zero is correct, not a fault.

AI is blocked, or spend spikes

Reading the block

Guardrail errors are not localized, so the raw English message reaches the toast. The ones that indicate a platform-level condition rather than a tenant limit:

Both are evaluated first, before anything else, even when aiGuardrailsMode is disabled, and they apply to platform-scoped pipeline runs as well as tenant calls (src/lib/ai/usage-guardrails.ts:400-439). Every block writes a blocked row to ai_usage_ledger.

The platform spend read itself fails open: any error reading the aggregate returns null and the call proceeds (usage-guardrails.ts:374-398).

Throwing the brakes

aiPaused, signupsPaused, waitlistEnabled, and both platform spend caps are enforced but have no control anywhere in the admin UI. AiConfigPanel.save() never sends them, so a hand-set value survives a later panel save. Flip them with a direct PUT, gated on the ai surface (super_admin only):

curl -s -X PUT https://ermisai.com/api/admin/ai/config \
  -H "Content-Type: application/json" \
  -H "Cookie: <your authenticated session>" \
  -d '{"aiPaused": true}'

The same body shape accepts platformDailySpendCapCents and platformMonthlySpendCapCents (integer USD cents, or null to clear), aiGuardrailsMode (disabled | observe | enforce), aiMeteringEnabled, signupsPaused, and waitlistEnabled (src/lib/contracts/ai-runtime.ts:158-191). Confirm the result at /api/health → controls.

Guardrails mode matters during an incident: observe records would-block events without rejecting requests, which is the right setting when you suspect the guardrail itself is the fault. A deployed environment running with guardrails disabled emits a one-time stderr WARNING naming /api/admin/ai/config and ERMIS_AI_GUARDRAILS_MODE.

Billing state is stale

The Polar webhook acks HTTP 200 for every non-retryable processing failure by design; only isRetryableDatabaseServiceError produces a 500. Unmapped products, a missing or foreign customer.externalId, and malformed payloads are all acked. A tenant can be stuck on the wrong plan indefinitely while Polar's delivery log is 100% green. Do not treat that log as a health signal.

A past_due or canceled standing reverts the entire workspace to Free entitlements — zero monitoring rules, the Free source cap, 20 deliveries per day, every paid capability off, and tenant-billable AI hard-blocked. So "the customer says they lost everything" is usually a standing problem, not a data problem.

Triage:

1. /admin/tenants shows the tenant scope id, plan, monthly AI capacity, and billing standing. The envelope displayed is the one enforcement actually uses, so a past_due tenant shows the Free envelope, not its contracted one.
2. GET /api/admin/billing/webhook-events?status=ignored&limit=50 (and status=failed) lists what the receiver dropped. The same log is on /admin/costs with all | processed | ignored | failed filters.
3. Repair with a resync — replaying the archived payload would just reproduce the original outcome:

curl -s -X POST https://ermisai.com/api/admin/billing/resync \
  -H "Content-Type: application/json" \
  -H "Cookie: <your authenticated session>" \
  -d '{"tenantScopeId": "org:org_..."}'

This pulls the live Polar subscription and upserts it through the same idempotent path the webhook uses. A paid database row with no active Polar subscription is downgraded to canceled. It needs the costs surface, so ops can run it. The UI equivalent is "Resync tenant from Polar" on /admin/costs.

Restore procedure after data loss

Redrives and manual operations

None of these have a button except where noted. All are curl-or-runbook operations.

Both flush routes also expose a GET variant scheduled every two minutes and authenticated with the CRON_SECRET bearer. The requeue button on /admin/costs is a known cross-surface mismatch: the page is ops-accessible but the endpoint requires the ai surface, so an ops operator sees the button and gets a 403 toast.

Cron health

Five crons are declared in vercel.json, all pinned to region arn1:

Every GET self-authenticates with the bearer secret and every handler is wrapped in runGuardedCronHandler, which converts a throw into a typed 500 routed to Sentry. reportCronDrainFailures raises a Sentry error whenever a drain reports failures — that is what turns a silent metering backlog into a page. Sentry's cron monitors ride on _experimental.vercelCronsMonitoring in next.config.ts; the webpack.* Sentry options are no-ops under Turbopack, so do not expect automaticVercelMonitors to do anything.

Smoke-test the bearer path:

curl -s -o /dev/null -w "%{http_code}" https://ermisai.com/api/admin/ai/metering/flush
# 401 without a bearer

curl -s -H "Authorization: Bearer $CRON_SECRET" https://ermisai.com/api/admin/ai/metering/flush
# 200

Leading indicators of under-billing

Two backlogs mean tenant AI spend is real but invisible to envelope checks and to Polar:

• Metering outbox. Visible on /admin/ai under "AI usage operations" as Pending metering (with Oldest <datetime> or "No pending rows") and Failed metering. A growing Failed metering count is unbilled overage waiting for a requeue.
• Ledger retry queue. readAiLedgerRetryQueueDepth() exists at src/lib/ai/ledger-retry-queue.ts:322 but has no call site — nothing in the app or the admin UI reads it. To check depth, inspect the Redis lists directly: ermis:ai:ledger:retry:queue, ermis:ai:ledger:retry:dlq, ermis:ai:ledger:retry:processing.

Screens that are not monitoring

Four admin surfaces read as telemetry and are not. Knowing this saves an hour on a bad night.

• /admin/queues "Queue health" derives four cards from story phase counts and source health (src/lib/platform/admin/dashboards.ts:149-196). Retries and Dead letter are hard-coded 0 for cluster-queue, synthesis-queue, and review-queue, and three cards share the same lagSeconds. Nothing in the codebase reads Vercel Queue depth or the <topic>-dlq mirror topics.
• /admin/costs is an all-time GROUP BY over ai_usage_ledger with no date range, in USD, while tenant envelopes on the adjacent screen are in EUR.
• /admin/deliveries is the 500-entry recent-N mirror described above.
• The platform review queue returns [] when the editorial-draft repository is unavailable (local-platform-data.ts:2733-2737). An empty queue can mean "database down", not "nothing pending".