Inbound webhooks: Clerk, Polar and Resend

ErmisAI runs three inbound webhook receivers in production. Two of them (Clerk and Polar) keep durable Postgres state in sync with an external system of record; the third (Resend) feeds the email suppression list. docs/implementation/webhook-setup.md:5-9 is the in-repo registration checklist for all three.

All three are listed in isPublicRoute in src/proxy.ts:141-143, so Clerk's middleware does not challenge them. Each route then authenticates itself — that is the whole reason tests/api-route-auth-guard.vitest.ts accepts a signature check as a gate rather than requiring a session.

Behaviour common to all three

The shared status ladder before any business logic runs:

Clerk

src/app/api/webhooks/clerk/route.ts. It is the sync path for clerk_users, the organization snapshot columns on that table, and tenant_memberships.

Idempotency

The event id is the svix-id header when present, otherwise <eventType>:<sha256(rawBody)> (deriveClerkWebhookEventId, clerk/route.ts:25-38). beginClerkWebhookEventProcessing (src/lib/db/clerk-user-repository.ts:774-827) inserts a clerk_webhook_events row with status processing, unique on provider_event_id (src/lib/db/schema.ts:401-424).

On unique conflict the row is only re-claimed when the prior attempt is failed, or when a processing row is older than 15 minutes (STALE_PROCESSING_WEBHOOK_AFTER_MS, clerk-user-repository.ts:772). A processed or ignored row is a true duplicate: the conditional setWhere fails, RETURNING is empty, and the route answers 200 { received: true, duplicate: true } (clerk/route.ts:217-229).

Statuses are processing | processed | ignored | failed (clerkWebhookEventStatusValues, schema.ts:45-50).

Handled events

Nine event types are mapped. Anything else is recorded ignored with the reason Event type <type> is not mapped for user sync (clerk/route.ts:359).

Ordering that is load-bearing

A failure to open either erasure request does not fail the webhook. It emits clerk_webhook_erasure_request_failed telemetry so the miss is observable, and an operator can open the request by hand — see GDPR erasure runbook.

Ownerless-organization recovery

After any of the three organizationMembership.* events, the route calls recoverOwnerlessOrganization(orgId) (clerk/route.ts:307-311). Clerk-native surfaces — the <OrganizationSwitcher> "leave" action, dashboard role edits — bypass the app's interactive owner guards, so a sole owner can vanish without the app ever seeing the request. The recovery promotes the most senior remaining member, writes the membership public metadata before the org role (metadata takes precedence in role normalization, so it must never lag), and posts two notifications: "You are now the workspace owner" to the promotee and "Workspace ownership transferred" to the org inbox (src/lib/auth/clerk-organization-management.ts).

Membership titles are preserve-on-absent

upsertTenantMembershipSnapshot treats title: undefined as "keep the stored value" and only an explicit null or a string as a write (clerk-user-repository.ts:175-207). Both callers pass resolveTenantTitle(metadata) ?? undefined.

This exists because the member title is self-service — it is typed during onboarding and never mirrored into Clerk metadata. Without the preserve rule, any unrelated Clerk webhook would blank the title, which fails the hasCompletedNewsroomSetup predicate and bounces the member back into the onboarding wizard.

Responses and telemetry

Success returns 200 { received, duplicate: false, processed, eventType }, plus ignoredReason when the event was not mapped. An unexpected throw records the row failed and returns 500 so Svix retries (clerk/route.ts:385-403).

Telemetry lines are prefixed [clerk-webhook] and carry one of clerk_webhook_received, clerk_webhook_signature_invalid, clerk_webhook_duplicate, clerk_webhook_processed, clerk_webhook_ignored, clerk_webhook_failed, clerk_webhook_erasure_request_failed (src/lib/telemetry/clerk-webhooks.ts). Failures and erasure-request failures go to stderr as error; signature-invalid, duplicate and ignored go to stderr as warn; the rest to stdout.

Polar

src/app/api/billing/webhooks/polar/route.ts. This is the only path that changes a tenant's plan or subscription status. The post-checkout browser redirect changes nothing — the in-app banner says so: "Checkout completed successfully … Subscription and queued-story release state are refreshing."

Idempotency

The event id is the first non-empty of webhook-id, svix-id, x-webhook-id, x-polar-webhook-id, otherwise <eventType>:<sha256(rawBody)> (derivePolarWebhookEventId, src/lib/services/polar.ts:208-227). billing_webhook_events is unique on (provider, provider_event_id) (schema.ts:318-344), with the same 15-minute stale-processing reclaim window as Clerk (billing-repository.ts:916-973).

Handled events

Anything else is recorded ignored with Event type <type> is not mapped for billing sync (polar/route.ts:237).

Tenant resolution

The tenant is resolved only from event.data.customer.externalId, lowercased, and only when it starts with org: or user: (normalizeTenantScopeExternalId, polar/route.ts:66-82). Anything else — including a customer created outside ErmisAI's provisioning path — yields ignored with Missing customer.externalId in subscription webhook payload (or … order webhook payload).

Status comes from the payload, not the event type

Two writes downstream are worth knowing about:

• The free plan always stores status = 'free', whatever Polar reports for the free product, so status stays a clean paid/free discriminator (billing-repository.ts:764-770).
• The whole subscription upsert runs in one transaction with the existing row SELECT … FOR UPDATE. A polar_modified_at older than the stored value skips everything, including the cross-row unique-index clears (billing-repository.ts:745-762). Invoice writes carry a separate monotonic status-rank guard (open 0 < paid 1 < failed 2) so a late order.paid cannot un-refund an invoice.

Unmapped products are acked as ignored, never downgraded

If a subscription event names a Polar product id with no seeded plan mapping, upsertBillingSubscriptionFromPolarWebhook throws UnmappedPolarProductError (billing-repository.ts:156-166, thrown at :729). The route catches only that class and records ignored with the reason unmapped_polar_product: <id> (polar/route.ts:208-218).

The rationale is in the code: an earlier version silently fell back to the previous plan or Free and marked the event processed, so a paying customer landed on the wrong entitlements with no operator signal. Now the event stays visible in the forensics list and the tenant is recoverable with an admin resync once the product id is seeded.

The error policy, and the trap it creates

Only isRetryableDatabaseServiceError(err) produces a 500. Everything else is acked 200 with the row recorded failed (polar/route.ts:264-303). The comment at :273-278 gives the reason: a malformed payload or an unparseable date would 500 on every retry, so Polar would hammer the endpoint and block the rest of the billing queue.

Forensics and recovery

GET /api/admin/billing/webhook-events (src/app/api/admin/billing/webhook-events/route.ts) lists the event log newest-first. It is gated on the costs admin surface — super_admin or ops. Query params: status (one of processing, processed, failed, ignored; anything else is a 400 Invalid status filter) and limit (default 50, clamped 1–200). Each row returns id, provider, providerEventId, eventType, status, errorMessage, payloadPreview, createdAt, processedAt — the preview only, never the archived body.

The same data backs the Webhook event log card in the Billing ops panel on /admin/costs, with filters all | processed | ignored | failed.

Recovery is manual, and replaying the archived payload would reproduce the same outcome. The fix is POST /api/admin/billing/resync (also gated on the costs surface), which pulls live Polar state and upserts it. Billing state also self-heals on the tenant's next sign-in through ensurePolarBillingAfterAuth (src/lib/auth/post-auth-redirect.ts:28-54).

Resend

src/app/api/webhooks/resend/route.ts. It exists for one purpose: writing permanent bounces and spam complaints into the hard-bounce suppression list so a dead address is never retried.

Signature verification is hand-rolled

Resend uses the Svix scheme, and the route implements it directly rather than adding a dependency (resend/route.ts:53-88):

1. Require svix-id, svix-timestamp and svix-signature; any missing header fails.
2. Parse the timestamp as seconds and reject a skew above 5 minutes (SVIX_TIMESTAMP_TOLERANCE_SECONDS, :17).
3. Strip a leading whsec_ from the secret and base64-decode the remainder.
4. HMAC-SHA256 over the literal string `${svix-id}.${svix-timestamp}.${rawBody}`, base64-encoded.
5. Split the header on spaces, take the part after the comma in each version,signature pair, and compare with timingSafeEqual. Any match passes.

What it acts on

resolveSuppressionSource (:98-110):

Recipients from data.to (string or array) are trimmed, and each is written as recordHardBounceSuppression({ token: buildEmailSuppressionToken(recipient), source }). The response reports how many records were newly written.

What this receiver does not have

There is no idempotency ledger, no payload archive, and no rate limit on this route. Re-delivery is harmless because the suppression write is an add-if-absent on a hash field, but there is no duplicate-detection response and no forensics table to inspect afterwards.

RESEND_WEBHOOK_SECRET is one of the ten launch-critical keys checked by collectLaunchConfigStatus (src/lib/platform/launch-readiness.ts:43-49). Its recorded impact: "Resend bounce/complaint webhook 503s; hard bounces and spam complaints are never suppressed, accelerating domain-reputation damage during a send." Read the live answer from GET /api/health → config.missing.

Payload archiving and retention

Clerk and Polar both archive the raw request body before processing. Resend does not.

archiveJsonPayload({ domain, payload }) (src/lib/storage/payload-archive.ts:42-73) computes a SHA-256 checksum, a preview capped at 2048 characters with a … suffix, and a date-partitioned path:

archives/<domain>/<YYYY>/<MM>/<DD>/<uuid>.json

The two domains in use are clerk-webhooks (clerk-user-repository.ts:781-784) and billing-webhooks (billing-repository.ts:926-929).

The private store

Webhook bodies carry PII — emails, names, billing ids — so archives go to a dedicated private Vercel Blob store via BLOB_ARCHIVE_READ_WRITE_TOKEN, under an archive/ prefix, with addRandomSuffix: true, and the returned url is nulled out so no fetchable location is ever persisted (src/lib/storage/object-storage.ts:281-319). The bucket column is stamped with the marker private-archive so deletes route back to the right store. On the Supabase provider the equivalent target is SUPABASE_STORAGE_ARCHIVE_BUCKET with visibility: 'private'.

Retention

Both repositories prune their own table opportunistically, at the top of begin…EventProcessing, at most once every 6 hours per process, deleting rows whose processed_at is older than 30 days and then deleting the matching blobs (clerk-user-repository.ts:17-18, 63-108; billing-repository.ts:106-107, 182-228). Blob deletion is best-effort; a failure writes [payload-archive] failed to delete archived payload blob … to stderr and the bucket lifecycle rule is the backstop.

A second consequence for erasure work: archives are partitioned by date, not by tenant, so a tenant purge cannot target them. They disappear only on the 30-day prune, and expediting that requires manual bucket work.

Configuration

Registration checklist, endpoint paths and the subscribe-to event lists per provider are in docs/implementation/webhook-setup.md.

Debugging a sync drift