Scheduled jobs and the queue subsystem

ErmisAI runs background work through three mechanisms, and only two of them are live:

1. Five Vercel cron jobs, declared in vercel.json and authenticated with CRON_SECRET. This is the entire scheduled surface. Everything that happens without a browser request happens here.
2. after()-deferred work, scheduled inside a request and run once the response has been sent.
3. A Vercel Queues pipeline, declared in vercel.json and wired end to end — but gated off by a feature flag that defaults to false, with three of its four consumers throwing on purpose.

If you are debugging staleness, start with the crons. If you are debugging a backlog, read the queue section and then read it again, because most of what /admin/queues displays is not queue telemetry.

The five cron jobs

All five are declared in vercel.json crons. The project pins a single region, "regions": ["arn1"].

Every one of these routes exposes a GET for the cron and a POST for a human operator. The GET is bearer-authenticated; the POST requires a Clerk session with a specific admin surface:

The surface-to-role map is src/lib/auth/platform-roles.ts:25-35.

Cron authentication

isAuthorizedCronRequest (src/lib/api/cron-auth.ts:46-65) compares the request's Authorization header against Bearer ${CRON_SECRET} with timingSafeEqual, after a length check that is safe because the expected length is fixed by the env value and does not vary with input.

The five cron paths are allow-listed past Clerk in src/proxy.ts (isPublicRoute), because the platform's cron request carries no Clerk session. The middleware matcher is not the gate; the handler is. tests/api-route-auth-guard.vitest.ts recognises isAuthorizedCronRequest as one of its accepted gate tokens, so a new cron route without it fails CI.

Two helpers live alongside it:

• runGuardedCronHandler(context, handler) (cron-auth.ts:23-36) wraps the handler body so an unexpected throw becomes a typed 500 routed through createInternalRouteErrorResponse, instead of an opaque crash the auto-created Vercel cron monitor cannot attribute.
• reportCronDrainFailures(context, failed) (cron-auth.ts:12-16) calls logServerError — and therefore Sentry — whenever a drain reports more than zero failures. The drains are durable, so one failure is not fatal, but a sustained outage would otherwise only be visible by reading the cron's JSON response by hand.

Sentry cron check-ins are wired at runtime through _experimental.vercelCronsMonitoring: true in next.config.ts:466-471. The webpack.* Sentry options are no-ops under Turbopack, which is why the check-ins go through that flag instead.

/api/admin/rss/refresh — the ten-minute tick

This is the load-bearing job. maxDuration = 300.

The GET handler does three things in a fixed order (src/app/api/admin/rss/refresh/route.ts:37-80):

Response body:

{
  "published": false,
  "pipelineEnabled": false,
  "inlineRefreshed": true,
  "tenantIngestion": {
    "tenantScopes": 0,
    "refreshed": 0,
    "failed": 0,
    "skippedTimeBudget": 0,
    "elapsedMs": 0
  },
  "alertSweep": {
    "tenantsWithRules": 0,
    "evaluated": 0,
    "skippedNoFeedCache": 0,
    "failed": 0
  },
  "refreshedAt": "..."
}

Nothing in the app renders this response. It is a curl/log artefact.

Tenant ingestion

ingestAllTenantScopedFeeds (src/lib/platform/local-platform-data.ts:1203-1256):

• Enumerates completed newsroom tenant scopes via listConfiguredNewsroomTenantScopeIds (src/lib/platform/newsroom-preferences.ts:1178-1193), which returns [] — not an error — when the newsroom-profile repository is unavailable or the query throws.
• Processes scopes serially, one AI build at a time, under ERMIS_TENANT_INGESTION_TIME_BUDGET_MS (default 240000). Scopes not reached before the budget expires are counted in skippedTimeBudget, not retried within the tick.
• Applies a deterministic rotation offset, Math.floor(startedAtMs / (10 * 60_000)) % tenantScopeIds.length, so the tail of a long tenant list is not permanently starved when the budget cuts a sweep short.
• Refreshes each scope with { forceRefresh: true, bootstrapRefresh: 'never' } and isolates failures per tenant (a failed counter plus a stderr line).

Spend stays bounded because feed caching is scoped by source set plus content locale: tenants sharing a catalog and locale share one underlying build, and the per-scope throttle and cross-instance Redis lock dedupe rebuilds. N default-catalog tenants cost one build per tick, not N.

The alert sweep

evaluateInstantAlertRulesForAllTenants (src/lib/platform/alerts/index.ts:813-861):

• Enumerates tenants with listStateKeysByPrefix('platform:alerts-rules:'). A tenant that has never written an alert-rules row is invisible to the sweep.
• Runs in batches of ALERT_SWEEP_CONCURRENCY = 5.
• Reads candidate stories through peekTenantScopedAggregatedStories, a build-free peek. A scope with no built cache is counted in skippedNoFeedCache and skipped. The sweep never triggers a synchronous fetch, cluster and synthesis pass from a cron tick.
• Isolates failures per tenant, writing [alerts] sweep evaluation failed for <scope>: <msg> to stderr.

New-story arrivals already fire evaluation during ingestion, after the response, via onNewStoriesSynchronized. The sweep exists to additionally cover rules created or edited since those stories arrived.

There is no dedicated alerts cron entry. This tick is the only scheduled alert evaluation in the system.

The manual POST

POST /api/admin/rss/refresh takes the other branch: if the queue publish did not happen, it calls listAggregatedStories({ forceRefresh: true }) inline, which refreshes the global catalog scope, not tenant scopes. It then runs the same alert sweep. No UI in the repository calls it — it is a runbook endpoint.

/api/admin/ai/metering/flush

flushAiMeteringOutbox(limit = 100) (src/lib/billing/ai-metering.ts:146-198) drains ai_usage_metering_outbox to Polar, one row at a time. The serialisation is deliberate: one Polar request at a time avoids stampeding the provider's rate limits, and the processed/failed mark lands before the next row is claimed so a crash re-delivers at most the in-flight row.

Eligible rows are status = 'pending', or status = 'failed' with attempt_count < MAX_AI_METERING_ATTEMPTS (src/lib/db/ai-usage-repository.ts:171, 178-193). That constant is 5.

Returns { processed, failed, skipped, reason }. When Polar is not configured, it returns immediately with reason: 'polar_not_configured' and zeroes.

/api/admin/ai/ledger/retry/flush

redriveFailedAiLedgerEntries (src/lib/ai/ledger-retry-queue.ts:147-281) drains AI usage ledger inserts that failed to reach Postgres, typically during a transient outage. This one is Redis-backed, not Postgres-backed. Three Upstash lists:

Mechanics worth knowing:

• Each tick first runs a reclaim pass, LMOVE-ing anything stranded on the processing list back onto the queue. Ledger persistence is idempotent on a stable client-supplied id, so re-processing a reclaimed entry cannot double-count — which is what makes the reclaim safe against a concurrent redrive.
• Entries are claimed with LMOVE into the processing list rather than RPOP, so a crash after the claim leaves the raw entry recoverable instead of losing it.
• Batch size defaults to DEFAULT_REDRIVE_BATCH_SIZE = 100.
• MAX_ATTEMPTS = 10. Past that the entry moves to the DLQ list and is counted in deadLettered, which reportCronDrainFailures('ai-ledger-redrive', …) escalates to Sentry — dead-lettered ledger rows are unrecoverable billable spend.

Returns { drained, succeeded, requeued, deadLettered, errors }. With no Redis configured it returns immediately with errors: ['redis_not_configured'] and drains nothing.

/api/admin/state/cleanup

Housekeeping for the Postgres KV store (src/app/api/admin/state/cleanup/route.ts). Reads already filter expired rows, but nothing deleted them, so abandoned scope keys — old per-source-set feed caches, multiple kB each — accumulated indefinitely.

Two deletes:

1. platform_state rows where expires_at IS NOT NULL AND expires_at < now().
2. platform_state_hash rows where the key matches platform:ai-envelope-threshold-firings:% and updated_at is older than THRESHOLD_FIRING_RETENTION_DAYS = 90.

Returns { deleted, deletedHashRecords, processedAt }.

This is the one cron route that does not use runGuardedCronHandler; it wraps its own body in try/catch and returns createInternalRouteErrorResponse directly.

/api/admin/erasure/purge

Executes GDPR erasure requests whose grace window has elapsed. It iterates listDueDataErasureRequests() serially — each confirm runs an irreversible multi-table purge transaction, and serial execution keeps one heavy transaction on the connection pool at a time.

A null return from confirmDataErasureRequest means the request stopped being active between the list and the confirm (an operator confirmed or cancelled concurrently). That is not counted as a failure.

Returns { due, purged, failed, results, processedAt }, and routes any failures to Sentry through reportCronDrainFailures('gdpr-erasure-purge', failed) — personal data that should be gone but is still present should page, not wait for tomorrow's tick.

Full detail on the lifecycle, what a purge deletes, and the manual backstops is on the GDPR erasure runbook.

Adding a new scheduled job

Deferred work with after()

Not everything background is a cron. runAfterResponse (src/lib/runtime/after-response.ts) wraps Next.js after() with an error sink and a microtask fallback for contexts where after() throws:

runAfterResponse(
  async () => {
    /* work that must not block the response */
  },
  { onError: (error) => logServerError('context', error) },
)

Three call sites use it: alert evaluation for newly synchronised stories and delivery-log appends (src/lib/platform/local-platform-data.ts:1152, 1965), and post-response AI usage accounting (src/lib/ai/providers.ts:403-413).

The queue subsystem

@vercel/queue at ^0.4.0. Four topics are declared in src/lib/platform/queues-adapter.ts:28-33:

export const rssQueueTopics = {
  ingest: 'rss.ingest',
  enrich: 'rss.enrich',
  cluster: 'rss.cluster',
  synthesize: 'rss.synthesize',
} as const

vercel.json binds each topic to a consumer route file with an experimentalTriggers entry of type queue/v2beta:

"src/app/api/queues/rss/ingest/route.ts": {
  "experimentalTriggers": [{ "type": "queue/v2beta", "topic": "rss.ingest" }]
}

Delivery is push-based — Vercel invokes the consumer route when a message arrives. The app never polls.

The honest state of it

Beyond the flag, four things are true and must not be softened:

• Only rss.ingest is implemented, and it is not a fan-out. publishRssAggregationRefresh emits exactly one message per refresh with sourceId: 'all-sources' and feedUrl: ''; the handler calls listAggregatedStories({ forceRefresh: true }) — the same function the inline path calls. Cross-source clustering makes aggregation a whole-catalog operation, so the stage is a single full-refresh trigger.
• rss.enrich, rss.cluster and rss.synthesize throw on purpose. Each handler writes a stderr line and then throws "<topic> consumer is not implemented; do not enable a producer for this topic". Nothing publishes to them. Failing loud beats acking with no work: if a producer is ever wired before the stage exists, the message retries and then dead-letters, surfacing the gap instead of dropping the article.
• The consumer never touches tenant scopes. It refreshes the global catalog scope only. Enabling the flag does not offload tenant ingestion; that stays on the cron's ingestAllTenantScopedFeeds.
• Nothing reads the -dlq topics. They are write-only forensics. There is no DLQ inspection surface anywhere in the codebase.

Per-stage staging backed by a persistence layer is described in-code as reserved for a future refactor. Treat the three stub topics as reserved names, not as capability.

Message authentication

@vercel/queue's handleCallback performs no inbound verification — it parses any well-formed CloudEvent POST and invokes the handler. The consumer routes are allow-listed past Clerk in src/proxy.ts (/api/queues/(.*)) because Vercel's push delivery carries no session. Without a second control, a crafted anonymous POST to /api/queues/rss/ingest would trigger a full AI aggregation run.

So the payload carries its own HMAC (queues-adapter.ts:138-182):

• The canonical signing input is every field except signature, as Object.entries sorted by key, JSON-stringified.
• HMAC-SHA256, keyed by ERMIS_QUEUE_SIGNING_SECRET, hex digest.
• Verification is timingSafeEqual after a length check, and fails closed: no secret configured means no message can be verified, so none is processed.
• publishRssMessage throws when the secret is unset, rather than emitting a message every consumer would reject.

Provisioning detection and the local path

isQueuesProvisioned() (queues-adapter.ts:112-123) checks for a non-empty VERCEL_OIDC_TOKEN or VERCEL_QUEUE_API_TOKEN.

• Provisioned: the message goes to client.send(topic, message, { idempotencyKey }), where the key is `${topic}:${batchId}:${first 96 chars of the JSON, non-alphanumerics replaced with _}`. If the QueueClient cannot be constructed, publishing throws. There is deliberately no in-process fallthrough: on serverless a detached in-process run can be killed when the response finishes, degrading "publish failed" into "maybe ran, maybe not, no error surfaced".
• Unprovisioned (local dev): the publish dispatches to the handler registered by registerRssInProcessHandler, with synthetic MessageMetadata. If no handler is registered the message is dropped with a stderr line. Consumer routes register themselves at import time, so importing the route is enough to get a working dev loop.

getQueueRegion() reads VERCEL_REGION and falls back to 'iad1' when it is empty — the fallback only applies off-platform, since the deployed project runs in arn1.

The consumer wrapper

createRssConsumer(topic, handler, options) (queues-adapter.ts:318-403) is what every consumer route exports as its POST. It:

1. Registers the handler in the in-process registry, so the same function body runs whether or not queues are provisioned.
2. Returns a plain () => new Response(null, { status: 204 }) if the QueueClient cannot be constructed, so the route stays importable.
3. Verifies the message HMAC first. A bad signature is rejected and acked — the wrapper returns rather than throwing, because retrying a forged message is pointless and would only mirror attacker payloads into the DLQ. The rejection is logged through logServerError, and therefore Sentry, with the messageId and deliveryCount.
4. Runs the handler. Errors go through logServerError too — queue consumers run outside the route try/catch funnels that feed Sentry elsewhere.
5. On the final attempt, mirrors a summary to a sibling <topic>-dlq topic, then swallows the error so the message is acked and stops retrying.

Retry backoff is Math.min(300, 2 ** deliveryCount) seconds. Defaults are visibilityTimeoutSeconds: 300 and maxAttempts: 5.

Vercel Queues has no platform-level dead-letter queue, which is why the wrapper builds one by hand. The DLQ payload is { originalMessageId, originalTopic, deliveryCount, failedAt, errorMessage } with the message truncated to 500 characters, and the send is awaited before the ack — a fire-and-forget send could be killed with the function instance after the ack response, losing the only record of the poison message.

Consumer routes

The ingest handler also re-checks rssQueuePipeline itself and acknowledges without work when it is off, so a message in flight during a flag flip does not run an unwanted build.

Turning the pipeline on

There is no automation for this. Enabling it is:

1. Set ERMIS_QUEUE_SIGNING_SECRET in the environment. Do this first — see the callout above.
2. Flip the flag. At /admin/flags (super_admin only) it renders with the label "RSS queue pipeline". By API it is PUT /api/admin/flags with { "key": "rssQueuePipeline", "enabled": true }; the same route takes an optional tenantScopeId for a per-tenant override, and clearOverride: true to revert one.
3. Confirm on the next ten-minute tick: published: true in the refresh response, and a [queues:rss.ingest] running aggregation refresh batch=<id> line in the consumer's logs.

Turning it off is the same flag. The inline path never stopped being authoritative for tenant scopes, so there is nothing to migrate back.

/admin/queues is not queue telemetry

The surface exists (src/app/(internal)/admin/queues/page.tsx, gated on the queues surface: super_admin and ops, with ops landing there by default). Heading "Queue health"; four cards with "Pending", "Retries", "Dead letter" counters and a {n} + "s lag" chip. Without the role, the page renders "Queue monitoring access is restricted" / "This operations surface is limited to platform users with queue monitoring access."

The four cards are named ingest-queue, cluster-queue, synthesis-queue and review-queue, and none of them reads a Vercel Queue. listQueueHealth (src/lib/platform/admin/dashboards.ts:149-196) derives all four rows from the global draft feed and the cross-tenant review queue:

All four share one lagSeconds, the age of the newest story. The retries and dead-letter zeroes are hardcoded literals, not measurements.

The first three pending counts are additionally misleading: the ingestion pipeline only ever assigns a story the status review or published (resolveStoryPhase), so collecting, clustering and synthesizing are not statuses real stories carry. Nothing in the codebase reads Vercel Queue depth, retry counts, or the -dlq topics.

Failure modes