Feature flags and runtime controls

ErmisAI has three separate control planes that change platform behaviour without a code change. They are easy to confuse, they are stored differently, and only one of them is fully editable from the admin console.

Platform feature flags

Exactly five keys exist, declared in src/lib/contracts/feature-flags.ts:3-9. The registry is closed — adding a flag is a code change. Labels and descriptions are English constants in that contract file, not i18n keys, so /admin/flags shows the same English text in every UI locale.

The exact label and description strings shown at /admin/flags come from featureFlagDefinitionByKey (src/lib/contracts/feature-flags.ts:15-45); defaults come from defaultFeatureFlagState (:47-55).

What "gated" means for each flag

The three default-on flags gate their API routes with a uniform refusal. When the flag resolves to false, the route returns 404 with body {"error":"Feature disabled","code":"feature_disabled"} — for example src/app/api/stories/queue/route.ts:35-41, src/app/api/stories/[storyId]/notes/route.ts:41-47, src/app/api/stories/[storyId]/review-nudge/route.ts:31-37. The client affordances disappear independently, driven by the same effective flag value.

rssQueuePipeline is the activation switch for the Vercel Queues path. publishRssAggregationRefresh() returns false without publishing anything while the flag is off (src/lib/platform/queues-adapter.ts:266-272), and the rss.ingest consumer acknowledges its message and does no work (src/app/api/queues/rss/ingest/route.ts:20-25). The inline aggregation path stays authoritative.

notificationsRestPolling is enforced server-side, not just as a client hint. GET /api/notifications/events returns 204 No Content when it is on (src/app/api/notifications/events/route.ts:50-55). An EventSource does not reconnect after a 204, so every subscriber falls through to its REST polling path cleanly.

Storage and resolution

Two platform_state keys back the whole system (src/lib/platform/feature-flags.ts:16-17):

• platform:admin:feature-flags — the global snapshot, one record per key holding enabled, updatedAt and updatedBy.
• platform:admin:feature-flag-overrides:<tenantScopeId> — a partial record of per-tenant overrides.

Resolution is a single line: isPlatformFeatureEnabled(key, tenantScopeId?) returns overrides[key] ?? globalEnabled (feature-flags.ts:114-125). A missing stored record falls back to defaultFeatureFlagState[key], with updatedAt defaulting to 1970-01-01T00:00:00.000Z and updatedBy to null (:19,100-102).

Persistence is mandatory. Any read or write failure throws [feature-flags] durable feature flag state (platform_state) is required in production after <stage> (:62-68) rather than silently defaulting.

The admin API

/admin/flags is gated on the flags surface, which is super_admin only (src/lib/auth/platform-roles.ts:31). The route re-checks the same surface (src/app/api/admin/flags/route.ts:25-30).

The panel's Inherit button sends exactly that last shape — enabled is required by the schema, so the client sends false and relies on clearOverride (src/lib/api/admin.ts:262-278, route branch at route.ts:73-81).

Tenant clients never read this route. They call GET /api/feature-flags, which requires canEditStories(appRole, tenantRole) and returns the effective value per key for the caller's tenant scope (src/app/api/feature-flags/route.ts:24-48).

Flag gotchas

• "Updated 1 Jan 1970" is normal. A flag that has never been toggled has no stored record, so the panel renders the DEFAULT_UPDATED_AT sentinel.
• Override keys are lowercased. getFeatureFlagOverridesKey applies .trim().toLocaleLowerCase('en') (feature-flags.ts:46-48), so Org:Org_ABC and org:org_abc resolve to the same override record.
• There is no tenant picker. The Per-tenant overrides panel is a free-text input with placeholder org:org_… or user:user_…. The operator must know the exact scope id.
• The client fails open. useTenantFeatureFlags swallows a failed /api/feature-flags fetch and keeps defaultTenantFeatureFlagMap (src/hooks/use-tenant-feature-flags.ts:25-27, src/lib/api/feature-flags.ts:8-14). If the fetch fails while a default-on flag is globally off, the UI shows the affordance and the API answers 404 feature_disabled.
• Toggling is one flag at a time. The panel switch issues one PUT per row; there is no bulk apply and no confirmation step.

The platform AI runtime config

Everything else operators can change at runtime lives in one zod-validated record at platform:admin:ai-runtime-config (src/lib/platform/ai-runtime-config.ts:63), shaped by platformAiRuntimeConfigSchema (src/lib/contracts/ai-runtime.ts:83-110).

The record also carries updatedAt and updatedBy, both stamped by updatePlatformAiRuntimeConfig from the calling Clerk user id.

Precedence: persisted config beats env, with one exception

Environment variables are seed defaults. resolveDefaultRuntimeConfig() reads them (ai-runtime-config.ts:515-590) to build the object used when nothing is stored yet; once a value is persisted, normalizeRuntimeConfig prefers the stored value and only falls back to the seed when the stored field is missing or the wrong type (:627-690).

aiUsageLedgerEnabled and enableProviderOptionsMatrix have no env seed at all — both are hardcoded true in resolveDefaultRuntimeConfig (ai-runtime-config.ts:530,540).

What the admin panel does and does not send

/admin/ai (also super_admin only, platform-roles.ts:32) renders controls for provider mode, profile, reasoning effort, chat stream mode, the ledger/metering/DeepSeek toggles, guardrails mode, the four stage models with per-stage effort, and the Azure model registry.

Its save() sends a fixed 13-field payload (src/components/features/admin/AiConfigPanel.tsx:364-378). Two of those fields have no UI control and are hardcoded on every save:

enableProviderOptionsMatrix: true,
chatAllowModelOverride: false,

The five launch controls are simply absent from that payload, which is why they need a hand-written request.

Stage model validation

PUT /api/admin/ai/config validates every patched stage model in four escalating steps, each returning 400 with a distinct code (src/app/api/admin/ai/config/route.ts:133-200):

Azure chat models supplied in the same PUT are added to the allowed set first, so a stage can target a deployment being registered in the same request (route.ts:124-131).

Runtime config gotchas

• An invalid persisted record falls back wholesale. If the normalised object fails platformAiRuntimeConfigSchema, normalizeRuntimeConfig returns the entire default config, not just the bad field (ai-runtime-config.ts:692-696). Every operator setting reverts at once.
• azureModels is a full-list replace (src/lib/contracts/ai-runtime.ts:164-165). A PUT that omits an entry deletes it. Entries are deduplicated by lowercased modelId, last wins, and the default Azure chat model is re-inserted if removed.
• Guardrails disabled in a deployment logs one loud warning. warnIfGuardrailsDisabledInDeployment writes a single stderr line naming /api/admin/ai/config and ERMIS_AI_GUARDRAILS_MODE when VERCEL_ENV is set (ai-runtime-config.ts:296-316). Monthly limits, overage gating, standing blocks and rate limits are all bypassed in that mode.
• Changing provider mode disables the stage-model selects until the change is saved, because the model catalog for the new mode has not been fetched.

Controls with no UI

These five fields are enforced in code and readable from /api/health, but nothing in /admin/flags or /admin/ai renders them.

Cap resolution also fails open: any error reading the ledger returns null and the invocation proceeds (usage-guardrails.ts:392-395).

Flipping them

There is no console path. The route is gated by Clerk auth() plus canAccessAdminSurface(role, 'ai') (src/app/api/admin/ai/config/route.ts:34-57), so the request must carry a signed-in super_admin session — in practice, the browser session cookie copied from an open /admin tab. The patch schema accepts any non-empty subset of fields (src/lib/contracts/ai-runtime.ts:150-196), so a single field is a valid body:

curl -X PUT https://ermisai.com/api/admin/ai/config \
  -H 'Content-Type: application/json' \
  -H "Cookie: ${ADMIN_SESSION_COOKIE}" \
  -d '{"aiPaused": true}'

curl -X PUT https://ermisai.com/api/admin/ai/config \
  -H 'Content-Type: application/json' \
  -H "Cookie: ${ADMIN_SESSION_COOKIE}" \
  -d '{"signupsPaused": true, "waitlistEnabled": false}'

curl -X PUT https://ermisai.com/api/admin/ai/config \
  -H 'Content-Type: application/json' \
  -H "Cookie: ${ADMIN_SESSION_COOKIE}" \
  -d '{"platformDailySpendCapCents": 5000, "platformMonthlySpendCapCents": 100000}'

Caps are non-negative integers in USD cents; null removes the cap. The response is the same payload the GET returns: { config, catalog }.

The alternative is a redeploy with the matching ERMIS_* seed, which only takes effect on a state store that has never persisted the field.

Reading the current values

GET /api/health is auth-free and reports the live values under controls (src/app/api/health/route.ts:98-131,151-164):

{
  "status": "ok",
  "aiGuardrailsMode": "enforce",
  "controls": {
    "aiGuardrailsMode": "enforce",
    "aiPaused": false,
    "signupsPaused": false,
    "waitlistEnabled": false,
    "platformDailySpendCapCents": null,
    "platformMonthlySpendCapCents": null
  }
}

Waitlist mode needs two switches

waitlistEnabled changes the form ErmisAI renders. It does not enforce anything: Clerk still accepts ordinary sign-ups unless the Clerk dashboard sign-up mode is also set to Waitlist (Configure → Restrictions → Waitlist), which is what performs server-side enforcement and batch approvals. Flipping only the Clerk dashboard leaves /sign-up rendering a <SignUp> that Clerk will reject. Both must be flipped together, as .env.example:138-140 states.

signupsPaused wins over waitlistEnabled in the render order (AuthShell.tsx:164-178) — pausing signups hides the waitlist form too. Neither affects /sign-in.

Env-backed flags that are not in the console

src/lib/platform/feature-flags-adapter.ts declares five entries on the Vercel Flags SDK. They are AI-layer switches, they are not visible or editable at /admin/flags, and they resolve from environment variables through each flag's decide function.

Accepted boolean strings are true/1/yes and false/0/no, case-insensitive; anything else falls back to the declared default (feature-flags-adapter.ts:35-40).

There are two access patterns, and they behave differently. ermisFlags.<name>() is request-scoped, dedupes per request and integrates with the Vercel Toolbar override cookie. getFlagValueSync(name) reads process.env directly and exists for module-init and boot paths where there is no request context — it applies the same defaults and parsing, but a Toolbar override cannot reach it (feature-flags-adapter.ts:1-19,128-150).

The discovery endpoint at GET /.well-known/vercel/flags publishes every declared flag to the Vercel Toolbar and the dashboard Flags tab. It verifies the request signature against FLAGS_SECRET and returns 401 when the secret is unset or the signature does not match (src/app/.well-known/vercel/flags/route.ts).

Two traps here:

• enableProviderOptionsMatrix has two conflicting defaults. The flag adapter default is false; the runtime-config default is true (ai-runtime-config.ts:530). The runtime config is what reaches the policy resolver in practice (src/lib/ai/providers.ts:1236), so the matrix is on. The env fallback only applies when resolveLanguageModelExecutionPolicy is called without an explicit providerOptionsEnabled (src/lib/ai/language-model-policies.ts:891-893).
• ERMIS_MODEL_OVERRIDE beats everything, including per-stage models and any per-request selection, and it is validated only for provider-prefix support — never against the live catalog (providers.ts:99-112,1176-1195).

One further switch is read straight from process.env and belongs to neither registry: ERMIS_DISABLE_AI_PIPELINE. Set to the exact string true, isAiPipelineConfigured() returns false and the aggregation pipeline behaves as if no provider were configured (src/lib/services/rss-aggregation.ts:1991-1996). It requires a redeploy.

How changes propagate

The two platform_state-backed planes propagate differently.

Feature flags propagate immediately. Every check reads platform_state directly; there is no in-process cache.

The AI runtime config is read through a per-process snapshot on globalThis.__ermisAiRuntimeConfigSnapshot (ai-runtime-config.ts:108-110,701-717). Hot paths read the synchronous getPlatformAiRuntimeConfigSnapshot(), which seeds itself from env defaults if nothing has loaded yet. The snapshot is refreshed only when something calls await loadPlatformAiRuntimeConfig(). Call sites that do so on every request:

The aggregation pipeline refreshes it at build start (rss-aggregation.ts:3568), so a kill switch flipped mid-day is picked up by the next build rather than mid-build.

Precedence summary