AI runtime: providers, stages and prompts

Every language-model call in ErmisAI goes through one function: getLanguageModel(modelId, { stage, executionPolicy }) (src/lib/ai/providers.ts:1248). There is no direct provider SDK usage anywhere in the product code. That single entry point resolves the effective model id, resolves an execution policy, pulls the raw model out of the provider registry, and wraps it in a middleware stack that carries telemetry, usage accounting, and the compose cache.

The layer is built on AI SDK v7. Runtime types are ErmisAiStage, ErmisAiProfile, ErmisAiProviderMode (src/lib/contracts/ai-runtime.ts:5-31), and every environment variable uses the ERMIS_ prefix.

The four stages

ErmisAiStage = 'chat' | 'cluster' | 'synthesis' | 'compose' (src/lib/contracts/ai-runtime.ts:7). There is no fifth stage and no way to add one without a code change — the stage union drives the policy table, the fallback chains, the per-stage model map, and the per-stage reasoning overrides.

Admin UI labels for the same four, in the order the panel renders them: Interactive chat, Story compose, Clustering, Synthesis (src/components/features/admin/AiConfigPanel.tsx:63-71).

Two provider modes

ERMIS_AI_PROVIDER_MODE selects one of gateway (default) or azure (src/lib/ai/models.ts:180-186). The modes are mutually exclusive, and the isolation is enforced rather than conventional.

Gateway mode. Every provider is reached through the Vercel AI Gateway. src/lib/ai/gateway.ts:17-19 builds createGatewayProvider({ baseURL: AI_GATEWAY_BASE_URL }), and each provider entry in the registry is a thin bridge onto gateway.languageModel('<provider>/<model>') (src/lib/ai/provider-registry.ts:54-67). Six providers are allowed: openai, anthropic, google, xai, deepseek, minimax (provider-registry.ts:18-25). Credentials come from AI_GATEWAY_API_KEY or a VERCEL_OIDC_TOKEN (gateway.ts:7-15).

Azure mode. Direct Azure OpenAI through @ai-sdk/azure. Exactly one provider is allowed, azure (provider-registry.ts:26). The bridge maps an azure/<model> id to a Foundry deployment name using the admin-managed registry, falling back to env (provider-registry.ts:69-90, src/lib/ai/azure.ts:116-160). Configuration is AZURE_API_KEY plus either AZURE_RESOURCE_NAME or a resolvable AZURE_BASE_URL (azure.ts:75-84).

isProviderModelAllowedForMode (provider-registry.ts:144-154) is the gate: in azure mode only azure/* ids resolve, in gateway mode only the six gateway providers. A model id for the wrong mode is not silently routed elsewhere — it fails the allow check and the resolver falls back to the mode default.

Model id resolution precedence

getStageDefaultModelId(stage) (providers.ts:1176-1195) resolves in this order:

resolveEffectiveLanguageModelId({ modelId, stage }) (providers.ts:1197-1216) applies the same global override first, then a caller-supplied model id if it is provider-valid, then the stage default.

Out of the box, all four stages run on the same model. Per-stage models are an admin choice, not a shipped default. Profile does not participate in model selection at all — it only modulates the execution policy.

Execution policy per stage

Static defaults live in src/lib/ai/language-model-policies.ts:132-170:

The global profile (fast, balanced, high-accuracy) modulates two of those (language-model-policies.ts:178-188):

• maxOutputTokens is multiplied by 0.78 / 1 / 1.22 and floored at 128.
• temperature is adjusted by -0.03 / 0 / +0.02 and clamped to [0, 2].

The middleware stack

buildLanguageModelMiddleware (providers.ts:1105-1174) pushes middleware in this order. Order is load-bearing — the telemetry middleware must see the params the earlier entries produced, and cache lookups must happen before the model is touched.

1. defaultSettingsMiddleware — added only if at least one of temperature, maxOutputTokens, seed, presencePenalty, frequencyPenalty, stopSequences, or providerOptions is set.
2. A reasoning settings middleware (createReasoningSettingsMiddleware, providers.ts:1094-1103) that injects the provider-agnostic top-level v7 reasoning param — only when the policy resolved one.
3. The compose response cache — only when stage === 'compose'.
4. createRuntimeTelemetryMiddleware — always. This is where usage accounting and the ledger hang.
5. extractReasoningMiddleware({ tagName: 'think' }) — only for MiniMax M2/M3 models. Every other reasoner returns native reasoning parts through the gateway, where tag extraction is dead weight (providers.ts:1164-1171).

Reasoning effort

A two-level, provider-agnostic control. Values: provider-default, minimal, low, medium, high, xhigh (src/lib/contracts/ai-runtime.ts:17-24). none is deliberately excluded from the union so the control can only scale effort, never silently disable an always-thinking model.

resolveEffectiveReasoningEffort(config, stage) (src/lib/platform/ai-runtime-config.ts:223-234): a per-stage value other than provider-default wins; otherwise the global value applies; if that is also provider-default, each provider resolver falls back to a profile-derived level.

Anthropic additionally gets speed: 'fast' on the fast profile with Opus 4.6, and either thinking: { type: 'adaptive' } (Opus/Sonnet 4.6 on high-accuracy) or thinking: { type: 'enabled', budgetTokens } for 4.5-era models — 6000 for chat, 12 000 elsewhere.

The top-level reasoning param is set only when the model is reasoning-active and the effective effort is not provider-default (language-model-policies.ts:925-926). SDK precedence means provider-specific options win where both exist; the top-level param exists so an explicit admin effort still reaches the model when the provider-options matrix is off.

Two distinct fallback mechanisms

They are easy to confuse and they run at different layers.

Gateway in-request fallback. providerOptions.gateway.models — a per-stage chain the Vercel AI Gateway executes inside a single request on 5xx, rate-limit, or timeout errors (language-model-policies.ts:241-254):

The primary model is filtered out of its own chain, and openai/gpt-4.1-nano additionally prepends openai/gpt-5-mini (resolveCuratedFallbackModels, :266-281).

Application-level pipeline retry ladder. pipelineStageRetryModelIds (language-model-policies.ts:226-229) is used only by the RSS pipeline, when a whole generateText attempt fails end to end — timeout, malformed structured output, provider error:

• cluster: openai/gpt-4.1-nano, anthropic/claude-haiku-4.5, anthropic/claude-sonnet-4.6
• synthesis: openai/gpt-4.1-nano, anthropic/claude-sonnet-4.6, anthropic/claude-opus-4.6

Maximum three attempts per stage (rss-aggregation.ts:218-219). If the last recorded pipeline error was for the same stage and the same configured model, the first attempt skips that model (resolveStagePrimaryModelId, rss-aggregation.ts:1165-1184).

Azure BYOK rides on the gateway path: when openai/gpt-4.1-nano is in the primary-plus-fallback set and both AZURE_API_KEY and AZURE_RESOURCE_NAME are present, providerOptions.gateway.byok.azure is populated along with a providerTimeouts.byok.azure of 4000 ms (AZURE_BYOK_TIMEOUT_MS). BYOK is skipped when only AZURE_BASE_URL is set (language-model-policies.ts:299-333).

Model catalog and discovery

The static fallback catalog (src/lib/ai/models.ts:10-95) holds 14 gateway base models across OpenAI, Anthropic, Google, and xAI. Three more — deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro, minimax/minimax-m2.5 — sit behind ERMIS_AI_EXPERIMENTAL_MODELS (models.ts:102-134). The Azure static catalog has exactly one entry, azure/gpt-5.4-mini.

In gateway mode, src/lib/ai/model-catalog.ts calls gateway.getAvailableModels() with a 4500 ms timeout, filters to modelType === 'language' and membership in the static gatewayAllowedModelIds set, scores and buckets the survivors to at most 36, and caches the result in a globalThis map for 10 minutes. In azure mode, discovery is bypassed entirely and the catalog is derived from the admin Azure registry (model-catalog.ts:366-369).

The admin-managed runtime config

Persisted in the Postgres KV table platform_state under the key platform:admin:ai-runtime-config (src/lib/platform/ai-runtime-config.ts:63), validated by platformAiRuntimeConfigSchema (src/lib/contracts/ai-runtime.ts:83-110). Fields cover provider mode, profile, provider-options matrix, chat model override, chat stream mode, ledger and metering toggles, guardrails mode, global and per-stage reasoning effort, the DeepSeek Flash reasoning toggle, per-stage models, the Azure registry, the signup/waitlist/AI kill switches, and the platform spend caps.

Read path: loadPlatformAiRuntimeConfig() reads, normalizes, and caches into a globalThis snapshot. Everything on the hot path reads the synchronous snapshot getPlatformAiRuntimeConfigSnapshot() (ai-runtime-config.ts:706-717), which seeds itself from env defaults if no load has run yet. Routes therefore await loadPlatformAiRuntimeConfig() early — see chat/route.ts:467, completion/route.ts:253, story-compose.ts:715. If you add a call site that reads the snapshot, add the load too, or you will silently get env defaults.

Write path: PUT /api/admin/ai/config (src/app/api/admin/ai/config/route.ts), gated on a Clerk session plus canAccessAdminSurface(role, 'ai') — super_admin only (src/lib/auth/platform-roles.ts:32). Beyond schema validation, each patched stage model is checked with four distinct 400 codes:

GET returns { config, catalog: { defaultModelId, items, providers } }.

The Azure model registry

Entries are validated by azureModelEntrySchema (ai-runtime.ts:65-74): modelId must match /^azure\/[A-Za-z0-9._:-]+$/, plus deploymentName, label, kind (chat or embedding), and reasoning. azureModels is a full-list replace on PUT — send the complete list, not a delta. The default chat entry azure/gpt-5.4-mini is always re-inserted if it is missing (ai-runtime-config.ts:361-367).

The per-entry reasoning flag is not a tuning knob. resolveAzureRegistryReasoningOverride (ai-runtime-config.ts:380-393) returns it for an azure/<model> id and it overrides the model-name heuristic entirely, because custom Foundry deployment names carry no capability signal and a misclassification 400s every call. For the same reason, Azure forceReasoning still ships even when the provider-options matrix is disabled — capability pinning is treated as correctness, not tuning (language-model-policies.ts:713-723).

Adding an Azure model requires no redeploy.

Prompts

Versioning

resolvePromptTemplate({ promptKey, versions, defaultVersion, requestedVersion }) (src/lib/ai/prompts/prompt-versioning.ts) returns the template plus { version, hash, requestedVersion, usedDefaultVersion, availableVersions }. The hash is the first 16 hex characters of the SHA-256 of the template string.

Exactly two prompt families are versioned, both keyed by content locale, both with two versions (2026-04-08.v1 and 2026-05-08.v1) and both defaulting to 2026-05-08.v1:

• story-compose — src/lib/ai/prompts/story-compose.ts, resolver at :417, env override ERMIS_PROMPT_VERSION_STORY_COMPOSE.
• story-refinement — src/lib/ai/prompts/story-refinement.ts, resolver at :397, env override ERMIS_PROMPT_VERSION_STORY_REFINEMENT.

The 2026-05-08.v1 delta is a source-language-independence instruction: the article is produced in the configured content language regardless of what language the sources are in.

What gets assembled

resolveStoryComposePrompt (story-compose.ts:417-481) builds, in order: the locale's persona template, the first-party-voice directive, the format/tone/length directives, and the story context block (story id, headline, localized category label, a bare confidence number, source names, body). Nothing else reaches the model — the publication name and primary vertical are not in the prompt.

The first-party-voice directive (src/lib/ai/prompts/first-party-voice.ts) is appended to both families and to the completion route's system text (completion/route.ts:290-295). English text:

Report the event itself - what happened - not an outlet’s coverage of it. Never build the article around a source or its act of publishing (avoid "X published/aired/reported…", "the segment/article examines…"). Cite a source inline only for a specific or contested claim, never as the subject.

Both families open with the Lyra persona line — in English, "You are Lyra by ErmisAI, a senior newsroom editor." — and both forbid model or provider disclosure.

Locale coverage

All 10 content locales (en, el, it, es, pt, pl, sv, da, nb, fi) have hand-written compose personas, refinement personas, first-party-voice directives, prompt phrase tables, and completion system lines. These are TypeScript constants, not messages/*.json entries, and they are not machine translations.

Coverage is enforced unevenly. The phrase tables, first-party-voice map, completion system lines, and draft placeholders carry satisfies Record<ContentLocale, …>, so a missing locale breaks the build. composeTemplateVersionsByLocale and refinementTemplateVersionsByLocale are plain as const (story-compose.ts:16, story-refinement.ts:11), so a missing persona silently falls back to English (story-compose.ts:428-431). That hole is exactly what tests/locale-parity.vitest.ts cases 2 and 3 exist to catch: they resolve the prompt for every content locale and assert that a non-en result does not start with the English persona line.

Adding a content locale therefore requires hand-written templates in both prompt files plus a branch in completionSystemLinesByLocale (src/app/api/stories/[storyId]/completion/route.ts:57-128). See Localization system for the full checklist.

Where prompt metadata goes

{ key, version, hash, requestedVersion, usedDefaultVersion } travels to four places: assistant message metadata as promptVersion / promptHash (src/lib/ai/types.ts:15-33), a transient data-chat-prompt stream part (chat/route.ts:747-754), the outbound correlation headers X-Ermis-Prompt-Hash and X-Ermis-Prompt-Version, and the compose result object. That is what makes a bad generation traceable to an exact template.

Caching

Anthropic prompt caching. buildSystemInstructions({ systemText, modelId }) (src/lib/ai/prompt-caching.ts) returns a plain string for most providers. For Anthropic models whose system text clears the model's minimum cacheable size it returns a SystemModelMessage with providerOptions.anthropic.cacheControl = { type: 'ephemeral', ttl: '1h' }. Thresholds, at roughly 4 characters per token: 4096 characters by default, 8192 for Haiku 3.x, 16 384 for Opus 4.5+ and Haiku 4.5+. In practice most ErmisAI system prompts are well under 4096 characters, so the marker rarely applies. The result must be passed as the top-level instructions option — a role: 'system' entry inside messages throws InvalidPromptError in v7.

Compose response cache. src/lib/ai/cache-middleware.ts is a wrapGenerate cache for the compose stage only. The key is a SHA-256 over { modelId, providerId, specificationVersion, responseAffectingParams }, where the params deliberately exclude headers (per-request correlation trace), abortSignal, and includeRawChunks, but include providerOptions — which is what keeps the cache tenant-scoped, since the gateway user and tags live there. Storage is Upstash Redis when configured, otherwise an in-memory LRU of 256 entries. TTL 6 hours, maximum payload 128 KB, key prefix ermis:ai-cache:compose:. A hit emits ai_model_invocation_cache_hit. Disable with ERMIS_DISABLE_COMPOSE_CACHE.

Telemetry, correlation and redaction

src/lib/telemetry/ai-runtime.ts writes single-line JSON to stdout and stderr prefixed [ai-runtime], emitting ai_model_invocation_started, …_completed, …_failed, …_warning, and …_cache_hit. Tenant scope ids are redacted to <scopeType>:<sha256-prefix-12> and user ids to user:<sha256-prefix-12>, because personal-tenant scopes embed the Clerk user id.

src/lib/ai/correlation-headers.ts sets X-Ermis-Surface, X-Ermis-Stage, X-Ermis-Profile, X-Ermis-Workflow, X-Ermis-Prompt-Hash, X-Ermis-Prompt-Version, X-Ermis-Tenant, X-Ermis-Invocation-Id, X-Ermis-Story-Id, and X-Ermis-Chat-Id, each capped at 256 characters with CR/LF stripped.

src/instrumentation.ts registers @vercel/otel and bridges AI SDK v7 spans. Sentry's built-in VercelAI integration is explicitly filtered out to avoid double-counted usage.

Usage accounting, the ai_usage_ledger table, guardrail admission, the kill switches, and Polar metering all hang off the telemetry middleware and are documented separately in AI usage accounting, guardrails and metering.

Configuration

Persisted admin config beats env seeds for everything except provider mode. Full variable list in Environment variable reference.

.env.example describes ERMIS_AI_EXPERIMENTAL_MODELS as a "comma-separated allowlist". That comment is wrong — the code treats it as a boolean (src/lib/ai/models.ts:123-129). .env.example is still the more complete reference; the README omits roughly twenty of the AI variables.

The five AI-related entries in the Vercel Flags SDK adapter — disableComposeCache, enableAiCompose, enableProviderOptionsMatrix, modelOverride, aiProfile — are env-driven only. They are not in the /admin/flags registry, which holds a different five flags. See Feature flags and runtime controls.

Wired but never called

Do not build on these without reading the code first; each is scaffolding with no consumer.

• Embeddings. DEFAULT_AZURE_EMBEDDING_MODEL = 'azure/text-embedding-3-small' is seeded into the Azure registry, the admin panel offers an Embedding kind, and the provider registry exposes embeddingModel() bridges for every provider. A repo-wide search finds zero call sites for embed, embedMany, or embeddingModel outside the registry and the Azure resolver themselves.
• Image models. imageModel() bridges exist at provider-registry.ts:62-64, 86-88. No callers.
• Reranking. Does not exist. The only mention is a comment about OTel span kinds in src/instrumentation.ts:8.
• Tools and tool calling. ChatMessage is typed UIMessage<MessageMetadata, CustomUIDataTypes, Record<string, never>> (src/lib/ai/types.ts:67) — an empty tool map. The chat route states in a comment that it has no tools.
• The user-facing model picker. StoryPromptInput can render an "Assistant settings" collapsible with a "Model" select, but only when the server reports chatAllowModelOverride: true (StoryChatPanel.tsx:718). That flag defaults to false and the admin panel writes false on every save, so the picker is unreachable in a normally-operated deployment.
• DeepSeek and MiniMax paths. Both providers are gated out of the catalog by default, so the DeepSeek Flash reasoning toggle and the MiniMax <think> extraction middleware are inert unless ERMIS_AI_EXPERIMENTAL_MODELS is on.

Where to change what