Outbound webhooks
Webhook delivery, the signature scheme and headers, retries, and failure handling.
A workspace can register one outbound webhook endpoint. ErmisAI POSTs a signed JSON body to it, records the outcome, and moves on. The whole dispatcher is src/lib/platform/integrations/webhooks.ts; there is no separate worker, queue or scheduler behind it.
What ErmisAI sends, and what it does not
Exactly two event types are ever dispatched:
X-Ermis-Event | Emitted by | Trigger |
|---|---|---|
webhook.test | sendWebhookTest, src/lib/platform/integrations/webhooks.ts:423-455 | The Send test webhook button in Settings → Integrations |
alert.triggered | Alert-rule evaluation, src/lib/platform/alerts/index.ts:679-706 | A monitoring rule matched a story |
There are no story lifecycle events. Approving, rejecting or publishing a story dispatches nothing to your endpoint — grep for eventType: across src/ returns only the two values above. If you need approved stories, poll the public integration API; it is the only surface that carries article content.
There is also no event subscription model. The stored configuration record is exactly { endpoint, secretMasked, secret, enabled, updatedAt } (webhooks.ts:33-41) — no event filter, no per-event endpoint, no versioning field.
Configuring the endpoint
The form lives at /app/settings?section=integrations under the heading Webhook configuration (src/components/features/integrations/WebhookConfigForm.tsx). It has three inputs — Endpoint, Signing secret, and the Enable webhook delivery switch — plus Save configuration and Send test webhook.
The signing secret is supplied by you. ErmisAI generates nothing and never displays a secret it created; the field is a password input with the placeholder whsec_..., and after saving, the helper line shows Current: followed by a mask.
The PUT contract
PUT /api/integrations/webhook (src/app/api/integrations/webhook/route.ts:12-16):
Egenskab
Type
All three fields are required on every save — the schema has no optional field and no partial update. The client blanks the secret box after each successful save (src/components/features/integrations/IntegrationsWorkspace.tsx:147-150), so changing only the endpoint means re-typing the secret; otherwise the client blocks the save with "Secret is required".
GET /api/integrations/webhook returns the config with the secret replaced by secretMasked. The raw secret is never returned to the browser.
The mask is the first 4 characters plus eight bullets, or - when empty (webhooks.ts:43-53). For a secret of four characters or fewer that rule concatenates the whole secret with the bullets, so a very short secret is displayed in full in the settings UI. Use a long random secret.
Endpoint validation at save time
updateWebhookConfig calls validateWebhookEndpoint with requireResolvedPublicAddress: false (webhooks.ts:379-382). That is a static URL check only. Rejections raise WebhookEndpointValidationError, which is a RouteError with status 400 and code invalid_webhook_endpoint (src/lib/platform/errors.ts:51-56), with these literal messages:
| Condition | Message |
|---|---|
Empty, while enabled is true | Webhook endpoint is required. |
| Not parseable as a URL | Webhook endpoint must be a valid absolute URL. |
Scheme is neither http: nor https: | Webhook endpoint must use HTTPS or HTTP. |
| URL contains a username or password | Webhook endpoint must not include embedded credentials. |
NODE_ENV=production and scheme is not https: | Webhook endpoint must use HTTPS in production. |
http: outside a development loopback target | HTTP webhook endpoints are only allowed for localhost during development. |
| Loopback hostname in production, or a private/reserved literal IP | Webhook endpoint must resolve to a public network address. |
Saving does not perform a DNS check. The strict resolution check runs only at dispatch time (webhooks.ts:292), so a configuration can save cleanly and then fail every single delivery with a recorded responseCode of 400. If deliveries are failing with 400 and the form accepted the URL, that is the reason.
The HTTP request
dispatchWebhookEvent (webhooks.ts:273-358) issues one POST with a JSON body and a hard 10-second timeout (WEBHOOK_DELIVERY_TIMEOUT_MS, webhooks.ts:30).
Only the response status code is used: status = response.ok ? 'success' : 'failed'. The response body is never read, parsed, or stored anywhere.
Headers
| Header | Value |
|---|---|
Content-Type | application/json |
User-Agent | ErmisAI-Webhooks/1.0 |
X-Ermis-Event | webhook.test or alert.triggered |
X-Ermis-Delivery-Id | Stable id for this delivery — see Delivery ids and deduplication |
X-Ermis-Timestamp | ISO 8601 timestamp generated immediately before the request |
X-Ermis-Signature | sha256=<hex> |
That is the complete set (webhooks.ts:314-322). There are no svix-* headers, no signature version list, no separate idempotency key, and no custom headers you can add.
Verifying the signature
X-Ermis-Signature: sha256=HMAC_SHA256(secret, "<X-Ermis-Timestamp>.<raw request body>")Hex-encoded, lower case, prefixed with sha256= (buildWebhookSignature, webhooks.ts:55-57). The signed string is the timestamp, a literal ., and the body — not the bare body. The secret is exactly the string you typed into the Signing secret field, byte for byte (it is .trim()-ed before use, webhooks.ts:282).
Verify against the raw bytes you received. Re-serialising the parsed JSON will change key order or spacing and the digest will not match.
import { createHmac, timingSafeEqual } from 'node:crypto'
/**
* @param {string} rawBody Exact request body as received, before JSON.parse
* @param {string} timestamp Value of the X-Ermis-Timestamp header
* @param {string} signature Value of the X-Ermis-Signature header
* @param {string} secret The signing secret configured in ErmisAI
*/
export function verifyErmisSignature(rawBody, timestamp, signature, secret) {
if (!timestamp || !signature) {
return false
}
const expected = `sha256=${createHmac('sha256', secret)
.update(`${timestamp}.${rawBody}`)
.digest('hex')}`
const received = Buffer.from(signature)
const computed = Buffer.from(expected)
return received.length === computed.length && timingSafeEqual(received, computed)
}ErmisAI does not enforce a replay window of its own — the timestamp is sent so that you can reject stale deliveries. If you want that protection, compare X-Ermis-Timestamp against your own clock before accepting the payload.
Event types
webhook.test
Sent by the Send test webhook button. The payload has three fields (webhooks.ts:441-445):
{
"type": "webhook.test",
"tenantScopeId": "org:org_2abcDEF",
"deliveredAt": "2026-07-23T09:14:02.113Z"
}alert.triggered
Sent when a monitoring alert rule matches a story and the webhook channel resolves for that rule — which requires the config to be enabled with a non-empty endpoint and a non-empty secret (src/lib/platform/alerts/index.ts:581-584, :119-140). Otherwise the alert is delivered in-app only.
{
"type": "alert.triggered",
"tenantScopeId": "org:org_2abcDEF",
"deliveredAt": "2026-07-23T09:14:02.113Z",
"rule": {
"id": "…",
"name": "…",
"alertMode": "instant"
},
"story": {
"id": "…",
"headline": "…",
"category": "…",
"confidence": 0,
"breaking": false,
"sensitive": false,
"sourceCount": 0,
"eventSummary": "…",
"sourceIds": [],
"sourceNames": []
}
}rule.alertMode is instant for every rule created through the product — the API enum accepts only instant, and the UI select has a single option. Legacy stored rows can still hold digest or both.
The payload carries no article body. story.headline and story.eventSummary are the only prose fields; there is no body, no language, no publishedAt, no source tier, and no claim provenance. Article content comes from the public integration API.
Note also what alert.triggered means: a story matched a monitoring rule, not that an editor approved anything. At ingestion, only stories the pipeline marked published are evaluated; separately, an editor approve re-runs evaluation with that story as the candidate. See Alert rules and notifications.
Delivery ids and deduplication
X-Ermis-Delivery-Id is stable per logical delivery, so it is the right key for consumer-side dedupe.
alert.triggered— the alert id, built asalert:<tenantScopeId>:<ruleId>:<storyId>:<channel>(alerts/index.ts:193-199). It is deterministic: a retry of the same alert arrives with the same id.webhook.test— a freshcrypto.randomUUID()on every press (webhooks.ts:439). Two test deliveries are never the same id.
Deduplicate on it. A delivery that your endpoint processed but answered with a non-2xx status, or that timed out after you had already committed the work, will be re-attempted with an identical id.
Story ids are content-derived — the cluster id is a deterministic hash of its earliest member article (src/lib/services/rss-aggregation.ts:2108) — so the same story.id can appear in more than one workspace. Always key your storage on tenantScopeId together with story.id. See Tenant scoping and isolation.
Failure codes
Every attempt writes a log record whose responseCode comes from this vocabulary (webhooks.ts:284-333, :191-197):
responseCode | Meaning | Network call made? |
|---|---|---|
422 | Webhook disabled, or endpoint empty, or secret empty | No |
400 | Endpoint validation failed at dispatch time (usually DNS resolving to a private/reserved address, or an unresolvable hostname) | No |
502 | The request threw — connection refused, TLS failure, DNS failure at connect | Attempted |
504 | The 10-second timeout fired (AbortSignal.timeout raises TimeoutError) | Attempted |
| Upstream status | Whatever your endpoint returned; anything outside 2xx is recorded as failed | Yes |
422 and 400 records are indistinguishable from network failures in the UI: they appear in Recent webhook deliveries and in /admin/deliveries with the same red failed pill. A stream of 422s means nobody enabled the webhook, not that your endpoint is down.
The persisted status enum has exactly two values. webhook_delivery_status is the Postgres enum ['success', 'failed'] (src/lib/db/schema.ts:52-53, 112-115) — there is no pending, retrying or dead state anywhere in the model.
There is no retry queue
No backoff, no dead-letter, no replay button, no route to redrive a delivery. dispatchWebhookEvent fires once and records the outcome.
What exists instead is opportunistic re-attempt, and only for alerts:
Before any network call, claimAlertDelivery atomically reserves the alert id in platform:alerts-delivery-claims:<tenantScopeId> under a row lock, so two concurrent evaluators (feed sync, approve, cron sweep) cannot both send (alerts/index.ts:454-490).
On success, the alert id is added to the dedup set and a history entry is written.
On failure, the claim is settled as failed and the alert id is deliberately left out of the dedup set (alerts/index.ts:709-720). The dedup set is seeded only from delivery-log records with status === 'success' (alerts/index.ts:578-580), so the next evaluation pass sees the alert as undelivered and tries again.
Consequences worth planning for:
- Re-attempts happen on the next alert evaluation, whenever that is — rule creation, new stories synchronised into the workspace, a story approval, or the cron sweep inside
/api/admin/rss/refresh, scheduled*/10 * * * *invercel.json. There is no fixed retry interval. - Re-attempts stop when the story ages out of the evaluated candidate set: the evaluator scans the 250 most recent
publishedstories (ALERT_STORY_SCAN_LIMIT,alerts/index.ts:562-565). A permanently broken endpoint does not accumulate an ever-growing backlog. - A
pendingclaim abandoned mid-dispatch (the function died) unblocks afterALERT_DELIVERY_CLAIM_TTL_MS = 5 minutes(alerts/index.ts:440). - A failed webhook writes no alert-history entry, so the per-rule/per-channel cooldown timer never starts for that attempt.
webhook.testis never retried.
SSRF constraints that can reject a working endpoint
At dispatch, validation re-runs with requireResolvedPublicAddress: process.env.NODE_ENV !== 'test' (webhooks.ts:292). That adds a DNS step on top of the static checks:
- The hostname is resolved with
dns.lookup(..., { all: true, order: 'verbatim' }). - If resolution fails or returns nothing → rejected,
Webhook endpoint hostname could not be resolved to a public address. - If any returned address is private or reserved → rejected,
Webhook endpoint must resolve to a public network address.The block list is the shared IANA-seededBlockListinsrc/lib/sources/network-guard.ts, andisPrivateOrReservedIpAddressreturnstruefor anything that is not a valid IP — fail-closed by design. - The surviving addresses are pinned into an undici dispatcher (
createPinnedNetworkDispatcher) so the socket dials exactly what passed validation, closing the validate-then-fetch DNS-rebinding window (webhooks.ts:294-299).
Endpoints that are legitimate but will still be rejected or fail:
- Split-horizon DNS, where your public hostname also resolves to an internal RFC 1918 address. One private answer in the set rejects the whole endpoint.
- A load balancer that rotates addresses between the lookup and the connect, since the socket is pinned to the resolved set.
- Anything behind a VPN, on
localhost, or on a private range in production. Use a public HTTPS endpoint, or a tunnel that terminates on a public address.
The dispatcher uses undici's own fetch, imported from the installed undici, not Node's global fetch. A pinned dispatcher Agent from the installed undici is rejected by Node's bundled undici with UND_ERR_INVALID_ARG, which previously failed every outbound webhook silently (webhooks.ts:10, 309-312). Do not "simplify" that import.
Where deliveries are recorded
One dispatch writes to two places (webhooks.ts:344-355):
| Record | Location | Cap |
|---|---|---|
| Webhook log | tenant_webhook_logs (src/lib/db/schema.ts:769-783), KV fallback platform:integrations-webhook-logs:<tenantScopeId> | WEBHOOK_LOG_LIMIT = 120 per tenant |
| Delivery log | platform:delivery-logs:<tenantScopeId> with method: 'webhook' | TENANT_DELIVERY_LOG_LIMIT = 500 per tenant |
The webhook log stores the resolved endpoint; the delivery log stores the raw configured endpoint. Neither stores the payload, the response body, or any header. WebhookLog is exactly { id, endpoint, status, responseCode, deliveredAt }.
A successful delivery also increments the per-tenant daily counter under platform:delivery-counters:<tenantScopeId> — only successes count (delivery-log.ts:165-187), retained 35 days. That is what feeds the WEBHOOK tile in Settings → Usage, so that tile under-reports attempts by design.
A failed delivery additionally appends a DeliveryFailedEvent to a bounded 500-entry global list (delivery-log.ts:113-138), which is what the delivery.failed SSE event carries. That event is filtered to the caller's own tenant unless the caller holds the platform deliveries admin surface (src/app/api/notifications/events/route.ts). See Realtime streams and polling fallback.
Surfaces where a human can see deliveries:
- Settings → Integrations → "Recent webhook deliveries" — the tenant's own last 120, newest first, rendered as a status pill plus
HTTP {responseCode} · {deliveredAt}. Empty state:"No webhook deliveries recorded yet." /admin/deliveries— a bounded 500-entry cross-tenant mirror, restricted to platform rolessuper_adminandops. The source file calls it "recent-N, not a ledger" (delivery-log.ts:14-16). It is not an audit trail./app/monitoring→ Delivery channels shows only a "Latest delivery failure" card, populated from live SSE events while the page is open. It is empty after a reload until a new failure arrives.
None of these is a durable audit log. If you need one, record deliveries on your side keyed by X-Ermis-Delivery-Id.
Sending a test delivery
POST /api/integrations/webhook/test (src/app/api/integrations/webhook/test/route.ts).
This route is bot-gated with verifyHumanRequest({ deepAnalysis: true }) before anything else runs. It is not callable from a script or a CI job — drive it from the button in the UI. It is additionally rate-limited on the integration:webhook-test key at 10/60s per IP and 15/60s per user.
When the dispatch does not succeed the route does not return 200 with a failure body. It maps the recorded code to an HTTP status and returns the shared error envelope with code webhook_test_failed and the message Webhook test failed with HTTP {responseCode}.:
- recorded
422, or any recorded 4xx → 400 - anything else (
502,504, upstream 5xx) → 502
Either way, an in-app notification titled "Webhook test executed" is written with the body "<STATUS> · HTTP <code>" (webhooks.ts:448-452). That notification, and "Webhook configuration updated", are hardcoded English regardless of the workspace locale.
Plan, role and standing gates
Every webhook route — GET/PUT /api/integrations/webhook, POST /api/integrations/webhook/test, GET /api/integrations/webhook-logs — goes through authorizeWebhookDeliveryAccess (src/lib/api/integrations-access.ts:88-93), which applies three checks in order:
- Session — no Clerk session →
401 Unauthorized. - Role —
canAccessTenantSurface(appRole, tenantRole, 'integrations'), which resolves tocanManageIntegrations=appRole === 'super_admin' || tenantRole === 'owner'(src/lib/auth/newsroom-roles.ts:51-53, 103-105). Tenant admins and members have no access at all; failure is a bare403 Forbidden. - Plan —
canAccessWebhookDeliveryForSubscription, failure returns403with the literal messageWebhook delivery requires Plus or above. ThewebhookDeliverycapability is false onindividual_freeand true on Plus, Pro, Business Plus, Business Pro and Enterprise.
The plan check is standing-aware: resolveEffectivePlanIdForStanding reverts a past_due or canceled subscription to individual_free entitlements (src/lib/billing/catalog.ts:476-483), so a lapsed payment 403s the whole webhook surface.
The dispatcher itself carries no plan check. What actually stops alert webhooks after a lapse is the monitoring-rules entitlement: resolveMonitoringRulesLimit also reverts to Free, whose limit is 0, so evaluation early-returns and nothing is dispatched.
Storage and secret handling
| Data | Postgres | platform_state fallback key |
|---|---|---|
| Config | tenant_webhook_configs, PK tenant_scope_id (schema.ts:750-767) | platform:integrations-webhook-config:<tenantScopeId> |
| Logs | tenant_webhook_logs (schema.ts:769-783) | platform:integrations-webhook-logs:<tenantScopeId> |
The fallback is reached through runWithIntegrationsRepository (src/lib/platform/shared/integrations-gateway.ts:62-81). In any deployed environment it is unreachable: isLocalStateFallbackAllowed() returns false whenever VERCEL_ENV is set, so a repository error throws instead of silently degrading to KV.
The secret is written through encryptSecret and read through decryptSecret (src/lib/db/integrations-repository.ts:95, 437) — AES-256-GCM, stored as enc:v1:<iv>:<authTag>:<ciphertext>, keyed by WEBHOOK_SECRET_ENCRYPTION_KEY.
WEBHOOK_SECRET_ENCRYPTION_KEY is optional, and when it is unset encryptSecret is a silent no-op that returns plaintext (src/lib/security/secret-encryption.ts:54-60). In that configuration tenant HMAC secrets sit in tenant_webhook_configs.secret in the clear. Conversely, once rows are encrypted, removing the key makes reads throw. Set it before the first webhook is configured, and never rotate it away without re-encrypting. Tenant outbound-webhook secrets are the only thing in the codebase this module protects.
Public integration API
The three published endpoints, key auth, and the story export contract.
WordPress connector handshake
The connect flow and the key-to-connection coupling.
API error and status reference
The shared error envelope and every domain error code.
Security controls
The SSRF guard, secret encryption, rate limiting and the published non-claims.
Alert rules and notifications
What makes a rule fire, and where the alert lands.
API keys, webhooks and WordPress
The owner-only integrations console, from the user's side.
