Skip to content
ErmisAI

Outbound webhooks

Webhook delivery, the signature scheme and headers, retries, and failure handling.

A workspace can register one outbound webhook endpoint. ErmisAI POSTs a signed JSON body to it, records the outcome, and moves on. The whole dispatcher is src/lib/platform/integrations/webhooks.ts; there is no separate worker, queue or scheduler behind it.

What ErmisAI sends, and what it does not

Exactly two event types are ever dispatched:

X-Ermis-EventEmitted byTrigger
webhook.testsendWebhookTest, src/lib/platform/integrations/webhooks.ts:423-455The Send test webhook button in Settings → Integrations
alert.triggeredAlert-rule evaluation, src/lib/platform/alerts/index.ts:679-706A monitoring rule matched a story

There are no story lifecycle events. Approving, rejecting or publishing a story dispatches nothing to your endpoint — grep for eventType: across src/ returns only the two values above. If you need approved stories, poll the public integration API; it is the only surface that carries article content.

There is also no event subscription model. The stored configuration record is exactly { endpoint, secretMasked, secret, enabled, updatedAt } (webhooks.ts:33-41) — no event filter, no per-event endpoint, no versioning field.

Configuring the endpoint

The form lives at /app/settings?section=integrations under the heading Webhook configuration (src/components/features/integrations/WebhookConfigForm.tsx). It has three inputs — Endpoint, Signing secret, and the Enable webhook delivery switch — plus Save configuration and Send test webhook.

The signing secret is supplied by you. ErmisAI generates nothing and never displays a secret it created; the field is a password input with the placeholder whsec_..., and after saving, the helper line shows Current: followed by a mask.

The PUT contract

PUT /api/integrations/webhook (src/app/api/integrations/webhook/route.ts:12-16):

Prop

Type

All three fields are required on every save — the schema has no optional field and no partial update. The client blanks the secret box after each successful save (src/components/features/integrations/IntegrationsWorkspace.tsx:147-150), so changing only the endpoint means re-typing the secret; otherwise the client blocks the save with "Secret is required".

GET /api/integrations/webhook returns the config with the secret replaced by secretMasked. The raw secret is never returned to the browser.

The mask is the first 4 characters plus eight bullets, or - when empty (webhooks.ts:43-53). For a secret of four characters or fewer that rule concatenates the whole secret with the bullets, so a very short secret is displayed in full in the settings UI. Use a long random secret.

Endpoint validation at save time

updateWebhookConfig calls validateWebhookEndpoint with requireResolvedPublicAddress: false (webhooks.ts:379-382). That is a static URL check only. Rejections raise WebhookEndpointValidationError, which is a RouteError with status 400 and code invalid_webhook_endpoint (src/lib/platform/errors.ts:51-56), with these literal messages:

ConditionMessage
Empty, while enabled is trueWebhook endpoint is required.
Not parseable as a URLWebhook endpoint must be a valid absolute URL.
Scheme is neither http: nor https:Webhook endpoint must use HTTPS or HTTP.
URL contains a username or passwordWebhook endpoint must not include embedded credentials.
NODE_ENV=production and scheme is not https:Webhook endpoint must use HTTPS in production.
http: outside a development loopback targetHTTP webhook endpoints are only allowed for localhost during development.
Loopback hostname in production, or a private/reserved literal IPWebhook endpoint must resolve to a public network address.

Saving does not perform a DNS check. The strict resolution check runs only at dispatch time (webhooks.ts:292), so a configuration can save cleanly and then fail every single delivery with a recorded responseCode of 400. If deliveries are failing with 400 and the form accepted the URL, that is the reason.

The HTTP request

dispatchWebhookEvent (webhooks.ts:273-358) issues one POST with a JSON body and a hard 10-second timeout (WEBHOOK_DELIVERY_TIMEOUT_MS, webhooks.ts:30).

Only the response status code is used: status = response.ok ? 'success' : 'failed'. The response body is never read, parsed, or stored anywhere.

Headers

HeaderValue
Content-Typeapplication/json
User-AgentErmisAI-Webhooks/1.0
X-Ermis-Eventwebhook.test or alert.triggered
X-Ermis-Delivery-IdStable id for this delivery — see Delivery ids and deduplication
X-Ermis-TimestampISO 8601 timestamp generated immediately before the request
X-Ermis-Signaturesha256=<hex>

That is the complete set (webhooks.ts:314-322). There are no svix-* headers, no signature version list, no separate idempotency key, and no custom headers you can add.

Verifying the signature

X-Ermis-Signature: sha256=HMAC_SHA256(secret, "<X-Ermis-Timestamp>.<raw request body>")

Hex-encoded, lower case, prefixed with sha256= (buildWebhookSignature, webhooks.ts:55-57). The signed string is the timestamp, a literal ., and the body — not the bare body. The secret is exactly the string you typed into the Signing secret field, byte for byte (it is .trim()-ed before use, webhooks.ts:282).

Verify against the raw bytes you received. Re-serialising the parsed JSON will change key order or spacing and the digest will not match.

import { createHmac, timingSafeEqual } from 'node:crypto'

/**
 * @param {string} rawBody   Exact request body as received, before JSON.parse
 * @param {string} timestamp Value of the X-Ermis-Timestamp header
 * @param {string} signature Value of the X-Ermis-Signature header
 * @param {string} secret    The signing secret configured in ErmisAI
 */
export function verifyErmisSignature(rawBody, timestamp, signature, secret) {
  if (!timestamp || !signature) {
    return false
  }

  const expected = `sha256=${createHmac('sha256', secret)
    .update(`${timestamp}.${rawBody}`)
    .digest('hex')}`

  const received = Buffer.from(signature)
  const computed = Buffer.from(expected)

  return received.length === computed.length && timingSafeEqual(received, computed)
}

ErmisAI does not enforce a replay window of its own — the timestamp is sent so that you can reject stale deliveries. If you want that protection, compare X-Ermis-Timestamp against your own clock before accepting the payload.

Event types

webhook.test

Sent by the Send test webhook button. The payload has three fields (webhooks.ts:441-445):

{
  "type": "webhook.test",
  "tenantScopeId": "org:org_2abcDEF",
  "deliveredAt": "2026-07-23T09:14:02.113Z"
}

alert.triggered

Sent when a monitoring alert rule matches a story and the webhook channel resolves for that rule — which requires the config to be enabled with a non-empty endpoint and a non-empty secret (src/lib/platform/alerts/index.ts:581-584, :119-140). Otherwise the alert is delivered in-app only.

{
  "type": "alert.triggered",
  "tenantScopeId": "org:org_2abcDEF",
  "deliveredAt": "2026-07-23T09:14:02.113Z",
  "rule": {
    "id": "…",
    "name": "…",
    "alertMode": "instant"
  },
  "story": {
    "id": "…",
    "headline": "…",
    "category": "…",
    "confidence": 0,
    "breaking": false,
    "sensitive": false,
    "sourceCount": 0,
    "eventSummary": "…",
    "sourceIds": [],
    "sourceNames": []
  }
}

rule.alertMode is instant for every rule created through the product — the API enum accepts only instant, and the UI select has a single option. Legacy stored rows can still hold digest or both.

The payload carries no article body. story.headline and story.eventSummary are the only prose fields; there is no body, no language, no publishedAt, no source tier, and no claim provenance. Article content comes from the public integration API.

Note also what alert.triggered means: a story matched a monitoring rule, not that an editor approved anything. At ingestion, only stories the pipeline marked published are evaluated; separately, an editor approve re-runs evaluation with that story as the candidate. See Alert rules and notifications.

Delivery ids and deduplication

X-Ermis-Delivery-Id is stable per logical delivery, so it is the right key for consumer-side dedupe.

  • alert.triggered — the alert id, built as alert:<tenantScopeId>:<ruleId>:<storyId>:<channel> (alerts/index.ts:193-199). It is deterministic: a retry of the same alert arrives with the same id.
  • webhook.test — a fresh crypto.randomUUID() on every press (webhooks.ts:439). Two test deliveries are never the same id.

Deduplicate on it. A delivery that your endpoint processed but answered with a non-2xx status, or that timed out after you had already committed the work, will be re-attempted with an identical id.

Story ids are content-derived — the cluster id is a deterministic hash of its earliest member article (src/lib/services/rss-aggregation.ts:2108) — so the same story.id can appear in more than one workspace. Always key your storage on tenantScopeId together with story.id. See Tenant scoping and isolation.

Failure codes

Every attempt writes a log record whose responseCode comes from this vocabulary (webhooks.ts:284-333, :191-197):

responseCodeMeaningNetwork call made?
422Webhook disabled, or endpoint empty, or secret emptyNo
400Endpoint validation failed at dispatch time (usually DNS resolving to a private/reserved address, or an unresolvable hostname)No
502The request threw — connection refused, TLS failure, DNS failure at connectAttempted
504The 10-second timeout fired (AbortSignal.timeout raises TimeoutError)Attempted
Upstream statusWhatever your endpoint returned; anything outside 2xx is recorded as failedYes

422 and 400 records are indistinguishable from network failures in the UI: they appear in Recent webhook deliveries and in /admin/deliveries with the same red failed pill. A stream of 422s means nobody enabled the webhook, not that your endpoint is down.

The persisted status enum has exactly two values. webhook_delivery_status is the Postgres enum ['success', 'failed'] (src/lib/db/schema.ts:52-53, 112-115) — there is no pending, retrying or dead state anywhere in the model.

There is no retry queue

No backoff, no dead-letter, no replay button, no route to redrive a delivery. dispatchWebhookEvent fires once and records the outcome.

What exists instead is opportunistic re-attempt, and only for alerts:

Before any network call, claimAlertDelivery atomically reserves the alert id in platform:alerts-delivery-claims:<tenantScopeId> under a row lock, so two concurrent evaluators (feed sync, approve, cron sweep) cannot both send (alerts/index.ts:454-490).

On success, the alert id is added to the dedup set and a history entry is written.

On failure, the claim is settled as failed and the alert id is deliberately left out of the dedup set (alerts/index.ts:709-720). The dedup set is seeded only from delivery-log records with status === 'success' (alerts/index.ts:578-580), so the next evaluation pass sees the alert as undelivered and tries again.

Consequences worth planning for:

  • Re-attempts happen on the next alert evaluation, whenever that is — rule creation, new stories synchronised into the workspace, a story approval, or the cron sweep inside /api/admin/rss/refresh, scheduled */10 * * * * in vercel.json. There is no fixed retry interval.
  • Re-attempts stop when the story ages out of the evaluated candidate set: the evaluator scans the 250 most recent published stories (ALERT_STORY_SCAN_LIMIT, alerts/index.ts:562-565). A permanently broken endpoint does not accumulate an ever-growing backlog.
  • A pending claim abandoned mid-dispatch (the function died) unblocks after ALERT_DELIVERY_CLAIM_TTL_MS = 5 minutes (alerts/index.ts:440).
  • A failed webhook writes no alert-history entry, so the per-rule/per-channel cooldown timer never starts for that attempt.
  • webhook.test is never retried.

SSRF constraints that can reject a working endpoint

At dispatch, validation re-runs with requireResolvedPublicAddress: process.env.NODE_ENV !== 'test' (webhooks.ts:292). That adds a DNS step on top of the static checks:

  1. The hostname is resolved with dns.lookup(..., { all: true, order: 'verbatim' }).
  2. If resolution fails or returns nothing → rejected, Webhook endpoint hostname could not be resolved to a public address.
  3. If any returned address is private or reserved → rejected, Webhook endpoint must resolve to a public network address. The block list is the shared IANA-seeded BlockList in src/lib/sources/network-guard.ts, and isPrivateOrReservedIpAddress returns true for anything that is not a valid IP — fail-closed by design.
  4. The surviving addresses are pinned into an undici dispatcher (createPinnedNetworkDispatcher) so the socket dials exactly what passed validation, closing the validate-then-fetch DNS-rebinding window (webhooks.ts:294-299).

Endpoints that are legitimate but will still be rejected or fail:

  • Split-horizon DNS, where your public hostname also resolves to an internal RFC 1918 address. One private answer in the set rejects the whole endpoint.
  • A load balancer that rotates addresses between the lookup and the connect, since the socket is pinned to the resolved set.
  • Anything behind a VPN, on localhost, or on a private range in production. Use a public HTTPS endpoint, or a tunnel that terminates on a public address.

The dispatcher uses undici's own fetch, imported from the installed undici, not Node's global fetch. A pinned dispatcher Agent from the installed undici is rejected by Node's bundled undici with UND_ERR_INVALID_ARG, which previously failed every outbound webhook silently (webhooks.ts:10, 309-312). Do not "simplify" that import.

Where deliveries are recorded

One dispatch writes to two places (webhooks.ts:344-355):

RecordLocationCap
Webhook logtenant_webhook_logs (src/lib/db/schema.ts:769-783), KV fallback platform:integrations-webhook-logs:<tenantScopeId>WEBHOOK_LOG_LIMIT = 120 per tenant
Delivery logplatform:delivery-logs:<tenantScopeId> with method: 'webhook'TENANT_DELIVERY_LOG_LIMIT = 500 per tenant

The webhook log stores the resolved endpoint; the delivery log stores the raw configured endpoint. Neither stores the payload, the response body, or any header. WebhookLog is exactly { id, endpoint, status, responseCode, deliveredAt }.

A successful delivery also increments the per-tenant daily counter under platform:delivery-counters:<tenantScopeId>only successes count (delivery-log.ts:165-187), retained 35 days. That is what feeds the WEBHOOK tile in Settings → Usage, so that tile under-reports attempts by design.

A failed delivery additionally appends a DeliveryFailedEvent to a bounded 500-entry global list (delivery-log.ts:113-138), which is what the delivery.failed SSE event carries. That event is filtered to the caller's own tenant unless the caller holds the platform deliveries admin surface (src/app/api/notifications/events/route.ts). See Realtime streams and polling fallback.

Surfaces where a human can see deliveries:

  • Settings → Integrations → "Recent webhook deliveries" — the tenant's own last 120, newest first, rendered as a status pill plus HTTP {responseCode} · {deliveredAt}. Empty state: "No webhook deliveries recorded yet."
  • /admin/deliveries — a bounded 500-entry cross-tenant mirror, restricted to platform roles super_admin and ops. The source file calls it "recent-N, not a ledger" (delivery-log.ts:14-16). It is not an audit trail.
  • /app/monitoring → Delivery channels shows only a "Latest delivery failure" card, populated from live SSE events while the page is open. It is empty after a reload until a new failure arrives.

None of these is a durable audit log. If you need one, record deliveries on your side keyed by X-Ermis-Delivery-Id.

Sending a test delivery

POST /api/integrations/webhook/test (src/app/api/integrations/webhook/test/route.ts).

This route is bot-gated with verifyHumanRequest({ deepAnalysis: true }) before anything else runs. It is not callable from a script or a CI job — drive it from the button in the UI. It is additionally rate-limited on the integration:webhook-test key at 10/60s per IP and 15/60s per user.

When the dispatch does not succeed the route does not return 200 with a failure body. It maps the recorded code to an HTTP status and returns the shared error envelope with code webhook_test_failed and the message Webhook test failed with HTTP {responseCode}.:

  • recorded 422, or any recorded 4xx → 400
  • anything else (502, 504, upstream 5xx) → 502

Either way, an in-app notification titled "Webhook test executed" is written with the body "<STATUS> · HTTP <code>" (webhooks.ts:448-452). That notification, and "Webhook configuration updated", are hardcoded English regardless of the workspace locale.

Plan, role and standing gates

Every webhook route — GET/PUT /api/integrations/webhook, POST /api/integrations/webhook/test, GET /api/integrations/webhook-logs — goes through authorizeWebhookDeliveryAccess (src/lib/api/integrations-access.ts:88-93), which applies three checks in order:

  1. Session — no Clerk session → 401 Unauthorized.
  2. RolecanAccessTenantSurface(appRole, tenantRole, 'integrations'), which resolves to canManageIntegrations = appRole === 'super_admin' || tenantRole === 'owner' (src/lib/auth/newsroom-roles.ts:51-53, 103-105). Tenant admins and members have no access at all; failure is a bare 403 Forbidden.
  3. PlancanAccessWebhookDeliveryForSubscription, failure returns 403 with the literal message Webhook delivery requires Plus or above. The webhookDelivery capability is false on individual_free and true on Plus, Pro, Business Plus, Business Pro and Enterprise.

The plan check is standing-aware: resolveEffectivePlanIdForStanding reverts a past_due or canceled subscription to individual_free entitlements (src/lib/billing/catalog.ts:476-483), so a lapsed payment 403s the whole webhook surface.

The dispatcher itself carries no plan check. What actually stops alert webhooks after a lapse is the monitoring-rules entitlement: resolveMonitoringRulesLimit also reverts to Free, whose limit is 0, so evaluation early-returns and nothing is dispatched.

Storage and secret handling

DataPostgresplatform_state fallback key
Configtenant_webhook_configs, PK tenant_scope_id (schema.ts:750-767)platform:integrations-webhook-config:<tenantScopeId>
Logstenant_webhook_logs (schema.ts:769-783)platform:integrations-webhook-logs:<tenantScopeId>

The fallback is reached through runWithIntegrationsRepository (src/lib/platform/shared/integrations-gateway.ts:62-81). In any deployed environment it is unreachable: isLocalStateFallbackAllowed() returns false whenever VERCEL_ENV is set, so a repository error throws instead of silently degrading to KV.

The secret is written through encryptSecret and read through decryptSecret (src/lib/db/integrations-repository.ts:95, 437) — AES-256-GCM, stored as enc:v1:<iv>:<authTag>:<ciphertext>, keyed by WEBHOOK_SECRET_ENCRYPTION_KEY.

WEBHOOK_SECRET_ENCRYPTION_KEY is optional, and when it is unset encryptSecret is a silent no-op that returns plaintext (src/lib/security/secret-encryption.ts:54-60). In that configuration tenant HMAC secrets sit in tenant_webhook_configs.secret in the clear. Conversely, once rows are encrypted, removing the key makes reads throw. Set it before the first webhook is configured, and never rotate it away without re-encrypting. Tenant outbound-webhook secrets are the only thing in the codebase this module protects.

On this page