Source catalog operations

ErmisAI keeps one canonical registry of news sources: the JSON file news-sources/catalog.v1.json, committed to the app repository. Everything downstream — the tenant source picker, the ingestion pipeline, the admin health screen — reads either that file or a Postgres table populated from it.

There is no admin UI for any of this. Editing the catalog is a code change; loading it into Postgres is a hand-issued POST. This page covers both, and the failure modes that make each one look like it worked when it did not.

The canonical file

news-sources/catalog.v1.json is a single document with six header fields and a sources array. Its current header, read from the file:

catalogChecksum is sha256(JSON.stringify(normalizedSortedSources)) and catalogVersion is literally "v1-" plus the first 16 hex characters of that checksum (src/lib/sources/source-catalog-loader.ts:107-111). Both are derived. Never hand-write either one.

metadata is a free-form Record<string, unknown> (source-catalog-loader.ts:29); today it carries only normalizedAt.

Per-source fields

All twelve fields are required and additionalProperties is false (news-sources/catalog.schema.json:26-55, mirrored by zod at source-catalog-loader.ts:14-27 and scripts/catalog/catalog-lib.ts:30-43).

The file validates itself at runtime

createSourceCatalogDataset (source-catalog-loader.ts:120-153) parses the document, re-normalizes every entry, recomputes the checksum, and then throws on any of three mismatches:

Source catalog sourceCount mismatch. Expected N, received M.
Source catalog checksum mismatch.
Source catalog version mismatch.

The file is imported at build time (source-catalog-loader.ts:10) and bundled — it is never read from disk at runtime — then memoized on globalThis.__ermisSourceCatalogCache.

What is in the catalog today

Counts recomputed from the shipped file:

The 13 sources that are both active and ingest-enabled are aljazeera-all, bbc-business, bbc-world, cnbc-world, dw-all, ft-home, npr-world, ie-silicon-republic, guardian-business, guardian-world, nytimes-business, nytimes-technology, nytimes-world — the twelve global-origin rows plus one curated exception.

That number is load-bearing. The global (unscoped) pipeline build calls listSourceCatalogFromRepository({ ingestEnabledOnly: true }) (src/lib/services/rss-aggregation.ts:3571), so it polls exactly those 13 feeds. The other 1280 active sources are selectable by tenants and get polled inside that tenant's scoped build. The 134 inactive rows appear nowhere except /admin/sources.

active versus ingest_enabled

Two independent booleans that operators routinely conflate.

Custom tenant-added sources have no catalog row, so the tenant-scope filter treats them as enabled (local-platform-data.ts:989-990).

Editing the catalog

pnpm exec tsx scripts/catalog/normalize-categories.ts

pnpm catalog:build

pnpm catalog:validate

Offline scripts

Getting the catalog into Postgres

POST /api/admin/sources/catalog/sync is the only in-app path. Its own docblock states the contract:

Manual-only admin endpoint. Triggered by operator runbooks (see README + cutover scripts), not by automatic in-app flows.

There is no button, no cron, and no deploy step. Grep for admin/sources/catalog/sync across src/ returns zero hits — only docs/ai-newsroom-product-spec.md:758 and scripts/db/cutover-to-supabase.sh:39 mention it.

Calling it

The route authenticates with auth() from Clerk and gates on canAccessAdminSurface(appRole, 'sources') — super_admin, editor, or ops (src/lib/auth/platform-roles.ts). There is no CRON_SECRET path and no API-key path, so the request must carry a signed-in operator session. In practice: open /admin/sources in a browser as an operator, then copy the request as cURL from devtools and change the method and path.

Responses (src/app/api/admin/sources/catalog/sync/route.ts:14-44):

What the route does

1. syncSourceCatalogRepositoryState() — resets the module-level memo flags hasSeededSourceCatalog and hasIngestedCuratedCatalog first, then seeds (src/lib/db/source-catalog-repository.ts:620-629). The reset exists because on a warm serverless instance the memo made this route a silent no-op that still replied ok.
2. invalidateGlobalNewsroomSourceCatalogCache() — clears the process cache, the in-flight promise, and the Redis key.
3. void scheduleSourceIconStamping() — fire-and-forget, not awaited.

ensureSourceCatalogSeeded() (source-catalog-repository.ts:504-550) is the actual work:

• Load the bundled dataset and stamp every entry with the file's catalogVersion.
• upsertSourceCatalogEntries — dedupe incoming rows by feed URL (first wins), drop feed-URL conflicts, then INSERT … ON CONFLICT (id) DO UPDATE in serial batches of 160 (SOURCE_CATALOG_UPSERT_BATCH_SIZE, line 13). icon_url and icon_resolved_at are deliberately excluded from the update set so previously stamped icons survive a resync (comment at lines 382-383).
• deactivateStaleManagedSources — every row with catalog_origin IN ('global','curated') whose catalog_version differs from the incoming version is set active = false, ingest_enabled = false (lines 406-422). This is how removed sources retire.
• recordSourceCatalogDatasetVersion — upsert into source_catalog_dataset_versions with metadata { schemaVersion, generatedAt, ...file metadata }.

ensureCuratedSourceCatalogIngested() is vestigial: it is a second memo flag that just calls ensureSourceCatalogSeeded() (lines 552-572). There is no separate curated ingest path.

The offline alternative

pnpm db:seed-bootstrap (scripts/db/seed-bootstrap.ts, seedSourceCatalog at lines 263-307) performs the same three steps in raw SQL. It is the only other writer of these tables.

What the sync deliberately does not do

• It never deletes. source_catalog_entries grows monotonically; retirement is a flag flip.
• It never overwrites a stamped icon.
• It skips feed-URL conflicts. feed_url is uniquely indexed but id is the primary key, so an incoming entry whose feed URL already belongs to a different id is dropped, not upserted (filterFeedUrlConflicts, lines 277-303). One stdout line per distinct conflict signature: [source-catalog-repository] skipped N source entries because feed_url already belongs to a different source id. Re-iding a source while keeping its feed URL therefore does nothing until you delete or rename the old row by hand.

How reads degrade

listSourceCatalogFromRepository(options?) (source-catalog-repository.ts:574-618) is the single read entry point. Options are includeInactive and ingestEnabledOnly; the default predicate is active = true and ordering is tier ASC, name ASC. It falls back to the bundled file in three distinct cases:

1. getDbContext() returns null — the DB module import failed, or the DB-backed catalog was already disabled in this process.
2. The read query throws → disableDbBackedCatalog('catalog read query', err) → fallback.
3. The query succeeds but returns zero rows (line 613) → fallback.

Case 3 is why the product works end to end on a deployment where the sync was never run.

dbBackedCatalogDisabled is a per-process latch (source-catalog-repository.ts:110-132, 167-182). Once one catalog query fails, that Node process serves the bundled file for every subsequent catalog read until the instance recycles, and emits exactly one Sentry event. Retryable database errors set the latch without a Sentry event; anything else logs to stderr and reports once.

Callers

The tenant-facing cache

getGlobalSourceCatalog() (newsroom-preferences.ts:576) is three layers deep: a process-global timed cache with SOURCE_CATALOG_CACHE_TTL_MS = 5 * 60_000 (line 186), an in-flight promise dedupe, and the Redis key platform:cache:newsroom-source-catalog (line 188) on the same TTL.

If the database read throws and local-state fallback is not permitted, it raises a 503 with code newsroom_source_catalog_unavailable and message "Newsroom source catalog is unavailable right now." Otherwise it falls back silently to the bundled file.

The two tables

Both live in src/lib/db/schema.ts.

source_catalog_entries (line 426)

id (PK, text), name, site_url, feed_url, type (source_type enum), tier (source_tier enum), region, language (default 'en'), categories (jsonb string[]), catalog_origin (plain text, default 'global' — not a pg enum), catalog_version (nullable text), ingest_enabled (bool, default true), active (bool, default true), icon_url, icon_resolved_at, last_fetched_at, last_fetch_status, last_error_message, created_at, updated_at.

Indexes (lines 454-458): unique on feed_url; plain indexes on active, ingest_enabled, catalog_origin, tier.

source_catalog_dataset_versions (line 462)

version (PK), catalog_checksum, source_count, metadata (jsonb), ingested_at; index on ingested_at. One row per catalog version ever synced — the audit trail for "which file is this database running".

Enums

export const sourceTypeValues = ['rss', 'scraper', 'wire'] as const   // pgEnum 'source_type'
export const sourceTierValues = ['wire', 'tier1', 'tier2', 'tier3'] as const // pgEnum 'source_tier'

Note that catalog_origin is not an enum, so the database will accept any string there. Only 'global' and 'curated' are produced by the sync, and only those two are swept by deactivateStaleManagedSources.

Source health and icons

/admin/sources renders SourceHealthTable — read-only, no buttons, no empty state. Per source it shows active/inactive, ingest on/ingest off, one of ok / failed / empty parse / not polled, then Last polled: … · 24h articles: … · Error rate: …%, the feed URL, and either the raw last error or Parser returned no RSS items or Atom entries.

listSourceHealth() (src/lib/platform/admin/dashboards.ts:96-124) merges the stored columns with live health from the global feed cache. Status resolution order (lines 126-147): live fetchStatus → stored last_fetch_status → failed if any error message exists → not_polled.

Custom tenant sources never appear here. updateSourceFetchStatus early-returns for any sourceId starting with custom- (source-catalog-repository.ts:684-696) because there is no catalog row to update; their health lives only in the tenant's scoped feed cache.

Icon stamping

icon_url is populated from the backlog active = true AND icon_url IS NULL, limit 200 (listSourceCatalogEntriesMissingIcon, source-catalog-repository.ts:656-682), with concurrency 4 (source-icon-stamper.ts:14-15).

The sync route fires scheduleSourceIconStamping() un-awaited — that runs one batch, and concurrent calls share the same in-flight promise, so a single sync stamps at most 200 sources. To clear a large backlog use pnpm db:backfill-source-icons, which calls runSourceIconStampingUntilDrained and repeats batches until one stamps nothing new. That script needs DATABASE_URL (or SUPABASE_DATABASE_URL) plus SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY.

Stamping is currently mostly wasted work: SourceCatalogTable, SourcePicker and MonitoredSourcesList all render <SourceFavicon> without passing iconUrl, so those three surfaces round-trip the authenticated /api/sources/icon route regardless of what is stamped.

What tier actually does

const sourceTierWeights: Record<SourceTier, number> = { wire: 1, tier1: 0.9, tier2: 0.72, tier3: 0.55 }

src/lib/services/rss-aggregation.ts:465-470. The weight is used in three places: ordering articles within a cluster, the confidence score (averageTierWeight * 25 of a 100-point score alongside unique-source count ×30, headline agreement ×25 and recency ×20), and the ordering of the eight source articles fed into the synthesis prompt.

Tier does not affect fetch scheduling, concurrency, or which sources get polled. The tier descriptions the tenant UI shows ("Highest intake priority", "the lightest intake weight") describe the clustering and confidence weight, not a fetch priority.

Things in the data that have no implementation

• type: 'scraper' — 95 rows, no scraper. The value is stored and displayed, but nothing branches on it: fetchFeed always performs an RSS/Atom fetch of feedUrl. All 95 are active: false, ingestEnabled: false, and for 93 of them feedUrl === siteUrl (a homepage), so activating one would produce an empty parse.
• type: 'wire' — zero rows. The enum value exists and is unused. Do not conflate it with tier: 'wire', which has 20 rows (news agencies); all but ft-home are ingestEnabled: false.
• listSourceCatalogVerticals() (source-catalog-loader.ts:195) and listCategories() (src/lib/platform/sources/catalog.ts:40) are exported with zero callers anywhere in src/, scripts/ or tests/.
• No admin UI for the sync, or for manual RSS refresh, state cleanup, or the erasure endpoints. All curl-and-runbook.

Test coverage

Thin, and worth knowing before you rely on CI to catch a catalog mistake:

• tests/catalog-lib.vitest.ts — 2 cases: reads the canonical document, asserts specific curated North American and Greek entries exist.
• tests/source-catalog-repository.vitest.ts — 2 cases, both on feed-URL conflict filtering.
• tests/feed-validation.vitest.ts — 3 cases.

Nothing asserts the source count, the tier distribution, or the sync's stale-deactivation behaviour. A bad edit that still checksums correctly will ship.