Discovery Agent — Technical Reference

Turns a raw scraped row into a decision-ready feed item.

Audience: engineers. Non-technical version: overview.md. Back to the agents index.

What it does

Two distinct jobs sit under "discovery":

Finding — deciding what URLs to pull in at all (search-first query generation, provider selection, article-shape filtering).
Enriching — reading each item that landed and writing the fields a human actually browses on: a clean summary, a category, a state, SEO keywords, and three scores with reasons.

Job 2 is the agent proper — it is the only part that runs an LLM per item, on a worker, unattended.

Where it lives

Concern	File
Enrichment worker loop	`app/src/lib/jobs/feed-enrichment-worker.ts`
Enrichment implementation	`app/src/lib/feed.ts` → `enrichFeedItem()`
Claim / fail / status	`app/src/lib/feed.ts` → `claimNextFeedItem()`, `failFeedItem()`
Search-first query building	`app/src/services/news/search-discovery.ts`
Provider orchestration	`app/src/services/news/discovery.ts`
Article-URL gate	`app/src/services/news/utils.ts` → `isArticleShapedUrl()`
Boot hook	`app/src/instrumentation.ts`
UI	`app/src/components/modules/discovery-feed-module.tsx`
Routes	`app/src/app/api/tenants/[tenantId]/feed/**`

Trigger

Automatic. No button.

An in-process worker polls every FEED_ENRICHMENT_WORKER_POLL_MS (default 2000 ms), started by instrumentation.ts at boot.
It walks configured tenants, atomically claims one pending item, enriches it, and moves on. inFlight guards the claim, not just the enrichment — claiming first would strand items in processing on every tick.
Stale processing items are swept back after 5 minutes; maxAttempts is 3.
Kill it process-wide with FEED_ENRICHMENT_WORKER_DISABLED=1.

A separate daily morning sweep (Coolify cron → /api/internal/news-fetch) queues the fetch runs that produce the pending rows in the first place.

Per-tenant mode

source_config.feed_enrichment_mode in news settings:

Mode	Behaviour
`full`	Default. LLM writes the summary plus everything else
`no_summary`	LLM call still runs; `feedSummary` stays the deterministic excerpt
`off`	No LLM at all — deterministic authority, keywords and persona only

Input

Per item: title, summary, first 5 000 characters of fullContent, source, current_category.

Per tenant, loaded alongside: news source record (priority + weight), audience personas, news settings, enabled source domains, tenant categories, brand guidelines, compliance rules, compliance intelligence rulebook.

Output

Written back onto the feed doc; enrichmentStatus flips pending → processing → ready (or failed). Enrichment version is pinned as feed-v6.

Field	Source
`feedSummary`	LLM — 2 plain-text sentences, no markdown/links/boilerplate
`category`	LLM, constrained to the tenant allow-list — invented labels are rejected and the crawl-time fallback is kept
`geoState`	LLM — one Indian state/UT or `National`; drives the statewise filter
`seoKeywords`	LLM — 5–10 phrases
`authenticityScore` + reason	LLM
`brand_fit` score + reason + `guideline_ids`	LLM, citations validated against the stored rulebook
`authorityScore` + reason	Deterministic — source priority/weight, trusted-source whitelist, enabled-domain match, article-shape penalty
persona matches	Deterministic — matched from text, never model-guessed

The deterministic/LLM split is deliberate: an article never states its reader's age or profession, so personas are matched, not inferred. Authority is a property of the source, not the text, so it is computed.

Scoring detail (authority)

known source     → 50 + (priority-5)*6 + (weight-1)*15   [+20 if trusted]
unknown source   → min(60, 40 +10 if enabled-domain +20 if trusted -10 if not article-shaped)

Model & prompt

OpenAI only, via generateOpenAiText — called directly, not through lib/ai/provider-chain.ts. Discovery must not silently fall through to Gemini or an NVIDIA-hosted model.
responseMimeType: "application/json", single call per item.
Usage tracked under feature feed for token analytics.
Query generation (generateSearchQueries) uses a second, smaller OpenAI call capped at 700 output tokens.

Failure mode

Fails open, never blocks the feed.

LLM unavailable or off mode → item still becomes ready with deterministic authority, keywords and personas. It just has a rougher summary.
Query generation fails → falls back to deterministicQueries() at the configured count, with a warning. (It used to collapse to a single query, which silently cut a tenant's discovered-URL volume to one query's worth.)
No discovery provider key → getDiscoveryProviderStatus() returns available: false with a message naming the missing env var.
Enrichment throws → failFeedItem(), retried up to 3 times.

Known gaps

No feedback loop. The agent never learns which enriched items a human actually queued. Page 6 of the roadmap tags this ES: "Learn from what we have done", "Scan competition to get insights", "Scan category to get insights" — none of it is wired.
Relevance is not personalised per reviewer, only per tenant.
One item at a time. Serial worker, ~2 s poll — a large morning sweep drains slowly. No batching, no concurrency.
5 000-char content budget. A restricted claim that appears only deep in a long article is not seen.
Competitor reel topics live in ops_ca_* and are deliberately not merged into the feed — they are read-only and never selectable for generation, because reels do not pass the compliance scan every news item gets.