From retrieval to citation: where evidence can be lost

Retrieval-augmented generation combines learned model behavior with external material. The original RAG paper studies a specific architecture using a Wikipedia index and a neural retriever; it is foundational research, not a blueprint for every modern assistant.

For diagnosis, separate four stages: forming the information need, retrieving candidates, selecting evidence for generation and attributing claims. Real products may loop through searches, use different tools or expose only part of this process. A final answer cannot reconstruct the entire hidden pipeline.

A question such as “Which tool fits a multilingual agency?” can create information needs about languages, account limits and reporting. Google documents possible query fan-out across subtopics. OpenAI describes targeted query rewriting and additional searches.

This makes useful coverage of underlying decisions important. It does not establish which rewrite happened in one observed answer. Avoid inventing an assistant’s search trace or naming a provider based only on the phrasing of the response.

Semantic retrieval can compare representations of a question and passages; keyword search and other methods may also contribute. Embedding similarity indicates relatedness in a representation space, not factual correctness, freshness or institutional authority.

Two pages can discuss the same integration while describing different versions. A retrieved page might be relevant but outdated, or accurate but never included in the final evidence. For a system you own, inspect candidate passages directly. For a public assistant, a citation list cannot tell you every page that was retrieved or rejected.

A generation step can omit a condition, merge facts from different sources or extend a supported statement into an unsupported conclusion. Even if retrieval succeeds, the output may not preserve the scope of the evidence.

For publishing, keep material qualifications attached to the claim. For an application you control, retain the supplied context and inspect the resulting claims. These are different interventions: rewriting a public page does not control how a third-party assistant packages evidence.

Illustrative scenario: a guide states that an automated audit identifies certain detectable accessibility issues in sampled pages. An answer says the tool “guarantees compliance for the entire site” and attaches the guide’s URL.

The source is relevant to the topic, but it does not establish the guarantee or exhaustive coverage. Split those claims and review each. Without a retrieval trace, you cannot tell whether the assistant omitted a supplied qualification or never received it. The defensible finding is an unsupported claim with a citation, not a proven diagnosis of the hidden cause.

• In your own system, record rewritten queries, retrieved candidates and supplied context when available.
• In public observations, retain full answers, citation positions and source passages.
• Separate retrieval relevance, factual correctness and citation support.
• Review uncited factual claims rather than scoring only visible links.
• Mark inaccessible sources unresolved and disclose the exclusion rule.

ALCE evaluates correctness and citation quality separately. Its historical benchmark motivates explicit evaluation, but its results are not a current assistant leaderboard or proof that any one publishing change fixes grounding.