How to audit whether AI citations support the answer

An answer can have many citations and still make unsupported claims. Citation support asks whether attached evidence establishes the associated statement. Factual correctness asks whether the statement is true. Citation completeness asks whether statements needing evidence have it. These are related but different review questions.

ALCE studies correctness and citation quality as distinct dimensions. OpenAI also recommends opening sources to check support. Neither establishes that a visible link should be treated as a complete accuracy score.

Keep the prompt, complete answer, visible citation placement, source URL and observation date. Open the destination and save the relevant passage or its version when you are permitted to do so. Distinguish original evidence from a page merely repeating another publisher’s claim.

Define the sample before seeing which answers look good. Reviewing only owned-domain citations can answer a question about owned evidence, but cannot represent all answer claims. Disclose that narrower scope. Keep sources that cannot be accessed as unresolved cases rather than silently assuming support.

For this educational workflow, use the following labels:

• Fully supported: the cited passage establishes the complete claim and its material qualifications.
• Partially supported: it establishes only part of the claim or misses an important condition.
• Unsupported: the passage does not establish the claim.
• Contradicted: the passage conflicts with the claim.
• Unresolved: the relevant evidence cannot be accessed or assessed.

Decide how compound statements are split before reviewing. If a sentence combines a feature and a guarantee, review those propositions consistently. Have another reviewer check ambiguous cases and preserve disagreement. This manual rubric is a GeckoCheck educational convention, not a provider metric.

Fictional review: an answer states that ExampleTool exports CSV on every plan. Its cited documentation confirms CSV exports but restricts them to one paid tier. The feature is supported, while the universal availability is contradicted by the stated limit. A compound claim should not receive full support because one clause is true.

In a second fictional sample, eight of ten accessible reviewed pairs are fully supported, one is partial and one is unsupported. Full-support rate is 8 ÷ 10 × 100 = 80%. Report the other labels and any unresolved cases separately. This is not collected product data or an ALCE benchmark result.

Show the count of fully supported pairs, accessible reviewed pairs and unresolved pairs, plus the selection rule. State whether partial support is excluded from the numerator. Separately review uncited factual claims; they are not represented by a rate calculated only over citation–claim pairs.

A high support rate can coexist with poor completeness if only a small portion of the answer has citations. It can also coexist with outdated sources. Record dates and identify claims that require more current evidence.

Automated graders can help identify candidate problems, but their judgments need validation against reviewed examples. Evaluate disagreements, ambiguous wording and source-access failures. Do not hide reviewer or grader uncertainty in a single confident score.

ALCE’s historical experiments use specific tasks, corpora and models. They provide methodological context, not a current ranking or expected error rate for consumer assistants. For a monitoring program, keep the prompt cohort and rubric stable enough that changes remain interpretable.