agentage.hu / Articles / Analysis

Why AI Drafts Invent Citations and How a Newsroom Blocks Them

AI models prioritize plausible text over factual truth, creating invisible errors that require strict human and automated verification steps.

By Zoltán Varga2026-11-056 min read

How Fabricated Citations Arise in Large Language Models

Large language models function by predicting the next word in a sequence based on training patterns. They do not access a database of facts during generation. Instead, they construct sentences that sound authoritative and plausible. This mechanism leads to the creation of references that follow correct formatting rules but point to non-existent documents. A study of commercial deep research agents found that 3 to 13 percent of generated URLs were hallucinated. These links often appear in search indexes as if they were real, yet they lead to dead ends or pages that never existed.

The problem stems from the training objective. Models optimize for coherence and style, not for truthfulness. Arvind Narayanan notes in his book that these systems are designed to produce plausible text, not true statements. Even if developers filtered training data to include only verified facts, the architecture would still generate new, unverified combinations. The model treats a citation as a string of characters that fits the context, not as a pointer to a specific external reality. This fundamental design choice makes fabrication an inherent risk rather than a rare bug.

Real-world cases demonstrate the severity of this issue. An Air Canada chatbot provided incorrect information about bereavement fares, leading to a legal ruling against the airline. In healthcare, transcription systems have added details to medical conversations that never occurred. These errors are not malicious acts but predictable outcomes of the technology. When a newsroom uses such tools without safeguards, it risks publishing stories built on foundations that crumble upon inspection. The fluency of the output masks the underlying lack of verification.

Chunk Selection and Misattributed Quotes

Retrieval-Augmented Generation systems rely on selecting specific text chunks to feed into the model. This selection process determines which information the AI uses to answer a question. If the retrieval mechanism picks irrelevant or fragmented text, the model may stitch together a quote that sounds real but is factually wrong. Hybrid search methods combine keyword matching with semantic similarity to find these chunks. However, even advanced reranking algorithms can miss the precise context needed for accurate attribution.

The way documents are split into chunks significantly impacts accuracy. Researchers have tested different overlap rates, such as 25 percent of the context window, versus fixed strides of 16 tokens. Too little overlap causes the loss of connecting information between sentences. Too much overlap introduces noise that confuses the retrieval system. When a quote is pulled from a chunk that lacks the surrounding speaker identification, the AI might attribute the statement to the wrong person. This is a structural failure in the data pipeline, not just a model hallucination.

Business questions often require semantic understanding, while technical queries need exact keyword matches. A system optimized for one type of data may fail on the other. If a newsroom uses a generic retrieval setup for diverse topics, the risk of misattribution increases. The model fills gaps in the retrieved context with its own predictions. Without a strict check on the source material, these predictions become part of the final draft. Editors must understand that the retrieval step is a critical point of failure where errors originate.

Why Fluent Text Hides the Error

Fabricated citations often possess perfect academic formatting. They include plausible DOIs, author names, and journal titles that follow standard conventions. This visual correctness tricks readers and editors into assuming the source exists. Google Scholar and similar indexing services may automatically create records for these ghost references because they cannot distinguish between a real document and a well-formed fake. The error persists because the citation looks legitimate on the surface. Human reviewers often skim these details, trusting the professional appearance of the text.

The illusion of authority is reinforced by the model's confidence. It presents false information with the same tone as verified facts. There is no hesitation or uncertainty in the generated output. This consistency makes it difficult to spot the error without external verification. In fields with less studied subjects, the fabrication rate rises significantly. For example, while depression-related citations might be mostly real, references on niche disorders can show error rates near 30 percent. The model fills the void of scarce data with invented details that fit the narrative.

Correction pathways for AI errors differ from traditional disinformation. Traditional falsehoods often spread through intentional networks. AI hallucinations emerge from architectural limitations and are harder to trace. A fake URL might not appear in search results initially, but once indexed by aggregators, it becomes part of the digital record. The Wayback Machine cannot recover a page that never existed. This creates a unique challenge for fact-checkers who must prove a negative. The absence of a source becomes the primary evidence of fabrication.

Workflow Rules That Block Invented Sources

Newsrooms must implement a strict link-or-delete policy. Every claim in a draft must be backed by a URL that resolves to a live page. If a link returns a 404 error or leads to a placeholder, the claim is removed. Editors should not accept a reference that requires manual reconstruction of the source. The burden of proof lies with the AI output. If the system cannot provide a working link, the information is treated as unverified. This rule forces the model to rely on retrieved data rather than its internal knowledge base.

Stored-source-only quoting prevents the model from generating new text for citations. The workflow requires that all quotes come directly from the retrieved chunks. No paraphrasing of the source metadata is allowed. If the source text does not contain the quote, the AI must state that it could not find it. This eliminates the possibility of the model inventing a speaker or a statement. The human editor verifies that the quote matches the original document exactly. This step ensures that the final article reflects the source material, not the model's interpretation.

Automated URL checks act as the first line of defense. Scripts should ping every link in the draft before publication. The system flags any URL that fails to resolve or redirects to an unexpected domain. Tools that cross-reference DOIs with known databases can identify ghost references. These checks run in parallel with the drafting process. If a link is invalid, the draft is halted until a valid source is found. This automation reduces the cognitive load on editors and catches errors that human eyes might miss during a quick review.

Automated Link and Quote Checks

Integration of cross-encoder rerankers improves the quality of retrieved chunks. These models re-evaluate the relevance of search results after the initial retrieval. They filter out low-quality or irrelevant text that might confuse the generator. By improving the input data, the system reduces the likelihood of hallucination. A hybrid pipeline combining BM25 and vector search provides a robust foundation. The system normalizes scores from both methods to select the most accurate context. This technical layer supports the human editorial process.

Verification scripts should compare the generated citation against the original source. If the AI cites a page, the script extracts the title and author from the live URL. It then compares these details with the text in the draft. Any mismatch triggers an alert. This automated audit trail documents every verification step. Editors can review the log to see which links passed and which failed. This transparency builds trust in the AI-assisted workflow. It turns the verification process into a measurable metric rather than a subjective judgment.

Regular audits of the retrieval pipeline ensure long-term reliability. The system should test its performance on a set of known facts weekly. If the error rate increases, the parameters for chunking or reranking need adjustment. Newsrooms must treat their AI tools as dynamic systems that require maintenance. The goal is not to eliminate the technology but to manage its risks. With consistent monitoring and strict rules, editors can leverage AI speed without sacrificing accuracy. The human element remains the final gatekeeper of truth.

• Enforce a link-or-delete rule for all claims.
• Require stored-source-only quoting for all citations.
• Run automated URL resolution checks on every draft.
• Cross-reference DOIs with external databases.
• Maintain an audit trail of all verification steps.

AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.