A theory of guarantee mechanisms in trust systems, drawn from a case: the bibliographies AI hallucinates.
In May 2026, a team led by Maxim Topaz (Columbia University) published in The Lancet an audit of nearly 2.5 million biomedical papers and 97.1 million references: one paper in 277 published in early 2026 contained at least one reference to a work that does not exist. The reflex it triggered, “let us check that the references exist,” is exactly the wrong one. A phantom reference is detectable, the DOI does not resolve, the machine notes it: that is what makes it the least of the dangers. The serious risk goes by less spectacular names, a real reference cited out of context, a real reference retracted and still invoked, a real reference drawn from non-reproducible science.
This text is not an article about hallucinated references. It is a theory of the mechanisms by which trust systems hold, and of what happens to them when automation dissolves the frictions that were holding them. Hallucinated references are not the subject; they are its measurable symptom. The case is not the foundation of the theory, it is its occasion and its first proof of existence: were the opening study revised tomorrow, the theory would survive almost intact.
Peer review checks the relevance of a citation, almost never its existence: it presumes it. That presumption rested on an asymmetry of costs, fabricating a credible reference, plausible authors, a real journal, a well-formed DOI, cost more than it returned. AI collapsed that cost without touching the other term. Yet the reward did not fall: a system that assesses careers by counting ensures that fabricating still pays. Zero cost, positive reward: the presumption could not hold.
The flaw is not new. As early as 2003, Simkin and Roychowdhury, tracking the propagation of typos copied from bibliography to bibliography, estimated that only 20% of citers had read the original. The reference was already a social object that circulates by copying, not a verified pointer. What we take for a crisis of existence is the visible worsening of an older crisis of correspondence between a claim and its proof. And self-correction does not absorb it: more than 75% of retracted articles remain cited the year following their retraction, almost all of those citations ignoring the fact. Science amortizes; it does not always correct. Worse, the systems that consume the literature have already left the documentary paradigm: in retrieval-augmented generation, no one reads the references, one queries a vector space, and the unit of proof ceases to be the document and becomes a fragment that no DOI designates.
The family to which the existence of a reference belonged must be named with care, because naming it badly ruins the argument. One would be tempted to join to it the calibration of a model, the applicability domain of a system, the traceability of an agent, and call the whole “invariants.” But these objects are not of the same nature, and no theorem unites them. I call a presumed invariant not a type of property but a status: that of being held as satisfied without active verification, because its violation was too costly to be produced at scale. The invariant is not a property, it is a status a property loses. What unites calibration and the existence of a reference is not a common essence but a single mode of guarantee: friction, not control. The category is functional, and that is why it survives the heterogeneity of its members.
From this, a rule, stated without dilution: a critical property guaranteed by a friction ceases to be presumable as soon as automation dissolves that friction. Its refutation condition is clean, that one exhibit a property whose guaranteeing friction has disappeared and which nonetheless remained reliable without any control replacing it. One point must be held firmly, which the first formulations missed: the need for control does not imply instrumentation. When friction falls, the property must be guaranteed otherwise, and “otherwise” has variants, regulation, legal liability, reputation, certification, organizational limitation, or measurement. Instrumentation has a single proper advantage, a decisive one: it is the only response that scales to the automation it corrects. That is what makes it dominant, not superior, and confusing “the most scalable” with “the right one” is the error this theory refuses at every step.
As stated, the theory is retrospective: it explains a rupture magnificently once it has occurred. For governance, that is insufficient, a leader does not want the fire explained to him, he wants the dry rooms pointed out. A property is an at-risk invariant when, simultaneously, it is today guaranteed mainly by a cost and not by a control, that cost is being dissolved by automation, and its violation is still unmeasured, hence invisible. The third condition is the most treacherous: an at-risk invariant makes no noise before it is measured. The absence of a documented incident is never proof of health; an invariant that has not yet manifested is more dangerous than one that has already broken, because it does not yet have an instrument.
The test applies today. The provenance of training data, long guaranteed by the cost of assembling a corpus, now trivial to obscure, still poorly measured. The authenticity of an author’s signature, once guaranteed by the effort of writing. The human character of a reviewer, a respondent, an account. The originality of a contribution, guaranteed by the difficulty of producing the plausible. None is demonstrated here; they are worth only as exercises in application, and that is the interest of a signature, that it applies before the proof.
If instrumentation is the dominant response, it must still not be reduced to verification. Instruments of verification establish that a link holds, a DOI resolver, a replication, a traceability audit: they answer “is it true of the link?”. Instruments of legitimation confer authority, an agency’s label, a protocol’s seal, acceptance by a community: they answer “who has the right to make proof?”. One would be wrong to oppose them to instruments as such, as if legitimacy escaped all machinery: peer review, the randomized trial, pharmacovigilance are devices. The distinction is not between instrument and institution, but between verifying and anointing. A third family is missing, whose absence would be culpable for anyone working on ontologies: instruments of coordination, standards, nomenclatures, ontologies, regulatory formats, persistent identifiers, which verify nothing and anoint no one, but make the true verifiable by several and the authorized transferable from one institution to another. A policy that instruments only verification, content to count phantom DOIs, leaves the other two exposed.
Applied to the case, verification unfolds in five degrees ordered by decreasing visibility, which is the trap: existence (does the reference designate a real work?), identity (the right work, intact, not retracted, a question become ontological once preprints and generative syntheses blur the canonical instance), relevance (does the reference support the claim, or only its theme? probably the heart of the risk), documentary traceability (can one trace the chain back?), computational traceability (can one link a synthesized conclusion to the fragments that produced it?). Level 1 is visible because it is simple; the following ones carry the essential risk and do not yet have an instrument at scale.
The integrity of the link is not everything. First edge, legitimacy: a reference becomes decisional proof only because an institution accepts it, and no verification instrument produces that acceptance. Second edge, validity: a proof can clear the five levels and remain false. When the Open Science Collaboration found only 36% significant replications where 97% of the original studies had been significant, a figure itself disputed, which tells the state of the field, it was not measuring a citation defect but a defect of truth. These two edges mark the boundary beyond which the verification instrument has, by construction, no purchase.
There remains the consequence the prudent versions treated as a reservation, and which is its culmination. If the presumed invariant is fragile, breaking as soon as its friction falls, and if the instrumented invariant becomes manipulable, a property measured and erected into a target falling under Goodhart’s law, and under Campbell’s law once the target serves to decide, then the question is no longer “how to repair?” but “does a repaired state exist?”. Instrumenting the verification of existence will produce pipelines that optimize the rate of resolved DOIs, and therefore fabrications designed to pass it, references that exist, point to a real document, and support nothing. Cost guaranteed in silence; the instrument guarantees in noise, and noise, one learns to produce. The instrument does not restore the lost guarantee, it shifts the failure mode by one notch and this new mode becomes the next invariant to monitor. The structure is cybernetic: every control mechanism engenders the space of its own circumvention. The challenge fits in one sentence, let someone show me a single trust mechanism that, once instrumented, ceased to engender new circumventions. I know of none.
The hallucinated bibliography was the door: the first place where a friction that silently guaranteed a critical property collapsed fast enough, and measurably enough, that one could make a number of it. Behind it, neither a problem of citations nor even a problem of proof, but a more general fact of which references are only an early instance: automation dissolves, across many systems at once, the costs that silently guaranteed properties no one had ever had to verify. The work to come is not to restore these guarantees, one does not restore a vanished friction. It is to choose, for each, how to replace it, by measurement when it scales, by the institution when one must anoint, by the standard when one must coordinate, knowing that each replacement will displace the problem without closing it. AI did not reveal a problem of generation, nor even of verification. It revealed that friction was the guarantee, and that it is disappearing. What remains to be built is not a system that has no more flaw, it is a discipline that knows the next one is already forming, silently, where nothing any longer costs enough to make it visible.
Doctrinal notes and explorations on AI in regulated systems. Once or twice a month. One-click unsubscribe.