Why equal error distributions produce unequal institutional drift
The reliability gate was proposed as a constructive device: a single inline place where the compromise between safety and utility stops being left to discretion and gets encoded by architecture instead. That proposal stands. This installment does the opposite thing to it. It turns the device back on itself and watches it work, because a gate that returns exactly the verdicts it was specified to return still produces a pathology, and that pathology is not an execution fault. It is a consequence of its shape. The validity domain is bounded and stated once: any system whose policy adjustment runs through a local decision loop.
The terrain of demonstration is the one this body of work has been instrumenting across several installments: the promotion port, the proof burden it demands, and the promotion gap that measures the distance between that burden and the one actually served. The question asked here sits upstream of that measure. It bears on what decides, before any measurement, which errors are allowed into the record at all.
Take two reliability gates with indistinguishable initial metrics: same precision, same recall, same distribution of errors across both classes. The classical reading calls them equivalent and moves on. The claim here is that they can diverge, and that the divergence is predictable from a property neither metric captures.
A reliability policy does not drift according to how often it errs, nor according to what its errors cost taken as a distribution. It drifts according to how its architecture distributes feedback: which errors return a signal binding enough, and soon enough, to bear on the decision, and which return nothing at the moment that matters. Policies do not evolve from error distributions. They evolve through the architecture that turns some errors into binding feedback and leaves others mute.
This article sits upstream of theories of organizational learning and organizational forgetting. It does not ask how an institution misreads what it receives, nor how it loses what it once knew. It asks which errors leave a trace at all, and holds that this filter is set before any learning and before any forgetting.
Claiming that the classical account is blind to this mechanism would be false, and the argument does not need it. A cost model can carry detection costs, delays, discovery probabilities, deferred costs, informational asymmetries. What it does not generally do is treat the asymmetry of feedback as an autonomous architectural mechanism of drift. It files the asymmetry as an exogenous parameter, a number handed in from outside, rather than as a structural property of the gate that generates drift from within.
Fields rarely forget variables. They file them in an annex. The contribution is to move this one from the annex to the mechanism, and to tie it to the architecture that produces it.
Three claims are held apart by status, and the separation is not a rhetorical precaution. The principle, that at equal error distribution feedback architecture governs policy divergence, is general and conjectural. The law, its falsifiable specialization to systems governed by a local decision loop, is demonstrated on the bounded terrain of AI reliability gates. The extension to other high-reliability domains is named as conjecture and nothing more. The theory is, at this stage, more mature than its evidence.
A theory is stable when it rests on few objects and derives the rest. Before reducing, the one boundary the argument turns on has to be fixed. Feedback architecture here denotes the local sociotechnical mechanism that turns a decision and its consequences into a signal addressed to the policy loop. An audit or a regulation is inside the architecture when it is part of that local mechanism, and outside when it is not. The boundary can include or exclude a given device depending on the system studied, but it is fixed for the length of an analysis, or endogenous and exogenous become movable to taste.
Three primitives carry the argument. The policy Π is the gate’s decision policy, the set of institutional parameters that govern its verdicts: the object that drifts. The binding force B is the constraint a feedback signal exerts on the local decision loop. Institutional memory M is the formal reference of record.
Everything else is derived and should never be treated as fundamental. Required, the proof burden a policy demands at a port in the sense of the hexagonal architecture, is a projection of Π. Constitutive traceability, the property of a system that records at the moment of the act rather than on an auditor’s request, is a property of M. The relative cost of transformation is a function of Π, M and B. Loud and Silent are two regimes of B. Down-typing is a transformation of Π. The ratchet is a dynamic property of Π under a certain structure of M. No fourth primitive is admitted.
B is the object most exposed to the charge of being a convenient notation, and the charge is better answered than awaited. B cannot be built from a list of incommensurable parts: latency is a time, attribution an information, recurrence a count, cost a currency. Those four are its determinants, not its coordinates, and B is monotone in the obvious directions.
A more serious ambiguity remains. B cannot be a property of the signal, the architecture and the organization all at once, stated interchangeably. The resolution is a distinction inside B rather than a fourth primitive. B_A denotes the architectural binding capacity, the constraint the architecture alone could impose, present wherever the architecture is. B_R denotes the realized binding force, what that capacity becomes in a given organization at a given moment, raised or lowered by a right of override or a lax authority. B_A is endogenous to the architecture; B_R is relational. The architecture owns the capacity, not the realization, and the article attributes only the capacity to it.
Defining B by its effect demands a handle that is not the effect it predicts. The candidate handle is the cost of suppression, and raw it measures the wrong thing: a weakly binding alert can be expensive to silence in a bureaucratic shop, a strongly binding one silenced in two clicks in a lax one. The usable quantity is normalized, C_s* = C_s / C_change, the suppression cost divided by the generic cost of changing the policy in that organization. This strips out the administrative surround and leaves the part attributable to the signal. Even normalized it remains a proxy for the realized constraint, and it is used as one.
Why does a binding signal produce a degradation of Required rather than an improvement of it? The arrow is real but not direct, and the mediation is where the argument earns its credibility.
A loud error, a false BLOCK or a REPAIR that halts the clinician in real time, produces a binding signal on the loop. The signal produces pressure for correction. The pressure raises the perceived institutional cost of holding the line, of keeping Required where it stands while the friction continues. That perceived cost enters a choice, and the choice is not binary. Between raising the bar and lowering it lies a family of containment responses: suspending the system, narrowing its domain, escalating the case to a human, falling back to manual handling, reducing the volume sent through the gate. Containment holds Required in place while limiting exposure.
The ratchet therefore appears not simply when degradation is cheaper than promotion. It appears when degradation is cheaper than promotion and cheaper than every containment response available. Where a cheap containment exists, the loud friction is relieved without touching Required. Where none exists, degradation is the least-cost path, and the drift is nothing other than the gradient of cost, followed.
The asymmetry is not a failing of the staff. It is the shape of the incentive field the architecture builds. A perfectly rational actor degrades Required under this field, because silencing the loud error is cheap and rewarded at once by the fall of friction, while raising the bar against the silent error is expensive and rewarded by nothing the loop can feel.
Here is the center of the argument. A local decision, taken once, is not yet a norm. A threshold lowered on a Tuesday is an exception so long as it stays an exception on the record. It becomes a norm at a precise moment: when the degraded state is inscribed as the reference of record, when the class-three attestation replaces the class-one verification as the artifact against which future compliance is judged. Institutional memory performs this inscription, and is therefore the level at which a local choice hardens into a standing rule.
The stronger claim is about what memory does not inscribe. An institution does not record what happens to it. It records what leaves a trace. The loud error leaves one by construction: it blocked someone, generated an override, produced an artifact that enters the record. The silent error leaves nothing. A false ALLOW, passed at runtime against a blind spot, produces no interruption and no document, and does not enter memory because it never produced anything for memory to hold.
The bias of institutional memory is therefore not decay and not amnesia. It is a non-inscription. The record skews toward the loud not because the silent was forgotten but because the silent was never written.
The distinction from two neighboring bodies of work has to be drawn plainly, the resemblance being close enough to be mistaken for identity. Organizational learning, after Levitt and March (1988), takes the received signal as given and studies its bad interpretation. Organizational forgetting, after de Holan and Phillips (2004), studies the loss of knowledge once acquired. Non-inscription is upstream of both: not a bad reading of an available signal, and not the loss of a signal once held, but the structural absence of the signal at the threshold of the record, an absence the architecture decides in advance through the binding force it assigns.
A precision the concept requires. A trace can be documentary, behavioral, metric, legal, narrative, or implicit in a routine, and a general theory of memory would admit them all. This article does not need them all. On its bounded terrain, the regulated reliability system, M is the formal reference of record: the audit artifact, the recorded threshold, the attestation kept as the standard against which compliance is judged. Restricting M to that record keeps it from swelling into an omnivore. The residue is what this body of work elsewhere calls reconstructibility: what cannot be reconstructed after the fact, because nothing was laid down to reconstruct from.
A first pass suggests memory makes return expensive. That is wrong, and correcting it names the remedy. A well-built memory makes return cheap, through versioning, decision history, rollback, a retained record of the old threshold. What makes return expensive is a particular structure of memory: the degraded state becomes the default reference, the prior state loses its standing, restoration requires a fresh authorization, and the burden of proof between lowering and raising stops being symmetric.
Irreversibility is not produced by memory as such. It is produced by the asymmetry of re-inscription and restoration that a given memory encodes.
A memory built against the ratchet would version its policies, retain the prior level rather than overwrite it, require a signed justification for each degradation, expire exceptions automatically rather than let them settle into the baseline, hold the burden of proof symmetric, and keep a rollback within reach. The aim is not to record more. It is to make degradations non-self-perpetuating, so that a lowered threshold decays back toward the standard unless it is actively renewed, rather than hardening into it by default.
The word attractor should be set aside: it promises a state space, transition equations and stability conditions the argument does not supply. Feedback regimes will do.
The loud regime is correction-seeking, and the term is chosen against self-correcting, which would claim too much. The signal creates a pressure for correction; it does not guarantee the correction succeeds. The system may be unmodifiable, the cause external, the users may circumvent without improving, the managers may ignore the overrides. What the loud regime guarantees is a signal that bears on the loop, not a repair. The pressure is real and points toward correction, which is precisely what the silent regime lacks.
The silent regime is not correction-seeking, and the exact claim must stay careful: the absence of correction does not by itself make the silent error grow, it makes it persist. Three states stay apart and separate two dynamics. Silent persistence is the error remaining at constant policy: invisible, stationary, a latent liability. Silent normative accumulation is the policy dynamic, successive down-typings each lowering the bar the next decision starts from, moving Π itself. Silent exposure amplification is the volume dynamic, more cases sent through the degraded policy without any further change to Π. The first is inert. The ratchet lives in the second and, through it, in the third.
The migration from the loud regime toward the silent one is what an earlier title named the observability ratchet. Remaining in the loud regime is costly, since remaining means paying the friction again and again. Moving toward the silent one is cheap, that one being quiet and unmonitored. The system does not fail to correct the silent error through negligence. It is pulled, by the shape of its feedback and the asymmetry of its record, away from the region it can correct and into the region it cannot see. The irreversibility of that migration is hysteresis in the strict sense of path dependence: the state after a sequence of down-typings is not a function of current pressures alone but of the record those down-typings laid down, and that record does not clear when the pressure does.
A theory becomes interesting when it predicts what the incumbent cannot, sharply enough to be wrong. The law predicts that two gates with identical initial performance, differing in the binding force their architecture assigns to the two error classes, will drift at different rates and to different extents.
The gap being measured needs its own definition, since a move from class one to class two is not a move from class two to class three, and an unweighted count would treat them alike. Let the promotion gap at time t be a weighted sum over ports, G(t) = Σ w_i d(R_i_target, R_i_actual), where d is a domain-defined distance between proof-burden classes, not a subtraction. The classes are ordinal, weights alone do not supply the missing metric, and d must be built from the costs or requirements the domain attaches to each class, or give way to an ordinal indicator such as the weighted count of degradation transitions. Drift velocity is v = ΔG / Δt; drift distance to a horizon H is D_H = G(H) − G(0). The formalization is illustrative, and its service is to fix what would be recorded.
This is better read as an experimental program than a single experiment, and three difficulties keep it from being clean. Sampling the ALLOW decisions changes more than binding force: it changes the audit cost, the human load, the availability of labels, the frequency of correction, possibly the model itself. The conditions want randomization or simulation, since operators who know they are watched adapt to being watched. The unit of analysis must be fixed, gate or port or decision or period or team or risk type, since the effect can appear at one grain and vanish at another. None of this makes the prediction untestable. It makes the test a designed program, which is the honest description of what a claim about institutional dynamics requires.
The single illustration available in the interim should be framed for what it is and no more. A reliability instance for toxicological prediction, built on a molecular featurization that encodes atoms by a one-hot atomic number, carries a structural blind spot for metallic coordination complexes, whose behavior depends on oxidation states and ligand chemistry the featurization does not represent. A confident false ALLOW on such a compound is a silent error in the exact sense used here: accepted at runtime, binding force near zero, no artifact, no entry in the record.
That instance was measured elsewhere, in the applicability domain audit establishing that a domain computed downstream of the featurization certifies compliance with no referent. It shows the silent class is not a theoretical convenience: it has a concrete instance with a concrete cause. One instance proves existence, not generality, which is why the program above is owed rather than assumed.
The comparisons are best read as a shift of causal priority rather than the discovery of empty ground. These bodies of work are not blind to the selectivity of feedback. Vaughan (1996) sees the non-event that fails to alarm; organizational safety has built whole apparatus around what gets reported and ignored. What this framework does is make that selectivity the primary object rather than one factor among many, and tie it to the architecture of the gate and the transformation of the policy.
With that framing, each distinction is a change of emphasis and mechanism, not a claim of absence. Levitt and March describe learning from a received signal; here the object is the condition under which the signal exists. De Holan and Phillips describe the loss of acquired knowledge; here it is knowledge never inscribed. Automation bias, after Parasuraman and Manzey (2010) and Mosier and colleagues (1998), sits in the operator’s cognition; here the level is institutional policy. Goodhart’s law, in Strathern’s (1997) formulation, describes a measure corrupted by becoming a target; here it is the port that degrades under a field the measure does not capture, which extends the distinction between measured performance and operational reliability.
The near miss needs an exact placement, and placing it strengthens the theory. A near miss usually does leave a trace; the organization simply weights it lightly, no serious consequence having followed. The problem there is not an absent signal but a weakly binding one, which places near misses not at the silent extreme but in the interior of a continuum: a signal can be absent, present but weakly binding, or strongly binding, and the drift is governed by where on that continuum an error’s feedback falls. The silent error is the limiting case where binding force reaches zero; the near miss is the instructive middle.
The clinical literature on alert fatigue belongs here as instance rather than rival. Reported override rates for medication alerts range from roughly forty-six to ninety-six percent, and institutions respond by suppressing alerts and reconfiguring thresholds. Read through the classical lens this is noise reduction. Read through the law it is the ratchet observed in the field under another name, the suppression of the loud signal being the mechanism by which the policy migrates toward the quiet regime. The data are strong as illustration and weak as proof, having been gathered to study fatigue and not drift.
The isolation of B remains to be defended. To call B structural and the rest perturbations would confound two distinctions: regulation can be as structural as architecture, a budget as durably inscribed as a port. The honest distinction is not structural against perturbation but endogenous against exogenous to the feedback architecture, on the boundary fixed above. B_A is endogenous, a property of how the gate turns errors into signals. Regulation, budget, culture, leadership are exogenous, and may explain more of the variance in a given case than B does. B is isolated not because it dominates but because it is the one force the architecture itself produces, and therefore the one an architect can change.
The mechanism has a cancellation condition, and naming it does more work than any defense of the mechanism could. Let promotion be signed by an independent party, ALLOW decisions sampled systematically, the point of adjustment moved off the runtime into a distinct audit cycle. Then the exogenous restoring forces meet the endogenous one, the cost of degradation rises toward the cost of promotion, the absences are instrumented, memory ceases to be biased, and the drift does not begin.
This is a hypothesis for design, not a clean refutation. A single system that satisfies the configuration and drifts anyway proves little, since the drift may come from a captured auditor, a falsified record, an undersampled review, a budget shock, any of the exogenous forces the law admits. Refuting the law needs a controlled comparison in which those forces are held still and only the feedback architecture varies.
The domains named as sites of extension deserve a sharper description, and the sharpness strengthens the argument rather than weakening it. Aviation’s near-miss reporting, nuclear precursor analysis, the just-culture reporting of high-reliability sectors are not empty ground awaiting the theory. They are the mechanism already engineered against, each an apparatus for giving a binding signal to the error that would otherwise pass silent. The comparison must then separate three quantities, or it proves nothing: the nominal device, the real rate of inscription it achieves, and the real binding force it carries into the decision. A near-miss system with a low true inscription rate is an immune configuration in name only. The falsifiable corollary survives the caveat and is sharper for it: sectors that achieve high real inscription and high real binding should drift more slowly than sectors whose instruments are nominal.
One temptation is firewalled. It is tempting to conclude that institutions converge not toward the truth but toward whatever binds hardest, and to state this as a law. It is a doctrinal formula, kept as one, useful as compression and disqualified as authority. The law is conditional and holds only where a local decision loop governs policy adjustment.
The limits are entered on the page. The theory is more developed than its evidence, which rests on one instance and one repurposed body of clinical data. The central program is specified but not run. The cost ratio is argued but not quantified. The binding force is given a proxy, the normalized cost of suppression, that is itself impure and unvalidated.
The reliability gate was offered, first, as a way to encode a compromise. It encodes something else as well, without anyone deciding that it should. By making one error loud and the other silent, it sets the binding force of their signals, and through that force it sets which errors reach the record and which never do. The policy then moves toward the errors it cannot ignore and away from the errors it cannot see, and the formal record, holding only what left a trace, turns the movement into a standing rule through the one asymmetry that matters: it is cheaper to write the degraded state down than to write it back.
The observability ratchet, which gave this work its first title, is the smaller part of what it now claims. The larger part is conditional and, being conditional, testable: at equal initial error distributions, institutional policies diverge when their feedback architectures impose different binding forces on the local decision loop.
An institution does not record what happens to it. It records what leaves a trace, and something decides in advance what is allowed to leave one. That deciding thing is the architecture of feedback. It is worth naming, because a drift located in the architecture can be engineered against rather than addressed through culture alone. A safety culture can change what gets reported, attributed and inscribed, and to that extent it changes the architecture too. The claim is only that the architecture is where the drift is written, and therefore where it can be met.
Doctrinal notes and explorations on AI in regulated systems. Once or twice a month. One-click unsubscribe.