Governability is not inventoried; it is tested under perturbation, and it is distinct from resilience
A human signature on an algorithmic decision grounds accountability, not governability. When Arizona requires, effective 1 July 2026, that a medical director personally sign any denial of coverage produced by an algorithm (HB 2175), and Colorado does the same as of 30 June (SB 24-205), the legislator obtains what it aims for: a designated responsible party, an open remedy, a grounded challenge. What it does not obtain is the system’s capacity to catch the decision if it turns out to be wrong three weeks later, when the patient has already left the circuit.
The thesis of this note holds in one sentence: the governability of a system is not demonstrated by the presence of safety mechanisms, but by their performance when they are put to the test. A circuit breaker that has never been tripped is not a guarantee, it is a decoration. The demonstration bears on automated systems endowed with an identifiable supervisor, and borrows its mature counterexamples from aviation, electrical grids and crisis management; it does not cover distributed orders without a central supervisor.
The risk is not the vagueness of one term but the confusion of three, which must be posited as a chain from the system toward its supervisor. Resilience is a property of the system: its capacity to absorb a perturbation without external intervention, through redundancy, containment, graceful degradation or margins. A resilient system takes the hit alone. Recovery is an operational property: the capacity to restore an acceptable space of states after the crossing of an unrepresented perturbation. It names three measurable objects, an acceptable space, an event that exits it, a return into the domain, and is instanced by the activation of a Disaster Recovery Plan after a disaster. Governability, finally, is not a property of the system but of the system/supervisor relation: the supervisor’s capacity to maintain the system within an acceptable space, or to bring it back, attested under test.
The decisive distinction reduces to this: resilience describes what the system does alone, governability what a third party can still do with it. The first is a virtue of the system, the second a virtue of the couple. It is this difference of object that separates the concept from the resilience of complex systems as Perrow, Hollnagel or Woods think it. Having governance resources is not being governable: the French army of 1940 had observation, authority and means, and collapsed for want of articulation to the rhythm of the event.
Recovery takes two forms that get confused. The first is reversibility, to go back, to undo, declined into non-equivalent registers: technical (rollback), decisional (annulment), legal (remedy), physical or clinical (repair of a real harm). This is where the illusions of safety lodge: one certifies a technical rollback while leaving intact a clinical harm already done, since a denial of reimbursement can remain legally reversible when the biological harm no longer is. The second form is resilience, surviving without going back: surgery, aviation and nuclear crisis management operate on largely irreversible decisions and remain governable because they absorb the harm.
Reversibility is therefore the most powerful case of recovery, not its sole modality; where it is impossible, resilience takes its place. What governability adds is the one question neither form poses of itself: are this return or this absorption within reach of a supervisor, at the right moment, and has it been verified?
What separates the systems that hold from those that fall is not luck, it is mobilizable recovery capacity. The list of collapses (Three Mile Island, Challenger, Long-Term Capital Management, the 2010 Flash Crash, Fukushima) contains only cases where the perturbation won; the symmetrical list exists nonetheless, that of air traffic control, electrical grids and civil aviation, which treat the unforeseen as routine. Catastrophes do not refute the thesis, they instance it: they are the cases where recovery was missing.
The word around which everything turns must still be named correctly. The pertinent category is not surprise, too vague, but the perturbation not represented in the design assumptions: a slow drift and a brutal shock belong to it equally, and the drift is often the more dangerous because it triggers no alarm. The limit is owned: one never demonstrates the handling of a specific unknown, only the performance, under test, of generic recovery capacities (width of the restorable state space, speed of return, independence of catch-up paths, margin before the irreversible). Governability does not suppress uncertainty about the unprecedented; it displaces the bet, from “did we foresee this event?” toward “do we have a catch-up machine generic enough, and have we run it?”.
Recovery capacity rests on four resources that form a graph of dependencies, not a pyramid: observability (seeing what the system does), intelligibility (understanding why), authority (acting without asymmetric sanction) and intervention capacity (having the means to act). Their dependencies are crossed: without sufficient observability recovery is unusable, but strong observability compensates weak intelligibility, as we have piloted the steam engine, antibiotics and deep learning well beyond what we explain of them. And without authority, intervention is fictional: a supervisor sanctioned for having blocked a flow learns to stop blocking. These four variables describe not the system alone but what a supervisor can see, understand, decide and do about it; that is why they fall under governability and not resilience alone.
Recovery does not disappear by decision, it erodes by dilution, under four mechanisms that reinforce rather than add to one another. The economic mechanism: automation collapses the marginal cost of the decision without touching that of serious examination, and the cost/benefit ratio of rigor inverts under volume. The organizational mechanism, the most tenacious: responsibility is asymmetric, one sanctions the supervisor who blocks a legitimate flow but rarely the one who lets a machine error through, so he learns that validating is risk-free. The cognitive mechanism: as the model goes on not erring, validation becomes reflex, and this automation complacency is all the stronger as the model is good. The agentic mechanism, the most recent: when a system itself orchestrates planning, delegation and execution, the chaining of decisions ceases to be legible and the power to undo bears on an object become opaque.
None of these mechanisms presupposes an intention: AI sets no stage, it displaces two constraints, cost and intelligibility, and this displacement alone suffices to make the stage set the spontaneous equilibrium. The consequence is that governability is a dynamic property: a system recoverable today may no longer be tomorrow, by simple accumulation of layers, without any decision ever having removed the circuit breaker. This is what no point-in-time audit will see, because it certifies a state when the problem is a trajectory.
Governability is not ascertained on the record, it is tested, and that is where everything is decided. The existing texts, GDPR Article 22, AI Act Article 14, the NIST AI RMF, ISO/IEC 42001, are already evolving toward notions of supervision effectiveness; the critique does not target them, it targets their operationalization. The root cause is an asymmetry of verifiability: the presence of a human is almost free to ascertain (a signature, a timestamp, a log), while recovery capacity must be simulated, exercised and timed. The signature wins because it is cheap to verify, not because it is effective.
Adopting recovery as a declarative indicator would produce circuit breakers on paper and procedures never tested: this is Goodhart’s law, and the riposte is not a better presence indicator but a test of effect, as one restores a backup instead of verifying that it exists. Four magnitudes then become opposable, on condition of being measured on a real exercise: the recovery time between detection and return, the fraction of states effectively restored relative to those targeted, the margin before the irreversible threshold, and the coverage of scenarios actually played. These figures are worth nothing outside the protocol that produces them: they prove that a catch-up functioned here within this time, not that it will function in the face of an unrepresented perturbation. The thesis stops at three owned limits: uncertainty about the unprecedented is irreducible, the passage from the individual supervisor to the collective (tumor board, board of directors, committee) displaces the problem toward the political, and designing a system recoverable by construction is a distinct undertaking.
One does not govern what one watches. One governs what one can recover, and only as long as one has put it to the test. Governability is not the existence of control mechanisms: it is the persistence of a power of action under perturbation. A system ceases to be governable not when it commits an error, but when no realistic intervention can any longer inflect its evolution. At that instant, and only then, supervision becomes a stage set: an observable presence, without effective power of orientation.
Doctrinal notes and explorations on AI in regulated systems. Once or twice a month. One-click unsubscribe.