Compliance frameworks are necessary. They are not sufficient to make a regulated system structurally governable.
A paradox has been running through the executive committees of the healthcare and life sciences sector: a large majority of organizations claim to have initiated structured AI governance programs, while an equally significant proportion expresses a persistent sense of inadequacy when confronted with concrete requirements of oversight, traceability and audit. This paradox reveals a confusion rarely articulated with sufficient clarity: the confusion between a governance policy and a governance architecture.
These two objects do not belong to the same register. A governance policy organizes responsibilities, rules and escalation mechanisms from outside the system. A governance architecture defines the structural properties that make certain behaviors traceable, bounded or structurally unacceptable by design, from inside the system.
The thesis of this article is that, in AI systems operating in regulated environments with high individual decision stakes, procedural governance frameworks are necessary but structurally insufficient. When traceability and bounding properties are not built into the design, governance remains retrospective, partial and vulnerable to the workarounds that any organization under stress will mechanically produce.
Procedural governance refers to the set of rules, committees, reference frameworks, evaluations and documentary processes applied from outside the system’s structure. Architectural governance refers to the structural properties that make the system’s behavior traceable, its validity perimeter explicable, and its modes of action controllable at the level of the system’s own functioning.
Two further distinctions are necessary. Reported auditability reconstructs a decision after the fact from artifacts produced around the system. Native auditability is the property of a system whose ordinary functioning itself produces the elements necessary for faithful reconstruction of the execution context. Native auditability does not mean complete explainability of the internal mechanism — it means constitutive traceability of the context, inputs, outputs and execution regime of the decision.
The major governance frameworks (NIST AI RMF, ISO 42001, EU AI Act) were built according to a principle of technological neutrality. This choice, understandable from a regulatory standpoint, has reinforced a reading whereby governance can be added after design, through procedural and documentary superimposition. To this regulatory bias, a cultural bias must be added: much of contemporary software engineering has been shaped by logics of rapid iteration and incremental correction, which become fragile when AI enters a decision chain whose effects must be justifiable, bounded and auditable. The paradox takes its precise meaning here: these organizations do not lack policies — they deploy systems that were not designed to be governed by those policies.
Retrospective traceability is a reconstruction, not a proof. In a standard ML pipeline, the output of an inference is not accompanied by a complete and immutable reconstruction of the operational context. Post hoc explainability methods (SHAP, LIME) provide useful indications but are too often mobilized as a substitute for a traceability the system was never designed to produce.
Inventory-based governance remains vulnerable to shadow AI. Architectural governance can make detectable by construction the integration of non-compliant components within the institutional perimeter — it does not eliminate lateral or extra-system uses, but it fundamentally changes the visibility regime of the problem.
Under operational pressure, procedure degrades faster than structure. An architectural property enjoys a different technical status: if inference requires effective persistence of trace elements, the failure of that persistence becomes a failure of the normal execution regime, not merely a documentary shortfall.
Constitutive traceability requires each relevant inference to produce, as a normal condition of its execution, a stable record of the elements necessary for its reconstruction. The trace must be a property of the system’s mode of functioning, not an optional byproduct.
Operational qualification of the validity domain requires the system to evaluate, at the point of execution, whether the input lies within the model’s operational validity domain. A system that predicts outside its validity space without signaling it is a system that lies about its own certainties.
Structural separation of decision regimes requires the system to explicitly distinguish what falls within effective automation and what falls within recommendation subject to human validation, encoded in the design of flows, interfaces, permissions and escalation mechanisms.
Certain architectures make native auditability more accessible, more stable and more economical. Event-driven architecture associated with event sourcing patterns occupies a particular place: relevant state changes are represented as explicit, timestamped events persisted in an append-only log. The event is not only the unit of information — it is the unit of audit. The same mechanism that makes components coordinable makes the system governable.
Architectural governance has a real upfront cost. It does not replace procedural governance — it makes it more credible. Third-party foundation models remain partially opaque: the native auditability of the integrating system is not the native auditability of the third-party model. The decision/recommendation boundary becomes more complex in multi-agent agentic systems — this is an open problem in the literature.
In the Sentinelle IA / PREDICARE program, the patient digital twin is modeled as a sequence of immutable timestamped events, enabling each predictive decision to be placed within a reconstructable chain by construction. In the ToxTwin V2.3 pipeline, the applicability domain is evaluated for each submitted molecule as a component of the inference pipeline, conditioning the presentation of the output — a form of protection against architectural disinformation. These two cases have value as illustrations of feasibility, not as general proof.
The most important consequence is a shift in sequencing. Questions of traceability, domain bounding and decision regime separation must enter architectural decisions much earlier than the compliance phase. The challenge is not to add more rules or committees — it is to bring into architectural decisions questions that are still too often treated as belonging exclusively to compliance.
The industry does not need to oppose procedural governance and architectural governance. It needs to stop believing that the former can durably compensate for the absence of the latter. The central question is not merely how to govern the AI we deploy — it is how to design systems whose governability is part of the architecture, and not of the documentation added around it.