And it costs us lives — anthropocentrism as an obstacle to patient safety
We demand that medical algorithms “explain themselves.” The AI Act enshrines it in law, ethics committees chant it, academic conferences dedicate entire tracks to it. Health AI start-ups boast: “Our AI is explainable: it relies on symbolic AI and Bayesian networks.” The implication being that it would know what it does — better than we do. Traceable, yes — at a computational overhead nobody prices into demos. Understandable, perhaps. But one question suffices to shake the edifice: can you explain your last important decision? Not the smoothed, coherent version constructed after the fact. The real process — the actual activation of your neural circuits, the causal sequence from stimulus to act. Nobody can.
Imagine a hypothetical MRI capable of imaging in real time the activity of every neuron, every synapse, every electrochemical gradient in your brain at the moment of a difficult clinical decision. Would you have explained that decision? No. You would have a map — exhaustive, precise, spectacular. But a map is not a law. Knowing what happened physiologically is not knowing why those activations produce this choice and not another. It is the distinction between syntax and semantics, between how and why, between observation and understanding. AI engineers, pressed for a technological answer, respond with traceability — as if documenting the path were equivalent to understanding why that path. Traceability and explainability are not synonymous.
The word “explainable” does not exist in the absolute. It exists for someone, within a mental model, in a given context. Explainable to a senior radiologist? An intern? A 72-year-old patient? An administrative judge? An ANSM regulator? An AI expert? These six people do not share the same conceptual space for receiving an explanation. Every explanation is a projection — a passage to a lower-dimensional space, calibrated to the recipient, never to the real process. What XAI produces (LIME, SHAP, attention maps, counterfactuals) are local approximations of global behaviour. Studies demonstrate it: mechanistically incorrect explanations generate confidence. The explanation fulfils a rhetorical function, not an epistemic one.
The double standard is not a scientific error. It is a defensive anthropocentric posture. We impose a standard of justification on the machine because we are deeply uncomfortable with a simple, disturbing idea: a non-conscious entity can make better decisions than we can without having to explain itself. An expert cardiologist who misses 25% of NSTEMI on ECG while telling a compelling clinical story is objectively less reliable than a deep learning algorithm achieving comparable or superior performance on the same task. Distrust of algorithmic decisions spans decades; the algorithm capable of outperforming the clinician is only five years old. We chose to trust the cardiologist not out of rationality, but out of ontological comfort. Out of vanity. Five decades of cognitive neuroscience have methodically demolished the assumption that humans could explain their decisions rationally and without bias.
If the real objective is patient safety, the question is not “why did the algorithm decide this” but “to what extent can we rely on it, and when might it be wrong.” These are measurable, auditable, actionable questions: inter-run variance on identical input, uncertainty calibration (does the model know when it doesn’t know?), stability against non-significant perturbations, error stratification by populations and clinical conditions, drift detection — post-market surveillance exactly as for a drug, pharmacovigilance concepts applied to AI medical devices.
True transparency is not mechanical, it is systemic. It does not ask the algorithm to explain each of its decisions. It asks developers to document training data, known limitations, validation populations; integrators to verify adequacy to the clinical context; clinicians to assume the final decision, informed by the system. This is a model of distributed responsibility — the same that prevails in aviation, nuclear, and pharmacology. The best analogy is not the autopilot (deterministic, formally verifiable). It is the drug: we do not always understand the molecular mechanism of action, we validate clinically, monitor post-market effects, and map contraindications. Just as computing the binding free energy between a ligand and its receptor does not predict a drug’s clinical efficacy, tracing the successive states of a neural network’s weights does not explain it. In both cases, emergent phenomena govern. This is all we should require of the algorithm: not explainability, but measured, audited, monitored reliability.