Systemic Risk Management for Distributed Probabilistic Systems
The question is not about agent autonomy per se. It is about propagation of uncertain assertions through delegated decision authority into interacting real-world effects.
A system can fail even when no component violates its local contract. A SafetyTriage agent stays within confidence thresholds. MedicalReview stays within escalation limits. RegulatoryReporting stays within filing authority. Yet the system produces outcomes none of them intended: cascading escalations overwhelming capacity, correlated errors amplifying across the fleet, feedback loops destabilizing the whole.
The problem is not defective agents. The problem is that distributed probabilistic systems exhibit emergence. Local decisions interact. Assertions compound. Effects accumulate. The system becomes a higher-order entity with properties not reducible to component properties.
Agentic systems require systemic governance not because agents are unreliable, but because emergence is real. Local containment is necessary but insufficient.
Systemic governance is a closed-loop control problem under partial observability. The control plane cannot perfectly observe the system it regulates. It works from noisy observations, constructs state estimates, and makes decisions based on estimates it knows are incomplete. Feedback determines whether the system stabilizes or diverges.
Local risk is baseline: an agent makes a probabilistic inference and is correct only with probability p.
Propagation mechanisms convert local risk into systemic risk through distinct channels:
Amplification: Assertions carry epistemic uncertainty, but downstream agents treat them as facts. A SafetyModel generates “high risk” with confidence 0.65, marked INFERRED. The Safety Committee escalates with 0.78. By RegulatoryReporting, it is OBSERVED fact with 0.85. Original epistemic status was laundered away. False certainty drives the system.
Correlation: If all agents depend on the same model, all agents amplify the same error simultaneously. This is not independence; it is synchronized failure across supposedly diverse fleet.
Interaction: One agent’s decision changes context for the next, not by amplifying error, but by changing environment. Agent A modifies a priority score. Agent B observes and reallocates resources. Agent C escalates a decision. None individually exceed authority. Together they form feedback loop no one caused. The system does.
Structural factors determine whether propagation becomes catastrophic: dependency concentration, capacity saturation, state-estimation uncertainty, temporal dynamics.
Current frameworks conflate three distinct concepts under “autonomy.” They must be separated.
Autonomy Envelope: What an agent is permitted to do. An action space constrained by policy.
Capacity Envelope: System’s ability to absorb and process effects without entering unrecoverable states. A dynamic property: Capacity_available(t) = ServiceRate(t) - Charge(t) - Backlog(t).
Risk Budget: Domain-specific allocation of tolerable loss. Effects are incommensurable—clinical escalations, financial exposure, regulatory incidents cannot be added into single score. But they can be tracked in parallel with separate thresholds.
The relationship: Capacity_available(t) constrains Risk_budget_available(t) Risk_budget_available(t) constrains Autonomy_envelope(t)
Action is allowed if: PolicyPermits(action) AND RiskCost(action) ≤ RiskBudget_remaining[domain] AND ServiceRate_available ≥ ServiceRate_required(action).
Three envelopes, three distinct mechanisms, three separate feedback channels.
The recursive insight: the governance plane produces assertions about system state. Those assertions carry epistemic uncertainty. The control plane is subject to the same epistemic limits it imposes on agents.
When the governance plane estimates “System at 87% capacity utilization,” this is an assertion with confidence 0.73 and validity window 5 minutes. But governance treats it as fact when making decisions. This is the exact problem diagnosed at agent level.
The governance plane must be transparent about its own epistemic status. State estimates carry explicit bounds. Control decisions are made knowing uncertainty.
Can the true system state be reconstructed from observable outputs? If not, governance is blind. State identifiability is precondition for effective control.
Here is the critical architectural gap: what happens if the control plane itself fails?
If the policy engine crashes, state estimator diverges, decision authority is compromised, the system must not collapse into paralysis or uncontrolled action.
The system requires a degraded mode:
ControlPlaneFailure detected (state estimate diverges, policy engine unresponsive) ↓ Switch to Cached Policy (last known good policy state) ↓ Reduce Autonomy Envelopes automatically (conservative defaults) ↓ Route high-impact decisions to Human Approval Gates ↓ Signal Anomaly (alert operators that system is degraded)
This is not failure; this is graceful degradation. System loses optimization but retains safety.
The control plane itself must be designed as fault-tolerant system with watchdog timers, fallback mechanisms, explicit failure detection. The degraded mode is not hope. It is architecture. Without it, control plane is common-mode failure for entire system.
Governance is not a maturity level to achieve. It is an operating point to maintain. Optimal governance exists, but it is neither unique nor stable.
G_t* = argmin[ ExpectedLoss_t(G) + GovernanceCost_t(G) ]
Governance operates at optimum that shifts as conditions change. Frameworks that resist recalibration become obstacles.
Continuous recalibration requires: Observable state (adequate instrumentation), Policy flexibility (ability to adjust constraints quickly), Feedback loops (consequences flow back), Adaptive thresholds (rather than fixed rules).
This is proportional response to changing conditions, not randomness.
Assertion_t → Decision_t → Effect_t → System-State_t+1 (observed with delay) ↓ State-Estimate_t+Δt_obs (observability requirement: can true state be reconstructed?) ↓ RiskBudget_t+Δt_policy (determines, with delay) ↓ Autonomy-Envelope_t+Δt_enforce (constrains, with delay)
Delays matter. Observation delay means estimate lags reality. Policy delay means constraints based on stale information. Enforcement delay means new assertions made before old constraints take effect.
Instability can arise not from bad policy but from delays exceeding system integration time. Control theory teaches: delays in negative feedback loops can cause instability and oscillation.
More fundamentally: if system is not state-observable, feedback loop is built on blind control. Governance cannot reconstruct what system is actually doing. Loop becomes reactive rather than predictive.
State identifiability is foundation. Observability is engineering requirement. Both must be designed in before governance can be effective.
Emergence is neutral. It can be protective or pathological.
Protective emergence: Redundancy prevents cascade, quorum requirements prevent single-model bias, graceful degradation preserves core function.
Pathological emergence: Cascade amplification, synchronized failures, feedback loops destabilizing, overgovernance paralyzing.
Governance task is to enhance protective emergence and suppress pathological emergence. You cannot prevent emergence. You can only shape it.
Optimal governance balances: sufficient constraint to prevent pathological emergence, sufficient freedom to allow protective emergence.
The trilogy closes on itself.
Article 1 asked: what is the primitive? Answer: Capability. Agent is implementation choice.
Article 2 asked: how do we know? Answer: Assertions carry epistemic status orthogonal to content. Authority independent of epistemology.
Article 3 asks: how do we control? Answer: Control plane itself is subsystem subject to epistemic limits and feedback. Must be designed as fault-tolerant infrastructure.
No governor has privileged access to ground truth. Governance operates through observation, inference, intervention—all integrated into governed system.
The architecture that emerges: Bound local authority. Preserve epistemic provenance. Govern system exposure. Control feedback loop. Make control plane fault-tolerant.
This is deeper than “manage agents better.” It is the architecture of control for distributed probabilistic systems under partial observability.
The conceptual shift is profound: from “is this agent safe?” to “is the system absorbing risk at the rate we can tolerate?”
Build accordingly.
Doctrinal notes and explorations on AI in regulated systems. Once or twice a month. One-click unsubscribe.