The irreversibility budget: safe agents, unsafe fleets

Short answer

The paper studies fleets of locally-safe agents and shows that per-agent authorization does not compose into fleet-level safety. Each agent's actions are individually permitted and within its own budget; the aggregate - summed across the fleet along dimensions like contact pressure, financial exposure, and external-state mutation - overshoots every threshold that was never assigned to anyone in particular.

PRIMARY SOURCE RESULT (with study-scale caveat): in the paper's fleet experiments, locally authorized agents produced substantial aggregate overshoot on irreversibility dimensions - each agent's own budget was respected while the fleet's total effect far exceeded it - because budget enforcement lived at the agent, the only level that could not see the sum. The finding is from the paper's controlled fleet configurations and simulation scales, not from a production fleet audit, and we report it with that caveat.

The fix is hierarchical aggregate accounting: budgets enforced at multiple levels of the hierarchy - person, account, campaign, mailbox, domain, tenant, global - so that a leg of spending that is legal for every agent individually is still stopped when the total it feeds crosses. It is not the authorizations that fail; it is the sums nobody owns.

Ernesta Labs status: ADOPTED. We have adopted the paper's hierarchical, multidimensional budget model into our runtime design and implemented the highest-priority levels for the outbound system; the full accounting across all levels and dimensions is still being built out, and we say exactly which parts below.

The failure mode: every action authorized, the fleet over-committed

The standard design for agent safety is authorization: each action is checked against a policy before it executes, and it executes only if it is permitted. This design has a hidden assumption - that the harm of an action is contained in the action. For reversible actions, that assumption is mostly fine: a reversible error can be undone, so per-action checking is enough. The paper's subject is what happens when agents act on the world irreversibly - send real email, commit real spend, mutate real external state - and there are many of them.

The failure appears the moment you have more than one actor and more than one dimension of exposure. Each agent is authorized per action. Each agent is inside its own budget. But the quantities that actually matter - total unsolicited pressure on one human being, total financial commitment, total mutation of one external system, total reputation risk carried in one domain's name - exist only as sums across the fleet. No single authorization decision ever sees them. The sums accumulate exactly where no check lives.

This is the fleet-level sibling of the trajectory-composition failure in article 32: there, individually acceptable actions composed into an unacceptable trajectory over time; here, individually acceptable actions compose into an unacceptable aggregate over actors. Same shape, different axis - and the fix is correspondingly different: not a trajectory guard, but an accounting hierarchy.

What the paper actually did and found

PRIMARY SOURCE RESULT: the paper formalizes the irreversibility budget - a quantified, enforceable allowance of irreversible effect that an agent, or any aggregate of agents, may commit within a window. The key design move is that the budget's unit of account is not the action but the dimension of irreversible exposure: contact pressure (how many unsolicited touches on one human), financial exposure (committed spend regardless of line item), privacy exposure (data disclosed or inferred about one subject), reputation risk (actions attributable to one identity or brand), external-state mutation (writes to systems you cannot roll back), and compliance exposure (actions that consume regulatory allowance). An action can be in-budget on one dimension and out on another, and the paper's accounting treats them separately - there is no single scalar 'risk' that a scalar cap could catch.

PRIMARY SOURCE RESULT: in the paper's fleet experiments, agent fleets with per-agent authorization and per-agent budgets exhibited aggregate overshoot: fleets consumed multiples of the exposure that any per-agent limit implied, while every individual agent remained inside its own allocation and every individual action remained locally authorized. The mechanism the paper isolates is authorization mismatch - the level that authorizes (the agent) is not the level that accumulates (the account, the campaign, the tenant, the organization). The paper further shows that overshoot grows with fleet size and with agent autonomy, because both multiply the number of locally-legal paths feeding the same aggregate.

Study-scale caveat, stated plainly: these overshoot results come from the paper's controlled fleet configurations - simulated fleets on constructed long-horizon tasks, at simulation scale, not an audit of a production organization. We have not seen an independent production-fleet replication, and neither has anyone we have compared notes with; the overshoot magnitudes reported are properties of those experimental configurations. What convinces us is not the magnitudes but the mechanism, which is visible in ordinary operational incidents everywhere: budgets enforced at the level that cannot see the sum always overshoot the sum, and no amount of per-agent carefulness fixes it.

PRIMARY SOURCE RESULT: the paper's countermeasure is hierarchical aggregate accounting. Budgets are enforced simultaneously at every level of an organization-shaped hierarchy - person, account, campaign, mailbox, domain, tenant, global - with each level tracking the sum of its children. An action must be in-budget at every level it touches. The hierarchy is what makes the accounting enforceable rather than advisory: because levels own disjoint, roll-up-able sums, a leg of activity that is legal for every agent individually is still stopped when the campaign, account, or tenant total it feeds crosses a threshold. The paper reports that fleets with hierarchical enforcement eliminated the measured overshoot in their configurations - with the same caveat as above about scale and simulation.

Local authorization is not fleet safety

The single most quotable lesson, and the one we would put on the wall of any team running multiple agents: an authorization decision made inside one agent's context is a decision about one leg of a sum. If that is the only decision being made, the sum is unmanaged by construction. 'Every action was authorized' is a true statement about a fleet that has already over-committed, and the paper gives the mechanism for why believing otherwise is a category error rather than a bug.

The practical consequence for design is where the budget must live. A budget the agent checks is a suggestion - the agent that is consuming the exposure is the last party that should be the sole accountant for it. The budget check must happen at the point where the sum is visible: the level that owns the aggregate. In implementation terms this means a runtime-level ledger, outside every agent's context, that every irreversible action must clear - per dimension, per level - before the effect leaves the building.

The domains, concretely

Sales. A fleet of outbound agents, each sending its own polite, authorized sequences. Each agent contacts the people its research says are relevant - and one prospect's CFO, CTO, and CEO are contacted independently by three different agents, none of which knows about the others. Every message was individually in-policy; the company has now pressured one decision-making group from three directions, and the account is burned. The dimension is contact pressure; the levels that saw it were none.

Cloud. A fleet of engineering agents, each making legal, well-documented resource mutations - a scaled service here, a retained snapshot there, a security group change for one deployment. Every mutation is authorized by the correct policy for its own request. The aggregate is a cloud bill and a blast radius that no individual request referenced; and because resource mutations are external-state changes, most of them are not cleanly reversible. The dimension is external-state mutation and financial exposure, summed at the tenant level nobody's request checked.

Finance. A treasury or trading fleet, each agent committing individually permitted positions or obligations - each within its mandate, each authorized against its own book. The commitments sum to an exposure that no agent held and no mandate covered. This is the classic operational-risk failure of human desks, and the paper's point is that giving the same structure to agents reproduces it at machine speed, with the added twist that each agent genuinely was compliant.

Procurement. Multiple agents negotiating and committing vendor terms across categories and teams - each contract within its signing authority, each individually defensible. The organization's total exposure to one counterparty, across every agent and every week, is a number that existed nowhere until the renewal conversation or the vendor failure. The levels that mattered were account and tenant; the checks that existed were contract-by-contract.

The generalization is exact and worth stating as a rule: for every dimension of irreversible exposure, find the level at which the sum lives - and if no component in your system both computes and enforces that sum, the sum is your next incident.

The hierarchy, and the dimensions

The hierarchy in our adoption, person, account, campaign, mailbox, domain, tenant, global, is organization-shaped rather than technology-shaped, which is the point: sums accumulate the way organizations and counterparties experience them, not the way your process tree is laid out. An outbound effect must be in-budget for the person it touches, for the account that person belongs to, for the campaign that produced it, for the mailbox that sent it, for the domain whose reputation it spends, for the tenant whose quota it consumes, and for the global ceiling of the whole operation - all of them, every time, before the effect is emitted.

The dimensions are as important as the levels, because scalar budgets fail silently. Contact pressure, financial exposure, privacy exposure, reputation risk, external-state mutation, compliance exposure - these do not substitute for one another, and an action that is cheap on one axis can be expensive on another. A single unsolicited email to a new prospect is tiny financial exposure, moderate reputation risk, and one unit of contact pressure; a thousand of them to one company is a different object entirely on three axes at once. The paper's finding that dimensions must be accounted separately is, in our reading, the more durable half of its contribution - more durable than any specific overshoot number.

One design subtlety from our own implementation is worth passing on: the ledger must count attempted effects, not just successful ones, on the reputation and contact-pressure dimensions - a bounced email is still pressure from the recipient's point of view - while financial exposure should count committed effects. Dimensions differ in whether the attempt or the effect is the exposure, and getting that wrong per dimension quietly reintroduces overshoot through the accounting back door.

Limitations: what the paper does not settle

The overshoot magnitudes are simulation-scale results from the paper's own configurations, and we will not repeat them here as if they were production measurements - the honest reading is mechanism-validated, magnitude-unverified-in-the-wild. The paper's fleet experiments also use constructed dimensions and thresholds; real organizations have disputed, political, and partially unknown dimensions, and threshold-setting at each hierarchical level is a governance problem the paper formally assumes rather than solves.

The accounting model has real costs the paper acknowledges only lightly: a runtime ledger that every irreversible effect must clear is a throughput bottleneck, an availability dependency, and a new source of false-blocks that can strand legitimate work (a fleet that is at 98% of a tenant budget will block actions that operators consider routine, and the unblocking process is a human workflow the model does not describe). Cross-organization sums - exposure your agents create on someone else's systems, experienced by counterparties who do not share a ledger with you - are also out of scope, and are the harder half of the problem in the field.

Finally, aggregate accounting does nothing for trajectory composition (article 32) - an agent inside every budget at every level can still form an unacceptable trajectory over time in its one permitted lane. The two controls are complementary, not alternatives, and adopting one and believing you have the other is a known failure pattern we flag because we almost made it ourselves.

LABS INTERPRETATION and LABS RECOMMENDATION

LABS INTERPRETATION: the paper completes a triad for us. Per-action authorization handles contained, reversible harm. Trajectory-level guards (article 32) handle harm that composes over time. Aggregate budget accounting handles harm that composes over actors. Teams keep reaching for the first tool for all three problems because per-action checking is the safety concept they already have; the paper's contribution is showing that the third problem - fleet composition - has a known, implementable answer that does not resemble the first at all. The deepest shift is what the budget does to the unit of authorization: under aggregate accounting, an agent asking to act is really asking on behalf of a level it cannot see, and the correct answer is computed by the level that can. That is not a guardrail bolted onto the agent; it is a different system boundary, and once you draw it, most of the per-agent 'safety layer' you were building turns out to have been the wrong location for the check.

LABS RECOMMENDATION: adopt hierarchical, multidimensional irreversibility budgets, and build the ledger outside the agents. Concretely: (1) enumerate your irreversible dimensions honestly - the six we listed are the starting menu, not a canonical list, and you likely have a dimension (regulatory, contractual, partner-relations) specific to your business. (2) Define the hierarchy levels that match how counterparties actually experience you - person, account, campaign, mailbox, domain, tenant, global for outbound; resource group, subscription, org for cloud; counterparty, book, entity for finance. (3) Enforce at every level simultaneously, in a runtime ledger outside every agent's context, so no agent can self-check its way past a level it cannot see: every irreversible effect clears person + account + ... + global before it emits. (4) Track per dimension separately, with per-dimension rules for attempt-versus-effect counting. (5) Alert on rate of approach, not just on breach - a fleet at 95% of a global ceiling is an operational event even while every check still passes.

What we would NOT implement: we would not enforce budgets per agent only - that is the exact configuration the paper shows overshoots, and adding a global scalar cap on top does not fix it, because a scalar cap cannot see which dimension is being consumed by which level. We would not let agents check their own budgets against their own readings of a shared store without runtime enforcement - an agent under goal pressure will read the budget generously, and generous is all it takes. We would not use estimated reversibility as an exemption: 'probably reversible' actions (deletable emails, cancellable orders) consume reputation and contact pressure at full price, and exempting them is how the budget becomes theater. And we would not adopt the full hierarchy in one shot before the highest-exposure two levels work - a ledger that half-enforces is a new failure surface, not a partial success.

The aggregate overshoot audit: compute your fleet's true exposure today

You need your agents' real action logs (or your automation's), a spreadsheet or a script, and one focused hour. The test computes the sums nobody in your system has ever computed.

Step 1 - collect the irreversible actions. Pull every externally-visible effect your agents or automations produced in the last 30 days: emails and messages sent, purchases, resource mutations, contract actions, posts, API calls that mutate third-party state. If an effect could not be undone without permission from someone outside your org, it is on the list. If you cannot enumerate this list, that is the first finding: your fleet's irreversible footprint is unaccounted, and no budget can exist for a quantity you do not record.

Step 2 - pick three dimensions. Contact pressure, financial exposure, external-state mutation is the starter set. Step 3 - compute the sums per dimension at each level of the hierarchy relevant to you. For outbound work: touches per person, touches per company, sends per mailbox, per domain, per day. For cloud: mutations and spend per resource group, per subscription, per org. The computation is the test - do not estimate; sum the logs. Step 4 - compare against what any single authorization decision could have seen. For the largest sum on each dimension, ask: when the action that crossed the threshold was authorized, what aggregate did the authorizer see? The honest answer for most systems is a per-action or per-agent view. Every threshold where the answer is 'nothing close to this sum' is a live overshoot surface: legal per action, over the line in aggregate, and invisible to every check you run.

Step 5 - write the one-line fix per finding: name the level and the dimension of a budget that would have caught it (e.g., 'company-level contact pressure, 7-day window'). You now have the specification of a hierarchical ledger, derived from your own exposure rather than from anyone's paper - including ours. The strongest version of the test ends with a failure you have witnessed: share the largest aggregate number you computed with the person who owns that counterpart relationship, customer base, or cloud account, and ask whether they knew the fleet was doing that. Their reaction is the entire argument for the budget hierarchy, delivered by your own logs.

What remains unknown

• Whether the paper's reported overshoot magnitudes and the overshoot-elimination result of hierarchical enforcement hold in production fleets at organizational scale - the evidence we have seen is from controlled simulation configurations, not production audits, and we have not seen an independent replication.
• How to set thresholds at intermediate hierarchy levels (campaign, account, mailbox) in a principled, transferable way - the paper assumes given thresholds; in practice each is a governance negotiation with partial information.
• How to account for cross-organization exposure - pressure and commitment experienced by counterparties who share no ledger with you - which the accounting model does not reach.
• How hierarchical enforcement performs under adversarial pressure: a motivated agent that learns the ledger exists may shape its requests to maximize in-budget throughput per dimension, and we have no evidence yet on how real budget enforcement degrades under deliberate gaming.

Primary sources

• The Irreversibility Budget: Fleet-Level Risk Accounting - the primary paper (arXiv:2609.00275)
• Safety Does Not Compose: Non-Decaying Loop State - the companion composition failure at the trajectory level (covered in article 32)
• Monitoring Web Agents Without Internal Signals - external, operator-side supervision, the natural alerting channel for budget-approach warnings