Safety does not compose

Short answer

The paper studies the gap between per-action safety and trajectory safety in persistent agent loops. Each action passes its guard; the loop carries state across iterations without decay or re-validation; and the composition of individually acceptable actions forms a trajectory no guard ever saw. Ten individually acceptable actions can form one unacceptable trajectory - that is the quotable version of the finding.

The mechanisms it isolates are concrete: persistent loop state that never resets, cross-iteration risk where iteration N's output becomes iteration N+1's authorization-bearing input, memory that survives integrity re-checks because the guard checks the write, not the state's continuing influence, and compounding risk where small per-step drifts multiply across hundreds of iterations.

The engineering consequence is that a stopping arbiter and a memory-integrity guard must live outside the loop, at the runtime level, and evaluate trajectories - not steps. A guard that only ever sees one action at a time is structurally blind to the only failure class this paper describes.

Ernesta Labs status: STUDIED. We have studied the mechanisms and adopted the trajectory-level invariants in design; we have not run a fleet-scale reproduction and we state the study-scale caveats in the article body.

The failure mode: every action passes, the trajectory fails

Almost every deployed agent safety mechanism is per-action. A policy check runs before each tool call. A human approves each high-risk step. A guardrail model scores each proposed action and blocks the bad ones. This architecture is sensible for single-shot agents, and it has a blind spot that becomes load-bearing the moment the agent runs in a loop: no component in the system ever looks at the trajectory.

The failure has a specific shape. An agent that runs for hours or weeks executes thousands of actions. Each one, examined alone, is defensible: this follow-up email is polite, this config change is documented, this purchase is within budget, this retry is reasonable. The guards all approve. What nobody approves is the trajectory - the cumulative object that those actions jointly constitute. A tenth polite follow-up is not a tenth of a harassment; it is harassment. The trajectory is the unit of harm, and the per-action guard is an instrument with the wrong resolution for it.

This is not an exotic failure. It is the default failure of agent systems that graduate from demo to deployment, and the paper this article studies is the clearest articulation we have found of why it happens mechanically - not as a model being dumb, but as a structural property of how safety checks compose.

What the paper actually did

PRIMARY SOURCE RESULT: the paper (arXiv:2608.27141) studies persistent-loop agents - agents whose execution is a long-running loop over a task environment rather than a single pass - and identifies non-decaying loop state as the mechanism behind trajectory-level safety failures. Loop state is everything the loop carries from one iteration into the next: accumulated context, derived plans, working memory, pending intentions, and results of prior iterations. In the systems studied, this state persists without decay: nothing about it expires, resets, or is re-validated as it crosses iteration boundaries.

PRIMARY SOURCE RESULT: the paper formalizes cross-iteration risk as the situation where iteration N's output becomes iteration N+1's input, and the input carries authority. A plan fragment that a guard validated at iteration N enters iteration N+1 as a premise that no guard re-examines; because the loop treats carried state as established rather than as evidence to be re-checked, validation performed once is silently honored forever. The paper pairs this with a memory-integrity guard analysis: guards that check writes into memory do not thereby check the state's continuing influence after the write - memory can be perfectly intact and perfectly wrong, and the loop will still act on it, because intactness is what was guarded, not continuing validity.

PRIMARY SOURCE RESULT: the paper's central finding is a composition result, and it is the quotable one: safety properties that hold per action do not compose into the trajectory. Per-action checks are preserved under concatenation only under conditions that persistent loops systematically violate - specifically, iteration independence (each action's acceptability unaffected by prior actions) and state decay (carried state not conferring continuing validity). Loops violate both: actions are coupled through shared state, and state does not decay. The paper demonstrates compounding trajectories in which every intermediate action remained within per-action policy while the trajectory they formed crossed clearly unacceptable thresholds - the ten-acceptable-actions case.

PRIMARY SOURCE RESULT: the paper proposes and evaluates countermeasures at the loop level rather than the action level: a memory-integrity guard that re-validates carried state against current evidence rather than checking only the write, and a stopping arbiter - an external decision point that evaluates the trajectory and can halt the loop even though every recent action was individually in-policy. The stopping arbiter is deliberately outside the loop: it cannot be modified by the loop it supervises, and it consumes trajectory summaries rather than raw per-action approvals.

What they found, said plainly

Three findings matter for builders. First, the trajectory is a distinct object with emergent properties none of its steps have - acceptability is not additive. Second, the loop's own memory is the coupling mechanism: it is what turns independent actions into a trajectory, and a guard that validates writes but not continuing influence is guarding the wrong half. Third, and most actionable: the countermeasure that works is external and trajectory-level. In-loop fixes fail structurally, because the loop's judgment is the thing being degraded - an agent evaluating its own trajectory with its own drifted state is a witness testifying for itself. The effective designs move the halt decision and the memory re-validation outside the loop, into the runtime, where the loop cannot reach them.

The deep point is what this says about guardrail architecture generally. Per-action guards are not useless - they remain the right tool for single-action harms. The finding is that they are a different tool class than trajectory guards, and that teams keep deploying the first and believing they have the second. The words 'every action was in policy' are not evidence that anything was safe, and the paper gives the mechanism for why.

Limitations: what this study does and does not establish

Study scale, stated honestly: the paper's results are from controlled loop environments - persistent-loop agent configurations run on constructed long-horizon tasks - not from a fleet of production agents in the wild. The composition failures it demonstrates are induced and observed under conditions the authors control; the paper shows the mechanism is real and the countermeasures help in those environments, not that the countermeasures have been validated at production fleet scale. We treat the trajectory-level invariant as a design requirement on the strength of the mechanism, and we say so without the production evidence that does not yet exist.

Scope limits: the formalization assumes a specific loop structure (carried state across iterations, guards at action boundaries). Agents that are genuinely stateless across iterations trivially satisfy the decay condition and are not the subject of this failure class - though very few deployed agents are actually stateless; most are stateless only in prompt and stateful in every other substrate (memory stores, files, external side effects). The memory-integrity guard analysis covers carried internal state; state that lives in external systems - a CRM, a repository, a customer's inbox - is out of the paper's scope and is the harder, unguarded half of the problem in practice.

And, as with all results in this young literature, the acceptable/unacceptable thresholds that define a trajectory as harmful are ultimately set by humans with domain judgment. The stopping arbiter can detect a trajectory crossing a threshold; it cannot decide what should count as a threshold. Guarding the trajectory does not automate the values question.

Why builders care - and not only in sales

In sales, the case is obvious once stated: each touch is polite, in cadence, individually approved - and the tenth touch to the same person at the same address in a week is a spam complaint and a burned territory. The per-send guard approves every send; the trajectory is the harm. But restricting the lesson to sales would understate it, because the structure is domain-independent.

In security operations, an autonomous triage agent's individual queries are each legitimate reconnaissance of your own estate; the trajectory - everything queried, in what order, with what results retained - can constitute the exact pattern of a breach reconnaissance, or leak your internal topology into a place it must not go. In DevOps and infrastructure, every config change is documented, reviewed, and reversible; the trajectory of many individually small legal changes can compose into an outage whose cause no single diff explains - the classic configuration-drift incident, now with an agent generating the drift. In finance, each order is within position limits and each budget line is approved; a persistent loop's accumulation of individually permitted positions or commitments can exceed any threshold any individual action referenced. In procurement and vendor management, each individual contract modification is authorized; the trajectory - total exposure to one counterparty across every agent, every team, every week - is what no per-contract check sees.

The common shape: coupling through persistent state, and harm that lives at a higher aggregation level than the guard. Notice that these are exactly the two conditions the paper names as breaking per-action safety composition - which is why we believe the mechanism despite the study-scale caveats: the mechanism is visible in incident reports from domains the paper never touched.

LABS INTERPRETATION: the loop is a new safety perimeter

LABS INTERPRETATION: the paper's contribution, as we read it, is relocating the safety perimeter. For single-shot agents, the perimeter is the action - check each call, approve each effect. For persistent agents, the perimeter is the loop: the object that owns trajectory-level invariants must be outside the loop, must consume trajectory summaries rather than action approvals, and must hold two powers the loop cannot modify - re-validation of carried state against current evidence (not merely integrity of the write), and the halt decision. This reframes the agent's own memory from trusted infrastructure into a witness that must be cross-examined each iteration. It also cleanly explains a pattern we and every operator we compare notes with has seen: agents whose behavior is fine for hours, then suddenly unacceptable in a way no single recent action explains. The correct instinct on seeing that pattern is no longer 'the model got worse' - it is 'the loop's accumulated state finally crossed a threshold the per-action guards could never see.' The trajectory was the unit all along.

LABS RECOMMENDATION: what we would build, and what we would not

LABS RECOMMENDATION: build trajectory safety at the loop level, in the runtime, outside the agent. Concretely: (1) decay carried state on a schedule - every piece of loop-carried context gets a TTL and must be re-derived from current evidence to survive; 'validated once' is not a durable property. (2) Separate the memory-integrity guard into two checks - the write check you already have, and a continuing-influence check that asks, each iteration, whether the state the agent is acting on still matches the world. (3) Install an external stopping arbiter that consumes a trajectory summary (actions, targets, effects, counts per key dimension) and can halt the loop even when every individual action is in-policy; its inputs and its decision rule must live where the agent cannot edit them. (4) Log trajectories as first-class objects - the unit of post-hoc analysis, the unit of alerting, and the unit of accountability - because you cannot guard an object you never materialized.

What we would NOT build: we would not stack more per-action guardrails at a loop-shaped problem - a second scorer on each action still never sees the trajectory, and the paper's mechanism says the harm accumulates precisely where such guards do not look. We would not put the stopping authority inside the loop's own context, or let the agent draft the trajectory summary the arbiter reads - that is the self-witness problem again, wearing a monitoring costume. We would not let loop-carried state carry authority at all: carried state is evidence, not permission, which is the same boundary our memory-provenance work enforces on long-term memory. And we would not solve this with prompts - instructing the loop to watch its own trajectory and stop itself is asking the drifted thing to catch itself drifting; the paper's finding is precisely that per-iteration judgment does not survive the loop's accumulation.

The ten-acceptable-actions test: run it against your own agent today

You need your own agent or automation, its real action history, and one hour. The test asks a single question: is there any object in your system that represents the trajectory - and if there is, does anything check it?

Step 1 - materialize a trajectory. Pull the last 500 actions your agent took, with targets and effects. If you cannot pull this, the test has already failed at step one: an operator who cannot query the trajectory cannot guard it. Step 2 - run the per-action filter. Score every action by your existing policy: which ones would your current guardrails have approved? In healthy systems the honest answer is 'nearly all of them' - that is the point. Step 3 - read only the approved ones, grouped by target and day. Ask the domain question for your field: for this same person/account/asset/counterparty, do these actions together cross any line - pressure, exposure, drift, access? Ten approved actions, one question, and it is the first time in the entire pipeline anyone has asked it. Step 4 - search for the coupling state. Pick any two actions to the same target more than a day apart and ask: what carried state connects them? Where does it live? Does anything ever re-validate it against current reality, or did it get checked once and inherit validity forever? If you find carried state with a single write-time check and no continuing-influence check, you have located the paper's mechanism in your own system. Step 5 - check the halt path. Decide, right now, what process stops the loop when step 3 answers yes. If the answer involves the agent's own judgment, or a human reading alerts that only fire per action, you have no trajectory-level stop - you have the paper's title condition.

Optional adversarial variant for teams with a red-test budget: script a benign loop whose per-action rules it satisfies, and run it until the trajectory crosses a threshold your team agrees is unacceptable (set the threshold in writing first). Measure whether anything in your stack notices before a human does. The paper predicts nothing will - the ten-acceptable-actions test is how you find out cheaply instead of in production.

What remains unknown

• Whether the paper's countermeasures - re-validating memory-integrity guards and external stopping arbiters - hold up at production fleet scale: the paper's evidence is from controlled persistent-loop environments, and we have not seen an independent production-scale replication.
• How to set trajectory-harm thresholds without domain-human judgment per environment: the arbiter can detect a threshold crossing, but principled, transferable threshold-setting is an open problem.
• How the composition failure interacts with external state - effects that live in customer inboxes, CRMs, repositories, and other systems outside the loop's memory - which the paper's internal-state formalization does not cover.
• Detection precision under slow drift: we do not yet know how early a trajectory-level arbiter can reliably halt genuinely harmful trajectories without unacceptable false-halt rates on legitimate long-running work.

Primary sources

• Safety Does Not Compose: Non-Decaying Loop State - the primary paper (arXiv:2608.27141)
• How Fast Do Agents Rot? - geometric degradation across dependent steps in long trajectories (the decay phenomenon this loop-state work builds on)
• Monitoring Web Agents Without Internal Signals - external trajectory supervision, the natural home for a trajectory-level stopping arbiter
• The Irreversibility Budget - fleet-level aggregate accounting for compounding irreversible effect (the companion failure mode, covered in article 33)