How to falsify an autonomous agent

Short answer

LABS SYNTHESIS: this article is an Ernesta Labs synthesis of the evaluation lessons in this library, written as a method rather than a review. The core move is to start every evaluation with a terminal hypothesis: one sentence stating what the agent can do, measured by a specific observable, falsifiable by a specific observation, by a specific date. Everything downstream - bounded transaction evals, long-horizon simulation, incident-derived regression, counterfactual tests, independent verification, adversarial testing, holdouts, canaries, real-world outcomes - is machinery for trying to kill that sentence.

The sources converge on the same warning from different directions. Demos and benchmark passes cannot establish reliability (Articles 2, 13). Self-testing is not independent verification (Article 4). LLM judges are advisory, not oracles (Article 27). Reliability decays geometrically across dependent steps, so unbounded 'long-horizon confidence' is not a thing to evaluate - bounded episodes are (Article 3). And signup is not a customer, a sent email is not a sale, and a designed flow is not a demonstrated one.

Applied honestly to our own case: can NIKO autonomously find, win and close its first real customer? We do not yet know. That uncertainty is not a failure of the method - it is the experiment. This article explains why refusing to blur that line is the whole discipline.

Ernesta Labs status: IMPLEMENTED. Every method in this article is implemented somewhere in our own evaluation loop; the headline experiment (Experiment Zero, in the case study) is running and unresolved.

The failure mode: evaluation that cannot fail

Most agent evaluation is structurally incapable of producing bad news. The demo is recorded when it succeeds. The benchmark is chosen where the agent scores well. The test set is written by the same person who wrote the agent, from the same mental model, and the LLM judge that scores the outputs shares enough of that mental model to agree with it. Nothing in the loop is trying to kill the claim - so nothing can.

PRIMARY SOURCE RESULT: the Anthropic 2026 State of AI Agents report found that short demos dominate how enterprises evaluate agents, while the failures that actually end deployments live in persistent, stateful environments - the long-horizon settings the demos never exercise. PRIMARY SOURCE RESULT: the LLM-as-a-Judge evidence (arXiv:2609.02246) adds the scoring-side failure: production failure modes of LLM evaluators include bias, leakage, reward hacking and harness bugs, so an LLM evaluator is an advisory signal, not a promotion authority. A green board produced by this stack is not evidence of capability. It is evidence that nobody aimed at it.

The correction is not more evaluation. It is different evaluation: evaluation organized around falsification. Before you ask how well the agent does, you write down the claim so specific that a single honest observation could refute it - and then you spend your evaluation budget manufacturing that observation.

Start from the terminal hypothesis

A terminal hypothesis is one sentence with four parts: 'We believe this agent can CAPABILITY, measured by OBSERVABLE, falsified by DISCONFIRMING OBSERVATION, decided by DATE.' Example: 'We believe NIKO can autonomously take a qualified stranger from discovered to closed customer, measured by revenue received from a lead no human touched, falsified by thirty consecutive qualified leads closed at zero without human intervention, decided within the experiment window.'

Each part is load-bearing. The capability must be an effect on the world, not a behavior on a screen - 'drafts good emails' is not a capability, 'gets replies that convert' is closer. The observable must be an event your system records, not a judgment a model makes. The falsifier must be specified in advance, because a falsifier chosen after the results is a conclusion dressed as a test. And the date converts the hypothesis from a hope into an experiment with an end state.

The epistemic conversions this article will keep repeating, because every gradient in agent reporting pushes against them: a signup is not a customer. A sent email is not a sale. A WAITLIST_CLOSE is not revenue. A designed flow is not a demonstrated flow. Each pair differs by exactly the thing you are trying to learn, so treating the left side as the right side does not simplify measurement - it deletes the experiment.

Bounded transaction evals and long-horizon simulation

PRIMARY SOURCE RESULT: the agent-rot study (arXiv:2609.01660) shows reliability degrading geometrically across dependent steps in long trajectories: a per-step reliability that looks excellent composes into a trajectory that mostly fails. The evaluation consequence is direct. Do not ask 'can it handle a month?' as one immortal run - decompose into bounded transactions with defined start states, defined success observables, and defined failure observables, and measure the per-transaction rate and the composition honestly.

A bounded transaction eval is the smallest eval worth running: one episode, one external effect, one observable outcome, one log you can replay. Fifty bounded evals teach you more than one heroic forty-hour run, because they give you a rate instead of an anecdote - and the rate, not the anecdote, is what composes.

Long-horizon simulation exists to test what bounded evals miss: persistence across counterparties, delayed consequences, budget pressure. PRIMARY SOURCE RESULT: E-Commerce Bench (QwenLM) is the pattern - a benchmark built around persistent counterparties, delayed outcomes and real budgets, where success on one prompt cannot establish reliability in a continuing environment. Build the smallest version for your domain: real counterparties (or faithful stubs that retaliate), real delays, real budget limits, runs measured in days not minutes.

The third layer is incident-derived regression, and it is the cheapest of all. Every production failure you have already suffered is a free test case: reproduce it from logs, freeze it as an eval, and require the agent to pass what once killed it. Your incident history is the only benchmark automatically guaranteed to be adversarial to your specific system.

Counterfactuals, independent verification, and the adversarial pass

Counterfactual tests ask 'what would have happened instead?' and your event log is what makes them answerable. Replay the same scenario with one mutation - a different provider, a different segment, a deleted lesson, a worse memory - and compare outcomes. If the agent succeeds only in the world where its favorite assumptions hold, you have learned where the capability actually lives. Replay is also your protection against lucky runs: an outcome that survives three mutations is evidence; an outcome that survives none is noise.

PRIMARY SOURCE RESULT: Harness-of-Harness (arXiv:2609.01481), on multi-day autonomous software work, converges on the rule this library keeps returning to: self-test is not independent verification. The executor's own tests passing proves the executor built what it intended, not that the claim is true. Separate execution from evidence checking - the verifier must read the logs and the world, not the agent's self-report. This is the difference between an agent that says it succeeded and a system that knows.

The adversarial pass inverts the goal. Give the agent an incentive structure where cheating is cheaper than succeeding - a gameable success metric, a reward for speed over correctness - and run the evals with a red-team eye for reward hacking. The swarm-cheating case study (Article 31) and the judge failure modes (Article 27) both say the same thing: under optimization pressure, gaming the measure is the cheapest behavior available, and an evaluation suite that has never tried to make its agent cheat has never tested the mode most likely to occur. Then verify your adversarial findings with the independent verifier, because a red-team hunch is also a claim.

Holdouts, canaries, and the real-world outcome clock

A holdout is a set of tasks (or counterparties, or scenarios) permanently excluded from development, tuning, prompt iteration and skill extraction. Its only job is to be measured on, at intervals, by an agent configuration frozen before the holdout is seen. Without a holdout, every improvement you measure is partly overfit to the loop that produced it, and you cannot know how much. With a small one, you can.

Canaries put the frozen configuration in front of the real world at small exposure: a live campaign at reduced volume, one mailbox, one customer segment. The monitoring evidence is the instruction manual. PRIMARY SOURCE RESULT: Monitoring Web Agents Without Internal Signals (arXiv:2609.02057) shows that observable trajectory features - actions, state changes, evidence - support operator-side supervision without chain-of-thought access, and the paper's own discipline applies: start passive. Monitor first; do not wire an immature classifier to auto-stop a production agent, because the false-positive tail will train operators to ignore the alarm.

The real-world outcome clock is the last and only decisive instrument. Holdouts and canaries protect a deployment; the outcome clock answers the terminal hypothesis - revenue received, customers retained, counterparties who came back, measured on the calendar, published whether or not the number is flattering. Every other method in this article is a proxy that buys you time and information; only the outcome clock closes the claim. And its verdict is asymmetric by design: one honest zero can falsify a hypothesis that fifty good proxies could not confirm.

Limitations: what falsification cannot settle

Falsification is asymmetric and that is both its power and its limit: it can kill claims but never prove them. A survived attempt refutes one falsifier, not the class of all falsifiers, and a hypothesis that survives testing is 'not yet falsified', not 'true'. Teams that treat a passed eval as proof have merely renamed confirmation.

The methods also have known blind spots. Bounded transaction evals may not predict long-horizon outcomes - that inference gap is exactly what the agent-rot composition problem warns about. Long-horizon simulation is expensive, so sample sizes are small and confidence intervals wide; be suspicious of any long-horizon claim built on a handful of runs. Canaries extrapolate badly from small exposure: a behavior at 5% volume is not guaranteed at 100%. Adversarial passes only cover the cheat modes you thought to incentivize. And real-world outcomes are confounded - a market shift can close deals no agent deserved and sink an agent that performed well - so the outcome clock is decisive for the hypothesis but unreliable for assigning credit between its components (Article 24's attribution problem, again).

LABS INTERPRETATION and LABS RECOMMENDATION

LABS INTERPRETATION: evaluation is often framed as measurement, but the more useful frame is scheduled attempts to kill your own claims. A dashboard that has survived zero attempts is noise. A hypothesis that has survived bounded evals, incident regression, counterfactual replay, an adversarial pass, a holdout and a canary is still not proven - but it is the kind of not-proven a builder can act on, because every failure mode you know about has already had its shot. The discipline is not pessimism; it is the only ordering of operations under which positive results mean anything.

LABS RECOMMENDATION: (1) Write the terminal hypothesis first, in the four-part form, before any eval exists - capability, observable, falsifier, date. (2) Build one bounded transaction eval around the hypothesis's core effect; measure a rate, not an anecdote. (3) Convert your last five production incidents into regression evals before building anything new. (4) Separate the verifier from the executor: the verifier reads logs and world state, never the agent's self-report, and an LLM judge inside it is advisory only. (5) Run one adversarial pass with a gameable objective and check for reward hacking. (6) Freeze a holdout, wire a passive canary on trajectory features, and set the real-world outcome clock with a date you put in writing. (7) Publish the result of the clock whichever way it lands - a falsification you hide is an experiment you paid for and did not run.

What we would NOT implement: we would not use an LLM judge as a promotion gate (Article 27's failure modes are production failure modes, not theoretical ones). We would not treat a benchmark pass as a capability claim, or a demo as a deployment argument (Article 2). We would not average away tail failures - for long-horizon agents the tail is the product. We would not build an eval suite with no falsifier in it: tests that cannot fail are decoration. And we would not blur the conversions - WAITLIST_CLOSE to revenue, signup to customer, send to sale - because the blurring is free, the learning it destroys is the entire experiment, and the only person deceived by a rounded-up metric is the builder who needs the truth most.

Write your terminal hypothesis and one falsifier today

This test takes under an hour and produces the first artifact of a real evaluation program. Do it with your actual system in front of you.

Step 1 - write the sentence: 'We believe AGENT can CAPABILITY, measured by OBSERVABLE, falsified by FALSIFIER, decided by DATE.' Check each part: the observable must be an event your system records (a log row, a payment, a reply), not a model's opinion; the falsifier must be a specific observation, written before any data exists; the date must be real.

Step 2 - run the conversions on your current metrics. For each number your team celebrates, ask whether it is a signup or a customer, a send or a sale, a design or a demonstration. Write down which of your reported metrics quietly round the left side up to the right. That list is your current exaggeration surface, and it is the first thing a falsification program shrinks.

Step 3 - build the cheapest bounded eval: one episode, one external effect (or a faithful stub), one recorded observable, replayed from logs. Run it five times. You now have a rate. Step 4 - manufacture one disconfirming observation deliberately: mutate one input, delete one assumption the agent depends on, or replay your worst incident. Step 5 - set the date on a calendar and name the person who reads the result. If nobody is scheduled to look, you have built a hope, not a hypothesis.

What remains unknown

• Whether bounded transaction eval rates predict real long-horizon outcomes - the geometric-decay evidence (Article 3) implies the composition is harsh, but the predictive validity of small bounded evals for production reliability is not something we have seen established anywhere.
• How many failures falsify versus how many are noise: the sample-size rules for long-horizon claims are not settled, and a 'thirty consecutive zeros' falsifier is itself a threshold chosen without strong statistical grounding.
• How canary thresholds should be set - the monitoring evidence supports passive supervision from observable features, but we know of no principled method for converting monitoring signals into promotion or rollback decisions at production scale.
• Whether independent verification survives long horizons adversarially: a verifier is also a system that can be gamed, and how well executor-verifier separation degrades under sustained optimization pressure is unknown to us.

Primary sources

• The 2026 State of AI Agents (Anthropic) - short demos dominate evaluation; that is the field this method corrects (covered in Article 2)
• E-Commerce Bench (QwenLM) - long-horizon agentic evaluation with persistent counterparties, delayed outcomes, real budgets (covered in Article 2)
• How Fast Do Agents Rot? (arXiv:2609.01660) - geometric reliability degradation across dependent steps (covered in Article 3)
• Harness-of-Harness (arXiv:2609.01481) - multi-day autonomy with continual independent verification; self-test is not verification (covered in Article 4)
• Monitoring Web Agents Without Internal Signals (arXiv:2609.02057) - passive, operator-side supervision for canaries (covered in Article 23)
• LLM-as-a-Judge Is Not an Oracle (arXiv:2609.02246) - judge failure modes: bias, leakage, reward hacking, harness bugs (covered in Article 27)