Monitoring agents without reading their minds

Short answer

PRIMARY SOURCE RESULT: 'Monitoring Web Agents Without Internal Signals' (arXiv:2609.02057) studies how to detect failing web agents using only observable trajectory features - the actions taken, the tool results, and the environment state changes - without access to chain-of-thought, self-report, or any internal reasoning signal.

The reason this matters: self-report is not evidence. A degraded agent can narrate confident, coherent progress while its actions loop, its tool calls fail, and the environment stays unchanged. The paper's approach makes supervision stand on what actually happened in the world, not on what the agent says happened.

The catalog of observable decay signals includes repeated identical actions, action loops, tool failures, grounding failures (claims about page content that the page does not support), unchanged environment state, contradictions between agent claims and observed results, fallback accumulation, UNKNOWN operations the observer cannot classify, strategy oscillation, permission-denied attempts, stale commitments declared done but never verified, and absence of verified progress.

Ernesta Labs' position: start PASSIVE. Run these classifiers in observation mode, log their alerts, and let humans review - do not auto-stop production agents on immature classifiers. The paper's limitations (web-agent/ReAct-specific evidence, cost of repeated samples, partly judge-derived labels, and false-cut rates) mean an automated kill-switch built on this today would fire on good runs and pass bad ones.

The failure mode: the fluent liar

Most agent supervision today is built on the wrong surface. Teams inspect chain-of-thought, ask the agent 'how is it going?', or require the agent to file self-assessments. All of these share one defect: they ask the suspect for testimony. Chain-of-thought is the agent's account of its own reasoning, not the reasoning itself; self-report is the agent's account of its own progress, not progress itself. Both can be fluent, confident, and detailed precisely when the underlying behavior is failing.

This is not a hypothetical worry. The agent-rot literature (article 3) shows that long-horizon degradation is invisible in the output: the model keeps producing confident, coherent steps long after its grounding in the actual world has decayed. If your supervision surface is the output, you have chosen the one channel that degrades in lockstep with the failure it is supposed to catch.

The alternative is to supervise on observables: what actions were taken, what the tools returned, and whether the environment actually changed. This treats the agent like any other production system. You do not debug a database by asking the database how it feels; you look at the query log. The paper we study here asks whether that is possible for agents - and what you can see.

What the paper actually did

PRIMARY SOURCE RESULT: 'Monitoring Web Agents Without Internal Signals' (arXiv:2609.02057) studies monitoring of web agents - agents that take actions on the web through tools - using only externally observable signals from the trajectory: the sequence of actions, the tool call results, and the changes (or non-changes) in the environment. It deliberately excludes chain-of-thought and self-reported state from the monitor's input.

PRIMARY SOURCE RESULT: the paper organizes supervision at two levels. Macro monitoring looks at the trajectory as a whole - is this run healthy or not - using aggregate features over the full action history. Micro monitoring looks inside the trajectory for specific decay events: repeated identical actions, loop structures, failed tool calls, and other localized pathologies. The two levels answer different operational questions: 'should a human look at this run' and 'where in the run did it start going wrong'.

PRIMARY SOURCE RESULT: the paper reports benchmark evidence that observable features alone can separate healthy from failing runs at useful rates, without any access to the agent's internal reasoning. We deliberately do not quote exact accuracy numbers: they are tied to the paper's specific web-agent harness, task set, and label sources, and copying them onto your system would be false precision. What transfers is the finding that the observable surface carries enough signal to be worth building on - not a guarantee that any particular classifier will transfer.

The label pipeline deserves its own sentence, because it bounds everything downstream: key-step labels in the paper's evaluation were partly derived using judge models. That means even the ground truth the monitors were scored against carries some model-derived uncertainty - a point we return to in the limitations section, and a direct instance of the LLM-as-a-judge problem we treat in article 27.

The catalog: what decay looks like from outside

The paper's most practically useful contribution is a vocabulary for externally observable failure. Ernesta Labs read it as a checklist you can instrument against any tool-using agent today. Repeated action: the same call issued again and again, verbatim or near-verbatim. Loops: a cycle in the action graph - A, then B, then A again, with nothing learned in between. Tool failures: calls that error, time out, or return empty, and their rate over time.

Grounding failures: the agent claims things about tool results or page content that the results do not support - a fetched page that does not contain the claimed button, a search result that does not contain the claimed fact. Unchanged environment: the agent narrates progress while the diff of environment state is empty; nothing it did moved the world. Contradictions: later claims that conflict with earlier observed results in the same trajectory.

Fallback accumulation: the agent silently substituting easier actions or degraded modes when the intended action fails - each individual fallback is often reasonable, but accumulation is a decay signal. UNKNOWN operations: actions the observer cannot classify into the expected vocabulary, which in production usually means a capability behaving in a way its registry entry did not promise. Strategy oscillation: flipping between two incompatible plans without completing either. Permission-denied attempts: repeated attempts at actions the agent is not allowed to take, which is both a health signal and a security signal. Stale commitments: tasks declared complete that were never verified complete. And the aggregate: no verified progress - a trajectory in which nothing checkable was achieved.

None of these requires reading the model's mind. All of them are derived from the action log, the tool results, and an environment diff - artifacts a competent harness already has or should have. That is the point: the supervision surface already exists in most stacks; it is just not being read.

Macro and micro: two questions, two monitors

Macro monitoring asks the run-level question: is this trajectory healthy, and should a human look at it now? Its inputs are aggregate features - action diversity, failure density, environment-change counts, repetition ratios - and its output is a triage decision. Macro monitors are cheap to run and high-leverage because they catch the fluent liar: the run whose self-report is a success story while its observable features are a pile of loops and empty diffs.

Micro monitoring asks the step-level question: where did this run start decaying, and what event marks the turn? Its inputs are localized pattern matches - the specific loop, the specific grounding failure, the specific permission-denied retry - and its output is an annotation on the trajectory. Micro output is what makes postmortems fast: instead of rereading a thousand-step trace, the reviewer starts at the flagged step.

Ernesta Labs studied the two levels against our own incident taxonomy and found the mapping straightforward. The 3am question - 'is anything wrong right now?' - is macro. The Monday question - 'what happened to this relationship over the last week?' - is micro over a long trace. A monitoring design that only answers one of them leaves the other to humans reading logs by hand, which is exactly the manual review this research tries to retire.

Limitations - preserved, not softened

The paper's own limitations matter more here than in most of our articles, because the natural engineering response - 'wire this to a kill-switch' - is exactly what the evidence does not support. First, the study is web-agent/ReAct-specific: the agents act on the web through tool calls in an observe-act loop. How much of the signal catalog transfers to non-web agents - coding agents, data pipelines, conversational systems with few observable actions - is untested.

Second, repeated samples are expensive. Several of the paper's detection approaches rely on running the same episode more than once or on sampling the trajectory repeatedly; in production, where a run is not a benchmark episode but real work on a real customer relationship, you often get exactly one sample. Detection methods that assume resampling have no direct commercial equivalent.

Third, the key-step labels used for evaluation were partly judge-derived - produced by LLM judges, not by humans or ground-truth instrumentation. A monitor's measured accuracy is therefore conditional on a judge's agreement, and judges carry their own failure modes (article 27). Fourth, false-cut rates - the rate at which healthy runs get flagged as failing - were, by the paper's own analysis, at levels unsuitable for direct commercial intervention. A classifier that stops production agents at those false-positive rates is a system that cancels good work at 3am and calls it safety.

Stated plainly: this paper is evidence for building observation. It is not evidence for building automatic intervention on top of that observation.

Why builders care

Because the alternative is worse and most teams are living in it: supervision by vibes. The common production stack has a dashboard of token counts, a folder of session transcripts nobody reads, and a human noticing problems when a customer complains. Chain-of-thought inspection does not scale past a handful of runs, and self-report is testimony from the party with the most to hide.

Observable monitoring scales, because it runs on logs you already produce. Every action your agent takes is a log line or should be; every tool result is a record or should be; every environment change is diffable or should be. The marginal cost of a passive monitor over an existing structured event log is small - far smaller than the cost of the incident it catches early, and far smaller than the human hours it retires.

It also composes with the rest of this research arc. Provider health (article 22) feeds it: tool failures become attributable to a provider or to the agent. The trace model (article 10) gives it a substrate: an append-only event ledger is exactly the input a macro/micro monitor wants. And it is the honest foundation for later intervention: you cannot cut a failing run reliably until you can detect a failing run reliably, and you cannot do either from the agent's own account of itself.

What we would implement, and what we would not

LABS INTERPRETATION: the paper's deepest contribution is not any single classifier but the repositioning of the supervision surface. Once you accept that chain-of-thought and self-report are narrative artifacts rather than evidence, your monitoring architecture writes itself: structured events in, observable features out, humans in the loop for anything consequential. The catalog of decay signals is a free gift from the paper - a checklist of what to instrument that costs an afternoon to log and months to rediscover without it.

LABS RECOMMENDATION: start PASSIVE, in this order. One: instrument the event log - every action, every tool result, every environment diff, every permission denial - because monitors are only as good as the substrate. Two: implement the cheap macro features first: repetition ratio, failure density, environment-change count, verified-progress presence. Three: implement micro annotations for the top failure modes in your own history - your system already has a favorite pathology; find it in the catalog and flag for it. Four: run everything in observation mode with alerts routed to a human queue, and score every alert against ground truth you establish by review. Five: only then discuss gating, and only on the narrowest class of runs, with human approval required for any irreversible stop. We have not reached step five and do not recommend anyone reach it on current evidence.

What we would not implement: no auto-stop of production agents on these classifiers today - the false-cut rates alone disqualify direct intervention, and resampling-based detection has no commercial equivalent. No chain-of-thought-as-supervision fallback: if the observable surface is thin for your agent type, the answer is more instrumentation, not less. And no treating UNKNOWN as failure: unclassifiable actions are a signal to extend the vocabulary, not a reason to page someone - the paper's own UNKNOWN category exists precisely because observers meet actions they cannot yet name.

The passive monitor sprint: run it this week

You need one agent in production or staging, its structured event log (or the ability to add one), and five signals. If your log does not exist yet, that is the first day's work: log every action, every tool result, and a hash or diff of environment state at each step. Do not start with the classifier; start with the substrate.

Implement five features over the last N completed runs. Repetition ratio: fraction of actions that exactly or nearly duplicate the previous action. Failure density: tool-failure count per hundred actions. Environment-change count: number of steps where the environment diff is non-empty. Stale commitments: tasks marked complete with no corresponding evidence record. Verified progress: whether the run ended with at least one independently checkable achievement. Each feature is a day or less of code over a structured log.

Then score your last month of runs by hand: pick the ten you know went well and the ten you know went badly, and see how the five features separate them. Two honest outcomes are acceptable. If the features separate the groups, you have a macro monitor - wire its alerts to a human queue and start counting false alarms. If they do not, you have learned your failure mode is not in the paper's web-agent vocabulary, which is itself a finding worth an afternoon. In neither case do you wire a kill-switch. The kill-switch question is what you earn after a month of counted alerts, not what you install on day one.

What remains unknown

• How much of the observable-signal catalog transfers beyond web/ReAct agents to coding agents, data pipelines, or low-action conversational systems - untested in the paper and untested by us.
• The cost-quality frontier in production: resampling-based detection has no commercial equivalent when a run is real work on a real customer, and no published result covers single-sample detection quality at production scale.
• How much of the reported accuracy survives judge-free labels: the paper's key-step labels were partly judge-derived, and we have no measurement of monitor quality against human-verified ground truth, in the paper or in our own estate.
• Whether any observable-feature threshold can ever justify automatic intervention in commercial settings - the false-cut rates and per-domain sensitivity are unknown, which is exactly why we run passive.

Primary sources

• Monitoring Web Agents Without Internal Signals (arXiv:2609.02057)