CivBench and the reflection-action gap

Short answer

CivBench (arXiv:2609.02459) evaluates agents over long trajectories with many tools under partial observability, using Civilization VI as the environment. The failure class it exposes is the reflection-action gap: agents that correctly diagnose a situation, state a commitment in their reasoning, and then never issue the tool calls that would fulfill it.

The benchmark also shows agents failing to query state that their next action depends on. They act on an assumed world instead of checking the real one, even when the check costs one cheap tool call. Knowing is not doing, and observation is not automatic.

Ernesta Labs treats this as a measurement problem before it is a model problem. We define two metrics - Proactive Monitoring Rate and Commitment Execution Rate - and put deterministic watchers behind obligations that cannot be allowed to fail silently.

The failure mode: agents that know and still do not do

Most agent demos measure whether the model can produce the right plan. Almost none measure whether the plan survives contact with a long run. The failure that shows up in long trajectories is stranger than a wrong plan: the agent produces the right diagnosis, writes the right commitment into its own reasoning, and then proceeds to do something else - or nothing at all.

We call this the reflection-action gap, following the benchmark we study in this article. The agent's reflective layer says 'I must move that unit before ending the turn' or 'I should check the provider status before retrying'. The action layer then ends the turn, or retries blind. Both statements live in the same trace. A reviewer reading the reflection would sign off. A reviewer reading the action log would not.

A second, related failure sits under it: failure to query important state. Long-horizon environments are partially observable. The world has facts the agent can only get by asking - a tool call, a lookup, a status check. The failure is not that the agent queries and gets bad data. It is that the agent never queries. It acts on a world model it built earlier and never verifies, even when the query is cheap and the action is expensive.

Both failures are invisible to the metrics most teams use. Task completion rate will not catch them, because the task sometimes completes anyway. Token-level evals will not catch them, because the reasoning reads beautifully. The failure lives in the space between what the agent said and what the agent did, and you have to be measuring that space to see it.

What the benchmark actually does

PRIMARY SOURCE RESULT: CivBench (arXiv:2609.02459, 'CivBench: Long-Horizon Benchmark for Tool-Mediated Agents') evaluates LLM agents inside Civilization VI, exposed through an MCP server (civ6-mcp, open source). The game is the point, not the gimmick: it provides genuinely long trajectories (hundreds of turns), a large tool surface (many queryable game systems), and hard partial observability (fog of war, hidden opponent state).

PRIMARY SOURCE RESULT: The benchmark scores agents on outcome and on trajectory-level process measures, which is what makes it useful for this article. Because every game action and query passes through the MCP server, the full stream of what the agent chose to look at versus what it chose to do is recorded and scorable.

PRIMARY SOURCE RESULT: The design deliberately separates what the agent says in reflection from what it does in action, making the reflection-action gap directly measurable rather than anecdotal. This separation is the benchmark's most valuable property for practitioner work, more than any leaderboard number.

What they found

PRIMARY SOURCE RESULT: Across evaluated models, the paper reports two trajectory-level patterns that matter to builders. First, agents under-query: they act on outdated or assumed state when a state-querying tool call was available and cheap, particularly as trajectories grow longer and the context fills with earlier reasoning. Second, stated commitments frequently do not convert into the corresponding actions: the agent's reflection names a next step, and the following action window does not contain it.

PRIMARY SOURCE RESULT: Both problems get worse with horizon length. Short trajectories look competent; the same agent late in a long game checks less and follows through less, even though each individual reasoning step still reads as coherent. This is consistent with the degradation pattern we covered in Article 3 of this library (Agent Rot, arXiv:2609.01660): reliability is not uniform across a long run.

We will not reproduce the paper's numbers here, because reading numbers off a demo-scale benchmark and treating them as production statistics is exactly the mistake this library exists to discourage. The finding that transfers is the shape of the failure, and the fact that it is measurable from the tool-call stream alone.

PRIMARY SOURCE RESULT: The tool surface matters to the monitoring result. CivBench's environment exposes many queryable systems - map state, unit status, diplomatic relations, production queues - through a shared MCP interface. The under-querying pattern is not that agents cannot reach state; the tools are right there. It is that they choose not to reach for it, and the choice gets worse as context accumulates. That is why this transfers: your runtime's tools being available says nothing about your agent consulting them before effects.

PRIMARY SOURCE RESULT: The benchmark's partial observability is load-bearing for the design. Because fog of war hides parts of the map and opponent intentions are inferred, an agent that skips state queries is not merely lazy - it is structurally deciding on fiction while the tools that would correct the fiction cost one call each. Long-horizon production agents sit in the same chair: the CRM, the status feed, and the ledger are all one read away, and the model has to be the kind of agent that looks.

Limitations

Be honest about what CivBench does not establish. It is a game environment. Civilization VI is adversarial, long, and partially observable, which is why it stresses agents usefully, but its actions are also reversible in ways production effects are not - a mis-moved unit can usually be corrected next turn, while a sent email cannot be unsent. Scores here do not translate to business-task reliability.

The benchmark covers a specific model set evaluated at a specific time. Agent behavior moves fast; rankings from one release cycle say little about the next. And the study is benchmark-scale: a demonstration that failure modes are real and measurable, not an estimate of how often they occur in your system.

Finally, the paper measures the gap; it does not close it. A benchmark that names a failure is not a mitigation for the failure. That step is left to the builder, which is the rest of this article.

Why builders care

Translate the game's failures into your system's language. Under-querying is your agent deciding to send the email without re-reading the subscriber's current status, because it read the status an hour ago. Unexecuted commitments are your agent writing 'I will follow up with this lead next week' into a memory entry that no scheduler, queue, or watcher will ever read. Both are silent. Both produce traces that look reasonable in review.

This is why the gap is an authority problem as much as a competence problem, which is why the article sits in this arc of the library. The agent's reflection is a claim about future action, and a claim is not an obligation. If your runtime treats a stated intention as a scheduled effect, you have delegated scheduling to the least reliable component in your system: free-text reasoning.

There is also a review problem. Teams evaluating agents by reading reasoning traces will systematically overestimate the agent, because the reasoning is where the agent is most articulate about its obligations. Teams evaluating by outcome will systematically underestimate the risk, because the task often completes despite the skipped check. Both evaluation styles miss the gap; only the action-versus-word comparison catches it, and that is the comparison the two metrics below operationalize.

The monitoring failure has the same structure. In a partially observable system - and every real system is, from your agent's seat - an unchecked assumption is a decision made against fictional state. Article 12 in this library (PlanFence) attacked this at the plan layer: validate dependencies before irreversible execution. CivBench shows the same discipline is needed at the observation layer, continuously, not just before the irreversible step.

Two metrics: Proactive Monitoring Rate and Commitment Execution Rate

LABS INTERPRETATION: CivBench makes the gap scorable, and Ernesta Labs carries that into two metrics a builder can compute from their own tool-call logs, with no game required. These are Labs definitions, not the paper's; they are our translation of the benchmark's measured failure shapes into production telemetry.

Proactive Monitoring Rate (PMR): of all decision points where an action has a queryable dependency - a fact the action depends on that the runtime can check with a read-only call - what fraction were preceded, in the same decision window, by an actual query of that dependency? PMR = monitored decision points / decision points with a queryable dependency. An agent that sends without re-reading status scores low. An agent that always looks first scores 1.0.

Commitment Execution Rate (CER): of all commitments the agent stated in reflection - 'I will do X' with an implied window - what fraction were followed by the corresponding action in the trace, verifiable against the tool-call log rather than the agent's word? CER = executed commitments / stated commitments. A low CER with a high-quality reflection stream is the reflection-action gap quantified.

Both metrics are cheap because both are computed from logs you already have if your tools are mediated. Neither requires reading the model's chain-of-thought as ground truth: CER takes the stated commitment only as a claim, and checks the action log. That independence matters; it is the same separation between self-report and evidence this library applies everywhere else.

What we would implement, and what we would not

LABS RECOMMENDATION: Log every tool call with kind (query or effect), the dependency identifiers the action touches, and a decision-window ID, so PMR is computable after the fact. Extract stated commitments from the agent's reasoning into a structured side channel - an explicit commitments list the agent is required to write, not a parsed scrape of prose - and match each against subsequent actions to compute CER. Report both per run and per model.

LABS RECOMMENDATION: Put deterministic watchers behind obligations that must not fail silently. If the system has promised a follow-up, a renewal, or a recheck, that promise belongs in a store that a scheduler owns - not in a paragraph the model wrote. A watcher that fires a check on schedule does not depend on the model remembering anything. Knowing is not doing; a watcher is doing, pre-committed.

What we would not implement: we would not try to fix the gap by prompting the model to be more conscientious. Prompting raises the reflection quality, which raises CER's numerator only if the action layer follows, and the benchmark's finding is precisely that it often does not. We would not treat a rising PMR as proof of correctness - a monitored decision can still be the wrong decision. We would not parse free-text reasoning as the system of record for commitments; a parse of prose is a guess about intent wearing a structured costume. And we would not auto-block actions on a low PMR score in production before the metric's false-positive rate is known; start by measuring, then gate.

Practical test: score your own reflection-action gap today

You can compute both metrics this week, with no new infrastructure, if your agent's actions pass through any mediated tool layer. Do this:

1. Pull your last 50 runs' tool-call logs. 2. PMR: pick your three most common effect actions (send, publish, write). For each occurrence, mark whether a read of the record the action depends on happened in the same decision window before the effect. Divide. That fraction is your Proactive Monitoring Rate. 3. CER: if your runtime captures any stated intentions - follow-ups, deferred checks, 'we will do X' - match each against the following actions in the log. Executed over stated is your Commitment Execution Rate. If you capture no stated intentions at all, that is itself the finding: your system has no way to know whether it keeps its word.

Then one watcher: take the single most load-bearing commitment your agent states, and move it out of prose into a scheduler-owned check with a completion record. Watch it for two weeks. If the watcher ever fires on a commitment the agent forgot, you have reproduced CivBench's finding in your own system, and you have the first data point for why deterministic watchers earn their keep.

What remains unknown

• How low PMR and CER run in real production systems - the benchmark demonstrates the failure shape; it does not estimate base rates outside a game environment, and our own measured baselines are not yet complete.
• Whether CER improvements are stable across model generations, or whether each new model reset the gap in a new way - the study covers a fixed model set at a fixed time.
• Which commitments are safe to leave in the model's hands versus which require a deterministic watcher - the boundary is currently a judgment call, not a measured threshold.
• Whether high reflection quality actively masks low CER in review, making teams less likely to notice the gap - plausible, unmeasured.

Primary sources

• CivBench: Long-Horizon Benchmark for Tool-Mediated Agents (arXiv:2609.02459)
• civ6-mcp - Civilization VI MCP server (GitHub)