Replay is not reality: three boundaries for learning agents

Short answer

PRIMARY SOURCE RESULT: Dream-RSI is a September 2026 arXiv preprint that revises executable exploration-policy code by replaying logged discovery trees while keeping the underlying coding model and evaluator fixed. The authors report lower discovery-agent-call counts and competitive or improved results across eight algorithm, math and GPU-kernel tasks. Replay can only traverse outcomes already present in the realized search space; it cannot generate an unseen branch.

AVAILABILITY CHECK: at the verified official Dream-RSI repository commit, code is being prepared for release. The repository contains the paper and assets but no runnable framework, dependency manifest, trace set, evaluation harness or reproduction scripts. The paper includes prompts and one discovered solver listing, but the full system is not runnable today from official artifacts.

PRIMARY SOURCE RESULT: Atria Dawn reports a verifiable-experience training pipeline and a 56-person observational study in which people retained most final method, goal and acceptance decisions. Android Bench 2.0 evaluates model-agent combinations on 30 private long-horizon tasks with five runs each, binary pass rate plus continuous completion scoring. Neither source turns partial progress or model-judged quality into a real-world commercial result.

Dream-RSI improves orchestration, not model weights

PRIMARY SOURCE RESULT: Dream-RSI makes exploration explicit as executable policy code — branching, batching, parallel allocation and stopping — around an otherwise unchanged discovery agent and evaluator. Completed discovery histories become replay simulators. A policy-development model proposes revisions, those revisions are scored over historical trees, and an accepted policy returns to live discovery, creating more histories. Calling this recursive self-improvement is the authors' framing; the mechanism studied is narrower than model self-training because the coding model, evaluator and interfaces stay fixed.

The v1 paper covers eight tasks: one Lasso-path algorithm-engineering task, three mathematical-optimization tasks, and four KernelBench tasks. On Lasso with Gemini-3.1-Pro, Dream-RSI used 317 discovery-agent calls versus 550 for the controlled recursive fixed-exploration baseline and reported lower held-out runtime. Its comparison to SimpleTES reaches about 162 times fewer calls, but that is not controlled: SimpleTES used GPT-OSS-120B. On the three math tasks, Dream-RSI was better than its fixed baseline on one, slightly worse on one, and tied on one. On four GPU kernels, the authors report lower generation counts or higher inverse-runtime scores in the paired comparisons shown.

The publication-safe conclusion is modest: the authors report that logged exploration can reduce additional discovery/evaluator executions when revising a search-allocation policy. The paper does not establish broad recursive intelligence, open-world learning, or automatic improvement of the underlying model.

The replay boundary is the realized search space

Replay can select recorded branches in different subsets, orders, batches and stopping positions. It cannot ask what a different stochastic model generation would have produced from the same workspace, and it cannot produce an outcome beyond what was logged. The paper's non-degradation property is therefore a replay property: the accepted policy's average score is no worse on the fixed historical pool. It is not a guarantee of improved live performance, transfer to a new task, or resistance to overfitting the replay pool.

The paper calls replay zero-execution-cost because no new discovery-agent or evaluator execution is required while traversing stored outcomes. That does not mean total cost is zero. Policy-development inference, replay sweeps, storage and orchestration still consume resources, and the paper's reported cost metric is cumulative discovery-agent calls rather than a full token, dollar, compute and wall-clock account.

The v1 text reports point results and curves, but we found no seeds, repeated-trial counts, error bars, confidence intervals or variance estimates for the headline results. The evaluation uses eight tasks with fixed automatic evaluators. No evidence is presented for noisy commercial environments, cross-domain policy transfer or changing counterparties.

Runnable code is not available yet

AVAILABILITY CHECK: the official repository README says the discovered programs, full codebase and reproduction scripts are being prepared. At verified main commit 4149ea9181ab1db80f85717ffda2c9f0f130e85b, the recursive tree contains the README, citation, assets and paper, but no Dream-RSI source implementation, dependency manifest, environment setup, evaluator, traces or reproduction command. GitHub showed no tags, releases or repository license at that check.

The paper appendices do provide the policy-development prompts and a full C++/Eigen/OpenMP listing for one discovered Lasso solver. That lets a reader inspect part of one output. It does not make the orchestration layer, replay simulator or reported benchmark reproducible. Until the announced release lands and can be executed, the framework's empirical claims remain readable but not independently reproducible from official code.

Atria Dawn: verifiable experience still depends on human authority

PRIMARY SOURCE RESULT: Atria Dawn Preview is described as a 744-billion-parameter mixture-of-experts model built on GLM-5.2, with a 256K context window and text-only input. The paper describes a Verifiable Experience Pipeline: tasks run in executable environments; trajectories create artifacts; tests, metrics, application state, geometry, source evidence or human criteria supply outcome signals; incomplete, contradictory, duplicate or behaviorally invalid trajectories are filtered. The paper reports 16 benchmarks and the highest reported result on five entries in its comparison table. Several rows use different harnesses, local modifications or LLM judges, and the headline table has no confidence intervals.

The human collaboration study contains 769 task records from 56 participants. Of 455 completed AI-assisted tasks with usable counterfactual responses, participants rated 151 — 33.2 percent — infeasible without AI. This is a participant counterfactual judgment, not a controlled causal estimate. Humans made the final selection in 85.5 percent of method or parameter decisions, 93.4 percent of goal or scope decisions, and 81.9 percent of acceptance-criteria decisions. Of 588 tasks with a major difficulty recorded, 76.0 percent moved forward through human intervention and 23.0 percent self-recovered.

Atria Dawn is therefore evidence for tool-mediated work with verifiable outcomes and substantial human judgment. It is not evidence that the model autonomously chooses safe goals, runs an open commercial process, or removes the need for accountable authority. The paper also omits training-token volume, total compute, optimizer details, full task inventory, ablations and a complete contamination audit, so the training result is not reproducible from the paper alone.

Android Bench 2.0 separates pass from progress

PRIMARY SOURCE RESULT: Google's official methodology defines 30 private long-horizon Android tasks intended to represent days or weeks of work: app creation, migrations, new features and cross-platform conversions. Each task starts in a fresh Docker container on a KVM-backed Android virtual device. Harbor standardizes environments and metrics. The benchmark evaluates a model-agent combination — such as a model inside Codex or Claude Code — because the harness changes what the system can do.

Each combination receives five independent runs per task, 150 attempts in total. The primary metric is pass rate. Passing requires a score of 1.0, all functional checks, full visual compliance and no constraint violation. The diagnostic completion rate is continuous: a weighted normalized base score across functional, regression, requirement and visual checks, multiplied by penalties. Build failure, cheating and foreign-language shortcuts can zero a run; legacy API use can halve it.

Verification combines deterministic instrumentation tests, database inspection and regression suites with scripted emulator walkthroughs, screenshots, accessibility trees and a Gemini 3.5 Flash judge. This makes the evaluation richer than one binary unit test, but still judge- and harness-dependent. The private dataset blocks full third-party reproduction today. Emulator mocks do not cover physical hardware, backends are local, and deterministic walkthroughs can make downstream screens unreachable after an early navigation failure.

Methodology drift is itself a finding

The official Android pages are not perfectly synchronized. At verification, the methodology page listed a 9/13/6/2 mix across app creation, migration, new feature and conversion, while the live leaderboard listed 10/13/5/2. The announcement said conversions reached at most 80 percent completion, while the methodology said the best runs approached 0.90 and the live table showed different category averages. We do not resolve those differences by choosing the most attractive number. They are documentation/version drift, and any quoted result needs the exact page and retrieval date.

Cost and latency also have a structural trap that Google states directly: an agent that fails early may look cheaper and faster because it did less work. Network path, price changes and provider-specific token accounting add more variance. Resource metrics should therefore be compared only between systems with similar completion and pass behavior, not ranked as independent proof of efficiency.

Implication: three ledgers, three meanings

LABS INTERPRETATION: maintain three distinct records. A REPLAY SCORE asks whether a proposed policy would select recorded history differently. A DIAGNOSTIC COMPLETION SCORE asks which verified parts of a long task are complete. A TERMINAL OUTCOME asks whether the external objective happened. These records can inform one another, but they cannot substitute for one another.

For Experiment Zero, repeated runs belong in offline simulation, replay and synthetic Action Brief tests because the production experiment has one historical trajectory and real irreversible effects. The terminal verdict remains external: a genuine customer, a genuine close and verified cash. A model judge, a replay improvement, 82 percent completion, a provider acceptance or a polished artifact cannot be renamed as that result.

Practical test: replay one failure without inventing a branch

Select one historical NIKO decision and enumerate only the observations and outcomes that were actually recorded. Let a candidate policy choose among those branches. If it asks for an unseen action or counterfactual outcome, mark LIVE EVIDENCE REQUIRED rather than fabricating a replay result. Report policy-development inference and storage cost separately from avoided external executions.

Then score the same trajectory diagnostically. Every point must link to deterministic evidence. Finally ask the terminal question: did verified cash from a genuine customer occur? The test passes only if the diagnostic score can rise while the terminal answer remains no, without either number being relabelled.

What remains unknown

• Whether Dream-RSI's reported savings reproduce after the full framework, traces and scripts are released.
• Whether replay-policy selection transfers to new tasks rather than overfitting the accumulated historical pool.
• Whether Atria Dawn's benchmark and collaboration findings reproduce independently; the paper is an author-run preprint and several evaluations are harness-dependent.
• How stable Android Bench 2.0 results are across judge, harness and documentation revisions while the long-horizon dataset remains private.
• Whether any of these methods improves NIKO's ability to acquire a customer; none of the sources studies autonomous selling.

Primary sources

• Dream-RSI: Recursive Self-Improvement through Evolving Worlds (arXiv:2609.14858v1)
• Dream-RSI official repository and release plan
• Atria Dawn: The Dawn of Agentic Superintelligence (arXiv:2609.15818v1)
• Atria Dawn Preview official repository
• Android Bench 2.0 official methodology
• Android Bench live leaderboard