Automated Research Produces Reviewable Papers, With Weak Evidence

Automated research agents are moving from isolated paper-writing demos toward systems that can choose topics, run code, and leave behind inspectable artifacts. The harder question is no longer whether an agent can produce a plausible manuscript once, but what happens when such a system is run continuously and judged across the whole output distribution. This paper introduces FARS, a Fully Automated Research System for AI-for-AI work, and evaluates its first public deployment rather than a selected set of successes.

Core Contribution

The main contribution is deployment evidence. FARS produced 166 complete research papers across 67 fine-grained AI/ML topics, with proposals, plans, code, logs, results, and manuscripts preserved in a shared workspace. The authors then collected 282 structured human reviews covering 140 of those papers. That setup makes the paper less a claim that automated research is solved than a measurement of where current agent systems already produce reviewable artifacts, and where they still fail under artifact-aware inspection.

Technical Approach

FARS decomposes research into four stages: Ideation, Planning, Experiment, and Writing. Ideation starts from nine seed AI/ML directions but can explore beyond them; accepted proposals are filtered through literature review, candidate direction generation, peer-style discussion, proposal drafting, and evaluation. Planning turns a proposal into a machine-readable experiment contract, with ordered tasks for environment setup, baselines, main experiments, effectiveness evaluation, and analysis.

Figure 1 shows the pipeline structure: multiple projects branch from ideation, then proceed independently through planning, experimentation, and writing while sharing infrastructure and persistent artifacts.

The experiment stage is designed around auditability rather than just task completion. Agents execute validated plans item by item, preserve progress records, commit outputs to version control, and run checks before continuing. The system uses layered software environments, curated ML skills, managed GPU allocation, unified model endpoints, and checkpoint recovery. In the public run, this infrastructure included a GPU cluster with 160 NVIDIA GPUs.

The writing stage separates evidence analysis from prose generation. An analysis agent determines which claims are supported by proposals, plans, experiments, and related work, while a writing agent drafts the manuscript from a blueprint tied to those artifacts. The paper argues that this separation reduces opportunistic claims and makes downstream review possible, though the human reviews show it does not eliminate unsupported results or integrity failures.

Results and Analysis

The deployment ran for 417 hours, consumed 21.6 billion model tokens, and cost about $186,000 in GPU and token costs, amortized to roughly 2.51 deployment hours, 130 million model tokens, and $1,120 per completed paper. Of the 166 papers, 72 remained in the initial seed topics and 94 emerged during autonomous exploration. The largest seed clusters were Reinforcement Learning from Verifiable Rewards and Automated Evaluation of Frontier LLMs, with 13 and 10 papers; the largest emergent cluster was AI Safety and Alignment, also with 13 papers.

The review outcomes are mixed. Across 282 reviews, the mean overall rating was 3.17 on an ICLR-style 0–10 scale, with 6 as the acceptance threshold; no review assigned 8 or 10. At the review level, 17.7% reached 6. At the paper level, the mean was 3.23, and 11.4% of the 140 reviewed papers reached a mean rating at or above 6. Figure 2 makes the distribution visible: the mass is below acceptance, while a small tail reaches review-worthy scores.

The sub-scores explain the pattern. Soundness averaged 2.15 and contribution 2.00, while presentation was higher at 2.46 and reviewer confidence averaged 3.78. Contribution had the strongest correlation with overall rating, Pearson r = 0.743, followed by soundness at 0.652 and presentation at 0.451. In other words, reviewers were not mainly penalizing surface fluency. They rewarded papers that had a meaningful contribution and enough evidence to support it.

The integrity audit is the sharper finding. Reviewers flagged integrity problems in 47 of 282 reviews, affecting 39 of 140 reviewed papers. These flags were concentrated in weaker papers: 43.5% of rating-0 reviews reported integrity issues, compared with 22.3% at rating 2, 10.2% at rating 4, and 2.0% at rating 6. The most common failure classes in review text were experimental-design pathologies, internal-consistency failures, language or reasoning pathologies, hallucinated methods or baselines, and fabricated experimental results.

Evidence and Caveats

Against prior automated research systems, FARS received the highest automated Stanford Agentic Reviewer mean score in the paper’s comparison: 5.00 across 165 papers, versus 4.13 for DeepScientist and 4.00 for ScientistOne. But the authors treat this comparison as a relative signal, not an acceptance estimate. The same FARS corpus scored 3.23 under human review, and the baseline corpora are small and uneven.

The paper’s strongest value is its audit corpus: a large, public run where weak outputs, negative results, code, logs, and reviews are preserved. Its weakest claim would be any broad inference that autonomous research is already dependable. The results instead support a narrower reading: FARS can generate a nontrivial tail of reviewable AI/ML papers, but the median output remains below acceptance quality, mainly because the evidence is too narrow or inconsistent to establish the claimed contribution.