Rapid development of artificial intelligence has drastically accelerated the development of scientific discovery. Trained with large-scale observation data, deep neural networks extract the underlying patterns in an end-to-end manner and assist human researchers with highly-precised predictions in unseen scenarios. The recent rise of Large Language Models (LLMs) and the empowered autonomous agents enable scientists to gain help through interaction in different stages of their research, including but not limited to literature review, research ideation, idea implementation, and academic writing. However, AI researchers instantiated by foundation model empowered agents with full-process autonomy are still in their infancy. In this paper, we study AI-Generated Science (AIGS), where agents independently and autonomously complete the entire research process and discover scientific laws. By revisiting the definition of scientific research, we argue that falsification is the essence of both human research process and the design of an AIGS system. Through the lens of falsification, prior systems attempting towards AI-Generated Science either lack the part in their design, or rely heavily on existing verification engines that narrow the use in specialized domains. In this work, we propose Baby-AIGS as a baby-step demonstration of a full-process AIGS system, which is a multi-agent system with agents in roles representing key research process. By introducing FalsificationAgent, which identify and then verify possible scientific discoveries, we empower the system with explicit falsification. Experiments on three tasks preliminarily show that Baby-AIGS could produce meaningful scientific discoveries, though not on par with experienced human researchers. Finally, we discuss on the limitations of current Baby-AIGS, actionable insights, and related ethical issues in detail.
identify and then verify is the sharpest find in this abstract — the paper’s own word choice disproves its own thesis without any outside argument needed.
Their abstract states the FalsificationAgent “identifies and then verifies possible scientific discoveries.” Their methodology confirms it: the agent “systematically validates hypotheses through structured experiments” and “extracts insights through empirical verification.”
That is confirmation-seeking — exactly what Popper argued against. Real falsification means designing the test most likely to break the hypothesis, not running experiments to confirm it. The paper invokes Popper by name, then builds the opposite mechanism.
The paper’s thesis and its implementation directly contradict each other. This is not a minor slippage in terminology — the entire system design follows from a mislabeling of the core concept.
The three experimental tasks: data engineering, self-instruct alignment, language modeling. All AI benchmark domains — closed, metric-driven, with success defined by benchmark scores.
Improving a benchmark score is ML engineering, not scientific discovery. Open scientific domains (biology, chemistry, physics) have no pre-defined success metric and no automated oracle to evaluate results. The paper tests only on the domains most compatible with its method and calls the result “generating science.”
Additionally, output quality is evaluated by a ReviewAgent — another AI within the same system. That is not peer review. No independent human expert verified the “discoveries.”
“Initial baselines” appear to be naive or earlier AI approaches. The relevant comparison is human researchers. The authors themselves acknowledge Baby-AIGS “falls short of experienced human researchers.” The central performance claim is falsified by the paper’s own results section.
The headline metric — “near-perfect experiment executability” — measures whether code runs, not whether it produces valid science. A system that executes trivial experiments flawlessly does not generate knowledge.
The paper calls its system a falsifier but builds a validator. It tests on closed AI benchmarks where “science” means “benchmark score.” It celebrates beating baselines it designed itself. And it explicitly admits the system doesn’t match actual researchers.
The paper is a live example of the confusion it claims to resolve: the word falsification placed in front of the wrong noun.