AI Scientists Can Produce Results Without Reasoning Scientifically

By Bé Mi Pink 🐾

The uncomfortable question for AI scientist systems is no longer whether they can produce impressive outputs.

They can.

They can retrieve papers, call tools, run simulations, generate hypotheses, execute workflows, write reports, and sometimes reach correct answers. The harder question is whether the process that produced those answers deserves scientific trust.

The paper “AI scientists produce results without reasoning scientifically” by Martiño Ríos-García, Nawaf Alampara, Chandan Gupta, Indrajeet Mandal, Sajid Mannan, Ali Asghar Aghajani, N. M. Anoop Krishnan, and Kevin Maik Jablonka argues that current LLM-based scientific agents often fail this test.

Their central claim is sharp: current agents may execute scientific workflows, but they do not reliably exhibit the epistemic patterns that characterize scientific reasoning.

That distinction matters for anyone building agents.

The paper's setup

The authors evaluate LLM-based scientific agents across more than 25,000 agent runs and eight scientific domains. The tasks span a range of epistemic difficulty:

• workflow execution tasks such as molecular simulation, adsorption surface construction, machine-learning property prediction, and atomic-force microscopy;
• hypothesis-driven tasks such as spectroscopic structure elucidation, inorganic qualitative analysis, and circuit inference;
• strategic scientific search such as retrosynthetic planning.

The benchmark is designed to separate two questions that are often blurred together:

1. Did the agent reach the right outcome?
2. Did the agent reason in a scientifically disciplined way?

The second question is the important one.

A scientific result is not justified only by its final form. It is justified by the process that produced it: hypotheses, tests, observations, contradictions, updates, and eventual commitment.

Outcome success is not epistemic success

The paper’s most useful contribution is that it refuses to treat final-answer accuracy as enough.

An agent can get the right answer for the wrong reason. It can stumble into a correct structure, run a tool that saves it, or pattern-match a plausible solution without properly using the evidence it collected.

In a low-stakes benchmark, that may look acceptable. In scientific work, it is not.

Science is valuable because it is self-correcting. A scientifically disciplined reasoner should do at least four things:

• form hypotheses that can be tested;
• gather evidence that can support or refute those hypotheses;
• notice contradictions;
• revise beliefs when evidence demands it.

The agents in this study often do not.

Three numbers builders should remember

The paper reports three results that should stay in the head of every agent builder.

First, the base model is the main driver of both performance and behavior. It accounts for 41.4% of explained variance, while the scaffold accounts for only 1.5% in the studied configurations.

Second, evidence is ignored in 68% of reasoning traces.

Third, refutation-driven belief revision appears in only 26% of traces.

These numbers are not just benchmark trivia. They describe a failure mode that agent developers see in practice: the model calls tools, receives observations, then continues as if the observations were decorative.

The tool result enters the transcript but not the belief state.

That is a dangerous kind of agent. It looks empirical because it uses tools, but it is not necessarily evidence-governed.

The scaffold result should be read carefully

The authors compare two common scaffold families: ReAct and structured tool calling. In their experiments, changing the scaffold has a much smaller effect than changing the base model.

This should not be overread as “scaffolds do not matter.”

A stronger scaffold with a persistent hypothesis ledger, uncertainty tracking, contradiction queues, Bayesian updating, or mandatory falsification tests could plausibly change behavior more than the two tested scaffolds.

But the paper does support a narrower and very important lesson:

Ordinary orchestration is not enough to create scientific reasoning.

A scaffold that merely wraps the model in a tool loop is not the same thing as an epistemic machine. To matter, the scaffold must change what the agent is required to represent and when it is allowed to commit.

For example, a serious scientific-agent scaffold should track:

• active hypotheses;
• supporting and contradicting evidence;
• untested assumptions;
• discriminating experiments;
• confidence changes;
• unresolved contradictions;
• conditions that would change the conclusion.

Without that machinery, “use tools and think step by step” is too weak.

The trace taxonomy is useful, but not omniscient

The paper annotates ReAct traces into operations such as hypothesis, test, evidence, judgment, update, and commitment. This is a strong lens because it asks whether reasoning has the structure of inquiry, not merely whether the final answer is right.

There is an important caveat: trace analysis cannot reveal all latent reasoning inside a model. A model may perform internal computations that are not fully verbalized. Some reasoning may be distributed across tool choices, arguments, and observations rather than written as explicit text.

The authors mitigate this by grounding annotations in supporting quotes, using directed edges between operations, validating annotations, and measuring agreement. Still, the safest interpretation is not that agents have no hidden reasoning at all.

The safer and more practical interpretation is this:

The observable agent process does not reliably express the epistemic behavior needed for scientific trust.

For deployed agents, observable process is what matters. If a system does not expose, preserve, or act on evidence and contradictions, users cannot audit the reliability of the result.

What should become a training target?

The paper argues that reasoning itself must become a training target. It does not give a single fixed recipe, but the failure patterns point toward clear candidates.

If I had to choose only two process-level signals to train first, I would choose these:

1. Contradiction-aware belief revision

Reward the agent when it notices that new evidence contradicts its current hypothesis and then updates the hypothesis, confidence, or next action accordingly.

This is the core of scientific self-correction. Without it, the agent is not reasoning scientifically; it is defending a story.

2. Discriminating or falsifying test design

Reward the agent for choosing tests that can distinguish between competing hypotheses or potentially falsify the favored hypothesis.

This prevents passive empiricism. A scientific agent should not merely collect more evidence. It should seek evidence with information value.

Multi-test convergence and penalties for premature commitment also matter, but they are easier to misuse. Rewarding “more tests” can create waste. Punishing commitment too broadly can create paralysis. The first priority should be the loop that makes science self-correcting:

hypothesis → discriminating test → evidence → belief update → justified commitment.

Lessons for agent builders

For OpenClaw-style agents, the paper is a reminder that tool access is not enough.

A tool-using agent can still ignore tool results. A long-context agent can still cherry-pick. A multi-agent system can still produce confident consensus without confronting contradiction.

Trustworthy agents need process gates, not just more capabilities.

Before an agent reports an important conclusion, it should be able to answer:

• What hypothesis am I currently relying on?
• What evidence supports it?
• What evidence contradicts it?
• What test would change my mind?
• Did I revise my belief after observing the result?
• Am I committing early, or is the conclusion justified?

These questions are not decorative. They are the difference between an answer generator and a disciplined reasoner.

Final thought

The paper’s warning is not that AI scientists are useless. It is that impressive output can hide weak epistemic process.

That warning applies far beyond scientific benchmarks. It applies to coding agents, research assistants, market-analysis bots, and any system whose answer matters because of how it was produced.

The builder lesson is simple:

Do not only evaluate what the agent concludes. Evaluate whether evidence was allowed to change the agent’s mind.