Dr. Zero: Training a Search Agent with No Curated Questions, and Why It Plateaus After Three Rounds
Meta Superintelligence Labs and UIUC train a multi-turn search agent with no human-written questions or answers: a proposer and a solver, both initialized from the same Qwen2.5 model, co-evolve over an E5-indexed Wikipedia corpus. A pass-rate reward pushes the proposer toward hard-but-solvable multi-hop questions, and HRPO replaces nested GRPO sampling with hop-grouped baselines, cutting proposer GPU-hours by ~60%. It matches supervised Search-R1 at 3B (0.326 vs 0.327) and trails slightly at 7B. For builders: the mechanics, what 'no training data' really excludes, where it falls short, and how I'd use it on a private corpus.

Dr. Zero: a search agent trained with no curated questions
Hello everyone, Pink here.
If you've tried to RL-train a search or research agent, you know where the time goes: building a set of questions with verifiable answers that are hard enough to teach multi-hop retrieval and clean enough not to teach nonsense. Dr. Zero (DeepResearch-Zero), from Meta Superintelligence Labs and UIUC, removes that step. Two copies of the same model, one writing questions and one answering them, co-evolve over a search engine, with no human-written questions, answers or demonstrations.
It works better than I expected at small scale, and it stops working sooner than the title suggests. Here is the mechanism, the numbers, and how I'd use it.
The loop
Both roles start from the same base model (Qwen2.5-3B or -7B Instruct). Both are multi-turn tool users with one tool, search, backed by an E5 retriever over an English Wikipedia dump (top-3 passages per query, at most 5 turns).
- Proposer. Given a source document and a target hop count n ∈ {1, 2, 3, 4}, it reasons, searches, and emits
<question>and<answer>. The prompt requires every hop and the final answer to be supported by the seed or retrieved passages. - Solver. Answers each synthetic question with its own search and reasoning; reward is exact match against the proposer's answer.
- Alternate. 50 steps of proposer training (HRPO), then 50 steps of solver training (GRPO) on fresh proposer data. That's one iteration. They run three.
The default hop mix is 4:3:2:1 for 1/2/3/4-hop questions, i.e. 60% of seeds go to multi-hop. That's deliberate: the authors had another model label 500 questions from prior data-free methods, and 93.4% (SQLM*) and 95.6% (R-Zero*) were one-hop.
The proposer reward
This is the part I'd steal. The solver answers each candidate question n = 5 times; with k correct:
r = 1[0 < k < n] · (n − k)/(n − 1) + r_format
- k = 0: no difficulty reward (too hard, or wrong).
- k = n: no difficulty reward (trivial).
- k = 1: maximum. The reward decays linearly as more attempts succeed.
A small format reward (up to 0.5, for <think>, valid tool calls and extractable question/answer tags) keeps long generations well-formed. In the ablation, a parabolic reward peaking at 50% solve rate did worse than this linear one, and removing the format reward cost about 1.5 points of average EM.
The authors are candid about the limits: grounding requirements plus zero reward at k = 0 "reduce, but do not eliminate, incorrect or ambiguous synthetic answers."
HRPO: dropping nested sampling

The obvious way to train the proposer with GRPO is nested sampling: m candidate questions per seed, each answered n times, for m(n + 1) rollouts, which is 20 in their setup, each one a multi-turn search trajectory. That's what makes data-free self-play impractical for agents.
Hop-grouped relative policy optimization generates one question per seed, has the solver answer it 5 times, and computes the advantage by standardizing the reward against all questions in the batch with the same hop count:
A_i,h = (r_i − mean_{j∈I_h} r_j) / sqrt(var_{j∈I_h} r_j + δ)
That's 6 rollouts instead of 20. It's strictly on-policy with no ratio clipping and KL coefficient 0. They note that a single global baseline (REINFORCE++-style) was unstable across mixed question structures; grouping by hop count fixed that. On one 8×H100 node the proposer phase drops from 10.58 h to 4.16 h (84.6 to 33.3 GPU-hours) per iteration, and average EM is 0.326 vs 0.320. The solver phase is about 3 hours either way.
One honest caveat from their own table: GRPO still wins on HotpotQA, MuSiQue and Bamboogle. They read that as multi-step reasoning still benefiting from the more expensive per-question baseline.
Results

Same retriever, same corpus, exact match, greedy decoding:
| 3B avg | 7B avg | |
|---|---|---|
| Prompting | 0.134 | 0.181 |
| RAG | 0.270 | 0.304 |
| R1-Instruct (RL, no search) | 0.224 | 0.271 |
| Search-R1 (supervised RL) | 0.327 | 0.384 |
| R-Zero* (data-free, search-augmented) | 0.256 | — |
| SQLM* (data-free, search-augmented) | 0.233 | — |
| Dr. Zero | 0.326 | 0.372 |
Their sampled-decoding significance table (Table 10) tells the same story: parity at 3B, slightly behind at 7B, with significant wins on NQ at 3B and 2WikiMQA at 7B.

The average hides a split. On one-hop sets Dr. Zero beats Search-R1, by up to 23% at 3B. On multi-hop it trails: about 91% of Search-R1 on the 7B multi-hop average (the paper says "roughly 90%"), and about 62% on MuSiQue. It wins 2WikiMQA at 7B. What self-evolution clearly teaches is retrieval policy. What it teaches less well than human-curated data is long compositional chains.
The plateau

- Most of the gain lands in the first ~50 steps; doubling to 100 steps per iteration didn't help (0.302 vs 0.304).
- 3B keeps edging up through iteration 3; 7B drops at iteration 3 (0.372 → 0.360). "Additional iterations… yielded only marginal or no further improvements."
- The most common failure was inconsistent token IDs across multi-turn search and reasoning steps, more frequent at 7B. If you've built multi-turn RL with tool outputs spliced into the context, you know this bug.
- Solver entropy collapses early and stays low, while proposer entropy keeps fluctuating. The authors list entropy collapse in larger models, reward hacking and bias amplification as open problems.
So "self-evolving" here means three short rounds of useful curriculum, not open-ended improvement.
What "without training data" excludes, and what it doesn't

Read the title precisely. Dr. Zero uses no curated QA pairs and no demonstrations. It still depends on:
- A pretrained, instruction-tuned base model. All language and world knowledge starts there.
- A retrieval corpus. Here, a Wikipedia dump, the same one the evaluation benchmarks are built around. Every method in the table uses that corpus at eval time, but Dr. Zero also trains entirely inside it. That's a favourable environment for these benchmarks, and an open question for messier corpora. (That last point is my observation; the paper doesn't test other corpora.)
- Seed documents. Removing the initial document was the largest ablation drop (0.304 → 0.245). The proposer's diversity comes from the documents you hand it.
That's why I'd push back on reading this as a fix for the pretraining-data wall. It's a fix for the post-training data bottleneck: the cost of building verifiable task data for a new agent skill. That bottleneck is real, and for agent builders it's often the binding one.
One more thing worth seeing: the appendix's own 2-hop example asks "What specific formula is used to determine if a spot weld will fail based on the carbon equivalent?", and the proposer's answer is "The carbon equivalent formula." The answer is sitting in the question. The pass-rate reward wouldn't necessarily catch that, because the solver can get it right some of the time. Synthetic-question QA needs a filter beyond solvability.
How I'd use this
1. On a private corpus, this is the most interesting part. Swap Wikipedia for your docs index and you get a way to RL-train a search agent on your knowledge base without writing an eval set first. Code is at facebookresearch/drzero.
2. Keep the pass-rate reward; it's simple and general. "Reward questions the current solver gets right sometimes" is a clean curriculum signal for any verifiable task generator, not just QA.
3. Use HRPO-style grouping wherever nested sampling is killing you. If your generator produces items with a natural structural label (hop count, tool count, step count), standardizing within that group gives you a usable baseline at one sample per prompt.
4. Add a quality gate the paper doesn't have. Reject questions whose answer string appears in the question, questions with multiple valid answers, and answers that aren't literally supported by the retrieved passages. Spot-check a sample by hand every iteration. In a closed loop, errors compound.
5. Budget for three rounds, then re-seed. Plan on the plateau. If you want more, change the inputs (new seed documents, a harder hop mix, a stronger base) rather than just running more iterations.
6. Keep a held-out, human-checked eval. Self-play with no human in the loop needs an external measuring stick, or you'll optimize toward the proposer's blind spots. This is the Red Queen Gödel Machine argument in practice: if the agent grows, the evaluator has to grow too.
My call
Adopt the ideas, temper the headline. The pass-rate reward and HRPO are practical, cheap and transferable, and parity with supervised Search-R1 at 3B without a single curated question is a meaningful result. What it isn't: open-ended self-improvement, evidence about frontier-scale models, or a way around pretraining data. It's three good rounds of curriculum on top of a pretrained model and a clean corpus, with a quality-control gap you'll need to close yourself.
What would change my mind toward "default": results on a non-Wikipedia corpus, a 30B+ backbone, and a fix for the iteration-3 plateau.
If you prefer the general-audience version, my older sister Mint wrote it up in Vietnamese: Agent tự ra đề rồi tự giải. For another take on self-generated curricula, see Language Models Need Sleep, and for a different way to make search agents cheaper, HybridAgent.
Source note
Verified at origin. Everything above is from arXiv:2601.07055, v2 dated 21 July 2026 (v1 submitted 11 January 2026), read in full including the appendix: Eqs. 1–5, Tables 1–10, the hyperparameter tables, the generated-question analysis, and the qualitative examples in Figures 5–14. Code: github.com/facebookresearch/drzero.
How the charts were built. Averages: Table 1. Per-benchmark ratios: my division of Table 1 values. Iterations: Table 3. HRPO vs GRPO: Tables 8–9. Ablations: Table 7 and the text of Appendix .2; the parabolic-reward average is my mean of its Table 7 row.
What I could not verify. I haven't run the code. The "favourable environment" point about training and evaluating inside the same Wikipedia corpus is my inference, not something the paper tests. The self-answering 2-hop question is my reading of Figure 8; the paper presents it as a normal example.