Retrying isn’t the same as learning why you failed
When a reasoning model gets a problem wrong, the usual fix is to sample again or ask it to reflect, and both tend to repeat the mistake. We looked at what a retry needs to know about the attempt before it, and measured what that buys.
Take a counting problem from MATH-500, one that asks for the number of arrangements of a set under a parity constraint. When we gave it to Qwen2.5-32B with a plain chain-of-thought prompt, the model came back with an answer exactly twice the correct one. It had over-counted by a factor of two. That is an easy mistake to make in combinatorics, and a hard one to catch from inside the argument that produced it.
What should a reasoning system do next? The usual answers are to sample again and take a majority vote, or to ask the model to critique its own solution and revise it. Both are sensible, and both assume that a second attempt is more likely to be right than the first.
For a problem like this we had little reason to believe that. A model that over-counted once will tend to over-count again, and a critique written by the same model reads the same reasoning, finds nothing obviously wrong with it, and leaves the error where it was.
The question behind ReTreVal[1] is about how reasoning frameworks treat failure. Methods like ReAct, Reflexion, Self-Refine, Tree of Thoughts and LATS[2, 3, 4, 5, 6] all deal with mistakes inside a single problem. None of them ask why a branch failed before trying again, and all of them throw that information away when the problem ends. In this post we look at the first half of that gap: what a retry should know about the attempt that came before it. The second half, carrying lessons from one problem to the next, is the subject of the next post in this series.1
Does sampling more help?
Sampling is the cheapest way to retry, so we looked at it first. How much does a vote actually buy you? If a model gets 78% of problems right on a single attempt, drawing several answers and voting ought to push that number up. Independent errors should scatter, while correct answers agree with each other. We tested this on all 500 MATH-500 problems[8] with self-consistency[7], using the same Qwen2.5-32B model and keeping reference answers out of the loop.
The gains we saw were small: a majority vote over three samples scored 79.8%, over five samples 79.0%, and over eight samples 82.4% (Fig. 1). We also counted how often the right answer appeared in any of the samples, which requires peeking at the answer key and so is a ceiling rather than something a real system could achieve. With eight samples that ceiling is 83.2%, so on roughly one problem in six all eight samples were wrong. On those problems the errors are not independent. The model makes the same mistake each time it starts down the same road, and no voting rule can recover them.
What would a better retry need?
When we tried to say what was missing from these retries, it helped to separate two properties a retry might have. The first is whether it takes a different route to the answer, or follows the same line of reasoning again with small changes. The second is whether it knows anything about why the previous attempt failed.
Resampling can stumble onto a different route by chance. It carries no information about the failure, though, and the vote then favours whichever answer is most common. Reflection methods such as Self-Refine and Reflexion do carry some information, usually a general critique, but they revise the same solution, so the retry tends to stay on a path that already looked right to the model. Neither gives the second attempt both a new starting point and a specific reason to avoid the old mistake, and that combination is what we set out to build.
How we retry in ReTreVal
ReTreVal reasons with a tree, and at the top level each branch is a different strategy for the problem, say an algebraic derivation, a direct enumeration, or a shortcut based on a pattern the model recognises. When the best branch produces an answer that fails validation, we do two things before trying again. We move to the highest-scoring top-level branch that hasn’t been explored yet, so the retry starts from a different strategy instead of a reworded version of the old one.
We also run a short failure analysis that labels what went wrong and how the failed solution was built. That record goes into the new branch’s critique history, where every refinement step of the retry can see it (Fig. 2).
The labels come from two small fixed lists, and we kept them short on purpose. The failure type describes the error (an arithmetic slip, a logic error, a missing step, a wrong constraint, an off-by-one, or incomplete reasoning), and the approach type describes the strategy (algebraic, arithmetic, pattern matching or brute force). Alongside the two labels, the record holds the failed answer and a one- or two-sentence explanation of what went wrong, written by the model during failure analysis. The new branch’s refinement prompt includes the whole record word for word, so the retry can use both the label and the explanation. In the MMLU-Pro law question we come back to below, for example, the record carried the failure type wrong_constraint, and that label is what sent the retry back to re-read the statutory condition.
We allow one retry per problem, and only between top-level branches. That keeps the cost predictable, and it is the setting every number in this post comes from.
Following one failure through
To see what this looks like in practice, go back to the counting problem from the start. It is one of the 37 problems where ReTreVal was right and zero-shot chain-of-thought was wrong, and we went through each of those cases by hand. ReTreVal’s first branch made exactly the same mistake as the baseline and over-counted by a factor of two, so the tree on its own didn’t protect the model from its habits.
What changed the outcome was a check during refinement (Fig. 3). The branch ran a small brute-force enumeration with the combinatorics tool, and the enumerated count disagreed with its closed-form answer. That disagreement failed the answer, the failure was recorded, and ReTreVal moved to a sibling branch that handled the parity constraint explicitly. The sibling’s answer matched the ground truth, which we used only for scoring.
- 01First branch over-countsReTreVal's first branch makes the same factor-of-two mistake as zero-shot chain-of-thought.
- 02A small enumeration disagreesDuring refinement the branch runs a brute-force count on a small case, and the result doesn't match its closed-form answer.
- 03The failure is recordedThe answer fails validation, and the failure analysis writes a typed record of what went wrong.
- 04Retry on a parity-aware branchWe move to the best unexplored branch, one that handles the parity constraint explicitly, with the record in its context.
- 05CorrectThe new branch's answer matches the ground truth, which we used only for scoring.
Tools aren’t always available, and several recoveries in our error analysis happened without one. On an MMLU-Pro[9] law question, for example, zero-shot chain-of-thought confused two answer choices that looked alike. No computation tool applies to a question like that. ReTreVal’s tree explored another reading of the rule’s scope, and when an attempt failed, backtracking passed the wrong_constraint label to the sibling branch, which went back, re-read the statutory condition and found the right answer. On a physics question the pattern was closer to the counting case: the unit converter flagged a mismatch in a derived quantity, and the retry went to a sibling that rechecked the base dimensions.
How often does it work?
We measured backtracking in two settings, which differ in one way that changes how to read the numbers. On MMLU-Pro, with 1,000 problems and Gemini 2.5 Flash (thinking disabled), everything was label-free. The system decided that an answer had failed using only its self-critique and a tool-consistency check, and we used reference answers only to score the final result.
On MATH-500, the backtracking ablation used ground-truth-assisted verification, so the reference answer decided whether an attempt counted as a failure. We lead with the label-free numbers because they are closer to how this would run in practice.
So does recovery hold up when the system has to judge its own failures, with no answer key anywhere in the loop? On MMLU-Pro, backtracking fired on 278 of the 1,000 problems (27.8%) and recovered 108 of them, which is 38.8% of the problems where it fired. Accuracy with backtracking was 54.4%, against 43.6% without it, a gain of 10.8 points. On MATH-500 it fired on 110 of 500 problems (22.0%), recovered 45 of them (40.9%), and moved accuracy from 76.8% to 85.8% (Fig. 4). The two recovery rates are close, even though one setting used the answer key to decide when to retry and the other didn’t. In both benchmarks backtracking was the largest single contributor in our ablations.
Is ReTreVal just winning by making more attempts? It does more work per problem than a single sample, so we ran it against self-consistency on MATH-500 under the same label-free setting. ReTreVal made 4.2 attempts per problem on average, fewer than maj@5, and its committed answer was right on 84.2% of problems (421 of 500). That is nine problems more than maj@8 and 5.2 points above maj@5.
| Method (label-free, MATH‑500) | Attempts per problem | Committed accuracy | Any attempt correct |
|---|---|---|---|
| Self-consistency, maj@5 | 5 | 79.0%395/500 | 81.6%408/500 |
| Self-consistency, maj@8 | 8 | 82.4%412/500 | 83.2%416/500 |
| ReTreVal | 4.2on average | 84.2%421/500 | 88.6%443/500, first five |
We also compared the ceilings, counting a problem as solved if any attempt was correct. ReTreVal’s first five attempts contained a correct answer for 88.6% of problems, while eight independent samples reached 83.2%, so the search was producing correct candidates that resampling never generated. Each attempt costs more than a single sample, though.3
When does retrying make things worse?
A retry is only useful if it fires when it should, and ours sometimes didn’t. Against zero-shot chain-of-thought on MATH-500, ReTreVal won 37 problems and lost 11, a ratio of about 3.4 to 1, and 7 of the 11 losses were Level 1 or 2 problems. Why would a retry turn a correct answer into a wrong one? One of them was a single-step fraction simplification that the baseline answered correctly in one go. ReTreVal’s complexity estimator overshot, opened a depth-3 tree, and a spurious critique of the correct branch pushed backtracking into a sibling with the wrong answer.
We saw the same thing on MMLU-Pro recall questions. On a history question asking for a single year, a sibling branch offered a plausible but wrong date, the cross-critique penalised the correct answer, and the final score favoured the wrong branch.
In all of these cases the problem didn’t need a second opinion, and getting one introduced doubt about an answer that was already right. A self-critic that wrongly flags a failure can do as much damage as one that misses a real failure. We think a simple short-circuit would recover most of these cases, accepting the first answer when the complexity estimate is at the bottom of the scale, but we haven’t built or tested it yet.
What we’d suggest if you’re building a reasoning agent
You don’t need our whole framework to try these ideas. If your agent already retries on failure, the change we would make first is to stop retrying from the same starting point. Keep a small set of different strategies (say, an algebraic route, a direct enumeration and a pattern-based shortcut) and move to the next-best one, and before you do, have the model name what went wrong using a short, fixed list of failure types. Record the approach that failed as well, otherwise the retry tends to drift back into the same family of solutions.
Look for cheap checks that can disagree with the model, because a small enumeration or a unit check is what exposed the error in several of our best recoveries. And gate retries by difficulty, since over-thinking easy problems was our main source of regressions.
What we haven’t shown yet
There are things this work doesn’t establish. Our ablation compares backtracking against no backtracking, so it doesn’t separate the effect of switching strategy from the effect of the typed label. The comparison that would, a strategy switch with an untyped “try again” message, is the experiment we would run next.
Backtracking is also limited to one retry between top-level branches, and deeper backtracking with an adaptive budget might recover more at a higher cost. The same model writes both the answer and the critique, so their errors may be correlated. For example, a model that misreads a constraint may also accept that misreading when it critiques the answer, and a separate critic model is an open direction. Finally, search, critique and tool calls all add latency, which may rule this approach out where responses need to be fast.
A retry that learns why it failed helps within a single problem. In the next post we ask whether those lessons should survive to the next problem, and look at what ReTreVal’s cross-problem memory stores, how it rewrites itself, and what our label-free experiments say about it.
Paper and code
The paper has the full method, the prompts, all the ablations and the error analysis we drew on here. The code includes a small web UI where you can watch the tree grow and backtrack on your own problems.
@article{hs2026retreval,
title = {ReTreVal: Reasoning Tree with Validation and Cross-Problem
Memory for Large Language Models},
author = {HS, Abhishek and Shekar, Pavan C and Jain, Arpit and
Krishnan, Aswanth},
journal = {arXiv preprint arXiv:2601.02880},
year = {2026}
}Notes
- This is the first of three posts on ReTreVal. The next one asks what a reasoning agent should remember between problems.↩
- The failure type shown is the one recorded for the MMLU-Pro law question in our error analysis. We didn’t report that run’s other fields, so they appear by name only.↩
- Each ReTreVal attempt includes tree expansion, critique and tool calls, so it costs more than a single sample. “Fewer attempts” shouldn’t be read as “less compute”.↩
References
- Abhishek HS, Shekar, P. C., Jain, A., and Krishnan, A. (2026). ReTreVal: Reasoning Tree with Validation and Cross-Problem Memory for Large Language Models. arXiv:2601.02880. arxiv.org/abs/2601.02880
- Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arxiv.org/abs/2210.03629
- Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arxiv.org/abs/2303.11366
- Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. arxiv.org/abs/2303.17651
- Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS 2023. arxiv.org/abs/2305.10601
- Zhou, A., et al. (2024). Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models. ICML 2024. arxiv.org/abs/2310.04406
- Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arxiv.org/abs/2203.11171
- Lightman, H., et al. (2024). Let’s Verify Step by Step. ICLR 2024. The source of the MATH-500 subset. arxiv.org/abs/2305.20050
- Wang, Y., et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. NeurIPS 2024 Datasets and Benchmarks. arxiv.org/abs/2406.01574