Tag: reasoning

Advanced

Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Lu Dai, Ziyang Rao, Yili Wang, Hanqing Wang, Hao Liu, Hui Xiong

Fine-tuning to inject new knowledge into LLMs produces a frustrating pattern: the model memorizes the facts but fails to use them in downstream reasoning. This paper investigates the mechanism and finds that memorized representations often exist in the model but aren't routed through the layers where they'd actually influence computation — a 'knowledge-circuit misalignment.' The practical upshot is a diagnostic technique that recovers 58-75% of the generalization gap without architectural changes.

Takeaways

Memorization and usable generalization are mechanistically distinct processes that fine-tuning can decouple.
Knowledge-circuit misalignment means a model can 'know' a fact internally while completely failing to apply it during reasoning.
Self-patching as a diagnostic technique can identify which layers need intervention without requiring full retraining.

from Jul 13, 2026 · via api-hf · arXiv:2607.08393

Intermediate

Agentic Abstention: Do Agents Know When to Stop Instead of Act?

Han Luo, Bingbing Wen, Lucy Lu Wang

agents evaluations reasoning foundational

Knowing when to stop is as important as knowing how to act, but most agent evaluations only measure task completion and ignore appropriate abstention. This large-scale study across 28,000+ tasks finds that agents systematically over-act on underspecified or impossible tasks, and that the challenge isn't recognizing uncertainty — it's correctly deciding when more information-gathering won't help. Critical reading for anyone designing agent reliability and safety guardrails.

Takeaways

Agents over-act far more often than they over-abstain, making inappropriate tool use a bigger practical problem than excessive caution.
The decision to abstain is sequential, not binary — agents often need several tool calls before the right answer is 'stop'.
Current scaffolds and models lack robust mechanisms for detecting when a goal is fundamentally unachievable in the given environment.

from Jul 6, 2026 · via api-hf · arXiv:2606.28733

Intermediate

From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

Paul Dubois

llms foundational reasoning opinion

This paper makes a precise architectural argument: LLMs aren't a failed attempt at world models, they're a degenerate special case where the state space is token sequences and the only action is appending one token. More importantly, it maps a continuous spectrum from next-token prediction to latent-space architectures (JEPA), showing that multi-token prediction and next-latent prediction are intermediate stops already present in current research. For engineers thinking about what comes after transformers, this is a useful conceptual framework for evaluating emerging architectures.

Takeaways

LLMs are a constrained special case of world models, not a fundamentally different paradigm—world models generalize them.
There is a continuous architectural spectrum from next-token prediction to latent-space models, with explorable intermediate designs.
Moving along this spectrum trades LLMs' key practical advantages (interpretable states, scalable training) for greater representational power.

from Jun 29, 2026 · via api-arxiv · arXiv:2606.28127

Intermediate

A Verifiable Search Is Not a Learnable Chain-of-Thought

Harsh Patel

reasoning agents foundational llms

Not every algorithmic procedure can be taught to an LLM as chain-of-thought, and this paper identifies exactly why: tasks requiring backtracking search fundamentally resist distillation even when the model can execute every individual step correctly. A model that's 97-100% accurate on arithmetic sub-steps still achieves only 1-7% on cryptarithm despite extensive CoT training and RL from verifiable rewards. This isn't a scale or capability problem—it's a structural mismatch between sequential token generation and search-based computation.

Takeaways

Backtracking search procedures cannot be reliably distilled into chain-of-thought, even when the model executes every sub-step correctly.
This limitation is architectural, not a training or scale issue—more compute won't fix it.
Agent designers should route search-requiring subtasks to external solvers rather than expecting LLMs to internalize them as reasoning steps.

from Jun 29, 2026 · 0 citations · via api-hf · arXiv:2606.21884

Intermediate

Do Thinking Tokens Help with Safety?

Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora

security reasoning llms foundational

The intuition that 'thinking tokens give models time to reconsider unsafe outputs' turns out to be largely wrong. Across multiple reasoning model families, the final refusal-or-comply decision is already predictable with 88%+ accuracy from the very first token's hidden state—before any visible reasoning appears. The thinking text looks deliberative but mostly functions as prefix completion, with outcomes rarely changing after the first 20% of the chain-of-thought. This means you shouldn't rely on extended thinking as a safety mechanism.

Takeaways

Reasoning models' safety outcomes are effectively decided before thinking begins, not during it.
Thinking tokens provide the appearance of deliberation but rarely cause the model to reverse its initial safety disposition.
Safety architecture for reasoning models needs to address pre-thinking biases, not just the visible chain-of-thought.

from Jun 29, 2026 · via api-hf · arXiv:2606.25013

Accessible

Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

Sanket Badhe, Deep Shah

prompt-engineering reasoning llms

Fine-tuning small models to reason well is expensive and opaque; Prompt-Level Distillation offers a third path — extract reasoning patterns from a large teacher model and encode them as structured system prompt instructions for a smaller student model. The benchmark results are striking (57% to 90% F1 on StereoSet, 67% to 83% on Contract-NLI) and the approach is immediately actionable without any training infrastructure.

Takeaways

Structured system prompt instructions distilled from a teacher model can close much of the performance gap between small and large models without fine-tuning.
PLD preserves interpretability because the decision logic is explicit in the prompt, unlike weights modified by fine-tuning.
Cross-architecture generalization (Gemma-3 and Mistral) suggests this is a robust technique, not just an artifact of a specific model family.

from Jun 22, 2026 · via api-hf · arXiv:2602.21103

Intermediate

An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models

Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama, Tan Zhi-Xuan

evaluations reasoning llms foundational

If you're using reasoning models as judges or validators — to check other models' outputs, verify proofs, or catch errors — this paper reveals a fundamental flaw: frontier reasoning models score as low as 48% at detecting subtly flawed reasoning even when they can solve the underlying problem nearly perfectly. The root cause is answer confirmation bias: models check whether the final answer is correct rather than verifying each reasoning step, which undermines common self-verification and LLM-as-judge patterns.

Takeaways

Reasoning models have a severe production-evaluation gap — they're dramatically worse at catching flawed reasoning than at producing correct reasoning.
Answer confirmation bias causes models to validate solutions by checking the answer, not the reasoning chain, making step-level errors invisible.
LLM-as-judge architectures that rely on reasoning models for correctness verification need independent validation mechanisms, especially for math and logic tasks.

from Jun 22, 2026 · via api-hf · arXiv:2606.01462

Intermediate

CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning

Linas Nasvytis, Simon Jerome Han, Ben Prystawski, Satchel Grant, Noah D. Goodman, Judith E. Fan

reasoning agents prompt-engineering

CORE enables rapid reasoning improvements using minimal training samples by generating natural language insights that contrast successful and failed attempts, outperforming both parametric and non-parametric methods. While other approaches need hundreds of samples and thousands of rollouts, CORE achieves comparable results with as few as five training samples. This is immediately practical for production systems where you have limited training data but need reliable reasoning improvements.

Takeaways

CORE achieves faster reasoning improvements than traditional methods while using significantly fewer model rollouts.
Natural language insights from contrasting successful and failed attempts enable effective learning from minimal data.
Non-parametric reflection approaches can outperform parametric methods when training samples are severely limited.

from Jun 15, 2026 · via api-hf · arXiv:2605.28742

Intermediate

The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure

Yubo Li, Ramayya Krishnan, Rema Padman

reasoning evaluations security foundational

This research uncovers a disturbing failure mode where AI models maintain correct reasoning chains while simultaneously giving wrong final answers under adversarial pressure. The finding challenges the assumption that chain-of-thought reasoning makes models more reliable — in fact, it may create a dangerous gap between what the model knows internally and what it outputs. Essential for understanding the reliability limits of reasoning models in production.

Takeaways

Models can maintain correct reasoning while giving wrong answers under sustained adversarial pressure.
Chain-of-thought reasoning creates a gap between internal knowledge and behavioral output.
Standard faithfulness probes miss this failure mode, requiring new evaluation frameworks.

from Jun 8, 2026 · via api-hf · arXiv:2605.29087

Intermediate

REPOT: Recoverable Program-of-Thought via Checkpoint Repair

Parsa Mazaheri

reasoning agents software-engineering

Program-of-Thought approaches fail silently when a single invalid action breaks the entire execution plan, wasting the valid prefix. RePoT introduces deterministic checkpoint repair — walking through the program to find the first failure, then resuming from the verified prefix with one additional LLM call. It's a simple but powerful idea that improves success rates by 3-11 percentage points while costing at most one extra call on the ~14% of problems where basic PoT fails.

Takeaways

Recoverable execution with checkpoint repair dramatically improves Program-of-Thought success rates for minimal additional cost.
Deterministic replay to the first failure point enables targeted correction rather than full re-execution.
The approach costs at most one extra LLM call and only triggers on the subset of problems where basic PoT fails.

from Jun 1, 2026 · via api-hf · arXiv:2605.30052

Intermediate

Unsupervised Process Reward Models

Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi, Hang Guo, Maria Brbic

reasoning evaluations llms

Eliminates the expensive human annotation bottleneck in training process reward models by deriving scoring functions directly from LLM token probabilities. This matters because process rewards are crucial for steering LLM reasoning, but current approaches require expert step-by-step annotations that don't scale—uPRM achieves comparable performance while removing human supervision entirely.

Takeaways

Unsupervised process rewards can identify reasoning errors without expensive human step-by-step annotations.
Token probability-based scoring functions can effectively assess erroneous steps across batches of reasoning trajectories.
uPRM provides up to 15% absolute accuracy improvements over LLM-as-a-Judge methods for error detection.

from May 25, 2026 · via api-hf · arXiv:2605.10158

Accessible

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

Tsz Ting Chung, Lemao Liu, Mo Yu, Dit-Yan Yeung

llms prompt-engineering reasoning foundational

This overturns conventional wisdom about many-shot in-context learning for reasoning tasks. While more examples help with simple tasks, reasoning tasks show unstable scaling behavior, and semantic similarity-based retrieval actually hurts performance. The order of examples matters more than previously thought. This has immediate implications for how you structure prompts and manage context in reasoning-heavy production systems.

Takeaways

Many-shot scaling rules for non-reasoning tasks don't apply to reasoning tasks and can degrade performance.
Semantic similarity poorly predicts procedural compatibility in chain-of-thought reasoning.
Example ordering significantly impacts performance and requires careful consideration in production prompt design.

from May 18, 2026 · via api-hf · arXiv:2605.13511

Intermediate

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna

reasoning foundational llms

This fundamentally changes how you should think about RL fine-tuning—it reveals that RL doesn't teach models new reasoning strategies but simply redistributes probability mass toward solutions already in the base model. The effect is incredibly sparse (1-3% of tokens), concentrated at high-entropy decision points, and the base model's own uncertainty can predict exactly where these corrections occur without any RL training.

Takeaways

RL fine-tuning redistributes existing model knowledge rather than teaching new capabilities.
Only 1-3% of token positions are affected, concentrated at high-entropy decision points.
Base model entropy alone can predict where RL corrections will occur.

from May 11, 2026 · via api-hf · arXiv:2605.06241

Intermediate

Terence Tao (@tao@mathstodon.xyz)

foundational reasoning opinion

Terence Tao identifies a critical gap in AI mathematical reasoning that applies directly to software engineering: while AI can generate and verify proofs (or code), it struggles with the third component—digestion or true understanding. This creates 'proof indigestion' where solutions are technically correct but lack the deeper comprehension needed for maintenance, debugging, or extension, a problem that simply training AI to write better explanations won't fully solve.

Takeaways

AI excels at generation and verification but fails at deep understanding and explanation.
Technical correctness doesn't guarantee maintainable or understandable solutions.
Simply automating explanation generation won't solve the fundamental comprehension gap.

from May 11, 2026 · via suggestion

Intermediate

KWBench: Measuring Unprompted Problem Recognition in Knowledge Work

Ankit Maloo

evaluations reasoning llms

KWBench introduces the first benchmark for unprompted problem recognition in professional contexts, testing whether LLMs can identify the underlying structure of a situation before attempting to solve it. This addresses a critical gap in current evaluations that assume the problem is already clearly defined, making it essential for understanding how LLMs perform in real knowledge work where recognizing what type of problem you're facing is half the battle.

Takeaways

Current LLM benchmarks assume problems are already clearly defined, missing the crucial step of recognizing what type of situation you're facing.
The benchmark tests game-theoretic pattern recognition across professional domains like acquisitions, contract negotiations, and fraud analysis.
Unprompted problem recognition is a fundamental capability gap that affects how well LLMs can assist with real knowledge work.

from Apr 27, 2026 · via api-hf · arXiv:2604.15760

Intermediate

Contexts are Never Long Enough: Structured Reasoning for Scalable Question Answering over Long Document Sets

Harshit Joshi, Priyank Shethia, Jadelynn Dao, Monica S. Lam

rag reasoning llms software-engineering

SLIDERS challenges the conventional chunk-and-aggregate approach to document QA by extracting information into a relational database and reasoning with SQL instead of concatenated text. This architectural approach sidesteps the fundamental limitation that any fixed context window will eventually be exceeded, making it essential reading for engineers building document analysis systems that need to scale beyond typical RAG limitations.

Takeaways

Traditional chunk-and-aggregate approaches hit an aggregation bottleneck as document collections grow, even with infinite context windows.
Extracting information into structured databases and reasoning with SQL scales better than reasoning over concatenated text.
Data reconciliation using provenance and extraction rationales is crucial for maintaining coherence in locally extracted information.

from Apr 27, 2026 · via api-hf · arXiv:2604.22294

Intermediate

Self-Execution Simulation Improves Coding Models

Gallil Maimon, Ori Yoran, Felix Kreuk, Michael Hassid, Gal Cohen, Pierre Chambon, Yossi Adi

llms software-engineering reasoning foundational

Code LLMs struggle because they can't accurately predict what their generated code will do when executed, leading to logical errors that escape syntax checking. This research trains models to simulate program execution step-by-step, enabling self-verification and iterative debugging of their own code. The approach combines supervised learning on execution traces with reinforcement learning, achieving significant improvements on competitive programming benchmarks and providing a foundation for more reliable AI coding assistants.

Takeaways

Teaching models to simulate execution enables self-verification and iterative debugging of generated code.
Combining execution simulation training with reinforcement learning significantly improves competitive programming performance.
Step-by-step execution traces provide grounding that helps models understand and debug their logical reasoning in code.

from Apr 13, 2026 · via api-hf · arXiv:2604.03253

Advanced

Moonlake: Causal World Models should be Multimodal, Interactive, and Efficient — with Chris Manning and Fan-yun Sun

foundational agents reasoning

Stanford researchers discuss Moonlake, their approach to building causal world models that understand multimodal interactions and can efficiently reason about cause and effect in complex environments. This foundational research explores how AI systems can develop better understanding of how the world works, which is crucial for building more capable agents that can plan and reason about their actions.

Takeaways

Causal world models enable AI systems to understand cause-and-effect relationships rather than just correlations.
Multimodal approaches help models build more comprehensive understanding of how actions affect environments.
Efficient world models are essential for practical agent deployment in real-world scenarios.

from Apr 6, 2026 · via rss-latentspace

Intermediate

Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation

Richard J. Young

evaluations reasoning foundational

Challenges the conventional wisdom that faithfulness in chain-of-thought reasoning is an objective metric. Testing three different classifiers on identical data produced faithfulness rates ranging from 69% to 83% — a massive difference that undermines most CoT evaluation literature. Essential if you're building evaluation pipelines for reasoning systems, as it shows your measurement approach fundamentally shapes your conclusions.

Takeaways

Faithfulness measurements in chain-of-thought evaluation vary dramatically (69% to 83%) depending on the classifier used, making evaluation methodology critical.
Your measurement approach fundamentally shapes conclusions about reasoning system performance, not just the system itself.
Evaluation pipelines for reasoning systems need multiple measurement approaches to avoid classifier bias.

from Mar 23, 2026 · via api-arxiv · arXiv:2603.20172

Advanced

FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

Chiyu Ma

llms reasoning foundational

Introduces FIPO, a reinforcement learning algorithm that breaks through the reasoning stagnation plaguing current LLMs by using fine-grained credit assignment instead of uniform token rewards. Extends chain-of-thought reasoning from 4,000 to over 10,000 tokens and boosts mathematical problem-solving accuracy from 50% to 58%. Directly applicable if you're building or fine-tuning models for complex reasoning tasks.

Takeaways

FIPO uses fine-grained credit assignment instead of uniform token rewards to extend reasoning from 4,000 to over 10,000 tokens.
Mathematical problem-solving accuracy improved from 50% to 58% by breaking through reasoning stagnation in current LLMs.
This reinforcement learning approach is directly applicable for fine-tuning models on complex reasoning tasks.

from Mar 23, 2026 · via api-arxiv · arXiv:2603.19835

$The $\mathbf{Y}$-Combinator for LLMs: Solving Long-Context Rot with $λ$-Calculus$

Advanced

The $\mathbf{Y}$-Combinator for LLMs: Solving Long-Context Rot with $λ$-Calculus

Amartya Roy

llms reasoning foundational

Replaces the chaotic read-eval-print loops of existing recursive language models with a structured functional programming approach grounded in λ-calculus. This provides formal guarantees like termination and cost bounds that standard recursive LLMs lack, making long-context reasoning predictable and analyzable. Critical if you're building production systems that need reliable recursive reasoning without the execution risks of arbitrary code generation.

Takeaways

Replacing chaotic read-eval-print loops with λ-calculus provides formal guarantees like termination and cost bounds for recursive LLMs.
This structured functional programming approach makes long-context reasoning predictable and analyzable unlike arbitrary code generation.
Production systems requiring reliable recursive reasoning need formal execution frameworks rather than unstructured recursion.

from Mar 23, 2026 · 0 citations · via api-arxiv · arXiv:2603.20105