Tag

rag

14 papers · across all editions

06 · llms Intermediate

Models Are Getting Dumber on Purpose

hruvhwe

This blog post challenges the assumption that benchmark improvements translate to general model capability gains. The author argues that while models are achieving stunning results on math and coding benchmarks with far fewer active parameters than before, they're simultaneously getting worse at basic factual recall — and this is a deliberate tradeoff. The implication for practitioners is significant: if you're building RAG systems or agents that rely on factual grounding, you can't assume a high benchmark score means the model will be reliable on your use case.

Takeaways3

Benchmark scores on math/code are skyrocketing, but factual recall (SimpleQA) is declining — models are being optimized for one at the expense of the other.
Smaller quantized models (e.g., Qwen3.5 9B in 6GB VRAM) are now competitive with models that required massive infrastructure just two years ago.
For production systems, always evaluate models on the specific capability your app depends on — general benchmark rankings can actively mislead you.

from Aug 24, 2026 · 326 points on HN · via api-hn

10 · rag Intermediate

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Pengyu Wang, Benfeng Xu, Shaohan Wang, Xin Zeng, Huarui Wu, Lei Zhang, Licheng Zhang

The conventional wisdom that dense retrieval or agentic search beats BM25 turns out to be scale-dependent — at corpus sizes above roughly 10 million tokens, BM25 consistently outperforms more sophisticated approaches by up to 20 points, while costing a fraction of the query tokens. This challenges the tendency to default to vector search or agents for RAG, especially at enterprise scale where corpus size is large. The key finding is that there's a crossover point, not a universal winner, so matching your retrieval strategy to your actual corpus size matters enormously.

Takeaways3

BM25 outperforms dense retrieval and agentic search at corpus sizes above ~10M tokens, reversing the results typically seen in small-scale benchmarks.
Agentic search uses up to 39x more query tokens than BM25 at scale and becomes less effective as the search space grows, making it a poor default for large corpora.
Retrieval strategy selection should be driven by your actual corpus size — there is no universally best paradigm, and evaluation at the wrong scale gives misleading results.

rag agents evaluations

from Aug 3, 2026 · via api-hf · arXiv:2607.26497

11 · evaluations Intermediate

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo

If you're building document extraction pipelines for enterprise workflows, the gap between 'works on demos' and 'works on real documents' is enormous — and most benchmarks don't capture it. ExtractBench evaluates schema-guided extraction across 370 real enterprise documents and 67 document types, measuring not just value accuracy but also source grounding and cost together. Notably, commercial VLMs do well on short documents but frequently fail on long ones with repeating records — a critical failure mode for invoices, purchase orders, and similar business documents.

Takeaways3

Commercial VLMs often truncate record lists in long documents, making them unreliable for high-volume enterprise extraction without mitigation strategies.
Grounding metadata (which page/word a value came from) is treated as a first-class metric here, which matters enormously for auditability in real enterprise deployments.
Cost is measured alongside accuracy, forcing an honest tradeoff analysis rather than optimizing for correctness alone.

evaluations agents rag llms

from Aug 3, 2026 · via api-hf · arXiv:2607.29677

08 · rag Intermediate

AutoIndex: Learning Representation Programs for Retrieval

Sam O'Nuallain, Nithya Rajkumar, Ramya Narayanasamy, Hanna Jiang, Shreyas Chaudhari, Andrew Drozdov

Instead of tuning your retrieval model, what if you optimized how you prepare documents before they ever hit the index? AutoIndex uses LLM agents to iteratively discover and refine document transformation programs — slicing, enriching, normalizing — that dramatically improve BM25 retrieval without touching the retriever itself. Gains of up to 43% nDCG@10 suggest that document representation is a massively underexplored lever compared to the industry's obsession with embedding models and rerankers.

Takeaways3

Document preprocessing is a first-class optimization target — treating it as fixed leaves significant retrieval quality on the table.
Agentic, validation-guided search over transformation programs outperforms static indexing strategies across diverse task types.
BM25 with learned document programs can rival or beat systems that rely on expensive neural retrievers.

rag agents evaluations

from Jul 27, 2026 · via api-hf · arXiv:2607.18603

09 · rag Intermediate

Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking

Kailin Jiang, Lei Liu, Jian Xi, Hui Xu, Junlin Liu, Baochen Fu, Shaoqing Ren, Bin Li, Vichwang, Yu Lu, Haibo Shi

Standard RAG evaluation scores documents in isolation using nDCG, but what actually matters to an LLM consumer is the quality of the document set as a whole — including redundancy, conflicts, and complementarity between documents. This paper argues that the entire retrieval evaluation paradigm is broken for agentic use cases, introduces a 28K-rubric benchmark covering nine dimensions of set quality, and shows that even the best rerankers top out at 45% coverage. If you're building RAG pipelines for agents, this reframes what 'good retrieval' even means.

Takeaways3

Inter-document interactions (redundancy, conflict, complementarity) are systematically ignored by current retrieval evaluation, creating a blind spot for agentic pipelines.
No existing reranker performs well across both short-form and long-form retrieval scenarios simultaneously.
Shifting to rubric-based set evaluation reveals failure modes invisible to nDCG-style scoring.

rag evaluations agents llms

from Jul 27, 2026 · via api-hf · arXiv:2607.19747

07 · rag Intermediate

GRASP: GRanularity-Aware Search Policy for Agentic RAG

Varun Gandhi, Jaewook Lee, Shantanu Todmal, Franck Dernoncourt, Ryan Rossi, Zichao Wang, Andrew Lan

One of the trickiest problems in agentic RAG isn't retrieval itself — it's deciding *how* to retrieve and at what level of detail, since dumping irrelevant context into the reasoning loop degrades performance significantly. GRASP trains an RL-based agent that dynamically picks between semantic search, keyword search, and paragraph reading, and only expands context when necessary. Engineers building multi-hop reasoning pipelines will find this a practical framework for reducing retrieval noise without sacrificing recall.

Takeaways3

Mixing semantic and keyword retrieval tools under a learned policy outperforms committing to either approach alone.
Retrieving at sentence granularity and expanding only on demand keeps the context window clean and reasoning accurate.
RL-based retrieval coordination can be trained with a reward that balances accuracy, grounding, and query efficiency simultaneously.

rag agents reasoning

from Jul 20, 2026 · via api-hf · arXiv:2607.10463

07 · llms Accessible

Ceci n'est pas une pipe: AI systems as semantic abstractions

Jade Alglave

This paper argues that we lack a precise vocabulary for reasoning about when AI system outputs are justified — and that this gap leads to sloppy evaluation. The authors propose a semantic framework distinguishing between what domain knowledge supports, what sources actually say, and what the system can access at inference time, giving precise definitions to failure modes like unsupported assertion, stale sources, and added hypotheses. Useful conceptual grounding for anyone designing RAG systems, agent tool-calling policies, or evaluation rubrics.

Takeaways3

Apparent fluency in AI outputs systematically obscures whether claims are actually grounded in reliable authority.
Distinguishing 'what sources say' from 'what the system can use' clarifies why RAG and fine-tuning have fundamentally different failure modes.
The framework provides a vocabulary for writing precise specifications for agent actions that must be justified by explicit evidence.

llms foundational evaluations rag agents opinion

from Jul 13, 2026 · via api-arxiv · arXiv:2607.09489

04 · rag Intermediate

Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation

Cedric Caruzzo

This paper exposes a dangerous blind spot in standard RAG evaluation: a system can score near-perfect on hallucination and faithfulness metrics while confidently presenting evidence about the wrong entity. The authors call this 'deceptive grounding' — every claim is sourced from a real document, just the wrong one — and find failure rates up to 87% under adversarial conditions. Critically, domain-specialized medical models are *worse* at this than general models, which should concern anyone building high-stakes RAG applications.

Takeaways3

Standard faithfulness and hallucination metrics cannot detect entity attribution failures, creating a false sense of RAG safety.
Domain-specialized fine-tuning amplifies deceptive grounding rather than mitigating it, making medical RAG systems particularly vulnerable.
Removing entity-specific conflicting evidence from retrieved documents eliminates the failure, pointing toward retrieval filtering as a mitigation.

rag evaluations security llms

from Jul 13, 2026 · via api-arxiv · arXiv:2607.09349

11 · rag Intermediate

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

Yubo Li, Rema Padman, Ramayya Krishnan

This research exposes a critical blind spot in multi-source RAG systems: the same question can yield different answers depending on which institutional source the system retrieves, even when both sources are authoritative. The work shifts evaluation focus from answer correctness to inter-source relationship analysis, revealing that better retrieval actually uncovers more disagreement than expected. Essential for anyone building RAG over institutional knowledge bases.

Takeaways3

Multi-source RAG systems can give different answers to identical questions based on source selection.
Traditional single-gold-answer evaluation paradigms miss source-dependence failure modes.
Better retrieval reveals more inter-source disagreement than conventional metrics suggest.

rag evaluations security

from Jun 8, 2026 · via api-hf · arXiv:2605.29084

09 · software-engineering Intermediate

Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie

This approach solves the repository context problem for code models without the inference overhead of RAG or the cost of per-repo fine-tuning. Code2LoRA generates lightweight adapters that inject repository-specific knowledge directly into the model weights, with an evolutionary variant that updates as codebases change. If you're building AI coding assistants that need deep repository understanding, this offers a practical path to scale beyond token limits.

Takeaways3

Eliminates inference-time token overhead for repository context while maintaining repository-specific knowledge.
Supports both static snapshots and evolving codebases through GRU-backed adapter updates.
Outperforms traditional parameter-efficient fine-tuning approaches on repository-level tasks.

software-engineering llms rag

from Jun 8, 2026 · via api-hf · arXiv:2606.06492

10 · rag Intermediate

Contexts are Never Long Enough: Structured Reasoning for Scalable Question Answering over Long Document Sets

Harshit Joshi, Priyank Shethia, Jadelynn Dao, Monica S. Lam

SLIDERS challenges the conventional chunk-and-aggregate approach to document QA by extracting information into a relational database and reasoning with SQL instead of concatenated text. This architectural approach sidesteps the fundamental limitation that any fixed context window will eventually be exceeded, making it essential reading for engineers building document analysis systems that need to scale beyond typical RAG limitations.

Takeaways3

Traditional chunk-and-aggregate approaches hit an aggregation bottleneck as document collections grow, even with infinite context windows.
Extracting information into structured databases and reasoning with SQL scales better than reasoning over concatenated text.
Data reconciliation using provenance and extraction rationales is crucial for maintaining coherence in locally extracted information.

rag reasoning llms software-engineering

from Apr 27, 2026 · via api-hf · arXiv:2604.22294

10 · rag Intermediate

Don't Retrieve, Navigate: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG

Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh

Corpus2Skill fundamentally reimagines RAG by giving AI agents a navigable map of your knowledge base instead of treating them as passive consumers of search results. Rather than hoping retrieval finds the right documents, agents can see the corpus structure, drill down through hierarchical summaries, and strategically combine evidence across different branches—solving the core limitation that RAG systems can't reason about what they haven't seen.

Takeaways3

Traditional RAG limits AI agents to passive consumption of search results without visibility into corpus structure or unexplored areas.
Hierarchical skill directories enable agents to navigate knowledge strategically and combine evidence across different topic branches.
Offline corpus compilation into navigable structures provides better performance than runtime retrieval-only approaches.

rag agents software-engineering

from Apr 20, 2026 · via api-hf · arXiv:2604.14572

08 · agents Intermediate

The Design of AI Memory Systems

Unable to provide detailed description due to missing content, but AI memory systems design is crucial for building production agents and RAG applications that need to maintain context and learn from interactions.

agents rag foundational

from Apr 6, 2026 · via api-lobsters

12 · software-engineering Intermediate

Show HN: Robust LLM extractor for websites in TypeScript

andrew_zhong

A practical TypeScript library that solves the common problem of extracting structured data from websites using LLMs, addressing real pain points like HTML noise, token budget management, and brittleness of traditional CSS selectors. This represents the kind of focused tooling that makes AI-powered data extraction reliable enough for production use.

Takeaways3

LLM-based extraction needs preprocessing to remove HTML noise and stay within token budgets for reliable results.
Focused tools that solve specific AI integration problems are more valuable than general-purpose solutions for production teams.
AI extraction can replace brittle CSS selectors but requires thoughtful engineering to handle edge cases and failures.

software-engineering how-we-work rag

from Mar 29, 2026 · 72 points on HN · via api-hn