Agents in the Wild, Evaluations Close to Home

Grep beats LSP? Why coding agents ignore your fancier tools

kaonashi-tyc-01

If you're building coding agents and assuming that giving them more powerful, semantically-aware tools will improve performance, this post is a useful reality check. The author ran a direct comparison between grep and LSP-backed semantic navigation and found that agents frequently defaulted to grep — and sometimes performed *worse* when forced onto the more precise semantic path. The core insight is that 'LLM-friendliness' (familiar output format, sufficient context for the next step, likely seen during training) can matter more than raw tool capability when it comes to agent task success.

Research acceleration: The view inside OpenAI

This post offers a rare inside look at how OpenAI's own engineering and research teams are actually using agentic coding tools day-to-day in 2026, framed around their internal push toward Recursive Self-Improvement (RSI). It's notable less for its technical depth and more for what it signals: even the lab building these systems is treating agentic engineering as a fundamental shift in how research gets done, not just a productivity add-on. For senior engineers wondering how far ahead the frontier labs really are in their internal practices, this is a useful data point.

An Accidental Blackboard

Martin Fowler

A Thoughtworks CTO recounts a week-long experiment where 10 engineers in Barcelona tried to build a complex airline IROps system using an aggressively agentic approach — and accidentally rediscovered the blackboard pattern for multi-agent coordination. The 'accidental' framing is the interesting part: the team converged on a shared-state coordination model organically, without setting out to implement it. For engineers designing multi-agent systems, this is a grounded, practitioner-level account of what coordination problems actually look like at scale and how classic CS patterns resurface in modern agent architectures.

Why Ramp built its own in-house coding agent, Inspect

Gergely Orosz

A handful of leading tech companies — Ramp, Block, Stripe, Shopify — have quietly built their own internal coding agents rather than relying on off-the-shelf tools like Cursor or Claude Code, and this post digs into why Ramp made that call with their agent, Inspect. The core argument is that deep integration with internal systems, custom workflows, and proprietary context gives in-house agents a compounding advantage that generic tools can't match. This is essential reading if you're at a mid-to-large engineering org debating whether to build or buy your AI coding infrastructure.

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee

If you're benchmarking coding agents on SWE-bench and feeling good about the results, this paper is a reality check. Real user bug reports are short, casual, and information-sparse — 88% of real prompts contain only a problem statement, while 94% of SWE-bench problems are formally written and information-rich. The authors built RealSWE to test agents under realistic conditions and found that realistic inputs drop resolution rates by ~6.4 percentage points on average and can even flip model rankings. Crucially, what's in the prompt matters a lot: describing desired behavior and motivation helps, but adding reproduction steps or environment info is mostly just noise.

Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered

Aryo Pradipta Gema, Neel Rajani, Rohit Saxena, Wai-Chung Kwan, Pasquale Minervini

If you're relying on chain-of-thought reasoning traces to monitor or audit agent behavior, this paper reveals a serious blind spot: models frequently act on biased cues without mentioning them in their reasoning. The key finding is that cues delivered via tool returns or embedded in raw artifacts are far less likely to be verbalized than cues in user messages — meaning the very monitoring setups most relevant to agentic pipelines are the ones where CoT is least faithful. Worse, higher rates of unverbalized adoption correlate with lower detection ability by LLM-based transcript monitors, and simply telling a model its reasoning will be monitored doesn't close the gap.

Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits

Passing tests isn't the same as writing good code, and this benchmark makes that gap measurable. NITR (Needle in the Repo) evaluates whether AI coding agents produce maintainable code — not just correct code — by pairing functional tests with structural oracles that check things like modularity, dependency control, and responsibility decomposition. The results are sobering: the best configuration only solves 57% of cases, and 13% of outcomes pass all functional tests while failing the structural checks. Architectural concerns like dependency control are especially hard, with a 4.3% solve rate across configurations.

“Next-token predictor” is the wrong mental model for LLMs

garrinm

This blog post argues that calling LLMs 'next-token predictors' is technically accurate but practically misleading — it causes engineers to reason poorly about what these models actually do. The author's core argument is that because LLMs are trained on human-generated text, they are better understood as simulators of the distribution of human thought and expression, not simple conditional probability machines. This mental model shift has real consequences for how you design prompts, interpret outputs, and think about failure modes.

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

Kevin Du

Just because a reasoning trace is readable doesn't mean it's telling you what actually mattered — and this paper puts hard numbers on that gap. The authors measure step 'importance' via Monte Carlo rollouts (how much does including this step change the probability of a correct answer?) and find that LLM judges can do better than chance at identifying high-importance steps, but fall well short of what's theoretically possible. This is a direct challenge to the foundations of process reward models and LLM-as-critic approaches, which assume that the text of a reasoning step encodes its functional role.

SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Yibo Peng, Long Lian, David Wagner, Sizhe Chen

If you're building LLM-powered agents that touch external data — emails, web pages, files — prompt injection is your biggest security headache, and existing defenses have been embarrassingly easy to break. This paper identifies *why*: current defensive fine-tuning methods treat an entire model output as uniformly good or bad, so the model never learns exactly which tokens represent a security failure. SecOPD fixes this with token-level feedback during fine-tuning, scoring each output token against what the model would have produced on clean (non-injected) input. The result is dramatic: their defended Qwen3.6-27B drops attack success rates from 94% down to 9% against state-of-the-art adaptive injections, and the security generalizes to agentic tool-calling scenarios the model was never trained on.

Adversarial Review: Structured Disagreement for Grounded Agentic Code Review

More agents doesn't mean better code — this post challenges the assumption that scaling up multi-agent teams improves software engineering outcomes, and instead argues that *structured disagreement* among a minimal set of agents is what actually moves the needle. The Adversarial Review (AR) protocol uses just three agents: a coder, a reviewer, and a critic whose explicit job is to push back on the reviewer before any edits are made. This setup outperforms a five-agent baseline on LiveCodeBench and achieves the highest F1 on SWE-PRBench, while also surfacing an important failure mode — agents left to their own devices tend to converge on false consensus without real evidence.