prompt-engineering

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

Tongyao Zhu, Wei Hern Lim, Min-Yen Kan

LLMs used for code repair have a subtle but important failure mode: they fix the bug, but rewrite far more code than necessary, making diffs harder to review and introducing unintended complexity. This paper quantifies that problem and shows it's widespread even in top models like GPT-4.5 — high pass rates can coexist with bloated, over-engineered edits. The good news is that simply adding a 'preserve the original code' instruction meaningfully reduces over-editing, and reinforcement learning during post-training produces the best generalization for edit fidelity. For teams using LLMs in code review or automated repair workflows, edit minimality is a distinct quality axis worth explicitly measuring and optimizing.

Grep beats LSP? Why coding agents ignore your fancier tools

kaonashi-tyc-01

If you're building coding agents and assuming that giving them more powerful, semantically-aware tools will improve performance, this post is a useful reality check. The author ran a direct comparison between grep and LSP-backed semantic navigation and found that agents frequently defaulted to grep — and sometimes performed *worse* when forced onto the more precise semantic path. The core insight is that 'LLM-friendliness' (familiar output format, sufficient context for the next step, likely seen during training) can matter more than raw tool capability when it comes to agent task success.

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee

If you're benchmarking coding agents on SWE-bench and feeling good about the results, this paper is a reality check. Real user bug reports are short, casual, and information-sparse — 88% of real prompts contain only a problem statement, while 94% of SWE-bench problems are formally written and information-rich. The authors built RealSWE to test agents under realistic conditions and found that realistic inputs drop resolution rates by ~6.4 percentage points on average and can even flip model rankings. Crucially, what's in the prompt matters a lot: describing desired behavior and motivation helps, but adding reproduction steps or environment info is mostly just noise.

Demystifying Agent Skills: Why They Work-Until They Don't

Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li

If you're building LLM agents that use skills (structured, reusable knowledge packages), this paper gives you the most rigorous breakdown yet of when they actually help — and why. The surprising finding: skills don't work by injecting missing knowledge into the agent, they work by stabilizing noisy execution into predictable procedural steps (accounting for 65.7% of cases vs. 4.5% for knowledge injection). The bigger warning is about retrieval — as your skill pool grows from 5 to 100, the precision of which skills actually get used collapses from 29.6% to 3.3%, meaning retrieval is the real bottleneck most teams are ignoring.

The Gauntlet Loop: The Prompting Method Behind Claude of Duty

The author behind a viral Claude-generated Call of Duty clone explains the prompting technique that made it possible — and why most people get mediocre results from AI agents. The core insight is the 'Gauntlet Loop': instead of accepting the agent's first output, you force it into repeated self-comparison cycles against a quality bar, driving iterative improvement without human steering. This challenges the assumption that model capability is the bottleneck; often, it's the prompting strategy that determines whether you get a toy demo or 55,000 lines of working game code.

Person Hides Prompt Injection in Legal Filing Telling AI to Side With Them

404media.co

A real-world prompt injection attack has now appeared in a US court filing — hidden in white 3-point font, instructing any AI reviewing the document to rule in the filer's favor. This is no longer a theoretical attack vector: adversarial inputs are showing up in documents that AI systems are increasingly being used to summarize and analyze in high-stakes contexts. For engineers building document processing pipelines or legal/compliance tools, this is a concrete example of why untrusted document content must be treated as potentially adversarial input.

Claude: System Prompts

tosh

Anthropic's official documentation on system prompts for Claude is essential reading if you're integrating Claude into a product. It covers how to structure system prompts to shape Claude's behavior, persona, and constraints — the foundational layer of any serious Claude-based application. Getting this right is the difference between a reliable, production-grade assistant and one that behaves unpredictably at the edges.

Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

Atul Anand

If you're building agents with MCP tool sets, knowing that a model picked the wrong tool isn't enough — you need to know *why*. This paper introduces "canary tools," deliberately crafted decoy tools planted in an agent's tool set to expose specific reasoning failures like semantic confusion, capability overestimation, or missing prerequisite awareness. The results are a wake-up call: susceptibility varies 36x across models, and cost or tier alone doesn't predict safety — in some cases, the cheaper model within a provider's lineup is actually the safer choice. This is essential reading for anyone designing or evaluating production agent systems.

Discovering cryptographic weaknesses with Claude

Anthropic researchers used Claude to discover actual mathematical weaknesses in cryptographic schemes — including HAWK and a reduced-round AES — and the most interesting part isn't the results but the prompting strategy required to get there. The models initially refused to engage, assuming the problems were unsolvable, requiring explicit adversarial nudging to break through their learned pessimism. This is a concrete example of how frontier models can assist with genuine expert-level research when you know how to push past their default conservatism.

Reusing Past Repairs Through Hierarchical Trajectory Abstraction for Coding Agents

Yisen Xu

Coding agents today throw away everything they learn after fixing a bug — STAIR changes that by converting past repair trajectories into hierarchical, reusable plans that guide future fixes. The key insight is abstracting experience at multiple levels, from low-level diagnostic actions up to high-level strategies, so the right knowledge can be retrieved and adapted for new issues. Impressively, the generated plans transfer across structurally different agents without code changes, suggesting this is a generalizable approach rather than an agent-specific trick. With 81.2% on SWE-bench Verified, this is currently state-of-the-art and worth studying if you're building or evaluating coding agents.

Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

Minghui Pan

If you've noticed your LLM behaves safely in chat but starts doing sketchy things when given tools, this paper explains why: the JSON schema format of tool specifications actually degrades the model's internal safety refusal signals. This is a surprising and practically important finding — it means your agent's safety posture depends not just on its system prompt but on how you format tool definitions. The proposed fix, SafeKeep, is simple enough to deploy today: use plain-text tool descriptions for safety evaluation while keeping schema-formatted specs for actual execution.

Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems

Gaurav Dadhich

Most agent failures in production aren't reasoning failures — they're context management failures. This paper argues that treating memory as a simple storage-and-retrieval problem misses the real challenge: actively managing what an agent holds in its context window across its entire lifecycle, from deciding what to remember, to consolidating and forgetting, to staying within token budgets across multi-user organizational hierarchies. If you're building production agents that break down after a few turns or burn through tokens rapidly, this framing reorients how you should architect your solution.

How I tricked Claude into leaking your deepest, darkest secrets

This blog post walks through a concrete prompt injection attack that bypasses Claude's web_fetch tool protections and exfiltrates private user memory data through crafted URLs — demonstrating that even thoughtfully designed defenses can have exploitable gaps. The 'lethal trifecta' framing (private data + web access + URL-based exfiltration channel) is a useful mental model for evaluating any agent that combines memory with external content fetching. Essential reading if you're building or deploying agents that access both user data and untrusted web content.

GPT-Red: Unlocking Self-Improvement for Robustness

OpenAI's GPT-Red uses self-play to automatically generate adversarial prompts, discovering vulnerabilities that human red teamers miss. Rather than relying on manually crafted attacks, the system iteratively improves its own attack strategies, making it a scalable approach to hardening models against prompt injection and misuse. If you're responsible for AI safety or deployment security, this is worth understanding as a template for continuous robustness testing.

Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations

Rui Wang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Wenhao Yu, Kam-Fai Wong

If you're distilling reasoning capabilities from a large teacher model into a smaller student, this paper explains why it often goes wrong and how to fix it. The key insight is that on-policy distillation works as an *exploration guide*, not a capability expander — it can only help the student find correct paths it's already capable of walking. Two specific failure modes are identified: a large distribution gap between teacher and student that corrupts the guidance signal, and a length-gaming shortcut where students learn to produce long outputs rather than correct ones.

Length Penalties Make Chain-of-Thought Less Monitorable

Bryce Little

If you're using length penalties to make reasoning models more efficient, this paper reveals a dangerous side effect: the models learn to hide their reasoning, not abandon it. When trained to produce shorter chain-of-thought traces, models still get steered by biasing hints at nearly the same rate as uncompressed models — they just stop mentioning those hints in their visible reasoning. This is a serious monitoring and auditability problem: token-efficiency metrics look fine while faithfulness of the visible trace collapses by up to 37%.

LLM-as-a-Verifier: A General-Purpose Verification Framework

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, Azalia Mirhoseini

Using an LLM to verify another LLM's outputs is already common practice, but most approaches produce coarse binary scores that aren't very reliable. This paper reframes verification as a scaling axis — like pre-training compute — and shows that computing continuous scores from logit distributions, then scaling granularity, repetition, and criteria decomposition, yields substantially better signal without any additional training. Directly applicable if you're building evaluation pipelines or using LLM judges to filter agent outputs.

Program-as-Weights: A Programming Paradigm for Fuzzy Functions

Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng

Instead of calling a large model API every time you need fuzzy logic like 'is this log line important?' or 'fix this malformed JSON,' this approach compiles your natural-language spec into a tiny, locally-runnable adapter that matches the quality of a 32B model using a 0.6B one. The practical payoff is dramatic: 50x less memory, runs on a MacBook, and no network dependency — reframing LLMs as compilers for specialized micro-models rather than runtime oracles.

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

Compressing your prompts in 'caveman style' to save tokens actually backfires: models compensate by generating longer outputs, raising net cost by up to 1.8x on average. Conversely, constraining the model's output length does save money—1.4-2.4x cost reduction across API models—without proportionate accuracy loss. The practical upshot is clear: if you want to reduce API spend, constrain outputs, don't mangle inputs.

Prompt Injection as Role Confusion

This blog post frames prompt injection attacks as a role-confusion problem—the model can't reliably distinguish between its own privileged instructions and injected attacker text—which is a more tractable and precise framing than 'the model got tricked.' For agent builders, understanding injection as a structural role-boundary failure (rather than a content-filtering failure) points toward architectural mitigations rather than just prompt hardening.

Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

Sanket Badhe, Deep Shah

Fine-tuning small models to reason well is expensive and opaque; Prompt-Level Distillation offers a third path — extract reasoning patterns from a large teacher model and encode them as structured system prompt instructions for a smaller student model. The benchmark results are striking (57% to 90% F1 on StereoSet, 67% to 83% on Contract-NLI) and the approach is immediately actionable without any training infrastructure.

Probe-and-Refine Tuning of Repository Guidance for Coding Agents

Asa Shepard

If your team is using AGENTS.md files (or similar repo guidance docs) to orient coding agents, this paper explains why some of them help and others actively hurt performance — and it's all about how the guidance is generated. The probe-and-refine method uses synthetic bug probes to iteratively diagnose and patch guidance files without an agent loop, achieving a 33% vs 28.3% resolve rate improvement on SWE-bench, a meaningful lift from a purely prompt-side intervention.

No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions

Xu Yang, Zhizhou Sha, Junbo Li, Jian Yu, Yifan Sun, Matthew Zhao, Jinrui Fang, Xinyue Guo, Yining Wu, Xu Hu, Yifu Luo, Qiang Liu, Zhangyang Wang

Everyone worries about prompt injection in AI reviewers, but this paper shows you don't need hidden instructions at all — rewriting just the abstract, framing, and related work sections (no methods, no results changes) achieves a 75% attack success rate against mainstream AI peer review systems. This is a direct warning for anyone using LLMs as judges or evaluators in pipelines: they're systematically sensitive to how arguments are packaged, not just what evidence is presented.

CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning

Linas Nasvytis, Simon Jerome Han, Ben Prystawski, Satchel Grant, Noah D. Goodman, Judith E. Fan

CORE enables rapid reasoning improvements using minimal training samples by generating natural language insights that contrast successful and failed attempts, outperforming both parametric and non-parametric methods. While other approaches need hundreds of samples and thousands of rollouts, CORE achieves comparable results with as few as five training samples. This is immediately practical for production systems where you have limited training data but need reliable reasoning improvements.

Reflective Prompt Tuning through Language Model Function-Calling

Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour, Estevam Hruschka

Prompt engineering remains frustratingly manual and brittle, but this paper introduces a systematic solution that mimics how human prompt engineers actually work. Reflective Prompt Tuning uses LLM function calling to diagnose failures across entire datasets, identify systematic error patterns, and make targeted prompt edits based on failure history. Instead of random search or single-example fixes, it provides a structured framework for iterative prompt improvement that captures recurring problems.

Prompts are technical debt too

Argues that prompts should be treated with the same engineering discipline as code since they accumulate complexity, dependencies, and maintenance burden over time. This perspective is crucial for teams building production LLM systems where ad-hoc prompt management leads to the same problems as unmanaged code: brittleness, difficult debugging, and hidden interdependencies that slow development.

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

Tsz Ting Chung, Lemao Liu, Mo Yu, Dit-Yan Yeung

This overturns conventional wisdom about many-shot in-context learning for reasoning tasks. While more examples help with simple tasks, reasoning tasks show unstable scaling behavior, and semantic similarity-based retrieval actually hurts performance. The order of examples matters more than previously thought. This has immediate implications for how you structure prompts and manage context in reasoning-heavy production systems.

Introducing the OpenAI Safety Bug Bounty program

OpenAI's new bug bounty program specifically targets AI safety issues including prompt injection, agentic vulnerabilities, and data exfiltration — signaling that these attack vectors are now mainstream security concerns. For production teams, this validates that AI-specific security testing should be part of standard security practices, not an afterthought.