Edition

Agent Security Cracks, Evals Need Rethinking

August 10, 2026 · 12 papers

This week's edition is heavy on two converging themes: the security fragility of AI agents in production, and the growing evidence that our benchmarks and evaluation frameworks are failing to keep pace with how agents are actually built and deployed. On the security side, you'll find a real active exploit against Atlassian Rovo, empirical data showing humans miss one-third of malicious agent commands, and a cryptographer's take on Anthropic's cryptanalysis results. On the evals side, multiple papers argue that benchmark scores for coding agents are misleading, tool-calling defaults are underperforming alternatives, and we lack even a shared vocabulary for attributing agent failures — with a new 41-failure-mode taxonomy aiming to fix that last problem.

01 · security Accessible

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

Wirbelwind

If you're building AI agents and relying on humans to catch dangerous commands, this data should worry you. Across 40,000 game runs simulating a human-in-the-loop for an AI coding agent, players missed 1 in 3 malicious commands — and this was a game where ~34% of commands were threats, far higher than real-world rates that would induce complacency. The data makes a strong case that 'human approval' is not the security guarantee most teams assume it is, especially under time pressure and alert fatigue.

Takeaways3

Human reviewers missed 33% of threats on average, and only 20.8% of players caught all threats without over-blocking safe commands.
Time pressure and high command volume are the enemy of meaningful human oversight — the 'human-in-the-loop' is a much weaker control than it appears.
7% of players approved every single command, suggesting a non-trivial portion of real users will rubber-stamp agent actions entirely.

338 points on HN · via api-hn

02 · agents Intermediate

The Bitter Lesson of Tool Calling

Ishan Patel

If you're designing tool-calling pipelines for LLM agents, you may be leaving performance on the table by defaulting to JSON-based tool calls. This paper benchmarks programmatic tool calling (PTC) — where models invoke tools by writing typed Python code rather than emitting JSON — against native JSON tool calling across 14 models, and finds PTC matches or beats JSON in 11 of 14 cases. Critically, PTC holds up better under parallel fan-out and context degradation scenarios that are common in real agentic workloads.

Takeaways3

Letting models call tools via Python code rather than JSON improves or matches performance in the vast majority of tested models, with up to 10.6% gains on GPT-class models.
PTC is significantly more robust under parallel tool calls and long-context 'context rot' conditions where JSON calling degrades noticeably.
Performance of PTC scales with underlying model capability, meaning this approach will likely get better as models improve.

agents llms evaluations software-engineering

via api-arxiv · arXiv:2608.06370

03 · security Accessible

Atlassian Rovo Exfiltrates Data, Bypassing Controls

spavel.bsky.social

This is a concrete, real-world prompt injection attack against Atlassian's Rovo AI agent that results in full data exfiltration — Jira tickets, Confluence docs — with no human approval required. The attack exploits a URL retrieval tool that persists even when web search is disabled, meaning Atlassian's own access controls don't actually close the attack surface. What makes this especially important for practitioners is that Atlassian was notified over two months ago and has not patched it, so this is an active risk for any organization using Rovo today.

Takeaways3

Indirect prompt injection via uploaded files can silently hijack an AI agent's actions and exfiltrate sensitive organizational data without any user interaction beyond the initial query.
Disabling a feature (web search) does not guarantee the underlying tool is removed — always verify what tools your agent actually has access to at runtime.
Atlassian has not patched this vulnerability after 2+ months, so organizations using Rovo should treat it as an unmitigated risk right now.

security agents

via api-bluesky

04 · evaluations Intermediate

Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

mgorinova.bsky.social

This position paper challenges the industry's reliance on coding benchmarks like SWE-bench to compare AI coding agents, arguing they were designed for a pre-agent world and are now actively misleading. The core insight is that a coding agent is a *system* — model, harness, context, environment, feedback loops — and collapsing all of that into a single end-to-end score makes it impossible to know what's actually driving performance differences. If you're using benchmark scores to make build-vs-buy or model-selection decisions for agentic coding tools, this is essential reading.

Takeaways3

Benchmark scores conflate model quality with harness quality, meaning two agents with the same score may have completely different underlying strengths and weaknesses.
Grading against a single reference solution systematically penalizes valid alternative implementations, distorting comparisons between agents.
Harness and environment choices can move benchmark scores by margins comparable to jumping an entire model generation, making leaderboard comparisons unreliable.

evaluations software-engineering agents opinion

via api-bluesky

05 · agents Intermediate

LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks

Ziyu Ma, Hailang Huang, Shun Zou, Yong Wang, Shidong Yang, Yiming Hu, Fei Wei, XiangXiang Chu

Long-horizon agentic tasks — the kind where an agent must plan, execute, and self-correct across many interdependent steps — break down largely because agents lose track of state and let early mistakes corrupt later decisions. LongHorizon-Harness addresses this by externalizing task state and introducing a three-role loop (manager, executor, auditor) where each role operates with a fresh context and independently verified facts. The benchmark gains are substantial and consistent across multiple models and domains, suggesting this is a structural fix rather than a model-specific trick.

Takeaways3

Keeping task state outside the execution context and verifying it independently before each step dramatically reduces error propagation in long-horizon tasks.
The Manage-Execute-Audit loop — separating planning, doing, and verifying into distinct roles with fresh contexts — is a reusable architectural pattern for any long-horizon agent system.
Gains were consistent across models (Qwen, Claude) and domains (terminal, OS, web), suggesting this harness design addresses a fundamental limitation rather than overfitting to one benchmark.

agents software-engineering reasoning

164 upvotes on HF · via api-hf · arXiv:2608.01964

06 · agents Intermediate

Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

Harsh Raj, Vipul Gupta, Anas Mahmoud, Razvan-Gabriel Dumitru, Darvin Yi, Aakash Sabharwal, Yunzhong He

When an AI agent fails, the hardest question isn't *what* went wrong — it's *who* should fix it. This paper tackles that repair-assignment problem head-on by introducing a taxonomy of 41 failure modes organized around the interactions between components (model, harness, tools, memory, environment, user) rather than just the end outcome. Each failure mode is pinned to a specific edge in the system and a fault side, so you know whether to reach for post-training, scaffolding changes, or benchmark redesign. If you're building or evaluating agents and tired of vague 'the agent failed' labels, this gives you a shared vocabulary and a structured debugging framework.

Takeaways3

Outcome-level failure labels are too coarse to drive improvements — you need to localize failures to the specific component interaction that caused them.
The taxonomy's 41 failure modes are organized by component edge and fault side, making it directly actionable for deciding between model fine-tuning, harness fixes, or eval redesign.
The framework is architecture-agnostic and applies across coding assistants, long-horizon agents, and multi-agent systems.

agents evaluations software-engineering how-we-work

10 upvotes on HF · via api-hf · arXiv:2607.28802

07 · agents Intermediate

Prime Agent: A self-improving RLM agent

Xeophon

Prime Agent is a bold bet that modern harness designs are already obsolete — built for weaker models and now actively getting in the way of frontier ones. The system introduces two core abstractions: a Recursive Language Model (RLM) that treats context as a variable and sub-agent calls as function calls inside a persistent REPL, and a Continual Harness that lets the agent modify its own prompts, skills, memory, and sub-agents at runtime. The result is an agent that can adapt its own scaffolding as it learns, rather than being locked into static, hand-engineered structures. Essential reading if you're designing agent harnesses and wondering whether your architecture is already a ceiling rather than a floor.

Takeaways3

Static, design-time harnesses are a bottleneck — the harness itself should be a mutable artifact the agent can update as it runs.
Treating context as a variable in a persistent REPL lets agents handle arbitrarily long sessions without losing access to prior state.
Sub-agent delegation modeled as function calls enables composable, programmatic reasoning patterns that fixed tool-calling schemas can't support.

agents software-engineering llms

253 points on HN · via api-hn

08 · evaluations Accessible

How do programming languages impact token efficiency and correctness?

ndr

This post challenges the growing assumption — now being laundered into AI search results as fact — that dynamic languages are significantly more token-efficient for LLMs than static ones. The author digs into the original cited research and finds the methodology shaky, while also surfacing a genuinely interesting outlier: array languages like J can be dramatically more token-efficient than either camp. For engineers making language choices in LLM-heavy workflows or thinking about how token costs might shape future language design, this is a useful corrective against cargo-culting a single benchmark.

Takeaways3

The widely cited claim that dynamic languages have 2-3x lower token cost than static languages is based on thin, potentially flawed benchmarking.
Array languages (like J) can be far more token-efficient than either dynamic or static mainstream languages, though at the cost of readability.
Token efficiency is an emerging and underexplored axis for evaluating programming language choices in LLM-assisted development contexts.

evaluations software-engineering how-we-work llms

via api-lobsters

09 · llms Intermediate

Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

sebg

If you've ever wondered what's actually happening inside vLLM when it serves thousands of requests, this deep-dive is the clearest architectural breakdown available. It walks through the full stack — scheduling, paged attention, continuous batching, chunked prefill, prefix caching, speculative decoding, multi-GPU execution, and the serving layer — in a layered way that builds intuition before drowning you in code. Whether you're running vLLM in production, evaluating it against SGLang, or considering contributing to the project, this gives you the mental model you need to reason about performance tradeoffs.

Takeaways3

Paged attention and continuous batching are the core innovations that make vLLM's throughput competitive — understanding them is prerequisite to tuning the system.
Features like chunked prefill and prefix caching have significant latency and cost implications that aren't obvious without understanding the scheduler's behavior.
The V1 engine represents a substantial architectural evolution from V0, and understanding the progression clarifies why certain design decisions were made.

llms open-source foundational

149 points on HN · via api-hn

10 · opinion Intermediate

Some thoughts about Anthropic’s new cryptanalysis results

Cryptographer Matthew Green weighs in on Anthropic's cryptanalysis results, bringing a rare outside perspective from someone who actually understands both the cryptography and the AI context. The post is worth reading because it cuts through hype in both directions — neither dismissing the results nor overstating them — and grounds the discussion in what these findings actually mean for the security properties of AI systems. For engineers building systems that depend on LLM security guarantees, Green's analysis is a useful sanity check from a credible domain expert.

Takeaways3

Cryptographic claims about AI systems deserve the same rigorous scrutiny applied to traditional cryptographic protocols — enthusiasm from AI labs isn't a substitute.
An expert outside the AI bubble can often spot where security arguments are sound versus where they rely on hand-waving.
The intersection of cryptography and LLM capabilities is an emerging area where practitioners should be skeptical of both overclaiming and underclaiming.

opinion security llms

via suggestion

11 · evaluations Accessible

promptfoo/promptfoo

promptfoo

If you're shipping LLM-powered features and not systematically testing them, you're flying blind — promptfoo fixes that. It's an open-source CLI and CI/CD-friendly framework for evaluating prompts, agents, and RAG pipelines across models like GPT, Claude, and Gemini, using simple declarative configs. Beyond standard evals, it includes red teaming and vulnerability scanning capabilities, making it one of the few tools that covers both quality and security in one place. The fact that OpenAI and Anthropic themselves use it is a strong signal this is production-grade tooling worth adopting.

Takeaways3

Declarative, CI/CD-integrated eval configs make it easy to catch prompt regressions before they hit production.
Built-in red teaming and pentesting features let you probe AI systems for security vulnerabilities, not just output quality.
Supports side-by-side benchmarking across major model providers, making model migration decisions data-driven.

evaluations security open-source

24103 stars · via api-github

12 · llms Accessible

https://openai.com/index/ten-advances-in-mathematics/

OpenAI's blog post outlines ten significant mathematical advances their models have contributed to, signaling that LLMs are moving from 'useful assistant' to genuine research collaborator in formal reasoning domains. For engineers building reasoning-heavy systems, this is a meaningful benchmark of where frontier models actually stand on hard, verifiable problems. It challenges the conventional wisdom that LLMs are fundamentally pattern-matchers that can't do novel reasoning — at least at the frontier level.

Takeaways3

Frontier LLMs are now contributing to unsolved or cutting-edge mathematical problems, not just solving textbook exercises.
Verifiable domains like mathematics are becoming a key proving ground for evaluating true reasoning capability vs. memorization.
This raises the bar for what 'reasoning' benchmarks should look like when evaluating models for complex problem-solving tasks.

llms reasoning

via suggestion