open-source

SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Yibo Peng, Long Lian, David Wagner, Sizhe Chen

If you're building LLM-powered agents that touch external data — emails, web pages, files — prompt injection is your biggest security headache, and existing defenses have been embarrassingly easy to break. This paper identifies *why*: current defensive fine-tuning methods treat an entire model output as uniformly good or bad, so the model never learns exactly which tokens represent a security failure. SecOPD fixes this with token-level feedback during fine-tuning, scoring each output token against what the model would have produced on clean (non-injected) input. The result is dramatic: their defended Qwen3.6-27B drops attack success rates from 94% down to 9% against state-of-the-art adaptive injections, and the security generalizes to agentic tool-calling scenarios the model was never trained on.

Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

sebg

If you've ever wondered what's actually happening inside vLLM when it serves thousands of requests, this deep-dive is the clearest architectural breakdown available. It walks through the full stack — scheduling, paged attention, continuous batching, chunked prefill, prefix caching, speculative decoding, multi-GPU execution, and the serving layer — in a layered way that builds intuition before drowning you in code. Whether you're running vLLM in production, evaluating it against SGLang, or considering contributing to the project, this gives you the mental model you need to reason about performance tradeoffs.

promptfoo/promptfoo

promptfoo

If you're shipping LLM-powered features and not systematically testing them, you're flying blind — promptfoo fixes that. It's an open-source CLI and CI/CD-friendly framework for evaluating prompts, agents, and RAG pipelines across models like GPT, Claude, and Gemini, using simple declarative configs. Beyond standard evals, it includes red teaming and vulnerability scanning capabilities, making it one of the few tools that covers both quality and security in one place. The fact that OpenAI and Anthropic themselves use it is a strong signal this is production-grade tooling worth adopting.

How Far Will They Go? Red-Teaming Online Influence with Large Language Models

Daniel C. Ruiz, Anna Serbina, Ashwin Rao, Emilio Ferrara, Luca Luceri

This research exposes systematic political biases in open-source LLMs and shows how simple jailbreaks can expand their 'Overton Windows' — the range of political opinions they'll express. The findings reveal that most open-source models lean left, smaller models are more politically constrained, and recent models are often more biased than older ones. Essential reading if you're deploying LLMs in contexts where political neutrality matters or if you need to understand the security implications of model political expressivity.

Benchmarking Ollama vs LM Studio vs MLX

A hands-on performance comparison of three popular local LLM inference tools (Ollama, LM Studio, MLX) that investigates why one tool felt laggy in practice. If you're choosing between local inference options or debugging performance issues with self-hosted models, this benchmarking approach shows how to systematically evaluate tools beyond just theoretical specs.

Show HN: Gemma Gem – AI model embedded in a browser – no API keys, no cloud

ikessler

This Chrome extension demonstrates practical browser-based AI deployment by embedding Google's Gemma 4 model locally via WebGPU, complete with webpage interaction capabilities like clicking, typing, and JavaScript execution. It proves that sophisticated AI agents can run entirely client-side without API dependencies, opening new possibilities for privacy-preserving AI tools. The implementation shows how to build truly local AI agents with real-world utility.

Streaming experts

Breakthrough technique allows running massive Mixture-of-Experts models (up to 1 trillion parameters) on consumer hardware by streaming only the necessary expert weights from SSD for each token. This could democratize access to state-of-the-art models for teams without enterprise-scale infrastructure, though with latency tradeoffs.