A significant and increasing portion of code contributions today come from LLMs. With the announcement of initiatives such as Anthropic's Project Glasswing and OpenAI's Project Daybreak, security patches are no exception to this trend. Today, both AI-assisted and fully AI-generated patches are making their way into codebases everywhere, with as-yet-unknown long term consequences.

Our talk presents research on the efficacy of frontier LLMs at patching novel vulnerabilities based on realistic bug reports. How many LLM-generated patches are actually FLAWED—Fix-Like Artifacts with Embedded Defects? It turns out, about three-quarters of them! We gauge success based on a model’s rate of fully mitigating the original vulnerability, as well as whether an LLM erroneously modifies application behavior and even introducing new vulnerabilities. We will showcase FLAWED, our recently-published testing toolchain that allows for evaluating arbitrary models’ patching capabilities at scale. FLAWED ingests vulnerability descriptions and runs models through a gauntlet of varying prompts, execution modes, and environmental constraints over numerous iterations, tasking different LLMs to fix each vulnerability and evaluating their performance. We will present our findings from a dataset of over 6,000 patching runs produced through FLAWED. We will also demonstrate a variety of real-world case studies in which AI-generated patches introduced vulnerabilities to major open-source projects, including the Linux kernel.

Our data strongly suggests that the adoption of highly-automated, LLM-based patching pipelines should be treated with serious caution, and FLAWED gives organizations a way to quantify the risk inherent in doing so.