GitHub reports that Copilot writes over 46% of code in the files where it is enabled. So I asked an AI to write an XML parser five times with the same exact prompt. Four times it hardened the parser against XXE attacks. One time it shipped a critical vulnerability. That's Security Roulette -- and it's the default state of every AI coding assistant in production today.

We tested 27 AI code generators on 730 prompts written like real developer requests -- never mentioning security. The median configuration scored 57.9%. The best of anything we tested -- a thin security-tuned wrapper over GPT -- scored 83.8%, which inverted means even the best AI we measured ships exploitable code in 1 out of every 8 cases. The failure distribution is brutally non-uniform: authentication, rate limiting, cryptography, infrastructure-as-code, mobile security, and CI/CD pipelines fail at rates between 70% and 99%. And running the same prompt through the same model multiple times -- full corpus, 56,629 scored files across 20 models x 5 runs x 730 prompts at temperature 1.0 -- code varies 72% of the time, security postures vary 30.4% of the time, and 13.5% of prompts show >=90pp extreme variation between runs. Same model, same prompt, same temperature -- sometimes secure, sometimes a working exploit.

The talk pairs the statistics with live demos. We crack hardcoded JWT secrets in AI-generated auth code in 0.24 seconds using jwt_tool against the SecLists wordlist that ships with Kali Linux. We exploit AI-generated XML parsers, SQL queries, and reentrancy bugs -- every demo runs verbatim AI output through a working exploit harness, drawn from over an over 30-demo collection in demos/ in the repo.

The climax is a Solidity reentrancy contract written by Claude Opus 4.6 that documents its own vulnerability in its own docstring -- "WARNING: This contract contains a known reentrancy vulnerability for educational purposes" -- and ships the bug anyway. The fix is two extra lines: one
import (@openzeppelin/contracts/security/ReentrancyGuard), one modifier (nonReentrant). The AI's exact line ordering becomes safe. The model knew. The fix was trivial. It didn't reach for it. We run the live exploit: 1 ETH stake, 31 ETH stolen, 30 honest users left with phantom balances on a drained contract.

The compounding demo follows: two unrelated AI files (an OAuth client storing tokens in localStorage, a React component using dangerouslySetInnerHTML) chain into a working token-theft primitive. Neither prompt asked the AI to think about the other. The vulnerability
is the composition -- and per-prompt failure rates understate real risk because bugs compound across files.

The talk closes with the wrapper-engineering finding (a +24.3 percentage-point delta from a thin security-tuned wrapper over the same
underlying model) and a defenses framework that survives the math: at the best wrapper-tuned configuration, a 1,000-developer enterprise still
ships ~1,780 insecure suggestions per day. That volume is unreviewable by the developers writing the code, which is why the headline
recommendation is expert security review of the residual after automation, plus red-teaming your own SAST pipeline against this benchmark to find the categories your scanner silently misses. The truncation caveat is stated directly: roughly 30% of codex-app outputs in both conditions are incomplete generations; the +24.3 pp gap survives because both conditions truncate at the same rate, but the absolute 83.8% should not be read as a coverage guarantee.

Every prompt, artifact, score, and demo harness is in the public repository -- github.com/miroku0000/AI-Security-Benchmark -- and independently verifiable.