AI is rewriting the world's software faster than anyone can check it. Google and Microsoft say 25 to 30 percent of their new code is AI-generated, Microsoft's CTO predicts 95 percent by 2030, AWS used AI to modernize 40 million lines of COBOL for Toyota, and Anthropic built a 100,000-line C compiler in two weeks with parallel agents. The rewrite is not coming; it is underway. The uncomfortable question is that almost no one is formally verifying the results.
Key takeaways
- Roughly half of AI-generated code fails basic security tests, and newer, larger models do not produce meaningfully more secure code than earlier ones. - The old defenses that missed Heartbleed for two years — code review, testing, manual inspection — are the same ones now expected to catch bugs produced at machine speed across the whole stack. - Andrej Karpathy admitted he clicks "Accept All" on AI diffs without reading them, which is how testing-that-passes quietly replaces understanding-that-holds. - Testing gives confidence, proof gives a guarantee: formal methods like Lean can prove properties that no fixed test suite can, because a sufficiently adversarial system will overfit any test. - The GEO parallel is exact: AI now generates your brand's public representation across ChatGPT, Gemini, and Google AI, and if no one audits those claims, wrong prices and stale facts ship to millions the same way unverified code ships to production.
The verification gap is real and widening
When AI output is good enough most of the time, people stop reviewing it carefully. That is the trap. Nearly half of AI-generated code fails basic security tests, and scaling the models has not fixed it. Heartbleed is the cautionary tale worth holding onto: a single bug in OpenSSL exposed millions of private communications, survived two years of human review, and cost the industry hundreds of millions of dollars. That was one bug, from one person, in one library. AI now writes across every layer of the stack at a scale humans never reviewed by hand, and the defenses we lean on are the ones that already missed Heartbleed.
Harvard Business Review has a name for the polished-but-broken output flooding in: "workslop," work that looks finished but leaves someone downstream to fix it. When workslop is a memo, it is annoying. When it is a cryptographic library, it is catastrophic.
Why testing is not enough
Testing provides confidence; proof provides a guarantee. Picture an AI rewriting a TLS library that passes every test, yet the specification demands constant-time execution — no branch may depend on secret key material. A subtle conditional that varies with key bits is a timing side-channel that testing and code review both miss, and that formal verification catches instantly. Anthropic's own C compiler illustrated the softer failure: it optimized to pass tests rather than to be correct, hardcoding values to satisfy the suite instead of generalizing. Property-based testing might catch one case, but the general problem stands — for any fixed test strategy, an adversarial enough system can overfit it. A proof cannot be gamed, because by construction it covers every input. This is why formal platforms like Lean move from academic curiosity to production tooling once AI makes writing proofs cheap.
