AI coding agents can solve increasingly difficult problems. But can they keep solving them after the codebase starts fighting back?

In this episode of AI That Works, Dex and Vaibhav dig into Slop Code Bench, a benchmark designed to test something traditional coding benchmarks often miss: what happens when an agent has to keep extending the same codebase without knowing what requirements are coming next. Instead of judging a single solution in isolation, the benchmark adds new challenges over time and measures whether models can make progress without accumulating defects, regressions, and unmaintainable code.

They break down new results across models including Sol, Fable, Kimi, and GPT, looking at strict pass rates, cost, defects, code complexity, and the surprising relationship between spending more and producing better results. The conversation also moves beyond the leaderboard into a bigger question: what would an AI coding agent actually have to prove before engineers could trust it to work without constant human supervision?

TIMESTAMPS
00:00 How Much AI-Generated Code Do You Need to Read?
00:28 Episode Overview: Putting Slop Code Bench to the Test
01:21 Why AI Coding Benchmarks Matter
03:10 AI That Works Unconference Announcement
04:59 How Slop Code Bench Tests Real Software Development
08:09 A Real Example of Requirements Getting More Complex
09:25 Strict Pass Rate vs. Solving the Current Problem
13:13 How Sol, Fable, Kimi, and GPT Actually Performed
15:20 What Happens If AI Gets the Problem Upfront?
18:18 Why Sol's Benchmark Results Don't Match Real-World Experience
20:37 Does Spending More Money Produce Better Code?
21:56 Why AI Planning May Matter Less Now
25:39 Comparing Cost, Defects, and Model Performance
28:05 Measuring the Slop AI Leaves Behind
30:03 Can AI Recognize When Its Architecture Needs to Change?
32:14 Comparing Code Quality Across Models and Harnesses
35:00 The Real Cost of Running AI Coding Benchmarks
37:10 How Should Slop Code Bench Evolve Next?
39:02 What Makes You Trust AI to Work Lights Off?
41:17 Three Ways to Make Coding Benchmarks More Realistic
43:07 How Do You Avoid Disaster When Humans Stop Reading?
44:00 How MCPs and Skills Change Coding Agent Performance
45:38 Should You Delete Agent Instructions for Every New Model?
49:16 When AI Skills Actually Belong in Your Codebase
51:32 Making Claude and Codex Review Each Other's Work

#AIThatWorks #AICoding #CodingAgents #AIEngineering #SoftwareEngineering #SlopCodeBench #CodeQuality #LLMs #DeveloperTools