Code:
https://github.com/ai-that-works/ai-that-works/tree/main/2025-10-07-anthropic-post-mortem
Vaibhav Gupta and Aaron (co-founder, former AWS EC2/Prime Video engineer) dissect Anthropic's detailed post-mortem of three critical bugs that affected their production systems. They explore the technical intricacies of how models select tokens across distributed GPUs, why longer context windows can degrade performance, and how compiler optimizations mixing 16-bit and 32-bit floating-point math led to incorrect token selection. The discussion extends to practical lessons for AI engineers: building observability into AI systems, using "vibe checks" from social media for anomaly detection, and the critical importance of rollback strategies. They also analyze OpenAI's new Agent Builder and the broader trend of visual workflow tools for non-technical users.
Chapters
Introduction and Technical Difficulties
Anthropic's Recent Downtime and Its Implications
Understanding Context Window Routing Issues
Output Corruption and Performance Optimization Bugs
Floating Point Precision and Model Output Challenges
Evaluating AI Systems and Debugging Strategies
Deployment Best Practices for Startups
Understanding Failures vs. Hallucinations in AI
The Importance of Continuous Evaluation in AI
The Rise of Agent Builders in AI
Navigating the Future of AI Integrations
Transcript
I've watched this one closely. Ask me anything about it and I'll point you to the exact moment.