Full source code: https://github.com/ai-that-works/ai-that-works/blob/main/2025-09-23-evals-for-classification

In this episode of AI That Works, hosts Vaibhav Gupta and Dex, along with guest Kevin Gregory, explore the intricacies of building AI systems that are ready for production. They discuss the concept of dynamic UIs, the challenges of large-scale classification, and the importance of user experience in AI applications. The conversation delves into the use of LLMs for enhancing classification systems, the evaluation and tuning of these systems, and the subjective nature of what constitutes a 'correct' classification. The episode emphasizes the need for engineers to focus on accuracy and user experience while navigating the complexities of AI engineering. The speakers also discuss model upgrades, user feedback, and the importance of building effective user interfaces, emphasizing iterative development and rapid prototyping for chatbot performance evaluation.

Chapters
00:00 Introduction to AI That Works
02:55 Dynamic UIs and Their Importance
05:57 Large Scale Classification Challenges
08:41 Building a Robust Classification System
11:38 Evaluating and Tuning the Classification Pipeline
14:29 Understanding Accuracy vs. Cost in AI
17:29 User Experience and Subjectivity in Classification
20:30 Final Thoughts and Future Directions
26:14 User Experience and Product Design
27:35 Error Analysis and Problem Definition
29:22 Prompt Engineering and Model Upgrades
34:00 Data Collection and Iteration
35:42 UI Development and Visualization Challenges
42:40 Understanding Data Quality and Test Cases
44:02 Building the UI: Time Estimates and Challenges
47:34 The Importance of Iteration in Development
48:09 Choosing the Right Tools for UI Development
50:09 Vibe Coding: A New Approach to Prototyping
53:56 Evaluating Model Performance: Insights and Adjustments
58:53 Dynamic Evaluation Systems for Chatbots
01:06:51 The Trade-offs of Using LLMs vs. Deterministic Code