Summary & Key Takeaways ​

• The article details using DSPy to evaluate and enhance Datasette Agent's SQL system prompts.
• It highlights a specific issue where schema listings without column names led to errors.
• The research task was conducted using Claude Code for web and Claude Fable 5.
• Testing with GPT 4.1 mini and nano identified promising directions for prompt improvements.
• A key finding suggests including column names in the schema or softening "don't call describe_table" advice.

Our Commentary ​

Simon Willison consistently delivers practical, insightful content. This exploration of DSPy for prompt evaluation is exactly what we need as LLM applications mature. The specific example of SQL prompt issues and the proposed solutions are incredibly valuable. It's a reminder that even advanced models need careful engineering to avoid subtle pitfalls. I'm always fascinated by the iterative process of refining these systems.