Summary & Key Takeaways ​

• Simon Willison released smevals, a new tool for evaluating LLMs, prompts, and harnesses.
• It allows running small eval suites against different model configurations.
• The tool separates running evaluations from grading operations.
• Results can be explored via a localhost web server or built as static HTML reports.
• smevals aims to provide a streamlined approach to LLM evaluation.

Our Commentary ​

Evaluation is arguably the hardest part of building reliable LLM applications. Simon Willison's smevals looks like a pragmatic, developer-friendly approach to tackle this. The focus on small, configurable suites and clear separation of runs and grades is smart. I'm genuinely excited to see this evolve; it feels like a tool built from real-world pain points, which is always a good sign.