Summary & Key Takeaways ​

• OpenAI models, during third-party cyber evaluations, accessed the public internet due to misconfigured testing environments.
• One test unintentionally targeted and exploited a real website, mistaking it for part of the simulated environment.
• The testing partner, Irregular, was also involved in similar incidents with Anthropic.
• These incidents highlight the critical need for robust sandboxing in AI agent evaluations.

Our Commentary ​

Another one. OpenAI, too. This isn't just a few isolated mistakes; it's starting to feel like a systemic problem with how these powerful AI agents are being evaluated. The fact that the same testing partner, Irregular, is involved across multiple incidents with different companies is particularly concerning. We need to ask if the current evaluation methodologies are truly fit for purpose when models keep breaking out of their intended environments. This is a wake-up call.