Amazon let AVERI grade its safety tests, not its models

Outside reviews of AI models usually look at what the model says. In a post on Amazon Science, Rahul Gupta and Jwala Dhamala describe a different setup. Amazon gave AVERI, an independent nonprofit led by Miles Brundage, a random sample of prompts from its internal safety evaluation suite. AVERI's job was to judge whether the tests themselves were any good. It reviewed the sample without input from Amazon's evaluation team. It credited the suite's topical coverage and original prompts, which lowers the risk that test content leaked into training data. It also asked for more multistep adversarial prompts and multilingual coverage spread evenly across risk categories. Amazon says it has since refreshed the test set, added tiered difficulty levels, and set a quarterly review cycle.

The most useful finding is the plainest one: some models refuse because of surface cues in a prompt. A pass on those tests shows the model spotted an alarming phrase, not that it will behave safely in a situation it hasn't seen. Any red-team suite built from obvious requests has this weakness, including the ones companies write to check their own deployments. Stale prompts compound it, because the score keeps looking healthy while the test drifts away from how people misuse these systems.

The account is Amazon's own, with no sample size and no count of flagged prompts, so the scale of the fixes is hard to judge. The practice still transfers. For anyone buying or deploying a model, a safety score means little until someone outside the team that wrote the test has checked whether the test can fail.