Imagine asking someone to inspect a building's safety, but only letting them look through the front door. They might spot a problem, but they cannot check the wiring. AI models pose a similar problem: outside researchers can test public behavior, while the company sees more of the training, safeguards, and incidents. On October 15, Anthropic said it had given a small team from the nonprofit METR deeper access to evaluate its systems.
What can the evaluators see?
Anthropic says six METR employees have company hardware, office space, and access to internal tools roughly comparable to its own risk teams. Some material remains off limits, including customer data, privileged legal information, and model architecture. METR is expected to publish assessments of risks, incidents, and practices. Anthropic says it cannot veto those reports, though it can require redactions in agreed categories and decides what falls into them.
Why is this different?
A public benchmark is a defined test, often focused on a model's answers. An independent evaluation can also examine how a company measures risk, responds to unexpected behavior, and applies safeguards. Better access may lead to more informative public findings. The evaluators still need to exercise their own judgment; access alone does not prove their conclusions or make a model safe.
The arrangement is voluntary and involves a small team. Anthropic describes other evaluator partnerships at earlier stages with narrower access or publication rights. At the time of the announcement, the promised METR assessments were future work, not a completed independent verdict. This is worth watching because the useful evidence will be in what the evaluators eventually publish, including any limits they report.
For a concrete example of why outside scrutiny matters, read about measuring AI work inside a lab or reports of unexpected model behavior.