
What an AI evaluation actually tested
NIST risk-management language: ask version, task and conditions — and what was out of scope.
Published

“Model X scored highly” is incomplete until someone says which version, which tasks, which datasets, who ran the test, and what failure modes sat outside the exam. NIST’s AI risk-management language treats evaluation as contextual: use and testing conditions define what a score can mean. A leaderboard row is not a universal certificate.
Questions that keep claims honest
Which model version and date were evaluated? Which tasks and datasets were in scope? Who commissioned and who performed the test? Were results audited or only self-reported? What harms, languages, domains or edge cases were out of scope?
Without those answers, marketing and measurement blur. An evaluation can be rigorous for coding benchmarks and silent on biosafety, or strong on English tasks and weak elsewhere. The gap is not a scandal by itself; hiding the gap is the reporting failure. Readers should also ask whether the evaluation was red-teamed, whether adversarial prompts were in scope, and whether the published summary omits failed tasks.
Scope
This explainer is not a scorecard for any named commercial model and not a claim that any vendor faked results. It is a method note aligned with risk-management practice: ask what was tested, under what conditions, and what remains untested before treating an AI evaluation as a general verdict.
Why RTN publishes method notes
Breaking news moves fast; definitions slip faster. These explainers exist so later stories can link a stable reading rule instead of re-arguing vocabulary under deadline pressure. They are not a substitute for opening the underlying report, guidance page or statute when a new claim appears.
Sources and limits
This RealTeaNews method piece is grounded in the primary links listed in the frontmatter. It does not invent unpublished statistics or unpublished decisions. Where a figure or outcome would require a document RTN has not opened, the article says so. Readers should refresh the linked official pages before relying on any single number in later commentary. Editorial practice on RealTeaNews is to prefer primary texts over paraphrase when the two conflict.
Production disclosure: AI-written for RealTeaNews house style; human-edited by @tonyscakes8g. Not investment, legal or medical advice.
Reading habit
When a new claim appears, open the primary page linked in the sources list, note the date on that page, and ask whether the claim still matches the document. If the document has moved or been superseded, update the citation rather than repeating yesterday’s paraphrase. That single habit prevents most method failures in fast news cycles.
Sources
- NIST, NIST. Accessed 9 November 2026.
Sources link to external sites. Their content is the responsibility of their publishers.