In plain words
Evals is short for evaluations: tests that give an AI system a task and judge its response or actions against defined success criteria.
A closer look
An eval combines a test case with a way to score the result. Grading can use code, human review, another model, or a combination. Useful test sets include realistic tasks and difficult cases.
Teams rerun evals when changing prompts, models, or tools to see what improved and what broke. Repeated trials help account for variation between runs. For agents, checking the actual outcome matters as well as reading the reply.
In practice
Before updating an AI writing assistant, a designer tests sample requests for shorter button labels. Each result is checked for length, preserved meaning, and tone, then compared with the previous version.
A useful distinction
Evals is shorthand for evaluation, not a separate technology. Passing a test set does not guarantee success everywhere; results depend on the cases and the quality of the grading.