← AI Glossary

Evals (evaluations)

Methods & visibility

Evaluations, or evals, are test suites that measure the quality of an AI system's answers: a set of representative questions, the expected answers and a scoring method, human or automatic. They are used to compare models, to check that a change has not made things worse and to track quality over time. Without evals, an AI is judged on a few tries and an impression.

In practice at Gensai

To set up evals, Gensai builds a set of real questions with their expected answers, replayed whenever the model, the instructions or the documents change.

When should evals be set up?

From the prototype onwards, with a few dozen real questions and their correct answers, then at every change of model, instructions or documents. The criteria should reflect what matters to the business: a correct answer, a cited source, a refusal when the information is missing.

Common mistake

Relying on public model leaderboards. A model at the top of the benchmarks may do worse on your documents and your questions: only an evaluation on your own cases tells you which one to choose.

Definition written with AI assistance, checked by Sébastien Lecocq, co-founder of Gensai.

Going beyond the definition?

From concept to project: Gensai builds custom, sovereign, GDPR-compliant AI solutions.

Related services: Custom AI solutions

55 boulevard de Strasbourg, 75010 Paris
Information used only to answer the request.