Evaluations, or evals, are test suites that measure the quality of an AI system's answers: a set of representative questions, the expected answers and a scoring method, human or automatic. They are used to compare models, to check that a change has not made things worse and to track quality over time. Without evals, an AI is judged on a few tries and an impression.
In practice at Gensai
To set up evals, Gensai builds a set of real questions with their expected answers, replayed whenever the model, the instructions or the documents change.
When should evals be set up?
From the prototype onwards, with a few dozen real questions and their correct answers, then at every change of model, instructions or documents. The criteria should reflect what matters to the business: a correct answer, a cited source, a refusal when the information is missing.
Common mistake
Relying on public model leaderboards. A model at the top of the benchmarks may do worse on your documents and your questions: only an evaluation on your own cases tells you which one to choose.
Definition written with AI assistance, checked by Sébastien Lecocq, co-founder of Gensai.
Going beyond the definition?
From concept to project: Gensai builds custom, sovereign, GDPR-compliant AI solutions.
Related services: Custom AI solutions