SimpleQA, presented on October 30, 2024, is a benchmark for evaluating factuality and hallucinations in models. Its dataset contains 4,000 human-written factual questions, each with a single indisputable answer.
An autograder assigns model answers to three categories: correct, incorrect, or not attempted. Two annotators verified the reference answers. The published description says engineers can use the benchmark to evaluate factuality with human-written questions and verified answers.
To try this approach through the Rota Nacional API, select a set of benchmark questions and keep their reference answers in an organization-controlled file. Send the questions to a model through the API, without including unnecessary personal data.
Compare each returned answer with its reference and record a label: correct, incorrect, or not attempted. Do not assume Rota Nacional supplies the benchmark or an integrated autograder; this comparison is part of your evaluation.
Check the results by counting each category and manually reviewing a sample of the labels. Consult the original SimpleQA material to verify its scope, method, and reference answers before drawing conclusions.