Preset Evaluation Sets and Templates

MMLU-Pro

MMLU is a key benchmark for testing LLMs. It includes 57 tasks, such as math, computer science, law, and history, to evaluate the models' world knowledge and problem-solving skills.

GPQA_Diamond

GPQA_Diamond is a multiple-choice question set created and checked by experts in biology, physics, and chemistry. It has 448 very hard questions. The set tests how well AI systems handle questions that mix different subjects, especially when they are outside the system's main area of expertise.

BoolQ

BoolQ is a set of "yes/no" questions. These questions come from real-world queries and have no specific guidelines, making them complex and varied.

AGIEval

The AGIEval benchmark features high-quality tests like the LSAT, college entrance exams (including China's and the US SAT), math competitions, and bar exams. The dataset only includes objective questions, such as multiple-choice and fill-in-the-blank. These tests measure a model's cognitive abilities, knowledge, and reasoning skills using official standards.

C-Eval

C-Eval is a Chinese dataset with 52 subjects and four difficulty levels. It tests how well LLMs understand Chinese.

GSM8K

GSM8K is a benchmark from OpenAI that tests how well LLMs can reason with data. It includes 8,500 elementary-level math problems to assess these models' math skills.

MathBench

It evaluates the math skills of LLMs, covering both theory and problem-solving.

ARC Challenge

ARC Challenge is a dataset for testing logical reasoning and problem-solving. It covers questions from different fields to assess advanced reasoning skills.

BBH

BBH is a big dataset with 204 tasks that test LLMs in areas like linguistics, child development, common sense, social bias, and software development. It checks how well these models tackle tough tasks.

CMMLU

CMMLU is the Chinese version of MMLU, covering areas like humanities, law, engineering, and math. It tests how well models know these subjects in Chinese.

OpenFinData

OpenFinData is an open-source financial evaluation dataset jointly released by EastMoney.com and Shanghai AI Lab. This dataset represents the most realistic industrial scenario needs and is currently the most comprehensive and professional financial evaluation dataset. It provides high-quality data resources for researchers and developers in the field of financial technology based on the diverse financial services of EastMoney.com.

FinEval

FinEval, a financial industry evaluation benchmark, is grounded in quantitative methodologies. Through long-term objective research and rigorous manual screening, it features over 26,000 diverse test items highly aligned with real-world application scenarios, including multiple-choice questions, subjective/objective short-answer questions, reasoning and planning tasks, and retrieval-augmented QA. Covering financial academic knowledge, financial industry expertise, financial safety regulations, financial agents, financial multimodality, and financial rigor, FinEval is designed to comprehensively evaluate the all-around application capabilities of LLMs within the financial sector.

MedMCQA

This large-scale MCQA dataset helps answer real-world medical school entrance exam questions.

PubMedQA

PubMedQA is a novel biomedical QA dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with "yes/no/maybe" using the corresponding abstracts (e.g., Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?). Each PubMedQA instance consists of four components: (1) a question, which is either an existing research article title or derived from it; (2) a context, which is the corresponding abstract without the conclusion; (3) a long answer, which is the conclusion of the abstract that typically answers the research question; and (4) a "yes/no/maybe" answer that summarizes the conclusion. PubMedQA is the first QA dataset that requires reasoning over biomedical research texts, especially their quantitative contents, to answer questions.