Help Center/ ModelArts/ Model Evaluation/ Model Evaluation/ Viewing the Model Evaluation Report
Updated on 2026-07-02 GMT+08:00
View PDF
Share

Viewing the Model Evaluation Report

After an evaluation task is created, you can view the task report. The procedure is as follows:

  1. Log in to the ModelArts console.
  2. In the navigation pane, choose Model Evaluation > Evaluation Tasks.
  3. Click Report in the Operation column. On the displayed page, you can view the basic information and overview of the evaluation task.

    For details about each evaluation metric, see LLM evaluation metrics.

  4. Export the evaluation report.
    1. In the Service Result Analysis area in the Report tab, click Export, select the evaluation report you want to export, and click OK.
    2. Click Export Record on the right to view the exported task ID. Click Download in the Operation column to download the evaluation report to the local PC.

LLM evaluation metrics

LLMs support automated evaluation and human evaluation. For details about the metrics, see Table 1, Table 2, Table 3, and Table 4.

Table 1 Automated LLM evaluation metrics (not using preset evaluation sets)

Evaluation Metric (Automated Evaluation – Custom Evaluation Set)

Description

ACC

The percentage of correctly predicted samples (exact matches). A higher score indicates a higher proportion of correct predictions and better model performance.

F1 Score

The harmonic mean of precision and recall. A higher score indicates a better balance between precision and recall.

BLEU_1

Matching degree between the sentence generated by the model and the actual sentence at the single-word level. A larger score indicates better model effectiveness.

BLEU_2

Matching degree between the sentence generated by the model and the actual sentence at the phrase level. A larger score indicates better model effectiveness.

BLEU_4

Weighted average accuracy of the model generation result and actual sentences. A larger score indicates better model effectiveness.

ROUGE_1

Recall rate calculated after the model generation result and labeling result are split by 1-gram (n-gram refers to a segment consisting of n consecutive words in a sentence). A larger score indicates better model effectiveness.

ROUGE_2

Recall rate calculated after the model generation result and labeling result are split by 2-gram (n-gram refers to a segment consisting of n consecutive words in a sentence). A larger score indicates better model effectiveness.

ROUGE_L

Recall rate calculated after the model generation result and labeling result are split by longest-gram (longest-gram refers to a segment consisting of n consecutive words in a sentence). A larger score indicates better model effectiveness.
Table 2 Automated LLM evaluation metrics (using preset evaluation sets)

Evaluation Metric (Automated Evaluation – Evaluation Template Used)

Description

Evaluation score

The score of each dataset is the pass rate of the model in the current dataset. If there are multiple datasets in the evaluation capability items, the weighted average pass rate is calculated based on the data volume.

Comprehensive Score

The comprehensive capability is the weighted average of the pass rates of all datasets.
Table 3 Human LLM evaluation metrics

Evaluation Metric (Human Evaluation)

Description

Accuracy

The answer generated by the model is correct and there is no factual error.

average

The model calculates the average score of the generated sentence and the actual sentence based on the evaluation metric.

goodcase

The model calculates the proportion of test cases whose score is 5 after the generated sentence and the actual sentence are compared based on the evaluation metric.

badcase

The model calculates the proportion of test cases whose score is less than 1 after the generated sentence and the actual sentence are compared based on the evaluation metric.

Custom metrics

Custom metrics, such as usability, logic, and security.
Table 4 Automated LLM evaluation metrics

Model Type

Evaluation Metric (Automated Evaluation – Rule-based – LLM-based)

Description

LLM

Score given by the judge model

Score given by the judge model for each case in the dataset.

Average

Average score of all test cases in the dataset.

Mid

Median score of all test cases in the dataset.

Standard deviation

Standard deviation of all test case scores in the dataset.

win

Number of models whose performance metrics (the definition of "good" needs to be specified in advance, for example, high accuracy rate or low error rate) are better than those of the benchmark model among all comparison models.

lose

Number of comparison models whose performance metrics are worse than those of the benchmark model.

tie

Number of comparison models whose performance metrics are the same as those of the benchmark model.

quantile

(win+tie)/(lose+tie)

Quantile (excluding tie_bad)

Exclude the tie_bad score: (win + tie_good)/(lose + tie_good)

Quantile (excluding tie_good)

Exclude the tie_good score: (win + tie_bad)/(lose + tie_bad)
Parent Topic: Model Evaluation