Creating a Model Evaluation Task

Prerequisites

• You have registered a Huawei ID and enabled Huawei Cloud services, performed real-name authentication, and ensure your account is not frozen or in arrears before using ModelArts. For details, see Signing Up for a HUAWEI ID and Enabling Huawei Cloud Services and Real-Name Authentication Introduction.
• You have configured an agency.

  Certain ModelArts functions require access to services like OBS. Before using ModelArts, ensure your account has been authorized to access these services.

Billing

If you store data in OBS, you will be billed for the storage resources used. For details, see OBS Billing Overview.

Constraints

Currently, only LLM evaluation is supported.

Creating a Rule-based Automated Evaluation Task

To create an automated evaluation task, follow these steps:

1. Log in to the ModelArts console.
2. In the navigation pane on the left, choose Model Evaluation > Evaluation Tasks. In the upper right corner of the Automated Evaluation tab, click Create.

3. On the displayed page, set parameters by referring to Table 1.

   Table 1 Parameters for a rule-based automated evaluation task

   Category	Parameter	Description
   Basic Information	Task Name	Name of an evaluation task. The task name must start with a letter and end with a letter or digit. It can contain 2 to 32 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
   	Description	Description of an evaluation task. This parameter is optional.
   Evaluation Object	Service Type	Currently, only the text generation type is supported.
   	Add Service	Select a model deployed on ModelArts for evaluation. A maximum of 10 models can be evaluated at a time.
   Evaluation Configuration	Evaluation Rule	Select Rule-based: Automatic scoring is performed based on rules. That is, scoring is performed based on similarity or accuracy, and the difference between the model's prediction and the labeled data is compared. It is applicable to standard multiple-choice questions or simple Q&A scenarios.
   	Evaluation Datasets	Preset evaluation set: Use a preset professional dataset for evaluation. Custom evaluation set: Upload an evaluation dataset and select evaluation metrics (F1 score, accuracy, BLEU, and Rouge). Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with .jsonl. It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/example.jsonl or /bucketname/example.jsonl. Upload a single .jsonl file. The file size cannot exceed 10 MB, and the maximum number of records is 1,000. Example: {"case_no":"1","corpus_no":"1","class_level1":"category1","class_level2":"category2","question":"hello","ref_answer":"What can I do for you today?","question_type":"common question type"}
   	Custom Evaluation Metrics	This parameter is required only when Evaluation Datasets is set to Custom evaluation set. You can select evaluation metrics as required. F1 score: The harmonic mean of precision and recall. A higher score indicates a better balance between precision and recall. Accuracy: The percentage of correctly predicted samples (exact matches). A higher score indicates a higher proportion of correct predictions and better model performance. BLEU: A metric used to measure the similarity between machine-translated text and reference text. A higher score indicates a better model machine translation. ROUGE: A set of metrics for evaluating summarization and machine translation by comparing generated output to references. A higher score indicates a better model effect.
   	Evaluation Results Storage	Path for storing the model evaluation result. Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/.

4. Click Create Now.
5. When the status is Completed, you can click Report in the Operation column to view the report and details of the evaluation job on the evaluation report page.

Creating an LLM-based Automated Evaluation Task

To create an automated evaluation task, follow these steps:

1. Log in to the ModelArts console.
2. In the navigation pane on the left, choose Model Evaluation > Evaluation Tasks. Click the automated evaluation tab and click Create.
3. On the Create Auto-Evaluation Task page, set parameters by referring to Table 2.

   Table 2 Parameters for an LLM-based automated evaluation task

   Category	Parameter	Description
   Basic Information	Task Name	Name of an evaluation task. The task name must start with a letter and end with a letter or digit. It can contain 2 to 32 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
   	Description	Description of an evaluation task. This parameter is optional.
   Evaluation Object	Service Type	Currently, only the text generation type is supported.
   	Add Service	Select a model deployed on ModelArts for evaluation. A maximum of 10 models can be evaluated at a time.
   Evaluation Configuration	Evaluation Rule	Select LLM-based.
   	Select Mode	Scoring: The judge model will automatically score inference results based on preset criteria. Comparison: The judge model compares each service against the baseline and outputs results as win, loss, or tie. At least 2 services must be selected for comparison mode.
   	Evaluation Datasets	Preset evaluation set: Use a preset professional dataset for evaluation. Only one preset evaluation set can be added. Custom evaluation set: Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with .jsonl. It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/example.jsonl or /bucketname/example.jsonl. Upload a single .jsonl file. The file size cannot exceed 10 MB, and the maximum number of records is 1,000. Example: {"case_no":"1","corpus_no":"1","class_level1":"category1","class_level2":"category2","question":"hello","ref_answer":"What can I do for you today?","question_type":"common question type"}
   	Evaluation Results Storage	Path for storing the model evaluation result. Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/.
   Judge Settings	Referee Model	Select a model deployed on ModelArts for evaluation.
   	Scoring Rules	You can select a preset or custom scoring prompt template. Preset prompts cannot be modified. To create a custom prompt template, click New in the dialog box on the right of Edit Custom Rule, enter the name, persona, task description, whether to include questions, whether to include reference answers, scoring policy, and evaluation metrics, and click Save.

4. Click Create Now. A single tenant can create a maximum of 2,000 evaluation tasks.
5. When the status is Completed, you can click Report in the Operation column to view the report and details of the evaluation job on the evaluation report page.

Creating a Human Evaluation Task

To create a human evaluation task, follow these steps:

1. Log in to the ModelArts console.
2. In the navigation pane on the left, choose Model Evaluation > Evaluation Tasks. In the upper right corner of the Human Evaluation tab, click Create.

3. On the Create Human Evaluation Task page, set parameters by referring to Table 3.

   Table 3 Parameters for a human evaluation task

   Category	Parameter	Description
   Basic Information	Task Name	Name of an evaluation task. The task name must start with a letter and end with a letter or digit. It can contain 2 to 32 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
   	Description	Description of an evaluation task. This parameter is optional.
   Evaluation Object	Service Type	Currently, only the text generation type is supported.
   	Add Service	Select a model deployed on ModelArts for evaluation. A maximum of 10 models can be evaluated at a time.
   Evaluation Configuration	Evaluation Metrics	You can set up to six custom evaluation metrics and define their standards.
   	Evaluation Datasets	Datasets for evaluation. Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with .jsonl. It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/example.jsonl or /bucketname/example.jsonl. Upload a single .jsonl file. The file size cannot exceed 10 MB, and the maximum number of records is 1,000. Example: {"case_no":"1","corpus_no":"1","class_level1":"category1","class_level2":"category2","question":"hello","ref_answer":"What can I do for you today?","question_type":"common question type"}
   	Blind Test	When enabled, model names are hidden during scoring, and the order of models is mixed up.
   	Evaluators	Only the assigned evaluators can score the task. The evaluation report is generated only after all evaluators have scored all cases.
   	Evaluation Results Storage	Path for storing the model evaluation result. Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/.

4. Click Create Now.
5. When the status changes to Pending evaluation, click Evaluate Online in the Operation column to go to the evaluation page.
6. Complete the evaluation as prompted. Then, click Submit.
   • Click Doubt or Nullify to doubt or nullify a case. To cancel this operation, click Cancel Doubts or Cancel nullify.
   • Score all evaluation metrics of the case and click Save and Next to save the scores and switch to the next case.
   • Click Previous to return to the previous user and rescore.
   • Click the Click to add a note area to add remarks.
   • On the evaluation page, hold down the left mouse button to select the text content to be marked and click Mark to mark the content as key content.
     Figure 1 Human evaluation
     

7. In the navigation pane, choose Evaluation Platform > Evaluation task > Manual evaluation. Click Assessment report in the Operation column to view the model evaluation result.

   After the evaluation is complete, go to the manual evaluation list page and click Human review to review the evaluation. After the review is complete, click Submit to submit the result.