Prerequisites

• Ensure that the account is not in arrears because resources are consumed when evaluation jobs are running.
• The model evaluation code has been compiled and uploaded to the OBS directory. The OBS directory you use and ModelArts are in the same region.
• The boot file of the model evaluation code is in .py format.
• You have prepared required data sources. A dataset of the image classification or object detection type has been created on ModelArts, and the dataset has been published. Or, you have uploaded the data used for image classification, object detection, or semantic segmentation to the OBS path.
• For details about how to compile the model evaluation code, see Model Evaluation API and Sample Code for Model Evaluation.

Creating a Job

1. Log in to the ModelArts management console and choose Model Management > Evaluation/Diagnosis from the left navigation pane.
2. On the Evaluation/Diagnosis page, click Create in the upper left corner. The Create Model Evaluation Job page is displayed.
3. On the Create Model Evaluation Job page, enter basic project information, job information, and resource pool information.
   1. Set basic information about the job.

      Table 1 Basic information

      Parameter	Description
      Billing Mode	The default value is Pay-per-use and cannot be changed.
      Name	Job name, which is used to distinguish jobs.
      Version	It is automatically generated by the system and named in the format of V0001 or V0002. It cannot be modified by users.
      Description	Brief description of the project.

      Figure 1 Basic information
      
   2. Set the job parameters.

      Table 2 Description of model evaluation job parameters

      Parameter	Description
      Model Source	The available model sources are as follows: Model Management: Select an available model and its version from the ModelArts model management list. Models that have been successfully imported under the current account are displayed in the drop-down list. Model Storage Location: Upload a model to OBS and select the model for evaluation from the corresponding OBS path.
      Job Type	Available values are Image Classification, Object Detection, and Semantic Segmentation. Select an evaluation type based on the application of your model.
      AI Engine	Select the AI engine used by the model evaluation job from the drop-down list on the right. TensorFlow|TF-1.13.1-python3.6, PyTorch | PyTorch-1.4.0-python3.6, and TensorFlow | TF-2.1.0-python3.6 are supported.
      Evaluation Code Directory	Path for storing the model evaluation code. The path is an OBS directory. You are advised to upload the model evaluation code to the OBS bucket before creating a job.
      Boot File	After the directory for storing the model evaluation code is set, select the boot file of the model to be evaluated in the directory. The boot file must be in .py format.
      Data Source	There are two types of data sources: Data Management and Data Storage Location. When Job Type is set to Image Classification or Object Detection, you can select data from Data Management or Data Storage Location. When Job Type is set to Semantic Segmentation, you can select data only from Data Storage Location. Data Management: Select a dataset and its version from ModelArts Data Management. The type of the selected dataset must be the same as the evaluation type. Data Storage Location: Select the OBS path where your desired dataset resides.
      Running Parameter	The system sets the model_url and data_url parameters by default based on the preceding configuration. You can click Add Running Parameter to add more parameter settings required for code evaluation.

      Figure 2 Setting job information
      
   3. Set the resource pool and its specifications for running the job. Due to the limitations of model evaluation, you can use the default values of resource pool parameters.

      Table 3 Resource pool parameters

      Parameter	Description
      Resource Pool	Currently, only public resource pools can be used in the evaluation job.
      Type	Model evaluation requires high-performance resources. Therefore, only the GPU type is supported.
      Specifications	Currently, 8 vCPUs 64 GiB GPUs are supported.
      Compute Nodes	The default value is 1. Currently, only the single-node mode is supported.

      Figure 3 Selecting a resource pool and its specifications
      
4. Ensure that the settings are correct and click Next.
5. On the Confirm Specifications page, confirm the parameters of the model evaluation job and click Submit.

   After the model evaluation job is created, the job starts to run. The running process takes several minutes. On the Evaluation/Diagnosis page, if the job status is Completed, the evaluation jobs are complete. You can click the job name and select a job version to view the evaluation result.

   A public resource pool is used. Therefore, when multiple users submit jobs at the same time, your jobs may be in the Queuing state, indicating that they are waiting for idle resources.

Follow-Up Procedure

After an evaluation job is created, you can perform the following operations to determine whether the model meets service requirements.

• Viewing evaluation results: After a model evaluation job is created and performed, you can view the evaluation result generated by the evaluation job based on the related metrics. For details, see Viewing Evaluation Results.
• Creating a version: If the result of the first evaluation does not meet service requirements, you can create another job version, change the dataset or evaluation code, and perform the evaluation again. For details, see Creating a Version.
• Deleting an evaluation job: If an evaluation job is no longer needed, you can delete it to avoid resource waste. On the Evaluation/Diagnosis page, locate the row where the target job resides and click Delete in the Operation column to delete the job. Deleted jobs cannot be recovered. Exercise caution when performing this operation.