Help Center> ModelArts> User Guide (Senior AI Engineers)> Model Management> Model Evaluation and Diagnosis> Model Evaluation Overview
View PDF

Model Evaluation Overview

After a model is imported to ModelArts, you can evaluate and diagnose the model to determine whether it meets service requirements.

Functions of Model Evaluation

The system provides evaluation metrics for evaluation tasks of different types of models. When the evaluation results are displayed, the system performs detailed evaluation on the models based on different data features to obtain the sensitivity of each data feature to the evaluation metrics and provides optimization suggestions. Users can fully understand the adaptability of models to different data features, so that model optimization can be targeted.

For a training job with a built-in algorithm or evaluation code, you can view the evaluation result after the training is complete. For details, see Viewing the Evaluation Result.

Figure 1 Model evaluation

Evaluation Jobs and Job Versions

  • An evaluation job can contain multiple versions.
  • In the same job, you can change the dataset or evaluation code to create a new version.
  • An evaluation result is generated after an evaluation job version is successfully executed. You can view the evaluation result of each evaluation metric on the evaluation result page under Version Manager.

Model Evaluation Restrictions

  • Currently, model evaluation and diagnosis support the following types of models and datasets: image classification, object detection, and semantic segmentation.
  • Models for evaluation and diagnosis use TensorFlow or PyTorch as AI engines. Only TF-1.13.1-python3.6, TF-2.1.0-python3.6, and PyTorch-1.4.0-python3.6 can be used to compile evaluation code.
  • Only GPU resources are supported. In addition, the resource pool supports only the single-node running mode and does not support the distributed mode.
  • Models generated by ExeML do not support model evaluation.
  • For the built-in TensorFlow algorithms of ModelArts, you can set parameters to start evaluation after training, or import the models to the model management module and create an evaluation job.

Operations Related to Model Evaluation