Creating a Training Job Using an API

Overview

When you need to perform machine learning training based on a specific dataset and algorithm model, you can use ModelArts APIs to create and configure a training job. After a training job is created, the platform automatically starts the training job based on the configured resource specifications. You can monitor the training progress and status based on the job ID.

Creating a training job using an API is suitable for the following scenarios and is the most recommended method for creating a training job in the production environment:

• Automated production pipeline: Integrates training jobs into the CI/CD pipeline or MLOps system. This setup manages data updates, triggers training automatically, evaluates results, and deploys them.
• Large-scale parallel hyperparameter search: Uses scripts to run dozens of training jobs with different settings at once. This helps find the best model faster by using a compute pool.
• Secondary development and integration: ModelArts also provides SDKs of multiple programming languages for you to use. If you need to integrate ModelArts into a third-party system for secondary development, APIs are the best access method.
• Refined resource scheduling: With the "training-inference shared pool" capability of Ascend supernodes, jobs can be batch delivered through APIs during off-peak hours, improving the cost-effectiveness of compute.

This section uses the PyTorch framework as an example to describe how to create a training job using an API.

Prerequisites

• You have obtained the endpoints of IAM and ModelArts Endpoint.
• You have confirmed the region where the service is deployed; you have obtained the project ID and name, obtained the account name and ID, and obtained the username and user ID.

• Data for training uploaded to an OBS directory. For details about how to create an OBS bucket and upload files, see Using OBS Console.
• OBS directory and ModelArts in the same region.

• The training code has been prepared. For example, to use the PyTorch framework, store the boot file test-pytorch.py in the OBS directory obs://cnnorth4-job-test-v2/pytorch/fast_example/code/cpu.
• You have created a path for storing trained models, for example, obs://cnnorth4-job-test-v2/pytorch/fast_example/outputs.
• A path for outputting the training job logs has been created, for example, obs://cnnorth4-job-test-v2/pytorch/fast_example/log.

API Call Process

The process for creating a training job using PyTorch is as follows:

1. Call the API for authentication to obtain a user token, which will be added in a request header for authentication. For details, see Step 1: Calling the Authentication API to Obtain a Token.
2. Call the API for creating a training job to create a training job using the UUID returned by the created algorithm and record the job ID. For details, see Step 2: Creating a Training Job.
3. Call the API for querying training job details to query the job status using the job ID. For details, see Step 3: Querying a Training Job.
4. Call the API for querying the logs of a specified task in a training job (OBS link) to obtain the OBS path of the training job logs. For details, see Step 4: Obtaining Training Job Logs.
5. Call the API for querying the running metrics of a specified task in a training job to view metrics of a training job. For details, see Step 5: Querying Metrics of a Training Job.
6. Call the API for deleting a training job to delete the job if it is no longer needed. For details, see Step 6: Delete a Training Job.

Step 1: Calling the Authentication API to Obtain a Token

Call the IAM authentication API to obtain the user token, which is used as the value of X-Auth-Token in all subsequent requests. Request body example (scope is set to project, indicating that the token only takes effect for resources in the specified project.)

URI: POST https://{iam_endpoint}/v3/auth/tokens

Request header: Content-Type: application/json

Request body

{
  "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "Your IAM username,"
          "password": "Your password,"
          "domain": { "name": "Your account name"}
        }
      }
    },
    "scope": {
      "project": { "name": "ap-southeast-1"  }
    }
  }
}

Expected result: A 201 Created message is returned. The X-Subject-Token value in the response header is the token value.

All subsequent request headers must contain the following information:

X-Auth-Token: MIIZmgYJKoZIhvcNAQcCoIIZizCCGYcCAQExDTALBglghkgBZQMEAgEwgXXXXXX...

Step 2: Creating a Training Job

Call the API for creating a training job to create a training job using the UUID returned by the created algorithm and record the job ID.

URI: POST https://{ma_endpoint}/v2/{project_id}/training-jobs

Request header

• X-Auth-Token: MIIZmgYJKoZIhvcNAQcCoIIZizCCGYcCAQExDTALBglghkgBZQMEAgEwgXXXXXX...
• Content-Type: application/json

Request body

{
    "kind": "job",
    "metadata": {
        "name": "test-pytorch-cpu01",
        "description": "test pytorch work cpu"
    },
    "algorithm": {
        "code_dir": "/cnnorth4-job-test-v2/pytorch/fast_example/code/cpu/",
        "local_code_dir": "/home/ma-user/modelarts/user-job-dir",
        "engine": {
            "image_url": "atelier/pytorch_cuda:pytorch_2.7.0-cuda_12.8-py_3.11.10-ubuntu_22.04-x86_64-20251215163925-4e5422a"
        },
        "command": "python ${MA_JOB_DIR}/cpu/test-pytorch.py"
    },
    "spec": {
        "resource": {
            "node_count": 1,
            "pool_id": "pool-maostest-train-06024304be00d5092fbdc0013d201342"
        },
        "log_export_path": {
            "obs_url": "/cnnorth4-job-test-v2/pytorch/fast_example/log/"
        }
    }
}

Parameters:

• Set kind to the type of the training job. The default value is job.
• Set name and description in the metadata field to the name and description of the training job.
• Set code_dir and local_code_dir in the algorithm field to the code directory and the local directory where the code is downloaded to the job, respectively.
• Set image_url in the algorithm field to the address of the training job image.
• Set command in the algorithm field to the command for starting the training job.
• In the spec field, pool_id indicates the ID of the resource pool on which the training job depends. node_count indicates whether to use multi-node training (distributed training). Set it to 1 for a single-node training by default. log_export_path specifies the OBS path to which logs are uploaded.

Response example: The status code 201 Created is returned, indicating that the training job has been created.

{
    "kind": "job",
    "metadata": {
        "id": "31318695-2011-4e48-9b90-9c9178c57951",
        "name": "test-pytorch-cpu01",
        "description": "test pytorch work cpu",
        "create_time": 1777545352008,
        "workspace_id": "0",
        "ai_project": "default-ai-project",
        "labels": {
            "training-job": "modelarts-os"
        },
        "user_name": "",
        "annotations": {
            "job_template": "Template DL",
            "key_task": "worker"
        },
        "training_experiment_reference": {},
        "tags": []
    },
    "status": {
        "phase": "Pending",
        "secondary_phase": "Creating",
        "pending_time": 1000,
        "duration": 0,
        "is_hanged": false,
        "retry_count": 0,
        "start_time": 0,
        "node_count_metrics": null,
        "tasks": [
            "worker-0"
        ],
        "metrics_statistics": {
            "cpu_usage": {
                "average": -1,
                "max": -1,
                "min": -1
            },
            "mem_usage": {
                "average": -1,
                "max": -1,
                "min": -1
            }
        }
    },
    "algorithm": {
        "code_dir": "/cnnorth4-job-test-v2/pytorch/fast_example/code/cpu/",
        "local_code_dir": "/home/ma-user/modelarts/user-job-dir",
        "command": "python ${MA_JOB_DIR}/cpu/test-pytorch.py",
        "engine": {
            "engine_id": "",
            "engine_name": "",
            "engine_version": "",
            "v1_compatible": false,
            "image_url": "atelier/pytorch_cuda:pytorch_2.7.0-cuda_12.8-py_3.11.10-ubuntu_22.04-x86_64-20251215163925-4e5422a",
            "non_swr_image": false,
            "run_user": "",
            "image_source": true,
            "image_repo_id": "",
            "image_id": ""
        }
    },
    "spec": {
        "resource": {
            "pool_id": "pool-maostest-train-06024304be00d5092fbdc0013d201342",
            "pool_resource_flavor": "",
            "node_count": 1,
            "pool_info": {
                "cpu_arch": "x86",
                "core_num": 5,
                "mem_size": 22,
                "cache_size": 0,
                "accelerator": "",
                "accelerator_num": 0,
                "accelerator_type": "",
                "accelerator_size": 0,
                "variant": "",
                "huge_pages": 0,
                "x_parameter_plane": "",
                "use_privileged": false,
                "use_host_network": false,
                "use_ib_network": false,
                "project_id": "",
                "pool_resource_flavor": "liumuqi-eni-test",
                "pool_id": "pool-maostest-train-06024304be00d5092fbdc0013d201342",
                "cluster_id": "",
                "maos_pool": true,
                "quota_id": "",
                "maos_migrated": false,
                "detect_all_in_int": false,
                "pool_type": "",
                "enable_cabinet": false,
                "enable_memarts": false,
                "enable_ems": false,
                "empty_dir_size": 0
            },
            "main_container_allocated_resources": {
                "cpu_arch": "x86",
                "cpu_core_num": 4,
                "mem_size": 20,
                "accelerator_num": 0,
                "accelerator_type": ""
            }
        },
        "log_export_path": {
            "obs_url": "/cnnorth4-job-test-v2/pytorch/fast_example/log/"
        },
        "is_hosted_log": true,
        "runtime_type": "production"
    },
    "ftjob_config": {
        "checkpoint_config": {
            "save_checkpoints_max": 0,
            "checkpoint_id": "",
            "skipped_steps": 0,
            "restore_training": 0
        },
        "task_env": {
            "envs": null
        }
    }
}

• Record the id value (training job ID) in the metadata field for subsequent steps.
• phase and secondary_phase under Status indicate the status and next status of the training job, respectively. In the example, Creating indicates that the training job is being created.

Step 3: Querying a Training Job

Call the API for querying training job details to query the job status using the job UUID.

Step 4: Obtaining Training Job Logs

Call the API for querying the logs of a specified task in a training job (OBS link) to obtain the OBS path of the training job logs.

Step 5: Querying Metrics of a Training Job

Call the API for querying the running metrics of a specified task in a training job to view metrics of a training job.

Step 6: Delete a Training Job

Call the API for deleting a training job to delete the job if it is no longer needed.

URI: DELETE https://{ma_endpoint}/v2/{project_id}/training-jobs/{training_job_id}

Request header

• X-Auth-Token: MIIZmgYJKoZIhvcNAQcCoIIZizCCGYcCAQExDTALBglghkgBZQMEAgEwgXXXXXX...

Set the italic parameters based on site requirements.

• Set training_job_id to the training job ID recorded in Step 2: Creating a Training Job.

Response description: Status code 202 No Content is returned, indicating that the job is successfully deleted.