Help Center/ ModelArts/ FAQs/ Training Jobs/ Compiling the Training Code/ How Do I Create a Training Job When a Dependency Package Is Referenced by the Model to Be Trained?

Updated on 2024-06-15 GMT+08:00

View PDF

How Do I Create a Training Job When a Dependency Package Is Referenced by the Model to Be Trained?

ModelArts allows you to install third-party dependency packages for model training. After the pip-requirements.txt file is stored in the training code directory, the system runs the command below to install the specified Python packages before the training boot file is executed.

pip install -r pip-requirements.txt

Only training jobs created using a preset image can reference dependency packages for model training.

Any one of the following file names can be used. This section uses pip-requirements.txt as an example.

pip-requirement.txt
pip-requirements.txt
requirement.txt
requirements.txt

For details about the code directory, see Storing the Installation File in the Code Directory.
For details about the specifications of pip-requirements.txt, see Installation File Specifications.

Storing the Installation File in the Code Directory

If you use My algorithm to create a training job, you can store related files in the configured Code Directory when creating an algorithm. The Boot Mode of the algorithm must be Preset image.
If you use Custom algorithm to create a training job, you can store related files in the configured Code Directory. The Boot Mode must be Preset image.

Before creating a training job, upload related files to OBS. For details about the file packaging requirements, see Installation File Specifications.

Installation File Specifications

The installation file varies depending on the dependency package type.

Open-source installation packages

Installation using the source code from GitHub is not supported.

Create a file named pip-requirements.txt in the code directory, and specify the name and version number of the dependency package in the file. The format is [Package name]==[Version].

Take for example, an OBS path specified by Code Dir that contains model files and the pip-requirements.txt file. The code directory structure would be as follows:
```
|---OBS path to the model boot file
     |---model.py               #Model boot file
     |---pip-requirements.txt   #Defined configuration file, which specifies the name and version of the dependency package
```
The following shows the content of the pip-requirements.txt file:
```
alembic==0.8.6
bleach==1.4.3
click==6.6
```
WHL packages
If the training background does not support the download of open source installation packages or use of user-compiled WHL packages, the system cannot automatically download and install the package. In this case, place the WHL package in the code directory, create a file named pip-requirements.txt, and specify the name of the WHL package in the file. The dependency package must be a .whl file.

Take for example, an OBS path specified by Code Dir that contains model files, the .whl file, and the pip-requirements.txt file. The code directory structure would be as follows:
```
|---OBS path to the model boot file
     |---model.py               #Model boot file
     |---XXX.whl                #Dependency package. If multiple dependencies are required, place multiple dependency packages here.
     |---pip-requirements.txt   #Defined configuration file, which specifies the name of the dependency package
```
The following shows the content of the pip-requirements.txt file:
```
numpy-1.15.4-cp36-cp36m-manylinux1_x86_64.whl
tensorflow-1.8.0-cp36-cp36m-manylinux1_x86_64.whl
```