How Do I Create a Training Job When a Dependency Package Is Referenced by the Model to Be Trained?
Store the pip-requirements.txt file in the training code directory.
Any one of the following file names can be used. This section uses pip-requirements.txt as an example.
- pip-requirement.txt
- pip-requirements.txt
- requirement.txt
- requirements.txt
Before the training boot file is executed, the system automatically runs the following command to install the specified Python packages:
pip install -r pip-requirements.txt
- For details about the code directory, see Storing the Installation File in the Code Directory.
- For details about the specifications of pip-requirements.txt, see Installation File Specifications.
Storing the Installation File in the Code Directory
ModelArts allows you to install third-party dependency packages during model training in either of the following ways:
- Method 1 (recommended): , store the required files or installation packages in the code directory.
Figure 1 Creating an algorithm
- Method 2: Before using a common framework to create a training job, store the required files or installation packages in the code directory. (This function will be unavailable soon.)
Figure 2 Using a common framework to create an algorithm
Installation File Specifications
The installation file varies depending on the dependency package type.
- WHL packages
If the training background does not support the download of open source installation packages or use of user-compiled WHL packages, the system cannot automatically download and install the package. In this case, place the WHL package in the code directory, create a file named pip-requirements.txt, and specify the name of the WHL package in the file. The dependency package must be a .whl file.
Take for example, an OBS path specified by Code Dir that contains model files, the .whl file, and the pip-requirements.txt file. The code directory structure would be as follows:
|---OBS path to the model boot file |---model.py #Model boot file |---XXX.whl #Dependency package. If multiple dependencies are required, place multiple dependency packages here. |---pip-requirements.txt #Defined configuration file, which specifies the name of the dependency packageThe following shows the content of the pip-requirements.txt file:
numpy-1.15.4-cp36-cp36m-manylinux1_x86_64.whl tensorflow-1.8.0-cp36-cp36m-manylinux1_x86_64.whl
Compiling the Training Code FAQs
- How Do I Create a Training Job When a Dependency Package Is Referenced by the Model to Be Trained?
- What Is the Common File Path for Training Jobs?
- How Do I Install a Library That C++ Depends on?
- How Do I Check Whether a Folder Copy Is Complete During Job Training?
- How Do I Load Some Well Trained Parameters During Job Training?
- How Do I Obtain Training Job Parameters from the Boot File of the Training Job?
- Why Can't I Use os.system ('cd xxx') to Access the Corresponding Folder During Job Training?
- How Do I Invoke a Shell Script in a Training Job to Execute the .sh File?
- How Do I Obtain the Dependency File Path to be Used in Training Code?
- What Is the File Path If a File in the model Directory Is Referenced in a Custom Python Package?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
more
