Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Creating an Algorithm Using a Custom Image

Updated on 2024-08-14 GMT+08:00

Your locally developed algorithms or algorithms developed using other tools can be uploaded to ModelArts for unified management.

Entries for Creating an Algorithm

You can create an algorithm using a custom image on ModelArts in either of the following ways:

  • Entry 1: On the ModelArts console, choose Algorithm Management > My algorithms. Then, create an algorithm and use it in training jobs or publish it to AI Gallery.
  • Entry 2: On the ModelArts console, choose Model Training > Training Jobs, and click Create Training Job to create a custom algorithm and submit a training job. For details, see Using a Custom Image to Create a CPU- or GPU-based Training Job.

Parameters for creating an algorithm

Table 1 Parameters for creating an algorithm

Parameter

Description

Boot Mode

Select Custom images. This parameter is mandatory.

Image

URL of an SWR image. This parameter is mandatory.

  • Private images or shared images: Click Select on the right to select an SWR image. Ensure that the image has been uploaded to SWR. For details, see How Can I Log In to SWR and Upload Images to It?.
  • Public images: You can also manually enter the image path in the format of "<Organization to which your image belongs>/<Image name>" on SWR. Do not contain the domain name (swr.<region>.xxx.com) in the path because the system will automatically add the domain name to the path. For example:
    modelarts-job-dev-image/pytorch_1_8:train-pytorch_1.8.0-cuda_10.2-py_3.7-euleros_2.10.1-x86_64-8.1.1

Code Directory

OBS path for storing the training code. This parameter is optional.

Take OBS path obs://obs-bucket/training-test/demo-code as an example. The content in the OBS path will be automatically downloaded to ${MA_JOB_DIR}/demo-code in the training container, and demo-code (customizable) is the last-level directory of the OBS path.

Boot Command

Command for booting an image. This parameter is mandatory. The boot command will be automatically executed after the code directory is downloaded.

  • If the training boot script is a .py file, train.py for example, the boot command can be python ${MA_JOB_DIR}/demo-code/train.py.
  • If the training boot script is an .sh file, main.sh for example, the boot command can be bash ${MA_JOB_DIR}/demo-code/main.sh.

Semicolons (;) and ampersands (&&) can be used to combine multiple boot commands, but line breaks are not supported. demo-code (customizable) in the boot command is the last-level directory of the OBS path.

Configuring Pipelines

A preset image-based algorithm obtains data from an OBS bucket or dataset for model training. The training output is stored in an OBS bucket. The input and output parameters in your algorithm code must be parsed to enable data exchange between ModelArts and OBS. For details about how to develop code for training on ModelArts, see Developing a Custom Script.

When you use a preset image to create an algorithm, configure the input and output pipelines.

  • Input configurations
    Table 2 Input configurations

    Parameter

    Description

    Parameter Name

    Set the name based on the data input parameter in your algorithm code. The code path parameter must be the same as the training input parameter parsed in your algorithm code. Otherwise, the algorithm code cannot obtain the input data.

    For example, If you use argparse in the algorithm code to parse data_url into the data input, set the data input parameter to data_url when creating the algorithm.

    Description

    Customizable description of the input parameter,

    Obtained from

    Source of the input parameter. You can select Hyperparameters (default) or Environment variables.

    Constraints

    Whether data is obtained from a storage path or ModelArts dataset.

    If you select the ModelArts dataset as the data source, the following constraints are added:

    • Labeling Type: For details, see Creating a Labeling Job.
    • Data Format, which can be Default, CarbonData, or both. Default indicates the manifest format.
    • Data Segmentation: available only for image classification, object detection, text classification, and sound classification datasets.

      Possible values are Segmented dataset, Dataset not segmented, and Unlimited. For details, see Publishing a Data Version.

    Add

    Multiple data input sources are allowed.

  • Output configurations
    Table 3 Output configurations

    Parameter

    Description

    Parameter Name

    Set the name based on the data output parameter in your algorithm code. The code path parameter must be the same as the training output parameter parsed in your algorithm code. Otherwise, the algorithm code cannot obtain the output path.

    For example, If you use argparse in the algorithm code to parse train_url into the data output, set the data output parameter to train_url when creating the algorithm.

    Description

    Customizable description of the output parameter,

    Obtained from

    Source of the output parameter. You can select Hyperparameters (default) or Environment variables.

    Add

    Multiple data output paths are allowed.

Defining Hyperparameters

When you use a preset image to create an algorithm, ModelArts allows you to customize hyperparameters so you can view or modify them anytime. After the hyperparameters are defined, they are displayed in the startup command and transferred to your boot file as CLI parameters.

  1. Import hyperparameters.

    You can click Add hyperparameter to manually add hyperparameters.

  2. Edit hyperparameters.
    NOTE:

    To ensure data security, do not enter sensitive information, such as plaintext passwords.

    For details, see Table 4.

    Table 4 Hyperparameters

    Parameter

    Description

    Name

    Hyperparameter name

    Enter 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.

    Type

    Type of the hyperparameter, which can be String, Integer, Float, or Boolean

    Default

    Default value of the hyperparameter, which is used for training jobs by default

    Constraints

    Click Restrain. Then, set the range of the default value or enumerated value in the dialog box displayed.

    Required

    Select Yes or No.

    • If you select No, you can delete the hyperparameter on the training job creation page when using this algorithm to create a training job.
    • If you select Yes, you cannot delete the hyperparameter on the training job creation page when using this algorithm to create a training job.

    Description

    Description of the hyperparameter

    Only letters, digits, spaces, hyphens (-), underscores (_), commas (,), and periods (.) are allowed.

Adding Training Constraints

You can add training constraints of the algorithm based on your needs.

  • Resource Type: Select the required resource types.
  • Multicard Training: Choose whether to support multi-card training.
  • Distributed Training: Choose whether to support distributed training.

Runtime Environment Preview

When creating an algorithm, click the arrow on in the lower right corner of the page to know the path of the code directory, boot file, and input and output data in the training container.

Follow-Up Procedure

After an algorithm is created, use it to create a training job. For details, see Using a Custom Image to Create a CPU- or GPU-based Training Job.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback