Developing Your First Workflow

This section provides an example of building a workflow with only one training phase based on the image classification algorithm. For details about the parameters for building a workflow with more phases, see Creating Workflow Phases.

Step 1: Installing the Development Environment

Two methods of installing the development environment are provided for you to choose as needed.

Method 1: Open a notebook instance in JupyterLab and install the environment.

Log in to the ModelArts console and choose Development Workspace > Notebook.
Click Create Notebook. On the page that is displayed, retain the default parameter settings and click Next. Confirm the information and click Submit. After the task is submitted, click Return Now to go back to the notebook list page. For more notebook instance parameters, see Creating a Notebook Instance.
In the notebook list, locate the instance created in 2 and ensure that its status is Running. Click Open in the Operation column. The JupyterLab page is displayed.

Create an IPYNB file.

Figure 1 Creating an IPYNB file
Click to enlarge

Run the following command. If the import is successful, the environment is ready.

from modelarts import workflow as wf

If the execution fails, run the following commands to manually install the environment:

!rm modelarts*.whl

!wget -N https://cn-north-4-training-test.obs.cn-north-4.myhuaweicloud.com/workflow-apps/v1.0.1/modelarts-1.4.18-py2.py3-none-any.whl
!wget -N https://cn-north-4-training-test.obs.cn-north-4.myhuaweicloud.com/workflow-apps/v1.0.1/modelarts_workflow-1.0.1-py2.py3-none-any.whl

!pip uninstall -y modelarts modelarts-workflow

!pip install modelarts-1.4.18-py2.py3-none-any.whl
!pip install modelarts_workflow-1.0.1-py2.py3-none-any.whl

If the import fails, run the installation commands again or restart the kernel and then run the installation commands again.

Method 2: Use a local IDE to remotely connect to a notebook instance.

If you use a local IDE to develop workflows, such as PyCharm, you only need to focus on local code development. For details about how to use PyCharm to access a notebook instance, see Connecting to a Notebook Instance Through PyCharm Toolkit.

Run the following commands on the local IDE terminal to install the environment. Python 3.7.x or later is required.

rm modelarts*.whl
wget -N https://cn-north-4-training-test.obs.cn-north-4.myhuaweicloud.com/workflow-apps/v1.0.2/modelarts-1.4.19-py2.py3-none-any.whl
wget -N https://cn-north-4-training-test.obs.cn-north-4.myhuaweicloud.com/workflow-apps/v1.0.2/modelarts_workflow-1.0.2-py2.py3-none-any.whl

pip uninstall -y modelarts modelarts-workflow

pip install modelarts-1.4.19-py2.py3-none-any.whl
pip install modelarts_workflow-1.0.2-py2.py3-none-any.whl

After you configure PyCharm for development using a local IDE, configure AK/SK in the code for session authentication. The following is an example of how this would be done:

from modelarts.session import Session
# Hard-coded or plaintext AK/SK is risky. For security, encrypt your AK/SK and store them in the configuration file or environment variables.
# In this example, the AK and SK are stored in environment variables for identity authentication. Before running this example, set environment variables HUAWEICLOUD_SDK_AK and HUAWEICLOUD_SDK_SK.
__AK = os.environ["HUAWEICLOUD_SDK_AK"]
__SK = os.environ["HUAWEICLOUD_SDK_SK"]
# Decrypt the information if it is encrypted.
session = Session(access_key=__AK, secret_key=__SK, project_id='***', region_name='***')

When downloading the ModelArts SDK and workflow SDK installation packages, you are advised to download the verification file as well to verify the integrity of the installation packages. This prevents possible service errors caused by installation package download issues.

Download the ModelArts SDK installation package and its verification file as well the workflow SDK installation package and its verification file in sequence.
Store the SDK installation package and its verification file in the same directory, and use OpenSSL to verify the integrity. The following is an example of workflow SDK verification:
```
openssl cms -verify -binary -in modelarts_workflow-*.*.*-py2.py3-none-any.whl.cms  -inform DER -content modelarts_workflow-*.*.*-py2.py3-none-any.whl  -noverify > ./test
```
If the following information is displayed, the verification is successful:
```
Verification successful
```

Step 2: Preparing Data

Prepare algorithms.

This section describes how to subscribe to an algorithm. You can also prepare your algorithm.

Go to AI Gallery, choose Asset Hub > Algorithm, search for an image classification algorithm, and go to its details page.
1. Click Subscribe on the right of the algorithm details page.
2. In the dialog box that appears, read and agree to Data Security and Privacy Risk and Service Agreement of AI Gallery, and click Continue.
After the subscription, click Access Console, select the region, and click OK. The system automatically directs you to Algorithm Management > My Subscriptions. Select the algorithm to show its basic information and version list at the bottom of the page.

Prepare a dataset.

Log in to the ModelArts console.
In the navigation pane on the left, choose Asset Management > Datasets and click Create.
Set parameters.
- Name: name of the dataset. The value can contain 1 to 100 visible characters and must start with a letter.
- Data Type: Select Images.
- Data Source: Select OBS or Local file.
  - OBS
    - Import Mode: Path
    - Import Path: Select an OBS path where your data is stored.
  - Local file
    - Storage Path: Select an OBS path to store your dataset.
    - Uploading Data: Click Upload data to upload your local data to the OBS path selected in the previous step.
- Labeling Status: Select Unlabeled or Labeled based on your needs.
- Output Dataset Path: Select an empty OBS path to store files such as dataset labeling information.
After setting the parameters, click Submit.

To match the subscribed algorithm (image classification), you must create an image dataset.

Step 3: Compiling a Workflow

This example builds a workflow with only one training phase based on the image classification algorithm.

For details about how to import other workflow phases required during workflow compilation, see Creating a Workflow.

After the development environment is installed, enter the following example code in JupyterLab of ModelArts notebook.

from modelarts import workflow as wf

# Define a unified storage object to manage the output directories.
output_storage = wf.data.OutputStorage(name="output_storage", description="Output directory configuration")

# Dataset object
dataset = wf.data.DatasetPlaceholder(name="input_data")

# Create a training job.
job_step = wf.steps.JobStep(
    name="training_job",
    title="Image classification training",
    algorithm=wf.AIGalleryAlgorithm(
        subscription_id="***", # Subscription ID of the image classification algorithm. Obtain the subscription ID on the algorithm management page. This parameter is optional.
        item_version_id="1.0.1", # Version number of the subscribed algorithm. In this example, the version number is 1.0.1. This parameter is optional.
parameters=[
                wf.AlgorithmParameters(name="task_type", value="image_classification_v2"),
                wf.AlgorithmParameters(name="model_name", value="resnet_v1_50"),
                wf.AlgorithmParameters(name="do_train", value="True"),
                wf.AlgorithmParameters(name="do_eval_along_train", value="True"),
                wf.AlgorithmParameters(name="variable_update", value="horovod"),
                wf.AlgorithmParameters(name="learning_rate_strategy", value="0.002"),
                wf.AlgorithmParameters(name="batch_size", value="64"),
                wf.AlgorithmParameters(name="eval_batch_size", value="64"),
                wf.AlgorithmParameters(name="evaluate_every_n_epochs", value="1.0"),
                wf.AlgorithmParameters(name="save_model_secs", value="60"),
                wf.AlgorithmParameters(name="save_summary_steps", value="10"),
                wf.AlgorithmParameters(name="log_every_n_steps", value="10"),
                wf.AlgorithmParameters(name="do_data_cleaning", value="True"),
                wf.AlgorithmParameters(name="use_fp16", value="True"),
                wf.AlgorithmParameters(name="xla_compile", value="True"),
                wf.AlgorithmParameters(name="data_format", value="NCHW"),
                wf.AlgorithmParameters(name="best_model", value="True"),
                wf.AlgorithmParameters(name="jpeg_preprocess", value="True"),
                wf.AlgorithmParameters(name="do_model_analysis", value="True"),
                wf.AlgorithmParameters(name="wf_metric_log", value="True"),
            ]

    ),
    inputs=[wf.steps.JobInput(name="data_url", data=dataset)],
    outputs=[wf.steps.JobOutput(name="train_url", obs_config=wf.data.OBSOutputConfig(obs_path=output_storage.join("/train_output/")))],
    spec=wf.steps.JobSpec(
        resource=wf.steps.JobResource(
            flavor=wf.Placeholder(
                name="training_flavor",
                placeholder_type=wf.PlaceholderType.JSON,
                description="Training flavor"
            )
        )
    )
)

# Create a workflow object.
workflow = wf.Workflow(
    name="image-classification-ResNeSt",
    desc="this is a image classification workflow",
    steps=[job_step],
    storages=[output_storage]
)

# By default, a workflow is created in the default workspace. You can specify the workspace to which the workflow belongs in the following way:
# workflow = wf.Workflow(
#     name="image-classification-ResNeSt",
#     desc="this is a image classification workflow",
#     steps=[job_step],
#     storages=[output_storage],
#     workspace=wf.resource.Workspace(workspace_id="***")
# )
# You can view workspace_id in ModelArts Workspace.

The preceding code example can be directly debugged and run in the notebook on the cloud.

To use the code in a local IDE, you need to add the related session authentication content and run the code in the local IDE. The code example is as follows:

from modelarts import workflow as wf
from modelarts.session import Session
# Hard-coded or plaintext AK/SK is risky. For security, encrypt your AK/SK and store them in the configuration file or environment variables.
# In this example, the AK/SK is stored in environment variables for identity authentication. Before running this example, set environment variables HUAWEICLOUD_SDK_AK and HUAWEICLOUD_SDK_SK.
__AK = os.environ["HUAWEICLOUD_SDK_AK"]
__SK = os.environ["HUAWEICLOUD_SDK_SK"]
# Decrypt the information if it is encrypted.
session = Session(access_key=__AK, secret_key=__SK, project_id='***', region_name='***') # Change the values based on your account information.

# Define a unified storage object to manage the output directories.
output_storage = wf.data.OutputStorage(name="output_storage", description="Output directory configuration")

# Dataset object
dataset = wf.data.DatasetPlaceholder(name="input_data")

# Create a training job.
job_step = wf.steps.JobStep(
    name="training_job",
    title="Image classification training",
    algorithm=wf.AIGalleryAlgorithm(
        subscription_id="***", # Subscription ID of the image classification algorithm. Obtain the subscription ID on the algorithm management page.
        item_version_id="10.1.0", # Version number of the subscribed algorithm. In this example, the version number is 10.1.0.
parameters=[
                wf.AlgorithmParameters(name="task_type", value="image_classification_v2"),
                wf.AlgorithmParameters(name="model_name", value="resnet_v1_50"),
                wf.AlgorithmParameters(name="do_train", value="True"),
                wf.AlgorithmParameters(name="do_eval_along_train", value="True"),
                wf.AlgorithmParameters(name="variable_update", value="horovod"),
                wf.AlgorithmParameters(name="learning_rate_strategy", value="0.002"),
                wf.AlgorithmParameters(name="batch_size", value="64"),
                wf.AlgorithmParameters(name="eval_batch_size", value="64"),
                wf.AlgorithmParameters(name="evaluate_every_n_epochs", value="1.0"),
                wf.AlgorithmParameters(name="save_model_secs", value="60"),
                wf.AlgorithmParameters(name="save_summary_steps", value="10"),
                wf.AlgorithmParameters(name="log_every_n_steps", value="10"),
                wf.AlgorithmParameters(name="do_data_cleaning", value="True"),
                wf.AlgorithmParameters(name="use_fp16", value="True"),
                wf.AlgorithmParameters(name="xla_compile", value="True"),
                wf.AlgorithmParameters(name="data_format", value="NCHW"),
                wf.AlgorithmParameters(name="best_model", value="True"),
                wf.AlgorithmParameters(name="jpeg_preprocess", value="True"),
                wf.AlgorithmParameters(name="do_model_analysis", value="True"),
                wf.AlgorithmParameters(name="wf_metric_log", value="True"),
            ]

    ),
    inputs=[wf.steps.JobInput(name="data_url", data=dataset)],
    outputs=[wf.steps.JobOutput(name="train_url", obs_config=wf.data.OBSOutputConfig(obs_path=output_storage.join("/train_output/")))],
    spec=wf.steps.JobSpec(
        resource=wf.steps.JobResource(
            flavor=wf.Placeholder(
                name="training_flavor",
                placeholder_type=wf.PlaceholderType.JSON,
                description="Training flavor"
            )
        )
    )
)

# Create a workflow object.
workflow = wf.Workflow(
    name="image-classification-ResNeSt",
    desc="this is a image classification workflow",
    steps=[job_step],
    session=session, # Add the authentication object.
    storages=[output_storage]
)

# By default, a workflow is created in the default workspace. You can specify the workspace to which the workflow belongs in the following way:
# workflow = wf.Workflow(
#     name="image-classification-ResNeSt",
#     desc="this is a image classification workflow",
#     steps=[job_step],
#     session=session, # Add the authentication object.
#     storages=[output_storage],
#     workspace=wf.resource.Workspace(workspace_id="***")
# )
# You can view workspace_id in ModelArts Workspace.

Step 4: Publishing a Workflow to the Runtime State

After the workflow is published to the runtime state, you need to configure it. For details, see 2. For more publishing modes, see Publishing a Workflow.

After the workflow is developed, run the following code to publish it to the runtime state. # If the log indicates that the workflow is published, you can go to the ModelArts workflow page to view the workflow.
```
workflow.release()
```
Figure 2 Published
Log in to the ModelArts console to view the newly published workflow. Go to the workflow details page and click Configure to set parameters.

(Optional) Clearing Workflow Resources

Delete workflows.

On the ModelArts console, choose Development Workspace > Workflow in the navigation pane on the left.
On the workflow list page, locate two generated workflows, one running and the other not running.
In the Operation column of those workflows, click More and select Delete.
In the Delete Workflow dialog box, confirm the information, enter DELETE, and click OK.

Delete a notebook instance.

On the ModelArts console, choose Development Workspace > Notebook in the navigation pane on the left.
In the notebook instance list, locate the target notebook instance, click More in the Operation column, and select Delete. In the displayed dialog box, confirm the information, enter DELETE, and click OK.

Delete an OBS bucket.

On the menu bar of the console, click and choose Object Storage Service.
In the navigation pane on the left of the OBS console, choose Buckets. In the bucket list, locate the created OBS bucket and go to its details page.
On the bucket details page, choose Objects in the navigation pane. In the right pane, select the object that is not needed and click Delete above the list. Alternatively, click More in its Operation column and select Delete.