Kubeflow Add-on

Basic Concepts

For more details, see Kubeflow Pipeline Concepts.

Prerequisites

• A CCE standard or Turbo cluster of v1.27 or later is available. For details, see Buying a CCE Standard/Turbo Cluster.
• Before installing the add-on, ensure that the cluster's pods can access the Internet to download required images. You can configure an SNAT rule for the cluster to ensure that pods can access the Internet. You will be billed for the SNAT rule. For details, see NAT Gateway Price Calculator.
• In the use case described in this section, a NodePort Service is required to access the Kubeflow web UI. Ensure an EIP is bound to any node in the cluster. The EIP will be billed. For details, see Elastic IP Price Calculator.

Notes and Constraints

• The Kubeflow add-on provides only pipeline capabilities.
• This add-on is being deployed. To view the regions where this add-on is available, see the console.
• This add-on is in the OBT phase. You can experience the latest add-on features. However, the stability of this add-on version has not been fully verified, and the CCE SLA is not valid for this version.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. On the displayed page, locate Kubeflow and click Install.
3. On the Install Add-on page, configure the add-on.

   Table 2 Add-on settings

   Parameter	Description
   Version	Select a version as needed.
   Add-on Specifications	Only the default specifications are supported.

4. Click Install in the lower right corner. If the status is Running, the add-on has been installed.

Components

• All preceding components are deployed in the kubeflow namespace.
• Components like Minio and MySQL provide dependencies for Kubeflow Pipelines and can be replaced as needed.

MySQL and Minio backend PVs are bound to emptyDir volumes. If pods are migrated or restarted, unexpected results may occur.

Use Case

This case describes how to use the Kubeflow add-on to run a simple pipeline, as shown in Figure 1. The pipeline includes a comp task that receives a string message and outputs it unchanged. This task is implemented by a Python 3.9 pod, which prints the message and writes it to a specified output file.

Figure 1 Execution process of the case

1. Install kubectl on an existing ECS and access a cluster using kubectl. For details, see Accessing a Cluster Using kubectl.
2. After the Kubeflow add-on is installed, CCE automatically creates the ml-pipeline-ui component to provide the frontend web interface. This component is automatically bound to a ClusterIP Service named ml-pipeline-ui. To access the UI, change the Service type of ml-pipeline-ui from ClusterIP to NodePort or LoadBalancer. The following uses a NodePort Service as an example.
   Run the following command to change the ClusterIP Service to a NodePort Service:

   kubectl edit svc ml-pipeline-ui -n kubeflow

   Set type to NodePort in the file and specify a node port that does not conflict with other services in the cluster. For details about NodePort Services, see NodePort.

   ...
   spec:
     ports:
       - name: http
         protocol: TCP
         port: 80
         targetPort: 3000
         nodePort: 30083 # Node port number
     selector:
       app: ml-pipeline-ui
       application-crd-id: kubeflow-pipelines
     clusterIP: 10.247.101.116
     clusterIPs:
       - 10.247.101.116
     type: NodePort # Change ClusterIP to NodePort.
     sessionAffinity: None
   ...

3. Enter http://<EIP of the target node>:<Node port>/ in the address bar of a browser to access the Kubeflow UI. On the UI, you can create and manage machine learning pipelines, define complete workflows including data preprocessing, model training, evaluation, and deployment. You can also configure and launch runs, monitor task execution in real time, and view logs and visualizations. This enables the automatic orchestration and execution of machine learning workflows.

   If the following page is displayed, the UI has been accessed.

   Figure 2 Kubeflow add-on UI
   

4. Create a pipeline.yaml file locally to configure a pipeline. This section provides a basic pipeline example that extracts input parameters during runtime, prints them, and outputs them to a specified file. For more information about pipelines, see Compile a Pipeline.
   pipeline.yaml file content:

   # PIPELINE DEFINITION
   # Name: my-pipeline
   # Description: My ML pipeline running on HuaweiCloud CCE.
   # Inputs:
   #    message: str
   # Outputs:
   #    Output: str
   components:
   comp-comp:# Define a component named comp-comp and specify its inputs (messages) and outputs.
       executorLabel: exec-comp   # Executor label, which is used to match the executor
       inputDefinitions:          # Component input definitions
         parameters:
           message:
             parameterType: STRING
       outputDefinitions:         # Component output definitions
         parameters:
           Output:
             parameterType: STRING
   deploymentSpec:
     executors:      # Configure executors.
       exec-comp:
         container:  # Container configuration, which specifies the tasks executed by pods in the CCE cluster
           image: python:3.9    # Use a Python 3.9 image.
           command:             # Container startup command
             - python3
             - -c
             - |
                 import sys, json      # Specify the Python script as follows: Extract messages from input parameters, print the messages, and write them to the output file (the path is provided by Kubeflow).
   
                 executor_input = json.loads(sys.argv[1])
                 inputs = executor_input.get("inputs", {})
                 parameters = inputs.get("parameterValues", {})
                 message = parameters.get("message", "Success Message From pipeline")
   
                 print("Received message:", message)
   
                 outputs = executor_input.get("outputs", {})
                 output_file = outputs.get("parameters", {}).get("Output", {}).get("outputFile")
                 if output_file:
                     with open(output_file, "w") as f:
                         f.write(message)
             - '{{$}}'
   pipelineInfo:
     description: My ML pipeline running on HuaweiCloud CCE.
     name: my-pipeline
   root:    
     dag:    # Define a DAG to determine the execution sequence and dependencies of multiple components, whether they are executed in parallel or sequentially.
       outputs:
         parameters:
           Output:
             valueFromParameter:
               outputParameterKey: Output   # The output of the comp task serves as the final output of the entire DAG.
               producerSubtask: comp  
       tasks:     # Define a task comp to call the comp-comp component, pass the input messages of the Python script to it, and use its output as the final output of the pipeline.
         comp:    
           cachingOptions:
             enableCache: true
           componentRef:
             name: comp-comp
           inputs:
             parameters:
               message:
                 componentInputParameter: message
           taskInfo:
             name: comp
     inputDefinitions:     # Declare the pipeline's input (message) and output parameters, both of which are strings.
       parameters:
         message:
           parameterType: STRING
     outputDefinitions:
       parameters:
         Output:
           parameterType: STRING
   schemaVersion: 2.1.0
   sdkVersion: kfp-2.12.1

   The overall execution logic of the preceding YAML file is as follows.

   Table 3 Execution Logic

   Phase	Step	Description	Associated YAML Field
   Input	Submit parameters.	Message parameters are input through the UI.	inputDefinitions.parameters.message
   	The pipeline receives parameters.	The pipeline centrally validates and standardizes the input parameters required for execution, and stores valid parameters persistently.	root.inputDefinitions
   	The DAG distributes parameters.	The DAG distributes parameters to the specified task.	tasks.comp.inputs.parameters.message
   Task execution	The task calls the component.	The task comp calls component comp-comp.	tasks.comp.componentRef
   	Associate the component with an executor.	The component finds containers by executorLabel and uses them for task execution.	components.comp-comp.executorLabel deploymentSpec.exec-comp
   	Run containers.	The API server of the CCE cluster receives the execution request and starts the pods to run the Python script task.	deploymentSpec.exec-comp.container.command
   Output	Define component outputs.	The component declares output parameter Output.	components.comp-comp.outputDefinitions
   	The DAG collects results.	The DAG collects the output from the comp task and passes it as the final output of the pipeline.	root.dag.outputs.parameters.Output
   	Provide the final output.	You can view the global output result.	outputDefinitions.parameters.Output

5. Return to the UI and click Upload pipeline in the upper right corner to upload the pipeline.yaml file. After the configuration is complete, click Create in the lower part to create a pipeline.
   Figure 3 Uploading a pipeline file
   

6. After the pipeline is created, the comp task is automatically displayed on the UI. Click Create run in the upper right corner to create a run. After the configuration is complete, click Start in the lower part to run the pipeline.
   Figure 4 Creating a run
   
   • Run parameters: automatically generated based on the pipeline.yaml file. Enter the message to be output.

   After you click Start, the API server of the CCE cluster receives the task request from the executor, launches the pods to execute it, and updates the execution status and logs in real time on the UI. You can monitor the progress and view the final result there. If the pods of the CCE cluster are stopped, the task execution is complete. In the navigation pane, choose Runs. If is displayed before the target run instance, the pipeline has been executed.

   Figure 5 Starting pods for task execution
   

7. Click the comp task and view its inputs and outputs. As shown in the following figure, the input message is My ML pipeline running on HuaweiCloud CCE., and the output is My ML pipeline running on HuaweiCloud CCE., which complies with the pipeline definitions.
   Figure 6 Input and output
   

Change History