Containerizing and Using QingTian Enclave on CCE

Solution Architecture

To deploy QingTian Enclave in a CCE cluster, install the add-on for containerizing QingTian Enclave and package a QingTian Enclave image file (.eif) into a container image. After completing these steps, you can create QingTian Enclave containers within the CCE cluster. kubelet allocates and schedules such containers through the /dev/qtbox_service0 API for managing QingTian Enclave instance lifecycle.

Compared with running QingTian Enclave instances on VMs, running QingTian Enclave containers splits services into untrusted and trusted components. The untrusted components include external service components of an application and components related to QingTian enclave instance management (such as the qt-enclave tool). The trusted components include services that involve encryption and decryption or sensitive information protection and run in a QingTian Enclave instance.

When running QingTian Enclave containers, qt-enclave is used to create a QingTian Enclave instance through a secure local channel (vsock) to run services that require encryption, decryption, or sensitive information protection.

Figure 1 Using QingTian Enclave in a CCE cluster

Prerequisites

A CCE standard or Turbo cluster is available.

Managing a QingTian Enclave Instance in a CCE Cluster

1. Log in to the ECS console, create an ECS that supports the QingTian Enclave feature, for example, a C7t ECS with 4 vCPUs and 8 GiB of memory or a higher flavor, and use Linux as the system image. (Huawei Cloud EulerOS 2.0 is recommended). In the Trusted Settings area, select Enclave.

   

2. Configure the ECS to be managed by CCE and pay attention to the following settings:
   • Storage: Add a data disk of at least 20 GiB of storage space to the ECS for the container runtime and kubelet components.

     

   • Network: Select the VPC where the cluster is located.

     

3. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Nodes. In the right pane, click the Nodes tab and click Accept Node in the upper right corner.
4. Click Select Cloud Server and select the created ECS.

   

5. Install the components that support the QingTian Enclave feature on the node by running the script and enter the following parameters in the Post-installation Command area in Advanced Settings:

   curl -k https://cce-statics.ap-southeast-3.obs.ap-southeast-3.myhuaweicloud.com/job-platform-package/cce-addons/addons-qt-enclave-20250417103811268.zip -o /tmp/addons-qt-enclave.zip
   cd /tmp
   unzip addons-qt-enclave.zip
   cd addons
   bash post_install.sh

   

6. Configure other parameters as required to complete node management. For details about the parameters, see Accepting Nodes for Management.

   After the node has been managed, locate the row containing this node and choose More > View YAML in the Operation column. You can see that there are schedulable QingTian Enclave resources in the YAML file.

   

Building and Using a QingTian Enclave Container Image

1. Prepare an ECS that runs Huawei Cloud EulerOS 2.0 and supports the QingTian Enclave feature and build a QingTian Enclave image file named hello-enclave.eif. For details, see Getting Started with QingTian Enclave.
   1. Log in to the ECS and install Docker for image builds.
      1. Confirm that the repository is configured properly.

         Check whether the parameters in the /etc/yum.repos.d/hce.repo file are configured properly. The proper configuration is as follows:

         [base]
         name=HCE $releasever base
         baseurl=https://repo.huaweicloud.com/hce/$releasever/os/$basearch/
         enabled=1
         gpgcheck=1
         gpgkey=https://repo.huaweicloud.com/hce/$releasever/os/RPM-GPG-KEY-HCE-2
         [updates]
         name=HCE $releasever updates
         baseurl=https://repo.huaweicloud.com/hce/$releasever/updates/$basearch/
         ......

      2. Install the Docker software package.

         yum install docker -y

   2. Install qt CLI, configure parameters for resource isolation in the configuration file as required, and enable resource isolation. To use qt CLI, install the necessary Python libraries. For details, see QingTian CLI (qt CLI).

      yum install qt-enclave-bootstrap
      yum install virtio-qtbox
      yum install qingtian-tool

      Install the required Python libraries.

      pip3 install docker knack

   3. Configure parameters for resource isolation in the configuration file and enable resource isolation. In this practice, 1 GiB of memory and 2 vCPUs are used.

      systemctl start qt-enclave-env

   4. Build a QingTian Enclave image. In this practice, the hello_enclave.sh script is used as the QingTian Enclave application.

      vi hello_enclave.sh

      The content is as follows:

      #!/bin/bash
      while true
      do
          echo "hello enclave!"
          sleep 2
      done

      Create a Dockerfile.

      vi Dockerfile

      The content is as follows:

      FROM ubuntu:latest
      COPY hello_enclave.sh /root/hello_enclave.sh
      CMD ["/root/hello_enclave.sh"]

   5. Check that the script has execution permissions.

      chmod +x hello_enclave.sh 

   6. Build a Docker image named hello-enclave.

      docker build -f Dockerfile -t hello-enclave .

   7. Run the qt enclave make-img command to convert the Docker image to a QingTian Enclave image file named hello-enclave.eif.

      qt enclave make-img --docker-uri hello-enclave --eif hello-enclave.eif

      The output is as follows:

      {
          "digest":   "SHA384",
          "PCR0": "63bf78ece7d2388ff773d0cad2ebc9a3070359db46d567ba271ff8adfb8b0b091be4ff4d5dda3f1c83109096e3656f3b",
          "PCR8": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
      }

2. Use hello-enclave.eif to build a container image. The built container image has the QingTian Enclave feature.
   1. Set up a container build environment.

      mkdir container
      cd container
      cp /usr/local/bin/qt .
      cp /etc/qingtian/enclave/qt-enclave-env.conf .
      cp /root/hello-enclave.eif .

   2. Create a Dockerfile in the container directory.

      vi Dockerfile

      The content is as follows:

      FROM ubuntu
      RUN apt update && \
              apt install -y python3 && \
              apt install -y python3-pip && \
              apt install -y python3-docker && \
              apt install -y python3-knack && \
              apt install -y libcjson-dev && \
              apt install -y libglib2.0-dev && \
              ln -s /usr/bin/python3 /usr/bin/python && \
              mkdir -p  /etc/qingtian/enclave && \
              mkdir -p /var/log/qingtian_enclaves && \
              mkdir -p /var/log/qlog
       
      COPY qt-enclave-env.conf /etc/qingtian/enclave/
      COPY qt /usr/bin
      COPY hello-enclave.eif /
      COPY entrypoint.sh /
      ENTRYPOINT /entrypoint.sh

   3. Create entrypoint.sh in the container directory.

      vi entrypoint.sh

      The content is as follows:

      #!/bin/bash
      qt enclave start --mem 4096 --cpus 2 --eif hello-enclave.eif --cid 4--debug-mode
      qt enclave console --enclave-id 0

   4. Build a container image with the QingTian Enclave feature.

      docker build -f Dockerfile -t enclave-demo .

   5. Push the container image to SWR and use it in a Kubernetes cluster. For details, see Pushing an Image.

Deploying a QingTian Enclave Container on CCE

1. Create an enclave-deployment.yaml file and deploy a workload.
   The following shows an example: (You need to replace the image address with the one pushed to SWR during the image build.)

   kind: Deployment
   apiVersion: apps/v1
   metadata:
     name: hello-enclave
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hello-enclave
         version: v1
     template:
       metadata:
         labels:
           app: hello-enclave
           version: v1
       spec:
         containers:
           - name: container-1
             image: swr.ap-southeast-3.myhuaweicloud.com/container/enclave-demo:latest
             resources:
               limits:
                 cpu: 250m
                 huawei.com/qt_enclaves: '1'
                 hugepages-1Gi: 4Gi
                 memory: 512Mi
               requests:
                 cpu: 250m
                 huawei.com/qt_enclaves: '1'
                 hugepages-1Gi: 4Gi
                 memory: 512Mi
             imagePullPolicy: IfNotPresent
             securityContext:
               seccompProfile:
                 type: Unconfined

2. Create the workload.

   kubectl create -f enclave-deployment.yaml

   Information similar to the following is displayed:

   deployment.apps/hello-enclave created

3. Check the pod status.

   kubectl get pod | grep hello-enclave

   Information similar to the following is displayed:

   hello-enclave-587b9f6d49-bvq22          1/1     Running                  0          38s

4. View the pod logs.

   kubectl logs hello-enclave-587b9f6d49-bvq22 --tail=10

   If information similar to the following is displayed, the QingTian Enclave container is functioning properly:

   hello enclave!
   hello enclave!
   hello enclave!
   hello enclave!
   ...