Creating a Custom CCE Node Image

Notes and Constraints

• Supported OSs and kernels are listed below.

  OS	Kernel Version
  CentOS 7.6	≥ 3.10.0-1160.15.2.el7.x86_64
  Ubuntu 18.04	≥ 4.15.0-136-generic
  Ubuntu 22.04	≥ 5.15.0-86-generic
  EulerOS 2.9 (x86)	≥ 4.18.0-147.5.1.6.h1071.eulerosv2r9.x86_64
  EulerOS 2.9 (Arm)	≥ 4.19.90-vhulk2103.1.0.h1060.eulerosv2r9.aarch64
  Huawei Cloud EulerOS 2.0 (x86)	≥ 5.10.0-60.18.0.50.r1002_48.hce2.x86_64
  Huawei Cloud EulerOS 2.0 (Arm)	≥ 5.10.0-60.18.0.50.r1002_48.hce2.aarch64

• Suggestions on using CCE node images:
  • You are advised to use the default node images maintained by CCE. These images have undergone strict tests and are updated in a timely manner. They provide better compatibility, stability, and security.
  • To use the custom image function in certain scenarios, you can use the node image IDs provided by CCE to create custom images. You cannot directly export an ECS public image or the image of a node that is currently running as a custom image.
• When customizing an image, exercise caution when modifying kernel parameters. Any improper kernel parameter modification will deteriorate the system efficiency. For details about the reference values, see Modifying Node Kernel Parameters.

  Modifying any of the following kernel parameters will affect the system performance: tcp_keepalive_time, tcp_max_tw_buckets, somaxconn, max_user_instances, max_user_watches, netdev_max_backlog, net.core.wmem_max, and net.core.rmem_max.

  To modify node kernel parameters, fully verify the modification in a testing environment before applying the modification to the production environment.

Precautions

• Before you create an image, prepare:
  • An ECS executor: An ECS x86 server is used as the Linux executor. You are advised to select Huawei Cloud EulerOS 2.0 and bind an EIP to the server so that it can access the Internet and be used to install Packer.
  • Authentication credentials: Obtain the AK/SK of the tenant or user with required permissions. For details, see How Do I Obtain an Access Key (AK/SK)?
  • A security group: Packer creates a temporary ECS and uses a key pair to log in to the ECS using SSH. Ensure that TCP:22 is enabled in the security group. For details, see Security Group Configuration Examples.
• When you create a custom node image, make sure:
  • You follow the instructions in this section to prevent unexpected problems.
  • You have the sudo root or root permissions required to log in to VMs created from base images.
• After the creation is complete:
  • The image creation process uses certain charging resources, including ECSs, EVS disks, EIPs, bandwidth, and IMS images. These resources are automatically released when the image is successfully created or fails to be created. Release the resources in time to ensure no charges are incurred unexpectedly.

Obtaining an Image ID

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Nodes.
2. Click Create Node in the upper right corner and set OS to Private image.
3. Click CCE Base Images. In the displayed dialog box, copy the image ID.
   

   The image ID varies depending on the region. If the region is changed, obtain the image ID again.

   Figure 1 Obtaining an Image ID
   

Creating a Node Image

1. Download cce-image-builder.
   Log in to the ECS executor and download and decompress cce-image-builder.

   wget https://cce-north-4.obs.cn-north-4.myhuaweicloud.com/cce-image-builder/cce-image-builder.tgz
   
   tar zvxf cce-image-builder.tgz
   cd cce-image-builder/

   

   The cce-image-builder contains:

   • turbo-node.pkr.hcl # Packer configuration template used for creating the image. For details about how to modify the template, see 3.
   • scripts/* # CCE image creation preset in the template. Do not modify it. Otherwise, the image might become unavailable.
   • user-scripts/* # Custom package script directory preset in the template. Take example.sh as an example. When you create a custom image, the image is automatically uploaded to the temporary server and executed.
   • user-packages/* # Custom package directory preset in the template. Take example.package as an example. When you create a custom image, the image is automatically uploaded to /tmp/example.package on the temporary server.

2. Install Packer.
   Manually download and install HashiCorp Packer. For details, see the official documentation.

   

   The Packer version needs to be 1.10.0.

   Install packer. A Huawei Cloud EulerOS 2.0 executor is used as an example.

   wget https://releases.hashicorp.com/packer/1.10.0/packer_1.10.0_linux_amd64.zip
   unzip packer_1.10.0_linux_amd64.zip
   cp -f packer /usr/bin/packer

3. Define Packer template parameters.
   The cce-image-builder/turbo-node.pkr.hcl file defines the process of building an image using Packer. For details, see Packer Documentation.

   

   • variables or variable

     turbo-node.pkr.hcl defines the parameters required in the process of image builds. You can configure the parameters based on the live environment. For details, see Table 1.

   • packer
     required_plugins defines the add-on dependency of Packer, including the add-on source and version range. When you run packer init, the add-on is automatically downloaded and initialized.

     packer {
       required_plugins {
         huaweicloud = {
           version = ">= 1.0.4"
           source  = "github.com/huaweicloud/huaweicloud"
         }
       }
     }

   • source

     The preceding defined variables are referred to automatically configure the parameters required for creating an ECS.

   • build

     The scripts are executed from top to bottom. Common modules such as the file upload module and script execution shell module are supported. The corresponding scripts and files are stored in the user-scripts and user-packages directories, respectively, in cce-image-builder.

     Example:

     build {
       sources = ["source.huaweicloud-ecs.builder"]
     
     # Example:
       provisioner "file" {
         source      = "<source file path>"
         destination = "<destination file path>"
       }
     
       provisioner "shell" {
         scripts = [
           "<source script file: step1.sh>",
           "<source script file: step2.sh>"
         ]
       }
     
       provisioner "shell" {
         inline = ["echo foo"]
       }
     }

4. Configure environment variables.
   Configure the following environment variables on the executor:

   export REGION_NAME=xxx
   export IAM_ACCESS_KEY=xxx
   export IAM_SECRET_KEY=xxx
   export ECS_VPC_ID=xxx
   export ECS_NETWORK_ID=xxx
   export ECS_SECGRP_ID=xxx
   export CCE_SOURCE_IMAGE_ID=xxx
   export PKR_VAR_ecs_flavor=xxx
   

   Table 1 Variables configuration

   Parameter	Description	Remarks
   REGION_NAME	Region to which the project belongs	To obtain the region information, go to My Credentials.
   IAM_ACCESS_KEY	Access key for user authentication	Apply for a temporary AK and delete it after the image has been built.
   IAM_SECRET_KEY	Secret key for user authentication	Apply for a temporary SK and delete it after the image has been built.
   ECS_VPC_ID	VPC ID	Used by the temporary ECS server, which must be the same as the one used by the executor
   ECS_NETWORK_ID	Network ID of the subnet	Used by the temporary ECS server. It is recommended that the value be the same as that of the executor. It is not the subnet ID.
   ECS_SECGRP_ID	Security group ID	Used by the temporary ECS. The public IP address of the executor must be allowed to pass through port 22 in the inbound direction of the security group to ensure that the executor can log in to the temporary ECS using SSH.
   CCE_SOURCE_IMAGE_ID	The latest CCE node image ID	For details, see Obtaining an Image ID.
   PKR_VAR_ecs_flavor	Specifications of a temporary ECS	Enter a node flavor supported by CCE. The recommended flavor is 2 vCPUs and 4-GiB memory or higher. For details about flavor names, see A Summary List of x86 ECS Specifications and A Summary List of Kunpeng ECS Specifications.

   Retain the default values for other parameters. To change the values, refer to the description in the variable definition in turbo-node.pkr.hcl and configure the value using environment variables.

   The ECS flavor variable ecs_az is used as an example. If no AZ is specified, select a random AZ. If you want to specify an AZ, configure an environment variable. The same applies to other parameters.

   # export PKR_VAR_<variable name>=<variable value>
   export PKR_VAR_ecs_az=xxx

5. Customize scripts and files.

   Write scripts and files by referring to the file and shell modules defined by the build field in the pkr.hcl file, and store the scripts and files in the user-scripts and user-packages directories in cce-image-builder.

   

   When customizing an image, exercise caution when modifying kernel parameters. Any improper kernel parameter modification will deteriorate the system efficiency. For details about the reference values, see Modifying Node Kernel Parameters.

   Modifying any of the following kernel parameters will affect the system performance: tcp_keepalive_time, tcp_max_tw_buckets, somaxconn, max_user_instances, max_user_watches, netdev_max_backlog, net.core.wmem_max, and net.core.rmem_max.

   To modify node kernel parameters, fully verify the modification in a testing environment before applying the modification to the production environment.

6. Create the custom image.
   After the parameter settings, create the image. This will take 3 to 5 minutes.

   make image

   

   In the encapsulation script packer.sh:

   • Automatic access of hashicorp.com by Packer is disabled by default for privacy protection and security purposes.

     export CHECKPOINT_DISABLE=false

   • The debugging detailed logs option is enabled by default for better visibility and traceability. The local Packer build logs packer_{timestamp}.log is specified so that the logs can be packed to the /var/log/ directory during build. If sensitive information is involved, remove the related logic.

     export PACKER_LOG=1
     export PACKER_BUILD_TIMESTAMP=$(date +%Y%m%d%H%M%S)
     export PACKER_LOG_PATH="packer_$PACKER_BUILD_TIMESTAMP.log"

   For details about Packer configuration, see Configure Packer.

   After the image is created, information similar to that shown below will display.

   

7. Clear build files.
   Clear the build files on the executor, mainly the authentication credentials in turbo-node.pkr.hcl.

   • If the authentication credentials are temporary, directly release the executor.
   • If they are built automatically, add post-processor in the configuration file to execute related operations.

Common Issues

• When Packer is used to create an image, the latest Huawei Cloud ECS open-source plugin is automatically obtained from GitHub. However, this process may fail due to the network environment.

  

  To solve the problem, apply either of the following methods:

  • Create an executor in a region such as CN-Hong Kong with a better network performance and connect the executor to the original region, for example, CN North-Beijing 4, to build a custom image.

    export REGION_NAME=cn-north-4

  • Download the corresponding plugin and initialize it to the local add-on path.

    To obtain the plugin releases, see GitHub.

    If the executor is x86, and the downloaded plugin is packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64.zip, initialize the plugin:

    PLUGIN_PATH="$HOME/.packer.d/plugins/github.com/huaweicloud/huaweicloud"
    mkdir -p $PLUGIN_PATH
    unzip packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64.zip -d /tmp/
    cp /tmp/packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64 $PLUGIN_PATH/
    sha256sum /tmp/packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64 | awk '{print $1}' > $PLUGIN_PATH/packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64_SHA256SUM
    ll $PLUGIN_PATH/*

    Then, run the make image command to create the image.

• If the error message "PublicIp type is invalid" is displayed, you need to check the EIP type in different environments, run export PKR_VAR_eip_type='xxx', and create an image. For details about EIP types, see Assigning an EIP.

  An example is as follows:

  export PKR_VAR_eip_type='5_bgp'

• If the error code Ecs.0019 and the error message "Flavor xxxx is abandoned" are displayed, the flavor may be unavailable in the current AZ. Try again or change the flavor in the turbo-node.pkr.hcl file.

  

• If the error message "no such host" is displayed, the current IAM domain name may fail to be resolved.
  Configure the environment variable below on the host. {IAM endpoint} specifies the IAM domain name of the current region.

  export PKR_VAR_auth_url='{IAM endpoint}'