Creating a Custom CCE Node Image

Constraints

• Suggestions on using CCE node images:
  • You are advised to use the default node images maintained by CCE. These images have passed strict tests and updated in a timely manner, providing better compatibility, stability, and security.
  • Use the base images provided by CCE to create custom images.
  • The component package on which nodes depend for running is preset in the base image. The package version varies with the cluster version. For custom images, CCE does not push component package updates.
• When customizing an image, exercise caution when modifying kernel parameters. Any improper kernel parameter modification will deteriorate the system running efficiency. For details about the reference values, see Modifying Node Kernel Parameters.

  Modifying the following kernel parameters will affect the system performance: tcp_keepalive_time, tcp_max_tw_buckets, somaxconn, max_user_instances, max_user_watches, netdev_max_backlog, net.core.wmem_max, and net.core.rmem_max.

  To modify node kernel parameters, fully verify the modification in a test environment before applying the modification to the production environment.

Precautions

• Before you create an image, prepare:
  • An ECS executor: An ECS x86 server is used as the Linux executor. You are advised to select CentOS7 and bind an EIP to it so that it can access the public network and install Packer.
  • Authentication credentials: Obtain the AK/SK of the tenant or user with required permissions. For details, see How Do I Obtain an Access Key (AK/SK).
  • Security group: Packer creates a temporary ECS and uses a key pair to log in to the ECS using SSH. Ensure that TCP:22 is enabled in the security group. For details, see Security Group Configuration Examples.
• When you create a custom node image, make sure:
  • You follow the instructions in this section to prevent unexpected problems.
  • You have the sudo root or root permissions required to log in to VMs created from base images.
• When the creation is complete:
  • The image creation process uses certain charging resources, including ECSs, EVS disks, EIPs, bandwidth, and IMS images. These resources are automatically released when the image is successfully created or fails to be created. Release the resources in time to ensure no charges are incurred unexpectedly.

Obtaining an Image ID

1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Nodes.
2. Click Create Node in the upper right corner and select OS to Private image.
3. Click View CCE base image information. In the displayed dialog box, copy the image ID.
   

   The image ID varies depending on the region. If the region is changed, obtain the image ID again.

   Figure 1 Obtaining an Image ID
   

Creating a Node Image

1. Download cce-image-builder.
   Log in to the ECS executor, download and decompress cce-image-builder.

   wget https://cce-north-4.obs.cn-north-4.myhuaweicloud.com/cce-image-builder/cce-image-builder.tgz
   
   tar zvxf cce-image-builder.tgz
   cd cce-image-builder/

   

   The cce-image-builder contains:

   • turbo-node.pkr.hcl # Packer configuration template used for creating the image. For details about how to modify the template, see 3.
   • scripts/* # CCE image creation preset in the template. Do not modify it. Otherwise, the image might become unavailable.
   • user-scripts/* # Custom package script directory preset in the template. Take example.sh as an example. When you create a custom image, the image is automatically uploaded to the temporary server and executed.
   • user-packages/* # Custom package directory preset in the template. Take example.package as an example. When you create a custom image, the image is automatically uploaded to /tmp/example.package in the temporary server.

2. Install Packer.
   Download and install the HashiCorp Packer. For details, see Install Packer.

   

   The Packer version needs to be 1.10.0.

   Take the CentOS 7 executor as an example. Run the following command to automatically install Packer (This example is for reference only. For detailed operations, see the official guide):

   # Configure the yum repository and install Packer.
   sudo yum install -y yum-utils
   sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
   sudo yum -y install packer-1.10.0
   
   # Configure an alias to avoid duplicate Packer binary in the OS and check the Packer version.
   rpm -q packer
   alias packer=$(rpm -ql packer)
   packer -v

3. Define Packer template parameters.
   The cce-image-builder/turbo-node.pkr.hcl file defines the process of building an image using Packer. For details, see Packer Documentation.

   

   • Parameters of variables or variable

     turbo-node.pkr.hcl defines the parameters required in the process of building an image. You can configure the parameters based on the live environment. For details, see Table 1.

   • Parameter of packer
     required_plugins defines the add-on dependency of Packer, including the add-on source and version range. When you run packer init, the add-on is automatically downloaded and initialized.

     packer {
       required_plugins {
         huaweicloud = {
           version = ">= 1.0.4"
           source  = "github.com/huaweicloud/huaweicloud"
         }
       }
     }

   • Parameter of source

     The preceding defined variables are referred to automatically configure the parameters required for creating an ECS.

   • Parameter of build

     The scripts are executed from top to bottom. Common modules such as the file upload module and script execution shell module are supported. The corresponding scripts and files are stored in the user-scripts and user-packages directories, respectively, in cce-image-builder.

     Example:

     build {
       sources = ["source.huaweicloud-ecs.builder"]
     
     # Example:
       provisioner "file" {
         source      = "<source file path>"
         destination = "<destination file path>"
       }
     
       provisioner "shell" {
         scripts = [
           "<source script file: step1.sh>",
           "<source script file: step2.sh>"
         ]
       }
     
       provisioner "shell" {
         inline = ["echo foo"]
       }
     }

4. Configure environment variables.
   Configure the following environment variables on the executor:

   export REGION_NAME=xxx
   export IAM_ACCESS_KEY=xxx
   export IAM_SECRET_KEY=xxx
   export ECS_VPC_ID=xxx
   export ECS_NETWORK_ID=xxx
   export ECS_SECGRP_ID=xxx
   export CCE_SOURCE_IMAGE_ID=xxx
   export PKR_VAR_ecs_flavor=xxx
   

   Table 1 Variables configuration

   Parameter	Description	Remarks
   REGION_NAME	Region to which the project belongs	To obtain the region information, go to My Credentials.
   IAM_ACCESS_KEY	Access key for user authentication	Apply for a temporary AK and delete it when the image is built successfully.
   IAM_SECRET_KEY	Secret key for user authentication	Apply for a temporary SK and delete it when the image is built successfully.
   ECS_VPC_ID	VPC ID	Used by the temporary ECS server, which must be the same as that of the executor
   ECS_NETWORK_ID	Network ID of the subnet	Used by the temporary ECS server. It is recommended that the value be the same as that of the executor. It is not the subnet ID.
   ECS_SECGRP_ID	Security group ID	Used by the temporary ECS. The public IP address of the executor must be allowed to pass through port 22 in the inbound direction of the security group to ensure that the executor can log in to the temporary ECS using SSH.
   CCE_SOURCE_IMAGE_ID	Latest CCE node image ID	For details, see Obtaining an Image ID.
   PKR_VAR_ecs_flavor	Specifications of a temporary ECS	Enter a node flavor supported by CCE. The recommended flavor is 2 vCPUs and 4 GiB memory or higher. For details about the flavor name, see A Summary List of x86 ECS Specifications.

   Note: Retain the default values of other parameters. To change the values, refer to the description in the variable definition in turbo-node.pkr.hcl and configure the value using environment variables.

   Use the ECS flavor variable ecs_az as an example. If no AZ is specified, select a random AZ. If you want to specify an AZ, configure an environment variable. The same applies to other parameters.

   # export PKR_VAR_<variable name>=<variable value>
   export PKR_VAR_ecs_az=xxx

5. Customize scripts and files.

   Compile scripts and files by referring to the file and shell modules defined by the build field in the pkr.hcl file, and store the scripts and files in the user-scripts and user-packages directories in cce-image-builder.

   

   When customizing an image, exercise caution when modifying kernel parameters. Any improper kernel parameter modification will deteriorate the system running efficiency. For details about the reference values, see Modifying Node Kernel Parameters.

   Modifying the following kernel parameters will affect the system performance: tcp_keepalive_time, tcp_max_tw_buckets, somaxconn, max_user_instances, max_user_watches, netdev_max_backlog, net.core.wmem_max, and net.core.rmem_max.

   To modify node kernel parameters, fully verify the modification in a test environment before applying the modification to the production environment.

6. Create a custom image.
   After custom parameter settings, create an image. The creation will take 3 to 5 minutes.

   make image

   

   In the encapsulation script packer.sh:

   • Automatic access of hashicorp.com by Packer is disabled by default for privacy protection and security purposes.

     export CHECKPOINT_DISABLE=false

   • The debugging detailed logs option is enabled by default for better visibility and traceability. The local Packer build logs packer_{timestamp}.log is specified so that the logs can be packed to the /var/log/ directory during build. If sensitive information is involved, remove the related logic.

     export PACKER_LOG=1
     export PACKER_BUILD_TIMESTAMP=$(date +%Y%m%d%H%M%S)
     export PACKER_LOG_PATH="packer_$PACKER_BUILD_TIMESTAMP.log"

   For details about Packer configuration, see Configuring Packer.

   After the image is created, information similar to the following will display.

   

7. Clean up build files.
   Clear the build files on the executor, mainly the authentication credentials in turbo-node.pkr.hcl.

   • If the authentication credentials are temporary, directly release the executor.
   • If they are built automatically, add post-processor in the configuration file to execute related operations.

Common Issues

• When the packer is used to create an image, the latest Huawei Cloud ECS open-source plugin is automatically obtained from GitHub. However, this process may fail due to the network environment.

  

  To solve the problem, apply either of the following methods:

  • Create an executor in a region such as CN-Hong Kong with a better network performance and connect the executor to the original region, for example, CN North-Beijing 4, to build a custom image.

    export REGION_NAME=cn-north-4

  • Download the corresponding plugin and initialize it to the local add-on path.

    To obtain the plugin releases, see GitHub.

    If the executor is CentOS 7.6 x86, and the downloaded plugin is packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64.zip, run the following command:

    PLUGIN_PATH="$HOME/.packer.d/plugins/github.com/huaweicloud/huaweicloud"
    mkdir -p $PLUGIN_PATH
    unzip packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64.zip -d /tmp/
    cp /tmp/packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64 $PLUGIN_PATH/
    sha256sum /tmp/packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64 | awk '{print $1}' > $PLUGIN_PATH/packer-plugin-huaweicloud_v1.0.4_x5.0_linux_amd64_SHA256SUM
    ll $PLUGIN_PATH/*

    Then, run the make image command to create an image.

• If the error message "PublicIp type is invalid" is displayed, you need to check the EIP type in different environments, run export PKR_VAR_eip_type='xxx', and create an image. For details about EIP types, see Assigning an EIP.

  An example is as follows:

  export PKR_VAR_eip_type='5_bgp'

• If the error code Ecs.0019 and the error message "Flavor xxxx is abandoned" are displayed, the flavor may be unavailable in the current AZ. Try again or change the flavor in the turbo-node.pkr.hcl file.

  

• If the error message "no such host" is displayed, the current IAM domain name may fail to be resolved.
  Configure the following environment variables on the host. {IAM endpoint} specifies the IAM domain name of the current region.

  export PKR_VAR_auth_url='{IAM endpoint}'