Updated on 2025-09-05 GMT+08:00

CCE Container Storage (Everest)

Introduction

Container Storage Interface (CSI) is a storage add-on standard recommended by the Kubernetes community. It is used for unified interconnection between the container orchestration platform and various storage systems. Based on the CSI standard, CCE provides CCE Container Storage (Everest), a self-developed storage add-on. This add-on can seamlessly interconnect with multiple IaaS storage services, including EVS (block storage), SFS, OBS, and SFS Turbo, to provide multiple types of persistent storage capabilities for containers.

CCE Container Storage (Everest) provides diverse persistent storage capabilities for clusters. It can adapt to storage requirements in various service scenarios, such as databases, high-performance computing, shared file access, and large file archiving. This add-on provides a unified storage interface, supports multiple types of storage media, and works closely with CCE scheduling policies, auto scaling strategies, and security systems. It can significantly improve the storage performance and service continuity of cloud native workloads.

Everest is a system resource add-on. It is installed by default when a cluster of Kubernetes v1.15 or later is created.

Notes and Constraints

  • In version 1.2.0 of the Everest add-on, key authentication is optimized when OBS is used. After the Everest add-on is upgraded from a version earlier than 1.2.0, restart all workloads that use OBS in the cluster. Otherwise, workloads may not be able to use OBS.

Installing the Add-on

This add-on has been installed by default. If it is uninstalled due to some reasons, you can reinstall it by performing the following steps:

  1. Log in to the CCE console and click the cluster name to access the cluster console.
  2. In the navigation pane, choose Add-ons. Locate CCE Container Storage (Everest) and click Edit.
  3. On the Install Add-on page, configure the specifications as needed.

    • If you selected Preset, you can choose between Small, Medium, or Large as needed. The system will automatically set the number of add-on pods and resource quotas according to the preset specifications. You can see the configurations on the console.

      The small specification is best for clusters with up to 50 nodes and 500 PVCs. The medium specification works well for clusters with up to 200 nodes and 2000 PVCs. The large specification is perfect for clusters with up to 1000 nodes and 10,000 PVCs.

    • If you selected Custom, you can adjust the number of pods and resource quotas as needed. The requested CPUs and memory can be adjusted based on the number of nodes and PVCs. For details, see Table 1.

      In non-typical scenarios, the formulas for estimating the limits are as follows:

      • everest-csi-controller
        • CPU limit: 250m for 200 or fewer nodes, 350m for 1000 nodes, and 500m for 2000 nodes
        • Memory limit = (200 MiB + Number of nodes x 1 MiB + Number of PVCs x 0.2 MiB) x 1.2
      • everest-csi-driver
        • CPU limit: 300m for 200 or fewer nodes, 500m for 1000 nodes, and 800m for 2000 nodes
        • Memory limit: 300 MiB for 200 or fewer nodes, 600 MiB for 1000 nodes, and 900 MiB for 2000 nodes
      Table 1 Recommended configuration limits in typical scenarios

      Configuration Scenario

      everest-csi-controller

      everest-csi-driver

      Nodes

      PVs/PVCs

      Add-on Pods

      CPU Cores (Limit = Request)

      Memory (Limit = Request)

      CPU Cores (Limit = Request)

      Memory (Limit = Request)

      50

      1000

      2

      250m

      600 MiB

      300m

      300 MiB

      200

      1000

      2

      250m

      1 GiB

      300m

      300 MiB

      1000

      1000

      2

      350m

      2 GiB

      500m

      600 MiB

      1000

      5000

      2

      450m

      3 GiB

      500m

      600 MiB

      2000

      5000

      2

      550m

      4 GiB

      800m

      900 MiB

      2000

      10000

      2

      650m

      5 GiB

      800m

      900 MiB

  4. Configure the add-on parameters.

    Table 2 Add-on parameters

    Item

    Configuration Method

    Description

    Reserved EVS Disks for Non-Container Workloads

    Visualized GUI configuration

    Number of disks on the node reserved for custom use. This parameter is supported when the add-on version is 2.3.11 or later.

    Assume that a maximum of 20 EVS disks can be attached to a node, and the value of this parameter is set to 6. Then 14 (20-6) disks can be attached to this node when the system schedules the EVS disk attachment workloads. The reserved six disks include one system disk and one data disk that are already attached to the node. You can attach four EVS disks to this node as additional data disks or raw disks for a local storage pool.

    Prohibit Global Secret from Mounting Object Storage

    Visualized GUI configuration

    Whether the default AK/SK can be used when an object bucket or parallel file system is mounted.

    • is: If you do not specify a custom secret for mounting OBS storage, the global secret you uploaded will be used.
    • No: If you do not specify a custom secret for mounting OBS storage, only the global secret you uploaded can be used. Otherwise, the mounting will fail.

    Concurrent EVS Disk Attaching Tasks

    Visualized GUI configuration

    Number of workers that can be concurrently processed by Everest for attaching EVS volumes. The default value is 60.

    Concurrent EVS Disk Detaching Tasks

    Visualized GUI configuration

    Number of workers that can be concurrently processed by Everest for detaching EVS volumes. The default value is 60.

    Distributed Volume Mounting

    Visualized GUI configuration

    When enabled, the everest-csi-driver component on each node is responsible for attaching or detaching EVS disks.

    If disabled, the everest-csi-controller component takes on this responsibility.

    flow_control

    Extended parameter settings

    The default configuration is used. After installing the CCE Container Storage (Everest) add-on, you can refer to ConfigMap everest-driver-th-config in the kube-system namespace for more details.

    over_subscription

    Extended parameter settings

    Overcommitment ratio of the local storage pool (local_storage). The default value for the local storage pool size is set to 80. If the pool size is 100 GiB, it can be overcommitted up to 180 GiB. (Total capacity after overcommitment = (1 + Overcommitment ratio) x Actual capacity)

    enable_local_autoexpander

    Extended parameter settings

    Whether to enable automatic scale-out for the local storage pool of thin volumes. If this function is enabled, automatic scale-out is triggered based on the scale-out threshold and scale-out step of the local storage pool.

    expansion_threshold

    Extended parameter settings

    Capacity expansion threshold of the local storage pool of the thin volume. When the usage of the local storage pool of the thin volume exceeds the threshold, the local storage pool is automatically expanded.

    expansion_step

    Extended parameter settings

    Capacity expansion step of a single EVS disk in the local storage pool of thin provisioning volumes (unit: Gi)

    expansion_max_evs_size

    Extended parameter settings

    Maximum capacity of a single EVS disk in the local storage pool of thin provisioning volumes (unit: Gi)

    volume_attaching_flow_ctrl

    Extended parameter settings

    Maximum number of EVS volumes that can be attached by the Everest add-on within 1 minute. The default value is 0, indicating that the performance of attaching EVS volumes is determined by the underlying storage resources.

    In the extended parameter settings, you can customize the advanced configurations that are not displayed on the GUI. If the settings in the extended parameters conflict with those on the GUI, the settings in the extended parameters will work.

  5. Configure deployment policies for the add-on pods.

    • Scheduling policies do not take effect on add-on pods of the DaemonSet type.
    • When configuring multi-AZ deployment or node affinity, ensure that there are nodes meeting the scheduling policy and that resources are sufficient in the cluster. Otherwise, the add-on cannot run.
    Table 3 Configurations for add-on scheduling

    Parameter

    Description

    Multi-AZ Deployment

    • Preferred: Deployment pods of the add-on will be preferentially scheduled to nodes in different AZs. If all the nodes in the cluster are deployed in the same AZ, the pods will be scheduled to different nodes in that AZ.
    • Equivalent mode: Deployment pods of the add-on are evenly scheduled to the nodes in the cluster in each AZ. If a new AZ is added, you are advised to increase add-on pods for cross-AZ HA deployment. With the Equivalent multi-AZ deployment, the difference between the number of add-on pods in different AZs will be less than or equal to 1. If resources in one of the AZs are insufficient, pods cannot be scheduled to that AZ.
    • Forcible: Deployment pods of the add-on are forcibly scheduled to nodes in different AZs. There can be at most one pod in each AZ. If nodes in a cluster are not in different AZs, some add-on pods cannot run properly. If a node is faulty, add-on pods on it may fail to be migrated.

    Node Affinity

    • Not configured: Node affinity is disabled for the add-on.
    • Specify node: Specify the nodes where the add-on is deployed. If you do not specify the nodes, the add-on will be randomly scheduled based on the default cluster scheduling policy.
    • Specify node pool: Specify the node pool where the add-on is deployed. If you do not specify the node pools, the add-on will be randomly scheduled based on the default cluster scheduling policy.
    • Customize affinity: Enter the labels of the nodes where the add-on is to be deployed for more flexible scheduling policies. If you do not specify node labels, the add-on will be randomly scheduled based on the default cluster scheduling policy.

      If multiple custom affinity policies are configured, ensure that there are nodes that meet all the affinity policies in the cluster. Otherwise, the add-on cannot run.

    Toleration

    Using both taints and tolerations allows (not forcibly) the add-on Deployment to be scheduled to a node with the matching taints, and controls the Deployment eviction policies after the node where the Deployment is located is tainted.

    The add-on adds the default tolerance policy for the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints, respectively. The tolerance time window is 60s.

    For details, see Configuring Tolerance Policies.

  6. Click Install.

Components

Table 4 Add-on components

Component

Description

Resource Type

everest-csi-controller

Used to create, delete, snapshot, expand, attach, and detach storage volumes. If the cluster version is 1.19 or later and the add-on version is 1.2.x, the pod of the everest-csi-controller component also has an everest-localvolume-manager container by default. This container manages the creation of LVM storage pools and local PVs on the node.

Deployment

everest-csi-driver

Used to mount and unmount PVs and resize file systems. If the add-on version is 1.2.x and the region where the cluster is located supports node-attacher, the pod of the everest-csi-driver component also contains an everest-node-attacher container. This container is responsible for distributed EVS attaching. This configuration item is available in some regions.

DaemonSet

Release History

Table 5 CCE Container Storage (Everest) add-on

Add-on Version

Supported Cluster Version

New Feature

2.4.134

v1.25

v1.27

v1.28

v1.29

v1.30

v1.31

Fixed some issues.

2.4.75

v1.23

v1.25

v1.27

v1.28

v1.29

v1.30

Fixed some issues.

2.4.28

v1.23

v1.25

v1.27

v1.28

v1.29

Fixed some issues.

2.3.14

v1.21

v1.23

v1.25

v1.27

v1.28

CCE clusters v1.28 are supported.

2.1.51

v1.19

v1.21

v1.23

v1.25

v1.27

Supported Huawei Cloud EulerOS 2.0.

2.1.38

v1.19

v1.21

v1.23

v1.25

Supported association between add-on specifications and cluster specifications.

2.1.30

v1.19

v1.21

v1.23

v1.25

  • Supported anti-affinity scheduling of add-on pods on nodes in different AZs.
  • Adapted the obsfs package to Ubuntu 22.04.

2.1.13

v1.19

v1.21

v1.23

v1.25

Optimized the performance of creating subPath PVCs in batches for SFS Turbo volumes.

2.1.9

v1.19

v1.21

v1.23

v1.25

  • Supported graceful exit of the controller.
  • CCE clusters v1.25 are supported.

2.0.9

v1.19

v1.21

v1.23

  • Rebuilt certain code and architecture of everest to improve its scalability and stability.
  • Enabled graceful exit.
  • Supported OBS process monitoring.

1.3.28

v1.19

v1.21

v1.23

  • Enabled graceful exit.
  • Supported OBS process monitoring.

1.3.22

v1.19

v1.21

v1.23

Handled occasional read and write failures after repeated disk mounting.

1.3.20

v1.19

v1.21

v1.23

Handled occasional read and write failures after repeated disk mounting.

1.3.17

v1.19

v1.21

v1.23

  • Changed the maximum number of unavailable everest-csi-driver pods during a rolling update from 10 to 10%.
  • Supported custom pod anti-affinity rules.
  • Counted the maximum number of SCSI volumes that can be managed by the CSI add-on on a node.
  • Drivers can be deployed based on custom resource specifications.

1.3.8

v1.23

CCE clusters v1.23 are supported.

1.3.6

v1.23

CCE clusters v1.23 are supported.

1.2.78

v1.15

v1.17

v1.19

v1.21

Supported anti-affinity scheduling of add-on pods on nodes in different AZs.

1.2.70

v1.15

v1.17

v1.19

v1.21

Optimized the performance of creating subPath PVCs in batches for SFS Turbo volumes.

1.2.67

v1.15

v1.17

v1.19

v1.21

  • Supported graceful exit of the controller.
  • Supported OBS process monitoring.

1.2.61

v1.15

v1.17

v1.19

v1.21

  • Enabled graceful exit.
  • Supported OBS process monitoring.

1.2.55

v1.15

v1.17

v1.19

v1.21

Handled occasional read and write failures after repeated disk mounting.

1.2.53

v1.15

v1.17

v1.19

v1.21

Handled occasional read and write failures after repeated disk mounting.

1.2.51

v1.15

v1.17

v1.19

v1.21

  • Changed the maximum number of unavailable everest-csi-driver pods during a rolling update from 10 to 10%.
  • Supported custom pod anti-affinity rules.
  • Counted the maximum number of SCSI volumes that can be managed by the CSI add-on on a node.

1.2.44

v1.15

v1.17

v1.19

v1.21

  • Enterprise projects can be selected for EVS and OBS volumes.
  • By default, the enable_noobj_cache parameter is no longer used for mounting OBS buckets.

1.2.42

v1.15

v1.17

v1.19

v1.21

  • Enterprise projects can be selected for EVS and OBS volumes.
  • By default, the enable_noobj_cache parameter is no longer used for mounting OBS buckets.

1.2.30

v1.15

v1.17

v1.19

v1.21

Supported emptyDir.

1.2.28

v1.15

v1.17

v1.19

v1.21

CCE clusters v1.21 are supported.

1.2.27

v1.15

v1.17

v1.19

v1.21

Supported extreme SSD (ESSD) and general-purpose SSD (GPSSD) EVS disks.

1.2.13

v1.15

v1.17

v1.19

Supported EulerOS 2.10.

1.2.9

v1.15

v1.17

v1.19

  • Enhanced the reliability of PV lifecycle maintenance.
  • Attach/Detach Controller can be used to attach or detach volumes in clusters v1.19.10.
  • Improved SFS mounting stability.
  • Changed the default EVS creation type of a new cluster to SAS.

1.2.5

v1.15

v1.17

v1.19

  • Improved the reliability of mounting-related capabilities.
  • Optimized the authentication function of using OBS, which requires you to upload the access key.
  • Improved the compatibility of the everest add-on with FlexVolume volumes.
  • Improved running stability of the add-on.

1.1.12

v1.15

v1.17

Enhanced the reliability of the everest-csi-controller component.

1.1.11

v1.15

v1.17

  • Supported security hardening.
  • Supported third-party OBS storage.
  • Switched to the EVS query API with better performance.
  • Disks can be created from snapshots using clone by default.
  • Optimized and enhanced disk status detection and log output for attaching and detaching operations.
  • Improved the reliability of determining authentication expiration.

1.1.8

v1.15

v1.17

Supported CCE v1.17. If CCE v1.13 is upgraded to v1.15, Everest can take over all functions of FlexVolume.

1.1.7

v1.15

v1.17

Supported CCE v1.17. If CCE v1.13 is upgraded to v1.15, Everest can take over all functions of FlexVolume.