Help Center/ Cloud Container Engine_Autopilot/ User Guide/ Add-ons/ CCE Secrets Manager for DEW

Updated on 2025-07-31 GMT+08:00

View PDF

CCE Secrets Manager for DEW

Introduction

The CCE Secrets Manager for DEW add-on is used to interconnect with Data Encryption Workshop (DEW). This add-on allows you to mount secrets stored outside a cluster (DEW for storing sensitive information) to service pods. In this way, sensitive information can be decoupled from the cluster environment, which prevents information leakage caused by program hardcoding or plaintext configuration. With this add-on, you can enjoy the following capabilities:

Mounting secrets: You can mount secrets in DEW to service pods, reducing information leakage risks.
Scheduled rotation: After a pod runs normally, if the secret declared in the SecretProviderClass object and stored in CSMS is updated, the latest secret values can be updated to the pod through scheduled rotation. When using this capability, set the secret version to latest.
Real-time detection of SecretProviderClass changes: After a pod runs normally, if a user modifies the secret declared in the SecretProviderClass object (for example, a secret is added or the version number is changed), the add-on can detect the change in real time and update the secret to the pod.

Constraints

DEW includes Key Management Service (KMS), Cloud Secret Management Service (CSMS), and Key Pair Service (KPS). Currently, the add-on can interconnect only with CSMS.
The cluster version must be 1.28.8-r0, 1.31.4-r0 or later. If the cluster version does not meet the requirements, you need to upgrade the cluster.
To install this add-on, you need to configure a VPC endpoint for accessing DEW in advance. For details, see Configuring a VPC Endpoint for Accessing DEW.
A maximum of 500 SecretProviderClass objects are allowed by this add-on.
When the add-on is uninstalled, related CRD resources are deleted accordingly. Even if the add-on is reinstalled, the original SecretProviderClass object is unavailable. If you want to use the original SecretProviderClass resources after the add-on is uninstalled and then reinstalled, manually create them again.

Billing

Operation	Billed Item	Billing Description	Reference
Installing the add-on	N/A	When the add-on is installed, pods for running it are not deployed. There are no expenditures for installing the add-on.	-
Using the add-on	Pod	When a secret in DEW is mounted to a service pod, a DaemonSet is automatically created for deploying the pods that are used to run the add-on. Each service pod is configured with a group of add-on pods. For details about the pod specifications for the add-on, see Table 1. When a secret in DEW is no longer mounted to a service pod in the cluster, the pods for running the add-on are automatically deleted. The billing will stop when the service pod is stopped. If a secret in DEW is mounted again, the pods for running the add-on will be automatically redeployed.	For details about pod price, see Billing Items of CCE Autopilot Clusters.

Operation

Billed Item

Billing Description

Reference

Installing the add-on

N/A

When the add-on is installed, pods for running it are not deployed. There are no expenditures for installing the add-on.

Using the add-on

Pod

When a secret in DEW is mounted to a service pod, a DaemonSet is automatically created for deploying the pods that are used to run the add-on. Each service pod is configured with a group of add-on pods. For details about the pod specifications for the add-on, see Table 1.

When a secret in DEW is no longer mounted to a service pod in the cluster, the pods for running the add-on are automatically deleted. The billing will stop when the service pod is stopped. If a secret in DEW is mounted again, the pods for running the add-on will be automatically redeployed.

For details about pod price, see Billing Items of CCE Autopilot Clusters.

Installing the Add-on

Log in to the CCE console and click the cluster name to access the cluster console.
In the navigation pane, choose Clusters > Add-ons. Locate the CCE Secrets Manager for DEW add-on and click Install.

In the Install Add-on dialog box, set Secret Synchronization Period.

Parameter	Field in YAML	Description
Secret Synchronization Period	rotation_poll_interval	Rotation interval, in minutes. The unit is m (not min). This is the interval for sending a request to CSMS and obtaining the latest secret. The interval range is 1 to 1440. The default value is 2.

Parameter

Field in YAML

Description

Secret Synchronization Period

rotation_poll_interval

Rotation interval, in minutes. The unit is m (not min).

This is the interval for sending a request to CSMS and obtaining the latest secret. The interval range is 1 to 1440. The default value is 2.

Click Install in the lower right corner. If the status is Running, the add-on has been installed.

Components

**Table 1** Add-on components
Component	Description	Resource Type	Resource specifications (single pod)
dew-provider	A component that obtains specified secrets from CSMS and mounts them to the pods	DaemonSet	vCPUs: 0.25 Memory: 0.5 GiB
csi-secrets-store	A component that maintains two CRDs, SecretProviderClass and SecretProviderClassPodStatus (spcPodStatus). SecretProviderClass is used to describe the required secret (such as the secret version and name). It is created by users and will be referenced in pods. spcPodStatus is used to trace the binding relationships between pods and secrets. It is automatically created by csi-driver and requires no manual operation. One pod corresponds to one spcPodStatus. After a pod is started, a spcPodStatus is generated for the pod. When the pod lifecycle ends, the spcPodStatus is deleted accordingly.	DaemonSet	vCPUs: 0.25 Memory: 0.5 GiB

Mounting Secrets

To mount a secret in DEW to a container, you need to take the following steps: First, create a ServiceAccount object as the secret for the cluster to access DEW. Then, create a SecretProviderClass object to define and obtain the secret in DEW. Finally, mount the ServiceAccount and SecretProviderClass objects to the service pod to ensure that the pod can securely access the secret. Currently, there are two methods to mount a secret to a pod:

Mounting a secret through a volume: Secrets are mounted to containers in the form of volumes. This method is suitable when there is a large amount of data.
Mounting a secret created from a secret in DEW: Secrets are mounted to containers as environment variables. This method is suitable when there is only a small amount of sensitive data.

The following describes the processes of the two methods.

Create a ServiceAccount object to declare the name of the secret that can be used by the service pod. Before you take this step, ensure that the corresponding secret exists in CSMS. If the secret does not exist, an error occurs when the secret is obtained from CSMS. As a result, the pod cannot run normally. For details about how to create secrets in CSMS, see Creating a Secret.

Figure 1 Secrets in DEW
1. Create a YAML file for the ServiceAccount object.
```
vim serviceaccount.yaml
```
  The following is the file content. You can declare the secret names that can be used by the service pod in the cce.io/dew-resource field. In this example, secret_1 and secret_2 can be referenced in the service pod. If you reference a secret that is not declared here, the mounting will fail. As a result, the pod cannot run normally.
```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-spc-sa
  annotations:
    cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]"  #secrets that allow pod to use
```
2. Create the ServiceAccount object.
```
kubectl apply -f serviceaccount.yaml
```
  Information similar to the following is displayed:
```
serviceaccount/nginx-spc-sa created
```
3. Check whether the ServiceAccount object has been created.
```
kubectl get sa
```
  If the following information is displayed, the ServiceAccount object named nginx-spc-sa has been created and will be referenced in the service pods.
```
NAME            SECRETS   AGE
default         1         18d   # This is the default ServiceAccount object of the system.
nginx-spc-sa    1         19s   # This is the newly created ServiceAccount object.
```

Create a SecretProviderClass object to define and manage secrets from DEW so that secrets can be securely mounted to pods in the CCE cluster.

Create a YAML file for the SecretProviderClass object.

vim secretproviderclass.yaml

The following is the file content. The parameters.objects field specifies the secrets to be mounted, in array format.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce     # The value is fixed at cce.
  parameters:
    objects: |
          - objectName: "secret_1"
            objectAlias: "secret_test"
            objectVersion: "v1"
            objectType: "csms"

**Table 2** Parameters in **objects**
Parameter	Type	Mandatory	Description
objectName	String	Yes	Secret name. Set this parameter to the secrets referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
objectAlias	String	No	File name of the secret written into the container. If this parameter is not specified, the name of the file that the secret is written to the container is objectName by default. If this parameter is specified, the value of objectAlias must be different from that of objectName and of objectAlias of other secrets and its own objectName. Otherwise, the mounting fails.
objectType	String	Yes	Type of the secret. Only csms is supported. Any value other than csms is invalid.
objectVersion	String	Yes	Version of the secret. Specify a version, for example, v1 or v2. Use the latest version, for example, latest. If objectVersion is set to latest and the corresponding secret in CSMS is updated, periodic polling is automatically enabled, and the secret will be updated to the pod after the interval specified by rotation_poll_interval.

Create the SecretProviderClass object.
```
kubectl apply -f secretproviderclass.yaml
```
Information similar to the following is displayed:
```
secretproviderclass/spc-test created
```
Check whether the SecretProviderClass object has been created.
```
kubectl get spc
```
If the following information is displayed, a SecretProviderClass object named spc-test has been created and will be referenced in subsequent pods.
```
NAME   AGE
spc-test   20h
```

Create a service pod. In this example, this pod is used to run an Nginx application.

Define a workload, reference the created ServiceAccount object in serviceAccountName, and reference the created SecretProviderClass object in secretProviderClass, specify the mount path of the container in mountPath. (Do not specify special directories such as / and /var/run. Otherwise, the container may fail to be started.)

vim deployment.yaml

Example file content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-spc
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: nginx-spc-sa  # Reference the created ServiceAccount for identity authentication of secrets.
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io  # Specify the Secrets Store CSI driver. The value is fixed.
            readOnly: true                       # Set the volume to read-only to ensure that the secret is not modified.
            volumeAttributes:
               secretProviderClass: "spc-test" # Reference the created SecretProviderClass.
      containers:
        - name: nginx-spc
          image: nginx:alpine
          imagePullPolicy: IfNotPresent
          volumeMounts:                            # Mount the volume defined above to the container.
            - name: secrets-store-inline           
              mountPath: "/mnt/secrets-store"  # Define the mount path of secrets in the container.
              readOnly: true                       # Read-only
      imagePullSecrets:
        - name: default-secret

Create a service pod.
```
kubectl apply -f deployment.yaml
```
Information similar to the following is displayed:
```
deployment/nginx-spc created
```

Check whether the pod has been created.

kubectl get pod

Information similar to the following is displayed:

NAME                     READY   STATUS   RESTARTS   AGE
nginx-spc-67c9d5b594-642np     1/1     Running    0            20s

Check whether the specified secret can be written.

Enter the container.

kubectl exec -ti nginx-spc-67c9d5b594-642np -- /bin/sh

Search for the specified secret.
```
cd /mnt/secrets-store/
ls
```
If the following information is displayed, secret_test declared in the SecretProviderClass object has been written to the pod.
```
secret_test
```

Obtain spcPodStatus to check whether the secret has been mounted to the pod.

Exit the container using Ctrl+d and obtain the spcps resource name.

kubectl get spcps

Information similar to the following is displayed:

NAME                                          AGE
nginx-spc-67c9d5b594-642np-default-spc-test   103s

Check whether the secret has been mounted to the pod.

kubectl get spcps nginx-spc-67c9d5b594-642np-default-spc-test -o yaml

If the following information is displayed, the secret has been mounted to the pod.

......
status:
mounted: true
objects:    # Mounted secret
- id: secret_test
version: v1
podName: nginx-spc-67c9d5b594-642np   # Pod that references the SecretProviderClass object
secretProviderClassName: spc-test               # SecretProviderClass object
targetPath: /mnt/paas/kubernetes/kubelet/pods/6dd29596-5b78-44fb-9d4c-a5027c420617/volumes/kubernetes.io~csi/secrets-store-inline/mount

Create a ServiceAccount object to declare the name of the secret that can be used by the service pod. Before you take this step, ensure that the corresponding secret exists in CSMS. If the secret does not exist, an error occurs when the secret is obtained from CSMS. As a result, the pod cannot run normally. For details about how to create secrets in CSMS, see Creating a Secret.

Figure 2 Secrets in DEW
1. Create a YAML file for the ServiceAccount object.
```
vim serviceaccount.yaml
```
  The following is the file content. You can declare the secret names that can be used by the service pod in the cce.io/dew-resource field. In this example, secret_1 and secret_2 can be referenced in the service pod. If you reference a secret that is not declared here, the mounting will fail. As a result, the pod cannot run normally.
```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-spc-sa
  annotations:
    cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]"  #secrets that allow pod to use
```
2. Create the ServiceAccount object.
```
kubectl apply -f serviceaccount.yaml
```
  Information similar to the following is displayed:
```
serviceaccount/nginx-spc-sa created
```
3. Check whether the ServiceAccount object has been created.
```
kubectl get sa
```
  If the following information is displayed, the ServiceAccount object named nginx-spc-sa has been created and will be referenced in the service pods.
```
NAME            SECRETS   AGE
default         1         18d   # This is the default ServiceAccount object of the system.
nginx-spc-sa    1         19s   # This is the newly created ServiceAccount object.
```

Create a SecretProviderClass to define and manage the secret from DEW so that the secret can be securely mounted to the pods in the CCE cluster. In a SecretProviderClass object, you can define the required secret information, such as the secret name and version.

Create a YAML file for the SecretProviderClass object.

vim secretproviderclass.yaml

The following is the file content. The parameters.objects field specifies the secrets to be mounted, in array format.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-spc-k8s-secrets
spec:
  provider: cce
  parameters: 
    # Reference a secret in CSMS.
    objects: |
      - objectName: "secret_1"
        objectType: "csms"
        objectVersion: "latest"
        jmesPath:
           - path: key1
             objectAlias: dbusername01
           - path: key2
             objectAlias: dbusername02
  # Create a CCE secret based on the CSMS secret and mount the secret to a pod.
  secretObjects:                
    - secretName: my-secret-01
      type: Opaque
      data:
        - objectName: dbusername01    # The value must be the same as that of objectAlias.
          key: db_username_01
        - objectName: dbusername02    # The value must be the same as that of objectAlias.
          key: db_username_02

**Table 3** Parameters in **objects**
Parameter	Type	Mandatory	Description
objectName	String	Yes	Secret name. Set this parameter to the secret referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
objectType	String	Yes	Type of the secret. Only csms is supported. Any value other than csms is invalid.
objectVersion	String	Yes	Version of the secret. Specify a version, for example, v1 or v2. Use the latest version, for example, latest. If objectVersion is set to latest and the corresponding secret in CSMS is updated, periodic polling is automatically enabled, and the secret will be updated to the pod after the interval specified by rotation_poll_interval.
jmesPath	Array of Object	Yes	jmesPath is used to extract key-value pairs from objects in JSON format. The CCE Secrets Manager for DEW add-on uses this tool to mount secrets to pods. If a secret has multiple keys, you need to enter multiple paths and object aliases. path: Enter the key in DEW. NOTE: The key cannot contain backslashes (\). There are no restrictions on other special characters. objectAlias is name of the file mounted to a pod. The value must be the same as the value of objectName defined in secretObjects.

**Table 4** Parameters in secretObjects
Parameter	Type	Mandatory	Description
secretName	String	Yes	Secret name.
type	String	Yes	Secret type.
data	Array of Object	Yes	objectName: name of the file mounted to the pod. The value must be the same as the value of objectAlias specified in objects. key: key of the secret. You can specify the key value as needed. The key value can be used to reference the encrypted content in the pod.

Create the SecretProviderClass object.

kubectl apply -f secretproviderclass.yaml

Information similar to the following is displayed:

secretproviderclass/nginx-deployment-spc-k8s-secrets created

Check whether the SecretProviderClass object has been created.
```
kubectl get spc
```
If the following information is displayed, a SecretProviderClass object named nginx-deployment-spc-k8s-secrets has been created and will be referenced in subsequent pods.
```
NAME                               AGE
nginx-deployment-spc-k8s-secrets   20h
```

Create a service pod. In this example, this pod is used to run an Nginx application.

vim deployment.yaml

Example file content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-k8s-secrets
  labels:
    app: nginx-k8s-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-k8s-secrets
  template:
    metadata:
      labels:
        app: nginx-k8s-secrets
    spec:
      serviceAccountName: nginx-spc-sa # Reference the created ServiceAccount.
      containers:
      - name: nginx-deployment-k8s-secrets
        image: nginx
        volumeMounts: # Mount the SecretProviderClass to a container.
          - name: secrets-store-inline   # Name of the volume to be mounted
            mountPath: "/mnt/secrets"    # Path of the container where the secret is to be mounted
            readOnly: true
        env: # Set environment variables so that the container can reference the secret during running.
          - name: DB_USERNAME_01     # Name of the environment variable for the workload
            valueFrom:
              secretKeyRef:
                 name: my-secret-01   # Secret name specified in the SecretProviderClass
                key: db_username_01   # The value of the key specified in the SecretProviderClass
          - name: DB_USERNAME_02
            valueFrom:
              secretKeyRef:
                name: my-secret-01
                key: db_username_02
      imagePullSecrets:
        - name: default-secret
      volumes:    # Use the secret specified in the SecretProviderClass to create a volume.
        - name: secrets-store-inline  # Custom volume name
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: nginx-deployment-spc-k8s-secrets   # Name of the SecretProviderClass created in the previous step

Create a service pod.

kubectl apply -f deployment.yaml

Information similar to the following is displayed:

deployment/nginx-deployment-k8s-secrets created

Check whether the pod has been created.

kubectl get pod

Information similar to the following is displayed:

NAME                                              READY   STATUS     RESTARTS     AGE
nginx-deployment-k8s-secrets-6c945565fb-s7smv     1/1     Running    0            20s

Verify the result.

kubectl get secrets

The output shows that a secret named my-secret-01 has been created using secret_1 specified in the SecretProviderClass object.

NAME             TYPE                             DATA   AGE
default-secret   kubernetes.io/dockerconfigjson   1      33d
my-secret-01     Opaque                           2      1h

Scheduled Rotation

After a pod runs normally, if the secret declared in the SecretProviderClass object and stored in CSMS is updated, the latest secret values can be updated to the pod through scheduled rotation. To use this capability, you need to set the secret version to latest. The following is an example:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce
  parameters:
    objects: |
          - objectName: "secret_1"
            objectVersion: "latest"  # change "v1"to "latest"
            objectType: "csms"

After the SecretProviderClass object is updated, the add-on periodically sends a request to CSMS to obtain the value of secret_1 of the latest version and updates the value to the pod that references the SecretProviderClass object. The interval for the add-on to periodically send requests is specified by rotation_poll_interval set in Installing the Add-on.

Real-Time Detection of SecretProviderClass Changes

After a pod runs normally, if a user modifies the secret declared in the SPC object (for example, a secret is added or the version number is changed), the add-on can detect the change in real time and update the secret to the pod. For example, add the secret_2 secret to the SecretProviderClass object. After the SecretProviderClass object is updated, secret_2 is quickly mounted to the pod that references the SecretProviderClass object.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce
  parameters:
    objects: |
          - objectName: "secret_1"
            objectVersion: "latest"
            objectType: "csms"
          - objectName: "secret_2"
            objectVersion: "v1"
            objectType: "csms"

Viewing Component Logs

You can use kubectl to view component logs.

View the pods for running the add-on.

kubectl get pod -n kube-system

Information similar to the following is displayed:

NAME                          READY   STATUS      RESTARTS   AGE
csi-secrets-store-76tj2       3/3     Running     0           11h
dew-provider-hm5fq            1/1     Running     0           11h

View the logs of the dew-provider component.

kubectl logs dew-provider-hm5fq -n kube-system

The command output is the detailed log information.

...
I0321 13:49:29.319708       1 dew_provider.go:43] object [secret_1]'s version [v1] doesn't change, read from local
I0321 13:49:29.319735       1 server.go:108] Start writing into mount point
I0321 13:49:29.319793       1 server.go:117] Write [secret_1/]'s value into file successfully
I0321 13:49:29.319801       1 server.go:123] Write secret file finished
I0321 13:49:29.320284       1 server.go:67] "Servicing mount request" Pod="nginx-deployment-k8s-secrets-6c945565fb-s7smv" Namespace="default" ServiceAccount="nginx-spc-sa"
I0321 13:49:29.617666       1 dew_provider.go:107] "Get secret successfully" name="secret_1" version="v1"
...

View the logs of the csi-secrets-store component. The pod of the csi-secrets-store component contains multiple containers. When viewing logs, run the -c command to specify a container. The secrets-store container is the major service container of the add-on and contains the majority of the logs.

kubectl logs csi-secrets-store-76tj2 -c secrets-store -n kube-system

The command output is the detailed log information.

...
I0321 10:17:29.314888       1 secrets-store.go:46] "Initializing Secrets Store CSI Driver" driver="secrets-store.csi.k8s.io" version="local-dev" buildTime=""
I0321 10:17:29.314987       1 reconciler.go:143] "starting rotation reconciler" rotationPollInterval="2m0s"
I0321 10:17:29.315141       1 server.go:126] "Listening for connections" address="//mnt/paas/kubernetes/kubelet/plugins/csi.sock"
I0321 10:17:30.933328       1 nodeserver.go:359] "node: getting default node info\n"
I0321 10:19:19.269337       1 nodeserver.go:353] "Using gRPC client" provider="cce" pod="nginx-spc-744b4c57dc-x74xq"
I0321 10:19:19.543221       1 secretproviderclasspodstatus_controller.go:224] "reconcile started" spcps="default/nginx-spc-744b4c57dc-x74xq-default-spc-test"
...

Configuring a VPC Endpoint for Accessing DEW

Log in to the VPC Endpoint console.
On the displayed page, click Buy VPC Endpoint.

Configure mandatory parameters as prompted. For details about more parameters, see Buying a VPC Endpoint.

**Table 5** Parameters for creating a VPC endpoint
Parameter	Description
Region	Select the region where the VPC endpoint is located. The VPC endpoint must be in the same region as the CCE Autopilot cluster.
Billing Mode	Only pay-per-use billing is supported. Specifies the billing mode of the VPC endpoint.
Service Category	Select Find a service by name.
VPC Endpoint Service Name	Submit a service ticket to obtain the service name in the current region from DEW engineers.
Create a Private Domain Name	Select this parameter. After the endpoint is created, you can access the endpoint using the private domain name.
VPC	Select the VPC where the CCE Autopilot cluster is running.
Subnet	Select a subnet.
IPv4 Address	Specifies the IPv4 address of the VPC endpoint. You can select Automatically assign IP address or Manually specify IP address.

Click Next to confirm the configuration.
- If the configuration is correct, click Submit.
- If any parameter is incorrect, click Previous to modify it as needed and then click Submit.
Go back to the VPC endpoint list. If the status of the VPC endpoint changes to Accepted, the VPC endpoint is connected to the VPC endpoint service.