CCE Secrets Manager for DEW
Introduction
The dew-provider add-on is used to interconnect with Data Encryption Workshop (DEW), which allows you to mount secrets stored outside a cluster (DEW for storing sensitive information) to pods. In this way, sensitive information can be decoupled from the cluster environment, which prevents information leakage caused by program hardcoding or plaintext configuration.
Notes and Constraints
- DEW includes Key Management Service (KMS), Cloud Secret Management Service (CSMS), and Key Pair Service (KPS). Currently, the dew-provider add-on can interconnect only with CSMS.
- A maximum of 500 SecretProviderClass objects can be created.
- When the add-on is uninstalled, related CRD resources are deleted accordingly. Even if the add-on is reinstalled, the original SecretProviderClass object is unavailable. If you want to use the original SecretProviderClass resources after the add-on is uninstalled and then reinstalled, manually create them again.
How the Add-on Works
- Basic mounting: After the dew-provider add-on is installed, you can create a SecretProviderClass object and declare and reference the volume in a pod. When the pod is started, the secret declared in the SecretProviderClass object is mounted to the pod.
- Scheduled rotation: After a pod runs properly, if the secret declared in the SPC object and stored in CSMS is updated, the latest secret values can be updated to the pod through scheduled rotation. When using this capability, set the secret version to latest.
- Real-time awareness of SPC changes: After a pod runs properly, if a user modifies the secret declared in the SPC object (for example, a secret is added or the version number is changed), the add-on can detect the change in real time and update the secret to the pod.
Installing the Add-on
- Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons, locate CCE Secrets Manager for DEW on the right, and click Install.
- On the Install Add-on page, configure parameters in the Parameters area, as listed in the following table.
Item
Parameter
Description
Secret Synchronization Period
rotation_poll_interval
Rotation interval, in unit of m (instead of min).
The rotation interval indicates the interval for sending a request to CSMS and obtaining the latest secret. The proper interval range is [1m, 1440m]. The default value is 2m.
- Click Install.
After the add-on is installed, select the cluster and choose Add-ons in the navigation pane. On the displayed page, view the add-on in the Add-ons Installed area.
Components
Component |
Description |
Resource Type |
---|---|---|
dew-provider |
A component that obtains specified secrets from CSMS and mounts them to the pods |
DaemonSet |
csi-secrets-store |
A component that maintains two CRDs, SecretProviderClass (SPC) and SecretProviderClassPodStatus (spcPodStatus). SPC is used to describe the secret that users are interested in (such as the secret version and name). It is created by users and will be referenced in pods. spcPodStatus is used to trace the binding relationships between pods and secrets. It is automatically created by csi-driver and requires no manual operation. One pod corresponds to one spcPodStatus. After a pod is started, a spcPodStatus is generated for the pod. When the pod lifecycle ends, the spcPodStatus is deleted accordingly. |
DaemonSet |
Mounting a Credential Using a Volume
- Create a ServiceAccount.
- Create a ServiceAccount object, which declares the secret names that can be used by services. If a user references a secret that is not declared here, the mounting will fail. As a result, the pod cannot run.
Create the serviceaccount.yaml file based on the template below, and declare the secret names that can be used by services in the cce.io/dew-resource field. Here, secret_1 and secret_2 are declared, indicating that the service is allowed to reference two secrets. In subsequent operations, if the user references secret_3 in the service, the verification fails. As a result, the secret cannot be mounted and the pod cannot run.
apiVersion: v1 kind: ServiceAccount metadata: name: nginx-spc-sa annotations: cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]" #secrets that allow pod to use
Ensure that the secrets declared here exist in CSCM, as shown in the following figure. Otherwise, even if the verification is successful, an error occurs when the corresponding secret is obtained from CSCM. As a result, the pod cannot run properly.
- Run the following command to create the ServiceAccount:
- Check whether the ServiceAccount object is successfully created.
$ kubectl get sa NAME SECRETS AGE default 1 18d # This is the default ServiceAccount object of the system. nginx-spc-sa 1 19s # This is the newly created ServiceAccount object.
A ServiceAccount object named nginx-spc-sa has been created. This object will be referenced in pods.
- Create a ServiceAccount object, which declares the secret names that can be used by services. If a user references a secret that is not declared here, the mounting will fail. As a result, the pod cannot run.
- Create a SecretProviderClass.
- The SecretProviderClass object is used to describe the secret information (such as the version and name) that users are interested in. It is created by users and will be referenced in pods.
Create the secretproviderclass.yaml file using the template below. Pay attention to the objects field in parameters, which is an array used to declare the secret to be mounted.
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: spc-test spec: provider: cce # The value is fixed at cce. parameters: objects: | - objectName: "secret_1" objectVersion: "v1" objectType: "csms"
Parameter
Type
Mandatory
Description
objectName
String
Yes
Credential name. Set this parameter to the secret referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
objectAlias
String
No
File name of the secret written into the container. If this parameter is not specified, the file name of the secret written into the container is the value of objectName by default. If this parameter is specified, the value must be different from objectName and from the objectAlias and objectName values of other secrets. Otherwise, the mounting fails.
objectType
String
Yes
Secret type. Only csms is supported. A value other than csms is invalid.
objectVersion
String
Yes
Secret version
- Specify a version, for example, v1 or v2.
- Use the latest version, for example, latest. When objectVersion is set to latest, if the corresponding secret in CSCM is updated, the secret will be updated to the pod after a certain interval (rotation_poll_interval).
- Run the following command to create a SecretProviderClass object:
- Check whether the SecretProviderClass object has been created.
$ kubectl get spc NAME AGE spc-test 20h
A SecretProviderClass object named spc-test is created. This object will be referenced in pods subsequently.
- The SecretProviderClass object is used to describe the secret information (such as the version and name) that users are interested in. It is created by users and will be referenced in pods.
- Create a pod.
The following describes how to create an Nginx application.
- Define a workload, reference the created ServiceAccount object in serviceAccountName, and reference the created SPC object in secretProviderClass, specify the mount path of the container in mountPath. (Do not specify special directories such as / and /var/run. Otherwise, the container may fail to be started.)
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-spc labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: serviceAccountName: nginx-spc-sa # Reference the created ServiceAccount. volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "spc-test" # Reference the created SPC. containers: - name: nginx-spc image: nginx:alpine imagePullPolicy: IfNotPresent volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" # Define the mount path of secrets in the container. readOnly: true imagePullSecrets: - name: default-secret
- Create a pod.
kubectl apply -f deployment.yaml
- Check whether the pod has been created.
$ kubectl get pod NAME READY STATUS RESTARTS AGE nginx-spc-67c9d5b594-642np 1/1 Running 0 20s
- Access the container and check whether the specified secret is written properly. For example:
$ kubectl exec -ti nginx-spc-67c9d5b594-642np -- /bin/bash root@nginx-spc-67c9d5b594-642np:/# root@nginx-spc-67c9d5b594-642np:/# cd /mnt/secrets-store/ root@nginx-spc-67c9d5b594-642np:/mnt/secrets-store# root@nginx-spc-67c9d5b594-642np:/mnt/secrets-store# ls secret_1
The command output shows that secret_1 declared in the SPC object has been written to the pod.
In addition, you can obtain spcPodStatus to check the binding relationship between pods and secrets. For example:
$ kubectl get spcps NAME AGE nginx-spc-67c9d5b594-642np-default-spc-test 103s $ kubectl get spcps nginx-spc-67c9d5b594-642np-default-spc-test -o yaml ...... status: mounted: true objects: # Mounted secret - id: secret_1 version: v1 podName: nginx-spc-67c9d5b594-642np # Pod that references the SPC object secretProviderClassName: spc-test # SPC object targetPath: /mnt/paas/kubernetes/kubelet/pods/6dd29596-5b78-44fb-9d4c-a5027c420617/volumes/kubernetes.io~csi/secrets-store-inline/mount
- Define a workload, reference the created ServiceAccount object in serviceAccountName, and reference the created SPC object in secretProviderClass, specify the mount path of the container in mountPath. (Do not specify special directories such as / and /var/run. Otherwise, the container may fail to be started.)
Mounting a Credential Using a Secret
CCE Secrets Manager for DEW must be 1.1.1 or later.
- Create a ServiceAccount.
- Create a ServiceAccount object, which declares the secret names that can be used by services. If a user references a secret that is not declared here, the mounting will fail. As a result, the pod cannot run.
Create the serviceaccount.yaml file based on the template below, and declare the secret names that can be used by services in the cce.io/dew-resource field. Here, secret_1 and secret_2 are declared, indicating that the service is allowed to reference two secrets. In subsequent operations, if the user references secret_3 in the service, the verification fails. As a result, the secret cannot be mounted and the pod cannot run.
apiVersion: v1 kind: ServiceAccount metadata: name: nginx-spc-sa annotations: cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]" #secrets that allow pod to use
Ensure that the secrets declared here exist in CSCM, as shown in the following figure. Otherwise, even if the verification is successful, an error occurs when the corresponding secret is obtained from CSCM. As a result, the pod cannot run properly.
- Run the following command to create the ServiceAccount:
- Check whether the ServiceAccount object is successfully created.
$ kubectl get sa NAME SECRETS AGE default 1 18d # This is the default ServiceAccount object of the system. nginx-spc-sa 1 19s # This is the newly created ServiceAccount object.
A ServiceAccount object named nginx-spc-sa has been created. This object will be referenced in pods.
- Create a ServiceAccount object, which declares the secret names that can be used by services. If a user references a secret that is not declared here, the mounting will fail. As a result, the pod cannot run.
- Create a SecretProviderClass.
- Create the secretproviderclass.yaml file using the template below.
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: nginx-deployment-spc-k8s-secrets spec: provider: cce parameters: # Reference a secret in CSMS. objects: | - objectName: "secret_1" objectType: "csms" objectVersion: "latest" jmesPath: - path: username objectAlias: dbusername - path: password objectAlias: dbpassword # Create a CCE secret based on the CSMS secret and mount the secret to a pod. secretObjects: - secretName: my-secret-01 type: Opaque data: - objectName: dbusername key: db_username_01 - objectName: dbpassword key: db_password_01
Table 2 objects parameter Parameter
Type
Mandatory
Description
objectName
String
Yes
Credential name. Set this parameter to the secret referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
objectType
String
Yes
Secret type. Only csms is supported. A value other than csms is invalid.
objectVersion
String
Yes
Secret version.
- Specify a version, for example, v1 or v2.
- Use the latest version, for example, latest. When objectVersion is set to latest, if the corresponding secret in CSCM is updated, the secret will be updated to the pod after a certain interval (rotation_poll_interval).
jmesPath
Array of Object
Yes
jmesPath is used to extract key-value pairs from objects in JSON format. CCE Secrets Manager for DEW uses this tool to support Secret mounting.
- path: Enter the key in a DEW secret. The key cannot contain special characters such as +, -, {}, and ().
- objectAlias: name of the file mounted to a pod. The value must be the same as the value of objectName defined in secretObjects.
Table 3 secretObjects parameter Parameter
Type
Mandatory
Description
secretName
String
Yes
Secret name
type
String
Yes
Secret type
data
Array of Object
Yes
- objectName: name of the file mounted to the pod. The value must be the same as the value of objectAlias specified in objects.
- key: key in a secret. The value can be used to reference the encrypted content in a pod.
- Run the following command to create a SecretProviderClass object:
- Check whether the SecretProviderClass object has been created.
$ kubectl get spc NAME AGE nginx-deployment-spc-k8s-secrets 20h
- Create the secretproviderclass.yaml file using the template below.
- Create a pod. The following describes how to create an Nginx application.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-k8s-secrets labels: app: nginx-k8s-secrets spec: replicas: 1 selector: matchLabels: app: nginx-k8s-secrets template: metadata: labels: app: nginx-k8s-secrets spec: serviceAccountName: nginx-spc-sa # Reference the created ServiceAccount. containers: - name: nginx-deployment-k8s-secrets image: nginx volumeMounts: # Mount the secret to a container. - name: secrets-store-inline # Name of the volume to be mounted mountPath: "/mnt/secrets" # Path of the container to be mounted readOnly: true env: # Reference the secret in environment variables. - name: DB_USERNAME_01 # Environment variable name in the workload valueFrom: secretKeyRef: name: my-secret-01 # Secret name specified in the SPC key: db_username_01 # The value of the key specified in the SPC - name: DB_PASSWORD_01 valueFrom: secretKeyRef: name: my-secret-01 key: db_password_01 imagePullSecrets: - name: default-secret volumes: # Use the secret specified in the SPC to create a volume. - name: secrets-store-inline # Custom volume name csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: nginx-deployment-spc-k8s-secrets # Name of the SPC created in the previous step
- Verify the result.
$ kubectl get secrets NAME TYPE DATA AGE default-secret kubernetes.io/dockerconfigjson 1 33d my-secret-01 Opaque 2 1h
The output shows that a secret named my-secret-01 has been created using secret_1 specified in the SPC object.
Scheduled Rotation
As described before, you can use this add-on to complete the mount secrets, that is, you can write the secrets stored in CSMS to a pod.
To change the secret version declared in the SPC object to latest, run the following command:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: spc-test
spec:
provider: cce
parameters:
objects: |
- objectName: "secret_1"
objectVersion: "latest" # change "v1"to "latest"
objectType: "csms"
After the SPC object is updated, the add-on periodically sends a request to CSMS to obtain the value of secret_1 of the latest version and updates the value to the pod that references the SPC object. The interval for the add-on to periodically send requests is specified by rotation_poll_interval set in Installing the Add-on.
Real-Time Detection of SPC Changes
SPC changes are already detected in real time in Mounting a Credential Using a Volume and Scheduled Rotation. For demonstration, add secret secret_2 to the SPC object as follows:
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: spc-test spec: provider: cce parameters: objects: | - objectName: "secret_1" objectVersion: "latest" objectType: "csms" - objectName: "secret_2" objectVersion: "v1" objectType: "csms"
After the SPC object is updated, the new secret_2 is quickly mounted to the pod that references the SPC object.
Viewing Component Logs
View the pod where the add-on runs.
$ kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE csi-secrets-store-76tj2 3/3 Running 0 11h dew-provider-hm5fq 1/1 Running 0 11h
View pod logs of the dew-provider component.
$ kubectl logs dew-provider-hm5fq -n kube-system ...Log information omitted... ...
View the pod logs of the csi-secrets-store component. As the pod of the csi-secrets-store component contains multiple containers, you must run the -c command to specify a container when viewing pod logs. The secrets-store container is the major service container of the add-on and contains the majority of the logs.
$ kubectl logs csi-secrets-store-76tj2 -c secrets-store -n kube-system
...Log information omitted...
...
Change History
Add-on Version |
Supported Cluster Version |
New Feature |
---|---|---|
1.1.33 |
v1.21 v1.23 v1.25 v1.27 v1.28 v1.29 v1.30 |
CCE clusters 1.30 are supported. |
1.1.3 |
v1.21 v1.23 v1.25 v1.27 v1.28 v1.29 |
Fixed some issues. |
1.1.2 |
v1.21 v1.23 v1.25 v1.27 v1.28 v1.29 |
|
1.0.31 |
v1.21 v1.23 v1.25 v1.27 v1.28 |
|
1.0.9 |
v1.19 v1.21 v1.23 v1.25 |
None |
1.0.6 |
v1.19 v1.21 v1.23 v1.25 |
None |
1.0.3 |
v1.19 v1.21 v1.23 v1.25 |
CCE clusters 1.25 are supported. |
1.0.2 |
v1.19 v1.21 v1.23 |
CCE clusters 1.23 are supported. |
1.0.1 |
v1.19 v1.21 |
Actively detects SecretProviderClass object changes. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot