CCE Secrets Manager for DEW

Introduction

The dew-provider add-on is used to interconnect with Data Encryption Workshop (DEW), which allows you to mount secrets stored outside a cluster (DEW for storing sensitive information) to pods. In this way, sensitive information can be decoupled from the cluster environment, which prevents information leakage caused by program hardcoding or plaintext configuration.

Constraints

• DEW includes Key Management Service (KMS), Cloud Secret Management Service (CSMS), and Key Pair Service (KPS). Currently, the dew-provider add-on can interconnect only with CSMS.
• The dew-provider add-on can be installed only on clusters v1.19 or later.
• The dew-provider add-on can be installed in CCE standard clusters and CCE Turbo clusters.
• A maximum of 500 SecretProviderClass objects can be created.
• When the add-on is uninstalled, related CRD resources are deleted accordingly. Even if the add-on is reinstalled, the original SecretProviderClass object is unavailable. If you want to use the original SecretProviderClass resources after the add-on is uninstalled and then reinstalled, manually create them again.

How the Add-on Works

• Basic mounting: After the dew-provider add-on is installed, you can create a SecretProviderClass object and declare and reference the volume in a pod. When the pod is started, the secret declared in the SecretProviderClass object is mounted to the pod.
• Scheduled rotation: After a pod runs properly, if the secret declared in the SPC object and stored in CSMS is updated, the latest secret values can be updated to the pod through scheduled rotation. When using this capability, set the secret version to latest.
• Real-time awareness of SPC changes: After a pod runs properly, if a user modifies the secret declared in the SPC object (for example, a secret is added or the version number is changed), the add-on can detect the change in real time and update the secret to the pod.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console. Click Add-ons in the navigation pane, locate CCE Secrets Manager for DEW on the right, and click Install.
2. On the Install Add-on page, configure parameters in the Parameters area, as listed in the following table.

   Parameter	Description
   rotation_poll_interval	Rotation interval, in unit of m (instead of min). The rotation interval indicates the interval for sending a request to CSMS and obtaining the latest secret. The proper interval range is [1m, 1440m]. The default value is 2m.

3. Click Install.

   After the add-on is installed, select the cluster and click Add-ons in the navigation pane. On the displayed page, view the add-on in the Add-ons Installed area.

Components

Mounting a Credential Using a Volume

1. Create a ServiceAccount.
   1. Create a ServiceAccount object, which declares the secret names that can be used by services. If a user references a secret that is not declared here, the mounting will fail. As a result, the pod cannot run.

      Create the serviceaccount.yaml file based on the template below, and declare the secret names that can be used by services in the cce.io/dew-resource field. Here, secret_1 and secret_2 are declared, indicating that the service is allowed to reference two secrets. In subsequent operations, if the user references secret_3 in the service, the verification fails. As a result, the secret cannot be mounted and the pod cannot run.

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: nginx-spc-sa
        annotations:
          cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]"  #secrets that allow pod to use

      Ensure that the secrets declared here exist in CSCM, as shown in the following figure. Otherwise, even if the verification is successful, an error occurs when the corresponding secret is obtained from CSCM. As a result, the pod cannot run properly.

      

   2. Run the following command to create the ServiceAccount:

      kubectl apply -f serviceaccount.yaml

   3. Check whether the ServiceAccount object is successfully created.

      $ kubectl get sa
      NAME       SECRETS   AGE
      default         1         18d   # This is the default ServiceAccount object of the system.
      nginx-spc-sa    1         19s   # This is the newly created ServiceAccount object.

      A ServiceAccount object named nginx-spc-sa has been created. This object will be referenced in pods.

2. Create a SecretProviderClass.
   1. The SecretProviderClass object is used to describe the secret information (such as the version and name) that users are interested in. It is created by users and will be referenced in pods.

      Create the secretproviderclass.yaml file using the template below. Pay attention to the objects field in parameters, which is an array used to declare the secret to be mounted.

      apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: spc-test
      spec:
        provider: cce     # The value is fixed at cce.
        parameters:
          objects: |
                - objectName: "secret_1"
                  objectVersion: "v1"
                  objectType: "csms"

      Parameter	Type	Mandatory	Description
      objectName	String	Yes	Credential name. Set this parameter to the secret referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
      objectAlias	String	No	File name of the secret written into the container. If this parameter is not specified, the file name of the secret written into the container is the value of objectName by default. If this parameter is specified, the value must be different from objectName and from the objectAlias and objectName values of other secrets. Otherwise, the mounting fails.
      objectType	String	Yes	Secret type. Only csms is supported. A value other than csms is invalid.
      objectVersion	String	Yes	Secret version Specify a version, for example, v1 or v2. Use the latest version, for example, latest. When objectVersion is set to latest, if the corresponding secret in CSCM is updated, the secret will be updated to the pod after a certain interval (rotation_poll_interval).

   2. Run the following command to create a SecretProviderClass object:

      kubectl apply -f secretproviderclass.yaml

   3. Check whether the SecretProviderClass object has been created.

      $ kubectl get spc
      NAME   AGE
      spc-test   20h

      A SecretProviderClass object named spc-test is created. This object will be referenced in pods subsequently.

3. Create a pod.

   The following describes how to create an Nginx application.

   1. Define a workload, reference the created ServiceAccount object in serviceAccountName, and reference the created SPC object in secretProviderClass, specify the mount path of the container in mountPath. (Do not specify special directories such as / and /var/run. Otherwise, the container may fail to be started.)

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-spc
        labels:
          app: nginx
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            labels:
              app: nginx
          spec:
            serviceAccountName: nginx-spc-sa   # Reference the created ServiceAccount.
            volumes:
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: "spc-test"  # Reference the created SPC.
            containers:
              - name: nginx-spc
                image: nginx:alpine
                imagePullPolicy: IfNotPresent
                volumeMounts:
                  - name: secrets-store-inline
                    mountPath: "/mnt/secrets-store"  # Define the mount path of secrets in the container.
                    readOnly: true
            imagePullSecrets:
              - name: default-secret

   2. Create a pod.

      kubectl apply -f deployment.yaml

   3. Check whether the pod has been created.

      $ kubectl get pod
      NAME                     READY   STATUS   RESTARTS   AGE
      nginx-spc-67c9d5b594-642np     1/1     Running    0            20s

   4. Access the container and check whether the specified secret is written properly. For example:

      $ kubectl exec -ti nginx-spc-67c9d5b594-642np -- /bin/bash
      root@nginx-spc-67c9d5b594-642np:/#  
      root@nginx-spc-67c9d5b594-642np:/# cd /mnt/secrets-store/
      root@nginx-spc-67c9d5b594-642np:/mnt/secrets-store# 
      root@nginx-spc-67c9d5b594-642np:/mnt/secrets-store# ls
      secret_1

      The command output shows that secret_1 declared in the SPC object has been written to the pod.

      In addition, you can obtain spcPodStatus to check the binding relationship between pods and secrets. For example:

      $ kubectl get spcps
      NAME                                                                    AGE
      nginx-spc-67c9d5b594-642np-default-spc-test   103s
      $ kubectl get spcps nginx-spc-67c9d5b594-642np-default-spc-test -o yaml
      ......
      status:
      mounted: true
      objects:    # Mounted secret
      - id: secret_1
      version: v1
      podName: nginx-spc-67c9d5b594-642np   # Pod that references the SPC object
      secretProviderClassName: spc-test               # SPC object
      targetPath: /mnt/paas/kubernetes/kubelet/pods/6dd29596-5b78-44fb-9d4c-a5027c420617/volumes/kubernetes.io~csi/secrets-store-inline/mount

Mounting a Credential Using a Secret

CCE Secrets Manager for DEW must be 1.1.1 or later.

1. Create a ServiceAccount.
   1. Create a ServiceAccount object, which declares the secret names that can be used by services. If a user references a secret that is not declared here, the mounting will fail. As a result, the pod cannot run.

      Create the serviceaccount.yaml file based on the template below, and declare the secret names that can be used by services in the cce.io/dew-resource field. Here, secret_1 and secret_2 are declared, indicating that the service is allowed to reference two secrets. In subsequent operations, if the user references secret_3 in the service, the verification fails. As a result, the secret cannot be mounted and the pod cannot run.

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: nginx-spc-sa
        annotations:
          cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]"  #secrets that allow pod to use

      Ensure that the secrets declared here exist in CSCM, as shown in the following figure. Otherwise, even if the verification is successful, an error occurs when the corresponding secret is obtained from CSCM. As a result, the pod cannot run properly.

      

   2. Run the following command to create the ServiceAccount:

      kubectl apply -f serviceaccount.yaml

   3. Check whether the ServiceAccount object is successfully created.

      $ kubectl get sa
      NAME       SECRETS   AGE
      default         1         18d   # This is the default ServiceAccount object of the system.
      nginx-spc-sa    1         19s   # This is the newly created ServiceAccount object.

      A ServiceAccount object named nginx-spc-sa has been created. This object will be referenced in pods.

2. Create a SecretProviderClass.
   1. Create the secretproviderclass.yaml file using the template below.

      apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: nginx-deployment-spc-k8s-secrets
      spec:
        provider: cce
        parameters: 
          # Reference a secret in CSMS.
          objects: |
            - objectName: "secret_1"
              objectType: "csms"
              objectVersion: "latest"
              jmesPath:
                - path: username
                  objectAlias: dbusername
                - path: password
                  objectAlias: dbpassword
        # Create a CCE secret based on the CSMS secret and mount the secret to a pod.
        secretObjects:                
          - secretName: my-secret-01
            type: Opaque
            data:
              - objectName: dbusername
                key: db_username_01
              - objectName: dbpassword
                key: db_password_01

      Table 2 objects parameter

      Parameter	Type	Mandatory	Description
      objectName	String	Yes	Credential name. Set this parameter to the secret referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
      objectType	String	Yes	Secret type. Only csms is supported. A value other than csms is invalid.
      objectVersion	String	Yes	Secret version. Specify a version, for example, v1 or v2. Use the latest version, for example, latest. When objectVersion is set to latest, if the corresponding secret in CSCM is updated, the secret will be updated to the pod after a certain interval (rotation_poll_interval).
      jmesPath	Array of Object	Yes	jmesPath is used to extract key-value pairs from objects in JSON format. CCE Secrets Manager for DEW uses this tool to support Secret mounting. path: Enter the key in a DEW secret. The key cannot contain special characters such as +, -, {}, and (). objectAlias: name of the file mounted to a pod. The value must be the same as the value of objectName defined in secretObjects.

      Table 3 secretObjects parameter

      Parameter	Type	Mandatory	Description
      secretName	String	Yes	Secret name
      type	String	Yes	Secret type
      data	Array of Object	Yes	objectName: name of the file mounted to the pod. The value must be the same as the value of objectAlias specified in objects. key: key in a secret. The value can be used to reference the encrypted content in a pod.

   2. Run the following command to create a SecretProviderClass object:

      kubectl apply -f secretproviderclass.yaml

   3. Check whether the SecretProviderClass object has been created.

      $ kubectl get spc
      NAME   AGE
      nginx-deployment-spc-k8s-secrets   20h

3. Create a pod. The following describes how to create an Nginx application.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment-k8s-secrets
     labels:
       app: nginx-k8s-secrets
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx-k8s-secrets
     template:
       metadata:
         labels:
           app: nginx-k8s-secrets
       spec:
         serviceAccountName: nginx-spc-sa   # Reference the created ServiceAccount.
         containers:
         - name: nginx-deployment-k8s-secrets
           image: nginx
           volumeMounts:   # Mount the secret to a container.
             - name: secrets-store-inline   # Name of the volume to be mounted
               mountPath: "/mnt/secrets"    # Path of the container to be mounted
               readOnly: true
           env:    # Reference the secret in environment variables.
             - name: DB_USERNAME_01      # Environment variable name in the workload
               valueFrom:
                 secretKeyRef:
                   name: my-secret-01    # Secret name specified in the SPC
                   key: db_username_01   # The value of the key specified in the SPC
             - name: DB_PASSWORD_01
               valueFrom:
                 secretKeyRef:
                   name: my-secret-01
                   key: db_password_01
         imagePullSecrets:
           - name: default-secret
         volumes:    # Use the secret specified in the SPC to create a volume.
           - name: secrets-store-inline  # Custom volume name
             csi:
               driver: secrets-store.csi.k8s.io
               readOnly: true
               volumeAttributes:
                 secretProviderClass: nginx-deployment-spc-k8s-secrets   # Name of the SPC created in the previous step

4. Verify the result.

   $ kubectl get secrets
   NAME             TYPE                             DATA   AGE
   default-secret   kubernetes.io/dockerconfigjson   1      33d
   my-secret-01     Opaque                           2      1h

   The output shows that a secret named my-secret-01 has been created using secret_1 specified in the SPC object.

Scheduled Rotation

As described before, you can use this add-on to complete the mount secrets, that is, you can write the secrets stored in CSMS to a pod.

To change the secret version declared in the SPC object to latest, run the following command:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce
  parameters:
    objects: |
          - objectName: "secret_1"
            objectVersion: "latest"  # change "v1"to "latest"
            objectType: "csms"

After the SPC object is updated, the add-on periodically sends a request to CSMS to obtain the value of secret_1 of the latest version and updates the value to the pod that references the SPC object. The interval for the add-on to periodically send requests is specified by rotation_poll_interval set in Installing the Add-on.

Real-Time Detection of SPC Changes

SPC changes are already detected in real time in Mounting a Credential Using a Volume and Scheduled Rotation. For demonstration, add secret secret_2 to the SPC object as follows:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce
  parameters:
    objects: |
          - objectName: "secret_1"
            objectVersion: "latest"
            objectType: "csms"
          - objectName: "secret_2"
            objectVersion: "v1"
            objectType: "csms"

After the SPC object is updated, the new secret_2 is quickly mounted to the pod that references the SPC object.

Viewing Component Logs

View the pod where the add-on runs.

$ kubectl get pod -n kube-system
NAME                          READY   STATUS      RESTARTS   AGE
csi-secrets-store-76tj2       3/3     Running     0           11h
dew-provider-hm5fq            1/1     Running     0           11h

View pod logs of the dew-provider component.

$ kubectl logs dew-provider-hm5fq -n kube-system
...Log information omitted...
...

View the pod logs of the csi-secrets-store component. As the pod of the csi-secrets-store component contains multiple containers, you must run the -c command to specify a container when viewing pod logs. The secrets-store container is the major service container of the add-on and contains the majority of the logs.

$ kubectl logs csi-secrets-store-76tj2 -c secrets-store -n kube-system
...Log information omitted...
...

Change History