Using KMS to Encrypt Secrets at Rest

Introduction to Secret Encryption at Rest

In a Kubernetes cluster, secrets are used to store sensitive information such as application passwords, TLS certificates, and Docker image credentials. By default, the data is stored in etcd in Base64 encoding format. Base64 encoding is only a data format conversion and does not provide any security. Data encoded using Base64 is stored in plaintext.

CCE clusters support automatic encryption of secrets using KMS master keys. Automatic secret encryption is implemented based on the KMS encryption provider mechanism of Kubernetes. The envelope encryption technology is used to ensure that secrets are stored in ciphertext when being written to etcd and are automatically decrypted when being accessed. This effectively reduces the exposure of unencrypted metadata and sensitive information and significantly shortens the potential exposure window. The encryption and decryption processes are as follows:

• Encryption process: When kube-apiserver is started, a DEK seed is generated. The seed is encrypted using the KMS master key of DEW and then cached. When a secret is created, kube-apiserver derives a DEK based on the DEK seed to encrypt the secret. The encrypted secret and derived DEK are stored in etcd.
• Decryption process: When reading a Kubernetes secret, the system obtains the encrypted secret and DEK from etcd. Then, the system calls the decrypt-datakey API of KMS to decrypt the DEK seed using the master key, and obtains the plaintext DEK using the decrypted DEK seed. Finally, the plaintext DEK is used to decrypt the encrypted secret, and then the original secret is returned to the user.

For details, see the Kubernetes official introduction. You can also refer to How Do Huawei Cloud Services Use KMS to Encrypt Data?

Prerequisites

• You have created a KMS key on Huawei Cloud DEW, and the key and your CCE cluster are in the same region. For details about how to create a key, see Creating a Key. For details about DEW billing, see Billing Overview.
• There is an agency named cce_trust_kms agency for CCE in your account. This agency is used by CCE clusters to obtain, encrypt, and decrypt keys. You can authorize CCE during the dependency check when using secret encryption at rest for the first time.

  Do not delete the agency after it is created, or clusters with secret encryption enabled will become unavailable.

Notes and Constraints

• At-rest encryption for secrets is only available during the creation of CCE standard and Turbo clusters of v1.27 or later. Only the KMS v2 API can be used to encrypt secrets. At-rest encryption for secrets cannot be disabled once it is enabled.
• This feature is in the initial rollout stage. To view the regions where this feature is available, see the console.
• If you have enabled at-rest encryption for secrets, do not disable or delete the key selected using the DEW console or open APIs during cluster creation. If you disable or delete the key, the cluster API server may become unavailable, which affects the applications.

Enabling At-Rest Encryption for Secrets

1. Log in to the CCE console.
2. On the Clusters page, click Buy Cluster in the upper right corner. Ensure the cluster version is v1.27 or later.
3. In the lower part of the page, expand Advanced Settings, locate Secret Encryption, and enable it.

   Choose a custom key or default key.

   

4. Configure other parameters by referring to Buying a CCE Standard/Turbo Cluster and create the cluster.
5. Click the cluster name to go to the cluster console. In the navigation pane, choose Settings. In the Cluster Settings area on the Dashboard page, verify that at-rest encryption for secrets has been enabled.

Using KMS Automatic Key Rotation to Encrypt Secrets at Rest

If the same KMS master key is used to encrypt secrets at rest for a long time, security risks will increase significantly. To enhance security, KMS automatically rotates keys based on the preset rotation period (365 days by default). This means a new key is automatically generated to replace an existing key.

CCE allows you to use KMS automatic key rotation to encrypt secrets at rest. If automatic key rotation is used, your existing secrets will still be encrypted with the existing key, but any new secrets will be encrypted with the new key. In this way, new data can be encrypted using the latest key, and old data can still be decrypted. This balances security and availability.

For details about how to enable automatic key rotation, see Enabling Key Rotation. You will be billed for automatic key rotation. For details, see How Is Automatic Key Rotation Billed?

To encrypt existing secrets with the updated key after the key is automatically rotated, run the following command:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

Helpful Links

• For more information about at-rest encryption for secrets, see Using a KMS Provider for Data Encryption.
• For details about envelope encryption, see How Do Huawei Cloud Services Use KMS to Encrypt Data?
• For details about how to enable automatic key rotation, see Enabling Key Rotation.