Help Center/ Data Encryption Workshop/ Best Practices/ Key Management Service/ Using KMS to Encrypt Secrets at Rest
Updated on 2026-09-20 GMT+08:00
View PDF
Share

Using KMS to Encrypt Secrets at Rest

At-rest encryption of secrets is a static data encryption approach provided by Kubernetes. You can specify EncryptionConfiguration to automatically encrypt secrets through envelope encryption when they are written to persistent storage (such as etcd). This ensures that sensitive information (such as passwords, certificates, and API keys) is not stored in plaintext. KMS provides out-of-the-box at-rest encryption for secrets in CCE clusters.

Scenario

In Kubernetes clusters, secrets are typically used to store sensitive information, such as database passwords, API keys, and certificates. Encrypting secrets at rest using KMS ensures that sensitive data is encrypted during storage, improving data security. Many industries and organizations, for example, in the finance and healthcare industries, have to follow strict data protection and compliance requirements. KMS can help enterprises meet the compliance requirements and ensure data security during storage and transmission. In multi-tenant environments, tenants need to isolate and protect their own sensitive data. Independent key management provided by KMS prevents data leakage among tenants. KMS supports automatic key management and rotation, which effectively reduces the complexity and risks of manual key management. Automatic management ensures more secure and efficient key lifecycle management. KMS provides detailed audit logs and monitoring functions to help enterprises track key usage, and detect and handle potential security threats in a timely manner. In conclusion, KMS can effectively improve the storage security of secrets in Kubernetes clusters and prevent unauthorized access to sensitive data.

Encryption Mechanisms

In Kubernetes clusters, secrets are used to store sensitive information, such as application passwords, TLS certificates, and Docker image credentials, which are encoded using Base64 and stored in etcd by default. Base64 encoding only converts data formats but does not ensure data security. Data encoded using Base64 is stored in plaintext.

KMS CMKs automatically encrypt secrets in CCE clusters. Automatic secret encryption is implemented based on the KMS Encryption Provider mechanism of Kubernetes. The envelope encryption technology is used to ensure that secrets are stored in ciphertext when being written to etcd and are automatically decrypted when being accessed. This effectively reduces the exposure of unencrypted metadata and sensitive information and significantly shortens the potential exposure window. The encryption and decryption processes are as follows:

  • Encryption: When kube-apiserver is started, a DEK seed is generated. The seed is encrypted using KMS CMKs and then cached. When a secret is created, kube-apiserver derives a DEK based on the DEK seed to encrypt the secret. The encrypted secret and derived DEK are stored in etcd.
  • Decryption: When reading a Kubernetes secret, the system obtains the encrypted secret and DEK from etcd. Then, the system calls the decrypt-datakey API of KMS to decrypt the DEK seed using the CMK, and obtains the plaintext DEK using the decrypted DEK seed. Finally, the plaintext DEK is used to decrypt the encrypted secret, and then the original secret is returned to the user.

Prerequisites

  • You have created a KMS key and the key is in the same region as the CCE cluster.
  • Your account has granted the cce_trust_kms agency permission to CCE. CCE clusters use this agency to perform operations such as key query, encryption, and decryption. You can authorize CCE during the dependency check when using secret encryption at rest for the first time.

    Do not delete the agency after it is created, or clusters with secret encryption at rest enabled will become unavailable.

Notes and Constraints

  • At-rest encryption for secrets is only available during the creation of CCE standard and Turbo clusters v1.27 or later. Only the KMS v2 API can be used to encrypt secrets. At-rest encryption for secrets cannot be disabled once enabled.
  • This feature is in the initial rollout stage. To view the regions where this feature is available, see the console.
  • If you have enabled at-rest encryption for secrets, do not disable or delete the key selected during cluster creation via the DEW console or OpenAPI. Otherwise, the cluster API server will be unavailable, affecting the proper running of service applications.

Procedure

  1. Log in to the CCE console.
  2. On the Clusters page, click Buy Cluster in the upper right corner. Set Cluster Version to v1.27 or later.
  3. In the lower part of the page, expand Advanced Settings, locate Secret Encryption, and enable it.

    Choose a custom KMS key or default key.

    Figure 1 Secret encryption

  4. Configure other parameters by referring to Buying a Standard/Turbo Cluster and complete the subsequent steps.
  5. Click the cluster name to access its details page after it is created. Choose Settings, in the Cluster Settings area on the Dashboard page, verify that at-rest encryption for secrets has been enabled.

Using KMS Automatic Key Rotation to Encrypt Secrets at Rest

If the same KMS CMK is used to encrypt secrets at rest for a long time, security risks will increase significantly. To enhance security, KMS automatically rotates keys based on the preset rotation period (365 days by default). This means a new key is automatically generated to replace an existing key.

KMS automatic key rotation allows at-rest secret encryption in CCE. If automatic key rotation is used, your existing secrets will still be encrypted with the existing key, but new secrets will be encrypted with the new key. In this way, new data can be encrypted using the latest key, and old data can still be decrypted. This balances security and availability.

For details about how to enable automatic key rotation, see Enabling Key Rotation. You will be billed for automatic key rotation. For details, see How Is Automatic Key Rotation Billed?

To encrypt existing secrets with the updated key after the key is automatically rotated, run the following command:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"
Parent topic: Key Management Service