Implementing Automatic E2E RDS Credential Rotation Using EG

Context

With the continuous evolution of digital services, databases, as the storage and management center of enterprises' core data, are facing increasingly severe challenges in security. The traditional static database password management mode has many potential risks. Once the password is disclosed, serious security accidents such as data leakage and malicious tampering may occur. To improve database security protection, the dynamic credential rotation mechanism is developed. It periodically changes the database password to effectively reduce the risk of password cracking and becomes an important method to ensure database security.

Huawei Cloud Data Encryption Workshop (DEW) periodically rotates and changes the Relational Database Service (RDS) credentials and updates the changes to the RDS instances in real time. This greatly enhances the security of database passwords. However, CAE, as a database client, cannot automatically detect the change after the RDS credentials are updated. As a result, CAE cannot update the credential information in time, and a new database connection cannot be established, affecting normal service running. This issue makes it difficult to implement E2E automation of the RDS credential rotation mechanism in actual applications, which is a bottleneck that restricts the efficient operation of the database security protection system.

Solution

To overcome the preceding difficulties and fully utilize the security of RDS credential rotation, Huawei Cloud builds an automatic E2E RDS credential rotation solution with EventGrid (EG) as the core. This solution uses the collaboration between Huawei Cloud services to form a complete closed loop of credential update and synchronization.

This section describes how to use EG to implement automatic E2E RDS credential rotation.

1. Step 1: Use DEW to Create an Event and a Secret, and Bind the Event

   DEW periodically rotates and changes RDS credentials based on preset security policies, and updates the new credentials to RDS instances in real time.

2. Step 2: Use EG to Create an Event Hub and an Event Subscription

   DEW sends credential update events through EG and pushes key information such as the RDS instance ID and new credential version to the EG topic.

   EG, as the event routing hub, identifies and filters events related to RDS credential update based on the preconfigured subscription rules and routes the events to the specified receiving endpoint of CAE.

3. Step 3: Use CAE to Create a Secret

   After receiving the event notification from EG, CAE immediately starts the credential update process. By parsing the credential information in the event, CAE injects the new credentials into the configuration of related components and triggers the component restart or connection pool update to establish a database connection with RDS using the new credentials.

4. (Optional) Step 4: Use AOM to Create an Alarm Notification Rule and an Event Alarm Rule

   You can create alarm notification rules and event alarm rules on AOM to monitor CAE credential update failures in real time. Once an exception occurs, AOM sends an alarm to O&M personnel in time based on the preset notification method (such as SMS and email). In this way, O&M personnel can quickly locate and rectify the fault, ensuring reliable running of the entire automatic credential rotation process.

Restrictions

The automatic E2E RDS credential rotation is available only in the AP-Singapore region.

Prerequisites

An RDS instance has been created on the RDS console, and a database has been created in the instance. This practice uses the PostgreSQL database.

Step 1: Use DEW to Create an Event and a Secret, and Bind the Event

Creating an Event

1. Log in to DEW.
2. Choose Cloud Secret Management Services > Events.
3. Click Create Event in the upper right corner, set parameters by referring to Table 1. Select Event Grid (EG) for Message Type and select all event types. For details, see Creating an Event.

   Table 1 Parameters for creating an event

   Parameter	Example Value
   Event Name	Enter report-eg.
   Status	Select Enabled.
   Message Type	Select Event Grid (EG).
   Channel	Select default.
   Event Type	Select all options, including Version creation, Version expiry, Secret rotation, and Secret deletion.

4. Click OK.

Creating a Secret and Binding the Event

1. On the DEW console, choose Cloud Secret Management Service > Secrets.
2. Click Create Secret and set parameters by referring to Table 2. Select the created event for Associated events. For details, see Creating a Rotation Secret.

   Table 2 Parameters for creating a credential - basic information

   Parameter	Example Value
   Type	Select Rotated secret and then RDS secret from the drop-down list.
   Secret Name	Enter test-case.
   Enterprise Project	Select default.
   Database	Select PostgreSQL.
   RDS DB Instance	Select the RDS instance corresponding to the database type.
   Secret Value	Select Dual account. After you enter a database account and password, an account with the same permissions is cloned. Select I understand the risks.
   KMS Encryption Key	Select Select from List and enter csms/default.
   Associated events	Select the created event report-eg.

3. Click Next and set parameters by referring to Table 3.

   Table 3 Parameters for creating a credential - rotation period

   Parameter	Example Value
   Automatic rotation	Enable .
   Rotation Period	Select 6 hours.
   Rotation Function	You can create or use an existing rotation function in FunctionGraph to rotate secret values.

4. Click Next > OK. The secret is created.

Step 2: Use EG to Create an Event Hub and an Event Subscription

Creating an Event Channel

1. Log in to EG.
2. In the navigation pane on the left, choose Event Channels. By default, a cloud service event stream named default is created for EG. The cloud service event stream receives events from cloud service event sources.

Creating an Event Subscription

1. On the EG console, choose Event Subscriptions.
2. Click Create Event Subscription.
   Figure 1 Event subscription page
   

3. Click Event Source and set parameters by referring to Table 4. For details, see Creating an Event Subscription.

   Table 4 Example event source parameters

   Parameter	Example Value
   Provider	Select Cloud services.
   Event Source	Select Cloud Secret Management Service.
   Event Type	Select all options, including DEW:CSMS:SecretRotated, DEW:CSMS:SecretVersionCreated, DEW:CSMS:SecretDeleted, and DEW:CSMS:SecretVersionExpired.
   Filter Rule	Retain the default value.

4. Click OK.
5. Click Event Target and set parameters by referring to Table 5.

   Table 5 Example event target parameters

   Parameter	Example Value
   Provider	Select Cloud services.
   Event Target	Select Cloud service API. To use this function, submit a service ticket.
   Cloud Service	Select Cloud Application Engine CAE.
   API	Select Modifying the Version of a Secret Registered with DEW.
   Agency	Select EG_TARGET_AGENCY. If no agency is available, click Create Agency on the right.
   X-Enterprise-Project-ID	Optional.
   secret_id	Enter the ID of the secret created on the DEW console. To obtain the secret ID, go to the secret list page of the DEW console.
   Body	Click Switch to Text Input and enter the following body: { "api_version": "v1", "kind": "Secret", "spec": { "name": "test", "version_id": "v2" } }

6. Click OK.

   

7. Click Save.

Step 3: Use CAE to Create a Secret

1. Log in to CAE.
2. Choose System Settings.
3. Click Edit on the secret configuration card.
4. On the Set Secret page, click Create Secret and set parameters by referring to Table 6.

   Table 6 Example parameters

   Parameter	Example Value
   Secret Name	Enter the name of the secret created on the DEW console. To obtain the secret name, go to the secret list page of the DEW console.
   Secret Version	Select the latest version.

5. Click OK.

(Optional) Step 4: Use AOM to Create an Alarm Notification Rule and an Event Alarm Rule

Viewing the CAE Component Credential Update Failure Event

1. Log in to AOM.
2. In the navigation pane, choose Alarm Center > Alarm List.
3. Click the Events tab. When the CAE component credential fails to be updated, a failure event is reported to AOM.

Creating an Alarm Notification Rule

1. On the AOM console, choose Alarm Center > Alarm Notification.
2. Click Create and set parameters by referring to Table 7. For details, see Creating an AOM Alarm Notification Rule.

   Table 7 Parameters for creating an alarm notification rule

   Parameter	Example Value
   Notification Rule Name	Enter a name. Example: CAE Notification Rule.
   Enterprise Project	Select default.
   Rule Type	Select Prometheus monitoring.
   Message Template	Select aom.built-in.template.zh.
   Topic	Select an SMN topic. For details about how to create a topic, see Creating a Topic.

3. Click OK.

Creating an Event Alarm Rule

1. On the AOM console, choose Alarm Center > Alarm Rules.
2. Click Create Alarm Rule and set parameters by referring to Table 8. For details, see Creating an AOM Event Alarm Rule.

   Table 8 Parameters for creating an event alarm rule

   Parameter	Example Value
   Original Rule Name	Enter a name. Example: CAE Component Credential Update Failure.
   Enterprise Project	Select default.
   Rule Type	Select Event alarm rule.
   Event Type	Select Custom.
   Event Source	Select CAE.
   Monitored Object	Event Name: Select CAE Component Credential Update Failure. Trigger Mode: Select Immediate Trigger. Alarm Severity: Select .
   Alarm Mode	Select Direct alarm reporting.
   Notification Rule	Select the created alarm notification rule.

3. Click Confirm. When this alarm is triggered, a notification will be sent to you based on the notification mode configured in SMN.