Credential Leakage Response Solution

Incident Type: Credential Leakage

Credential leakage means that the identity authentication information, such as the username, password, API key, and access token, of an individual or organization is obtained or disclosed by an unauthorized third party when the individual or organization uses online services, such as cloud services, social media platforms, and emails. This may occur in many cases, including but not limited to phishing, malware, social engineering, and system vulnerabilities. Once credentials are disclosed, attackers may use them to access sensitive data, perform illegal transactions, or damage the system, which severely affects services.

Response Solution

To address the preceding issues, Huawei Cloud launched the SecMaster service. SecMaster is a next-generation cloud native security operation platform. Based on years of cloud security experience of Huawei Cloud, it enables integrated and automatic security operations through cloud asset management, security posture management, security information and event management, security orchestration and automatic response, cloud security overview, simplified cloud security configuration, configurable defense policies, and intelligent and fast threat detection and response.

Incident Response Process

1. Identify whether identity credentials are compromised or disclosed.
   1. If you receive any of the following messages, check your identity credentials for damage or leakage:
      • Alerts or metrics from cloud services (such as CCM, SecMaster, and CTS) and external monitoring systems;
      • Prompt messages from the contractor or third-party service provider;
      • Findings through internal or external security inspections;
      • Internal system messages;
      • Information anonymously reported to you;
      • Information you receive in other ways. For example, an attacker may use leaked credentials to steal your data and modify your public-facing resources.
   2. Make sure a service ticket or case has been submitted for the incident. If no service tickets or cases are submitted, submit one manually.
   3. Determine and record the impact on end users.

      No matter whether this scenario has direct impact on users, record the survey result in the ticket or case related to the incident.

   4. For automatically created service tickets or cases, identify real alerts or abnormal metrics.

      For example, an alert or abnormal metric in CTS may be generated because there are unsafe IAM settings or IAM credential leaks. It may also be a billing alert triggered when your billing cost exceeds a predefined threshold.

   5. Locate the leaked credential set.
      • If a service ticket or case is created, check whether the user/role name, user/role ID, or access key ID has been recorded in the ticket or case.
      • If the alert is generated by SecMaster baseline inspection, you can view the baseline inspection result on the console and find the access key ID of the affected credential. For details, see Viewing the Baseline Inspection Result.
      • If the alert is reported by CTS, you can view the result in the incident list on the console. The resource name is the access key. The Credential field contains access_key_id, account_id, user_name, and other information.
   6. Determine when credentials may be compromised or leaked. Any API operation performed after this time shall be considered malicious, and any resource created after this time shall be considered leaked.
   7. If your application is interrupted, identify the possible incidents that cause the interruption. If the interruption is not related to a credential leak, check the deployment pipeline to see if any changes were made before the incident. You can use CTS to view logs of all account activities.
   8. Communicate incidents:
      • Determine stakeholder roles based on the organization's incident response plan.
      • Notify relevant stakeholders, including legal personnel, technical teams, and developers, and ensure that they are added to the service ticket and war rooms for continuous updates.
   9. Communicate incidents with external parties.
      • Ensure that your legal counsel is aware of the situation and keep them in the loop of internal stakeholders for any updates, especially updates for external communications.
      • Add colleagues responsible for public or external communication to the service ticket so that they can receive regular status updates about the incident and fulfill their communication responsibilities.
      • If your jurisdiction has regulations that require the reporting of such incidents, ensure that the person responsible for notifying the local or federal law enforcement agency in your organization also receives a notification of the incident/is added to the service ticket. Consult your legal counsel and law enforcement for guidance on the collection and preservation of evidence and regulatory authorities. Your report may help analyze similar activities or help others, even if it is not required by regulations to report to open databases, government agencies, or non-governmental organizations.

2. Contain incidents.

   You can disable compromised credentials or revoke permissions related to these credentials to prevent the use of compromised credentials to call APIs.

   1. Disable the compromised credentials identified in 1.
      1. If the user credentials are permanent IAM user credentials, delete them on the IAM console. For details, see Deleting an IAM User.
      2. If the user credentials are temporary credentials obtained from IAM, they can be associated with the IAM role. To disable these functions, perform the following steps:
         1. Cancels all sessions of the current role. If the attacker obtains a new temporary security credential and continues the attack, go to 2.a.ii.2.
         2. Delete all IAM policies added to the role, modify existing policies to block all access, or modify the policy of the role to prevent attackers from taking the role.

            Credentials are still valid for a specified period of time after being issued. After the trust policy is modified, the credentials can still be used within the validity period. 2.a.ii.1 and 2.a.ii.2 will prevent all users, including any legitimate users or applications, from using credentials obtained by taking on roles.

   2. You can view the credentials that are continuously used on the CTS console within about 30 minutes, regardless of access keys, IAM users, or roles, and confirm that the compromised credentials have been disabled.

3. Eradicate incidents.

   You need to check which API operations are performed and which resources are created, deleted, or modified after the credential is damaged, and take measures to eliminate the impact.

   1. Use your preferred monitoring tool to access CTS and collect all API operations performed by the damaged credential. The log collection time is from the damage time to the current time.
      • If you use a third-party tool (such as Splunk) to collect CTS logs, obtain logs from the tool.
      • If you send logs to OBS instead of using a third-party tool, you can use LTS to collect, query, and store logs.
   2. On the LTS console, query all API operations performed after the credential is damaged or leaked.
   3. Pick out the API calls that may:
      • Access sensitive data, for example, OBS Object.
      • Create resources, such as databases and cloud servers.
      • Creates resources, including EC auto scaling groups.
      • Create or modify permissions, and check API methods including but not limited to CreateUser, CreateRole, AssumeRole*, Get*Token, Attach*Policy, *Image*, *Provider, Tag*, Create*, Delete* and Update*.
      • Delete affected cloud resources.
      • Modify affected cloud resources.
   4. Based on the result of the previous step, identify any applications that might have been affected. Obtain the ID or tag information of each affected resource and notify the resource owner.
   5. If additional credentials (such as IAM users and roles) are created, disable and delete all credentials of these resources based on the 2.a.
   6. Repeat 3.a to 3.e to check for and handle all additional credentials.

4. Recover from incident.
   1. Restore the modified resource.
      • If the resource can be destroyed and replaced, a new resource is added.
      • If the resource cannot be replaced, perform either of the following operations:
        • Restore a resource from a backup.
        • New resources are prepared and configured in the application infrastructure, and damaged resources are isolated and removed from the application infrastructure.
        • Destroy damaged resources or continue to isolate them for evidence.
      • Restore the deleted resource.
        1. Check the application to which the resource belongs. If the resource tag is not listed in the CTS service item and the resource is supported by the cloud, check the configuration.
        2. If the deleted resource can be restored from a backup, restore it directly. If the deleted resource cannot be restored from a backup, obtain the resource configuration from CMDB, provision the resource again, and configure it in the infrastructure of the application.

5. Complete post-incident activities.
   • Investigate and collect evidence on certain compromised resources, analyze the attack methods used by attackers on the compromised resources, and check whether additional risks and risk mitigation measures need to be taken for related resources or applications.
     1. For any compromised resources that have been segregated for further analysis, forensic activities are performed on those resources and the findings are incorporated into ex post facto reporting.
     2. Ensure that the CMDB is correctly updated to reflect the current status of all affected resources and applications.
   • Review the incident itself and its response, determine which measures are effective and which are not, update the improvement process based on the information, and record the investigation results.