Credential Leakage Response Solution

Incident Type: Credential Leakage

Credential leakage means that the identity authentication information, such as the username, password, API key, and access token, of an individual or organization is obtained or disclosed by an unauthorized third party when the individual or organization uses online services, such as cloud services, social media platforms, and emails. This may occur in many cases, including but not limited to phishing, malware, social engineering, and system vulnerabilities. Once credentials are disclosed, attackers may use them to access sensitive data, perform illegal transactions, or damage the system, which severely affects services.

Response Solution

To address the issues mentioned above, Huawei Cloud launched SecMaster. SecMaster is a next-generation cloud native platform that enables integrated and automatic security operations. You can manage cloud assets, security posture, security information, and incidents in one place and enjoy intelligent threat detection, easy security orchestration, and automatic response.

Incident Response Process

1. Identify whether identity credentials are compromised or disclosed.
   1. If you receive any of the following messages, check your identity credentials for damage or leakage:
      • Alerts or metrics from cloud services (such as CCM, SecMaster, and CTS) and external monitoring systems;
      • Prompt messages from the contractor or third-party service provider;
      • Findings through internal or external security researchers
      • Internal system messages;
      • Information anonymously reported to you;
      • Information you receive in other ways. For example, an attacker uses leaked credentials to steal your data and modify your public-facing resources.
   2. Make sure a service ticket or case has been submitted for the incident. If no service tickets or cases are submitted, submit one manually.
   3. Determine and record the impact or experience of the problem on end users.

      From the perspective of users, this scenario may have no direct impact on users. Record the survey result in the ticket or case related to the incident.

   4. For automatically created tickets or cases, determine which alerts are real ones or which metrics are abnormal.

      For example, if an alarm or metric is triggered, the CTS service metric may indicate that some aspects of your IAM configuration are not compliant, or the IAM service alarm may indicate that credentials may be leaked. It may also be a billing alert that triggers an alert and notification when your billing cost exceeds a predefined threshold.

   5. Determine the set of credentials that have been leaked.
      • If a ticket or case is created, check whether the user/role name, user/role ID, or access key ID is recorded in the ticket or case.
      • If the alert is generated by SecMaster baseline inspection, you can view the baseline inspection result on the console and find the access key ID of the affected credential.
      • If the alert is reported by CTS, you can view the tracing result in the event list on the console. The resource name is the access key. The Credential field contains access_key_id, account_id, user_name, and other information.
   6. Determine when credentials may be compromised or leaked. Any API operation performed after this time shall be considered malicious, and any resource created after this time shall be considered leaked.
   7. If your application is interrupted, determine the possible incidents that cause the interruption. If the interrupt event is not related to a credential leak, check the deployment pipeline to determine whether any changes were made before the event occurred. You can use CTS to view logs of all account activities.
   8. Communicate incidents:
      • Determine stakeholder roles based on the organization's incident response plan.
      • Notify relevant stakeholders, including legal personnel, technical teams, and developers, and ensure that they are added to the work order and war room for continuous update.
   9. Communicate incidents with external parties.
      • Ensure that your legal counsel is aware of the situation and incorporate it into the status update for internal stakeholders, especially external communications.
      • Add colleagues responsible for public or external communication to the work order so that they can receive regular status updates about the incident and fulfill their communication responsibilities.
      • If your jurisdiction has regulations that require the reporting of such incidents, ensure that the person responsible for notifying the local or federal law enforcement agency in your organization also receives a notification of the incident/is added to the service ticket. Consult your legal counsel and law enforcement for guidance on the collection and preservation of evidence and regulatory authorities. Even if regulations do not require reporting to open databases, government agencies, or non-governmental organizations, your reports may help track similar activities or help others.

2. Contain incidents.

   You can disable compromised credentials or revoke permissions related to these credentials to prevent the use of compromised credentials to call APIs.

   1. Disable the compromised credentials identified by 1.
      1. If the user credentials are permanent user credentials, delete them on the console.
      2. If the user credentials are temporary credentials, they can be associated with the IAM role. To disable these functions, perform the following steps:
         1. Cancels all sessions of the current role. If the attacker obtains a new temporary security credential and continues the attack, go to 2.a.ii.2.
         2. Delete all IAM policies added to the role, modify existing policies to block all access, or modify the policy of the role to prevent attackers from taking the role.

            The credential is still valid for a specified period of time after being issued. Therefore, after the trust policy is modified, the credential can still be used within the validity period. 2.a.ii.1 and 2.a.ii.2 will prevent all users from using credentials obtained by taking on roles, including any legitimate users or applications.

   2. You can view the credentials that are continuously used on the CTS console within about 30 minutes, regardless of access keys, IAM users, or roles, and confirm that the compromised credentials have been disabled.

3. Eradicate incidents.

   You need to check which API operations are performed and which resources are created, deleted, or modified after the credential is damaged, and take measures to eliminate the impact.

   1. Use your preferred monitoring tool to access CTS and collect all API operations performed by the damaged credential. The log collection time is from the damage time to the current time.
      • If you use a third-party tool (such as Splunk) to collect CTS logs, obtain logs from the tool.
      • If you send logs to OBS instead of using a third-party tool, you can use LTS to collect, query, and store logs.
   2. On the LTS console, query all API operations performed after the credential is damaged or leaked.
   3. Pick out the API calls that may:
      • Access sensitive data, for example, OBS Object.
      • Create resources, such as databases and cloud servers.
      • Creates resources, including EC auto scaling groups.
      • Create or modify permissions, and check API methods including but not limited to CreateUser, CreateRole, AssumeRole*, Get*Token, Attach*Policy, *Image*, *Provider, Tag*, Create*, Delete* and Update*.
      • Delete affected cloud resources.
      • Modify affected cloud resources.
   4. Based on the result of the previous step, identify any applications that might have been affected. Obtain the ID or tag information of each affected resource and notify the resource owner.
   5. If additional credentials (such as IAM users and roles) are created, disable and delete all credentials of these resources based on the 2.a.
   6. Repeat 3.a to 3.e to check whether additional credentials are found until all credentials are handled.

4. Recover from incident.
   1. Restore the modified resource.
      • If the resource can be destroyed and replaced, a new resource is added.
      • If the resource cannot be replaced, perform either of the following operations:
        • Restore a resource from a backup.
        • New resources are prepared and configured in the application infrastructure, and damaged resources are isolated and removed from the application infrastructure.
        • Destroy damaged resources or continue to isolate them for evidence.
      • Restore the deleted resource.
        1. Check the application to which the resource belongs. If the resource tag is not listed in the CTS service item and the resource is supported by the cloud, check the configuration.
        2. If the deleted resource can be restored from the backup, restore it directly. If the deleted resource cannot be restored from the backup, obtain the resource configuration from CMDB, create the resource again, and configure it in the infrastructure of the application.

5. Post-incident activities.
   • Investigate and collect evidence on certain compromised resources, analyze the attack methods used by attackers on the compromised resources, and determine whether additional risks and risk mitigation measures need to be taken for related resources or applications.
     1. For any compromised resources that have been segregated for further analysis, forensic activities are performed on those resources and the findings are incorporated into ex post facto reporting.
     2. Ensure that the CMDB is correctly updated to reflect the current status of all affected resources and applications.
   • Review the incident itself and its response, determine which measures are effective and which are not, update the improvement process based on the information, and record the investigation results.