Guide for Fixing the Apache Log4j2 Remote Code Execution Vulnerability (CVE-2021-44228)

Prerequisites

• You have downloaded the patch tool package MRS_Log4j_Patch.tar.gz from the OBS path.
• You have determined the active OMS node in the cluster.
  

  Generally, OMS is deployed on two nodes, master1 and master2. You can use the following commands to determine the active OMS node. The node whose command output contains active is the active OMS node, and the node whose command output contains standby is the standby OMS node.

  For clusters whose version is earlier than MRS 3.x, use the following command:

  sh /opt/Bigdata/*/workspace0/ha/module/hacom/script/get_harole.sh

  For clusters whose version is later than MRS 3.x, use the following command:

  sh /opt/Bigdata/om-server*/OMS/workspace0/ha/module/hacom/script/get_harole.sh

Installing the Patch on Existing Cluster Nodes

1. Upload MRS_Log4j_Patch.tar.gz to the /home/omm directory on the active OMS node. For details, see How Do I Upload a Local File to a Node Inside a Cluster?.
2. Run the following commands to log in to the active OMS node as user root, modify the permission of the patch tool, switch to user omm, and decompress the patch tool package to the current directory:

   chown omm:wheel -R /home/omm/MRS_Log4j_Patch.tar.gz

   su - omm

   cd /home/omm

   tar -zxf MRS_Log4j_Patch.tar.gz

3. In the /home/omm/MRS_Log4j_Patch/bin/ips.ini file, configure the IP addresses (IP addresses of all nodes in the current cluster) of the nodes where the patch is to be installed.
   

   Configure one IP address in each line. No empty line is allowed.

4. Run the following scripts to install the patch:

   cd /home/omm/MRS_Log4j_Patch/bin

   nohup sh install.sh upgrade &

   Run the tail -f nohup.out command to check the execution status. If "upgrade patch success." is displayed, the execution is complete.

5. Log in to Manager, restart the affected components (you are advised to perform this operation during off-peak hours). For details, see List of Affected Components.
6. (Optional) If you want to install the patch for the newly downloaded client, run the following commands to install the patch for the component package first:

   su - omm

   cd /home/omm/MRS_Log4j_Patch/bin

   nohup sh install.sh upgrade_package &

   Run the tail -f nohup.out command to check the execution status. If "upgrade_package patch success." is displayed, the execution is complete.

   After the execution is complete, the client downloaded is the one with the patch installed.

   

   • This step takes a long time, and you do not need to restart the component after performing this step.
   • After the patch is installed, do not delete the files related to the patch directory. Otherwise, the patch cannot be uninstalled.

Installing the Patch on New Nodes

1. Disable Enable Component during the scale-out.

   

2. Upload MRS_Log4j_Patch.tar.gz to the /home/omm directory on the active OMS node. For details, see How Do I Upload a Local File to a Node Inside a Cluster?.
3. Run the following commands to log in to the active OMS node as user root, modify the permission of the patch tool, switch to user omm, and decompress the patch tool package to the current directory:

   chown omm:wheel -R /home/omm/MRS_Log4j_Patch.tar.gz

   su - omm

   cd /home/omm

   tar -zxf MRS_Log4j_Patch.tar.gz

4. In the /home/omm/MRS_Log4j_Patch/bin/ips.ini file, configure the IP addresses (IP addresses of the new nodes in the current cluster) of the nodes where the patch is to be installed.
   

   Configure one IP address in each line. No empty line is allowed.

5. Run the following scripts to install the patch:

   cd /home/omm/MRS_Log4j_Patch/bin

   nohup sh install.sh upgrade &

   Run the tail -f nohup.out command to check the execution status. If "upgrade patch success." is displayed, the execution is complete.

6. Log in to Manager and start instances on the new node.

Uninstalling the Patch

1. Log in to the active OMS node as user root and run the following commands to uninstall the patch:

   su - omm

   cd /home/omm/MRS_Log4j_Patch/bin

   nohup sh install.sh rollback &

   Run the tail -f nohup.out command to check the execution status. If "rollback patch success." is displayed, the execution is complete.

2. Log in to Manager, restart the affected components (you are advised to perform this operation during off-peak hours). For details, see List of Affected Components.
3. Perform the following operation if you have performed 6 in Installing the Patch on Existing Cluster Nodes during patch installation and you want to roll back the modification in the component package:

   Log in to the active OMS node as user root and run the following commands:

   su - omm

   cd /home/omm/MRS_Log4j_Patch/bin

   nohup sh install.sh rollback_package &

   Run the tail -f nohup.out command to check the execution status. If "rollback_package patch success." is displayed, the execution is complete.

(Optional) Upgrading the ECS Password Reset Plug-in

Huawei Cloud ECS provides the one-click password reset function. If the password of an ECS is lost or expires, you can use this function to reset the password with a few clicks. The password reset plug-in is a client process running in the ECS and does not provide any external network services. The password reset plug-in CloudResetPwdUpdateAgent uses the Apache Log4j2 component. According to the analysis and verification of Huawei Cloud security lab, the ECS password reset plug-in has no security risks.

To upgrade the Log4j2 version of this plug-in, perform the following steps:

1. Upload MRS_Log4j_Patch.tar.gz to the /home/omm directory on the active OMS node. For details, see How Do I Upload a Local File to a Node Inside a Cluster?.
2. Run the following commands to log in to the active OMS node as user root, modify the permission of the patch tool, switch to user omm, and decompress the patch tool package to the current directory:

   chown omm:wheel -R /home/omm/MRS_Log4j_Patch.tar.gz

   su - omm

   cd /home/omm

   tar -zxf MRS_Log4j_Patch.tar.gz

3. In the /home/omm/MRS_Log4j_Patch/bin/ips.ini file, configure the IP addresses (IP addresses of all nodes in the current cluster) of the nodes where the patch is to be installed.
   

   Configure one IP address in each line. No empty line is allowed.

4. Perform the following steps based on the node login mode:
   • Password login

     Run the following command:

     nohup sh install.sh upgrade_resetpwdagent passwd:Login password &

     For example, if the password is xyz123, run the following command:

     nohup sh install.sh upgrade_resetpwdagent passwd:xyz123 &

     Run the tail -f nohup.out command to check the execution status. If "upgrade_resetpwdagent patch success." is displayed, the execution is complete.

   • Key login
     1. Upload the private key file of user root to the /home/omm/MRS_Log4j_Patch/bin directory and ensure that the owner group of the file is root:root. Then, run the following commands:

        chown root:root /home/omm/MRS_Log4j_Patch/bin/Key file

        chmod 644 /home/omm/MRS_Log4j_Patch/bin/Key file

     2. Run the following commands:

        su - omm

        cd /home/omm/MRS_Log4j_Patch/bin

        nohup sh install.sh upgrade_resetpwdagent privatekey:Path of the private key file &

        For example, if the private key file path is /home/omm/MRS_Log4j_Patch/bin/abc.pem, run the following command:

        nohup sh install.sh upgrade_resetpwdagent privatekey:/home/omm/MRS_Log4j_Patch/bin/abc.pem &

        Run the tail -f nohup.out command to check the execution status. If "upgrade_resetpwdagent patch success." is displayed, the execution is complete.

List of Affected Components