ALM-12062 OMS Parameter Configurations Mismatch with the Cluster Scale

Alarm Description

The system checks whether the OMS parameter configurations match with the cluster scale at each top hour. If the OMS parameter configurations do not meet the cluster scale requirements, the system generates this alarm. This alarm is automatically cleared when the OMS parameter configurations are modified.

Alarm Attributes

Alarm Parameters

Impact on the System

If the parameters configured for the current cluster are smaller than the configuration standard required by the cluster scale, problems such as job running delay and slow service page response may occur. In severe cases, the Agent or OMS process on the cluster node is abnormal. As a result, component jobs fail to be submitted and OMS data fails to be synchronized.

Possible Causes

The OMS parameter configurations mismatch with the cluster scale.

Handling Procedure

Check whether the OMS parameter configurations match with the cluster scale.

1. Log in to the active management node of the MRS cluster as user root.

   For details about how to log in to a cluster node, see Logging In to an MRS Cluster Node.

   Run the following command to switch to user omm:

   su - omm

2. Run the following command to open the Manager log:

   vi $BIGDATA_LOG_HOME/controller/scriptlog/modify_manager_param.log

   Search for log "Current oms configurations can not support xx nodes", where xx indicates the number of nodes in the current cluster.

3. Optimize the current cluster configuration by referring to Optimizing Manager Configurations Based on the Number of Cluster Nodes.
4. One hour later, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 5.

Collect fault information.

5. On FusionInsight Manager, choose O&M > Log > Download.
6. Select Controller from the Service and click OK.
7. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
8. Contact the O&M personnel and send the collected log information.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

Optimizing Manager Configurations Based on the Number of Cluster Nodes

1. Log in to the active management node of the MRS cluster as user root.

   Run the following command to switch to user omm:

   su - omm

2. Run the following command to switch the directory:

   cd ${BIGDATA_HOME}/om-server/om/sbin

3. Run the following command to view the current Manager configurations.

   sh oms_config_info.sh -q

4. Run the following command to specify the number of nodes in the current cluster:

   Command format:

   sh oms_config_info.sh -s Number of nodes

   Example:

   sh oms_config_info.sh -s 1000

   Enter y as prompted.

   The following configurations will be modified:
        Module       Parameter         Current               Target 
        Controller   controller.Xmx    4096m             =>  16384m
        Controller   controller.Xms    1024m             =>  8192m        Controller   controller.node.heartbeat.error.threshold     30000                      =>   60000                   
        Pms          pms.mem           8192m             =>  10240m 
   Do you really want to do this operation? (y/n):

   The configurations are updated successfully if the following information is displayed:

   ...
   Operation has been completed. Now restarting OMS server.                  [done]
   Restarted oms server successfully.

   

   • OMS is automatically restarted during the configuration update process.
   • Clusters with similar quantities of nodes have same Manager configurations. For example, when the number of nodes is changed from 100 to 101, no configuration item needs to be updated.