ALM-12085 Service Audit Log Dump Failure

Alarm Description

The system dumps service audit logs at 03:00 every day and stores them on the OMS node. This alarm is generated when the dump fails. This alarm is cleared when the next dump succeeds.

Alarm Attributes

Alarm Parameters

Impact on the System

If the component audit logs fail to be dumped and the audit logs are aged locally, the audit logs cannot be retrieved, affecting the service analysis and troubleshooting of the component.

Possible Causes

• The service audit logs are oversized.
• The OMS backup storage space is insufficient.
• The storage space of a host where the service is located is insufficient.

Handling Procedure

Check whether the service audit logs are oversized.

1. In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click to view the IP address of the host for which the alarm is generated and additional information.
2. Log in to the host where the alarm is generated as user root.
3. Run the vi ${BIGDATA_LOG_HOME}/controller/scriptlog/getLogs.log command to check whether there is the keyword "LOG SIZE is more than 5000MB."
   • If yes, go to Step 4.
   • If no, go to Step 5.

4. Check whether the oversized service audit logs are caused by exceptions.

The OMS backup storage space is insufficient.

5. Run the vi ${BIGDATA_LOG_HOME}/controller/scriptlog/getLogs.log command to check whether there is the keyword "Collect log failed, too many logs on."
   • If yes, obtain the host IP address following the keyword "Collect log failed, too many logs on" and go to Step 6.
   • If no, go to Step 11.

6. Log in to the host with the IP address obtained in Step 5 as user root.
7. Run the vi ${BIGDATA_LOG_HOME}/nodeagent/scriptlog/collectLog.log command to check whether there is the keyword "log size exceeds".
   • If yes, go to Step 9.
   • If no, go to Step 8.

8. Check whether the additional information of the alarm contains the keyword "no enough space".
   • If yes, go to Step 9.
   • If no, go to Step 11.

9. Perform the following operations to expand the disk capacity (only for MRS 3.1.2 and earlier versions) or reduce the maximum number of audit log backups:
   • Expand the capacity of the OMS node.
   • Run the following command to decrease the value of MAX_NUM_BK_AUDITLOG, which specifies the maximum number of audit log backups:

     vi ${CONTROLLER_HOME}/etc/om/componentsauditlog.properties

10. Wait until the next execution period (03:00) and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 11.

Check whether the space of the host where the service is located is insufficient.

11. Run the vi ${BIGDATA_LOG_HOME}/controller/scriptlog/getLogs.log command to check whether there is the keyword "Collect log failed, no enough space on hostIp."
    • If yes, obtain the IP address of the abnormal host and go to Step 12.
    • If no, go to Step 15.

12. Log in to the host with the IP address obtained as user root, and run the df "$BIGDATA_HOME/tmp" -lP | tail -1 | awk '{print ($4/1024)}' command to obtain the remaining space of the host log directory. Check whether the value is less than 1000 MB.
    • If it is, go to Step 13.
    • If it is not, go to Step 13.

13. Expand the capacity of the node
14. Wait until the next execution period (03:00) and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 15.

Collect fault information.

15. On FusionInsight Manager, choose O&M > Log > Download.
16. Select Controller for Service and click OK.
17. Click in the upper right corner, set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively, and click Download.
18. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None