ALM-17003 Oozie Service Unavailable

Alarm Description

The system checks the Oozie service status in every 5 seconds. This alarm is generated when Oozie or a component on which Oozie depends cannot provide services properly.

This alarm is automatically cleared when the Oozie service recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

Oozie cannot be used to schedule jobs.

Possible Causes

• The DBService service is abnormal or the data of Oozie stored in DBService is damaged.
• The HDFS service is abnormal or the data of Oozie stored in HDFS is damaged.
• The Yarn service is abnormal.
• The Nodeagent process is abnormal.

Handling Procedure

Query the Oozie service health status code.

1. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster >Services > Oozie. Click oozie (any one is OK) on the oozie WebUI. to go to the Oozie WebUI.
   

   By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.

2. Add /servicehealth to the URL in the address box of the browser and access again. The value of statusCode is the current Oozie service health status code.

   For example, visit https://10.10.0.117:20026/Oozie/oozie/130/oozie/servicehealth. The result is as follows:

   {"beans":[{"name":"serviceStatus","statusCode":0}]}

   If the health status code cannot be displayed or the browser does not respond, the service may be unavailable due to Oozie process fault. See 13 to rectify the fault.

3. Perform the operations based on the error code. For details, see Table 1.

   Table 1 Oozie service health status code

   Status Code	Description	Error Cause	Solution
   0	The service is running properly.	None	None
   18002	The DBService service is abnormal.	Oozie fails to connect to DBService or the data stored in DBService is damaged.	See 4.
   18003	The HDFS service is abnormal.	Oozie fails to connect to HDFS or the data stored in HDFS is damaged.	See 7.
   18005	The MapReduce service is abnormal.	The Yarn service is abnormal.	See 11.

Check the DBService service.

4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services, and check whether the DBService service is running properly.
   • If yes, go to 6.
   • If no, go to 5.

5. Resolve the problem of DBService based on the alarm help and check whether the Oozie alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 18.

6. Log in to the Oozie database to check whether the data is complete.
   1. Log in to the active DBService node as user root.

      On the FusionInsight Manager page, choose Cluster > Name of the desired cluster > Services > DBService > Instance to view the IP address of the active DBservice node.

   2. Run the following command to log in to the Oozie database:

      su - omm

      source ${BIGDATA_HOME}/FusionInsight_BASE_8.1.0.1/install/FusionInsight-dbservice-2.7.0/.dbservice_profile

      gsql -U Username -W Oozie database password -p 20051 -d Database name

   3. After the login is successful, enter \d to check whether there are 15 data tables.

      The Oozie service has 15 data tables by default. If these data tables are deleted or the table structure is modified, the Oozie service may be unavailable. Contact the O&M engineers to back up the data and perform restoration.

Check the HDFS service.

7. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services, and check whether the HDFS service is running properly.
   • If yes, go to 9.
   • If no, go to 8.

8. Resolve the problem of HDFS based on the alarm help and check whether the Oozie alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 18.

9. Log in to HDFS to check whether the Oozie file directory structure is complete.
   1. Download and install an HDFS client..
   2. Log in to the client node as user root and run the following commands to check whether /user/oozie/share exists.

      If the cluster uses the security mode, perform security authentication.

      kinit admin

      hdfs dfs -ls /user/oozie/share

   • If yes, go to 18.
   • If no, go to 10.

10. In the Oozie client installation directory, manually upload the share directory to /user/oozie in HDFS, and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 18.

Check the Yarn and MapReduce service.

11. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services, and check whether the Yarn and MapReduce services are running properly.
    • If yes, go to 18.
    • If no, go to 12.

12. Resolve the problem of Yarn and MapReduce based on the alarm help and check whether the Oozie alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 18.

Check the Oozie process.

13. Log in to each node of Oozie as user root.
14. Run the ps -ef | grep oozie command to check whether the Oozie process exists.
    • If yes, go to 15.
    • If no, go to 18.

15. Collect fault information in prestartDetail.log, oozie.log, and catalina.out in the Oozie log directory /var/log/Bigdata/oozie. If the alarm is not caused by manual misoperation, go to 16.

Check the Nodeagent process.

16. Log in to each node of Oozie as user root. Run the ps -ef | grep nodeagent command to check whether the Nodeagent process exists.
    • If yes, go to 17.
    • If no, go to 18.

17. Run the kill -9 The process ID of nodeagent command, wait 10 minutes, and check whether alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 18.

18. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.