On this page

Show all

Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Alarm Reference/ ALM-17003 Oozie Service Unavailable

ALM-17003 Oozie Service Unavailable

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The system checks the Oozie service status in every 5 seconds. This alarm is generated when Oozie or a component on which Oozie depends cannot provide services properly.

This alarm is automatically cleared when the Oozie service recovers.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
17003	Critical	Error handling	Oozie	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	Details	Supplement to the alarm information.

Impact on the System

Oozie cannot be used to schedule jobs.

Possible Causes

The DBService service is abnormal or the data of Oozie stored in DBService is damaged.
The HDFS service is abnormal or the data of Oozie stored in HDFS is damaged.
The Yarn service is abnormal.
The Nodeagent process is abnormal.

Handling Procedure

Query the Oozie service health status code.

On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster >Services > Oozie. Click oozie (any one is OK) on the oozie WebUI. to go to the Oozie WebUI.

NOTE:

By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.
Add /servicehealth to the URL in the address box of the browser and access again. The value of statusCode is the current Oozie service health status code.

For example, visit https://10.10.0.117:20026/Oozie/oozie/130/oozie/servicehealth. The result is as follows:
```
{"beans":[{"name":"serviceStatus","statusCode":0}]}
```
If the health status code cannot be displayed or the browser does not respond, the service may be unavailable due to Oozie process fault. See 13 to rectify the fault.

Perform the operations based on the error code. For details, see Table 1.

**Table 1** Oozie service health status code
Status Code	Description	Error Cause	Solution
0	The service is running properly.	None	None
18002	The DBService service is abnormal.	Oozie fails to connect to DBService or the data stored in DBService is damaged.	See 4.
18003	The HDFS service is abnormal.	Oozie fails to connect to HDFS or the data stored in HDFS is damaged.	See 7.
18005	The MapReduce service is abnormal.	The Yarn service is abnormal.	See 11.

Check the DBService service.

On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services, and check whether the DBService service is running properly.
- If yes, go to 6.
- If no, go to 5.
Resolve the problem of DBService based on the alarm help and check whether the Oozie alarm is cleared.
- If yes, no further action is required.
- If no, go to 18.
Log in to the Oozie database to check whether the data is complete.
1. Log in to the active DBService node as user root.
  On the FusionInsight Manager page, choose Cluster > Name of the desired cluster > Services > DBService > Instance to view the IP address of the active DBservice node.
2. Run the following command to log in to the Oozie database:
  su - omm
  
  source ${BIGDATA_HOME}/FusionInsight_BASE_8.1.0.1/install/FusionInsight-dbservice-2.7.0/.dbservice_profile
  
  gsql -U Username -W Oozie database password -p 20051 -d Database name
3. After the login is successful, enter \d to check whether there are 15 data tables.
  The Oozie service has 15 data tables by default. If these data tables are deleted or the table structure is modified, the Oozie service may be unavailable. Contact the O&M engineers to back up the data and perform restoration.

Check the HDFS service.

On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services, and check whether the HDFS service is running properly.
- If yes, go to 9.
- If no, go to 8.
Resolve the problem of HDFS based on the alarm help and check whether the Oozie alarm is cleared.
- If yes, no further action is required.
- If no, go to 18.
Log in to HDFS to check whether the Oozie file directory structure is complete.
1. Download and install an HDFS client..
2. Log in to the client node as user root and run the following commands to check whether /user/oozie/share exists.
  If the cluster uses the security mode, perform security authentication.
  
  kinit admin
  
  hdfs dfs -ls /user/oozie/share
- If yes, go to 18.
- If no, go to 10.
In the Oozie client installation directory, manually upload the share directory to /user/oozie in HDFS, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 18.

Check the Yarn and MapReduce service.

On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services, and check whether the Yarn and MapReduce services are running properly.
- If yes, go to 18.
- If no, go to 12.
Resolve the problem of Yarn and MapReduce based on the alarm help and check whether the Oozie alarm is cleared.
- If yes, no further action is required.
- If no, go to 18.

Check the Oozie process.

Log in to each node of Oozie as user root.
Run the ps -ef | grep oozie command to check whether the Oozie process exists.
- If yes, go to 15.
- If no, go to 18.
Collect fault information in prestartDetail.log, oozie.log, and catalina.out in the Oozie log directory /var/log/Bigdata/oozie. If the alarm is not caused by manual misoperation, go to 16.

Check the Nodeagent process.

Log in to each node of Oozie as user root. Run the ps -ef | grep nodeagent command to check whether the Nodeagent process exists.
- If yes, go to 17.
- If no, go to 18.
Run the kill -9 The process ID of nodeagent command, wait 10 minutes, and check whether alarm is cleared.
- If yes, no further action is required.
- If no, go to 18.
Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.

Parent topic: Alarm Reference

Previous topic: ALM-16051 Percentage of Sessions Connected to MetaStore Exceeds the Threshold

Next topic: ALM-17004 Oozie Heap Memory Usage Exceeds the Threshold

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot