ALM-45616 CDL Job Execution Exception

Alarm Description

The system checks whether a CDL job is normal every 60 seconds. This alarm is reported when the CDL job is abnormal. This alarm is cleared when the job is restored or stopped.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
45616	Major	Quality of service	CDL	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm was generated.
	ServiceName	Specifies the service for which the alarm was generated.
	JobName	Specifies the job for which the alarm was generated.
	Username	Specifies the username of the job for which the alarm was generated.

Impact on the System

CDL tasks fail, and real-time data integration is interrupted.

Possible Causes

The CDL task fails to be executed due to incorrect parameter settings or other reasons. On the Job Management page of the CDL web UI, locate the row where the job is located and click Failed/Abnormal running in the Status column to view the failure cause, or view the failure cause in the logs.

Handling Procedure

Log in to FusionInsight Manager as a user who has the CDL job creation or administrator permission.
Choose O&M. In the navigation pane on the left, choose Alarm > Alarms, click in the row where Alarm ID is 45616, and view the name of the job for which this alarm is generated in Location.
Choose Cluster > Services > CDL and click the link next to CDLService UI to go to the CDL web UI.
Locate the failed job based on the task name in 2 and check whether Source is Hudi.
- If yes, go to 5.
- If no, go to 6.
On Manager, choose Cluster > Services > Yarn and click the link next to ResourceManager Web UI to go to the Yarn web UI. Search for the ID of the latest failed task in 2, select the task ID, click Logs, search for stdout, and go to 8.
Click Abnormal running or Failed in the Status column.
On the page that is displayed, view the error information and rectify the fault. For example, Figure 1 shows that the task running on Yarn is manually killed. For details, see trace error information, as shown in Figure 2.

Figure 1 CDL job exception

Figure 2 Trace error information
Rectify the fault based on the error information, execute the task again, and check whether the task can be executed successfully.
- If yes, no further action is required.
- If no, go to 9.

Collect the fault information.

On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Select CDL in the required cluster for Service.
Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.