ALM-45636 Number of Consecutive Checkpoint Failures of a Flink Job Exceeds the Threshold

Alarm Description

The system checks the number of consecutive checkpoint failures at intervals defined by the parameter metrics.reporter.alarm.interval (default value: 30s). This alarm is generated when the number of consecutive checkpoint failures of a FlinkServer job reaches the configured threshold specified by job.alarm.checkpoint.consecutive.failures.num (default value: 5). This alarm is cleared when checkpoints are recovered or the job is successfully restarted.

Alarm Parameter Configuration

Configure alarm parameters for jobs submitted on FlinkServer.

1. Log in to MRS Manager, choose Cluster > Services > Flink, click Configurations and then All Configurations, and search for the alarm parameters to be modified.
2. Change the values of the alarm parameters for FlinkServer(Role) and click Save.
3. Click the Instances tab, select the FlinkServer instances, and choose More > Restart Instance. The new alarm parameter values take effect after the instances restart.

Configure alarm parameters for jobs submitted on the Flink client.

1. Run the following command to go to the Flink client installation directory, for example, /opt/hadoopclient/Flink/flink:

   cd /opt/hadoopclient/Flink/flink

2. Run the following command to edit the flink-conf.yaml configuration file on the client and modify the alarm parameter values:

   vim conf/flink-conf.yaml

3. After the modification is complete, run :wq to save the modification and exit.

Alarm Attributes

Alarm Parameters

Impact on the System

The Flink job may fail. You need to check the status and logs of the Flink job to locate the fault. This is a job-level alarm and has no impact on FlinkServer.

Possible Causes

You can view failure causes in specific logs.

Handling Procedure

1. Log in to Manager as a user who has the FlinkServer management permission.
2. Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
3. Locate the failed task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the native Yarn page.
   Figure 1 Application ID of a job
   
   • If yes, go to Step 4.
   • If no, go to Step 6.

4. Click the application ID of the failed job to go to the job page.
   1. Click Logs in the Logs column to view JobManager logs.
      Figure 2 Clicking Logs
      
   2. Click the ID in the Attempt ID column and click Logs in the Logs column to view TaskManager logs.
      Figure 3 Clicking the ID in the Attempt ID column
      
      Figure 4 Clicking Logs
      
      

      You can also log in to Manager as a user who has the FlinkServer management permission. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.

5. View the logs of the failed job to rectify the fault, or contact the O&M engineers and send the collected fault logs. No further action is required.

If logs are unavailable on the Yarn page, download logs from HDFS.

6. On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/logs/Application ID of the failed job directory.
7. View the logs of the failed job to rectify the fault, or contact the O&M engineers and send the collected fault logs.

Alarm Clearance

This alarm is cleared when FlinkServer job checkpoints are recovered or the job is successfully restarted.

Related Information

None.