Quota for Submitting Flink Jobs to ZooKeeper Exceeds the Threshold

Symptom

The quota for submitting Flink jobs to ZooKeeper exceeds the threshold.

Possible Causes

• The number of submitted jobs exceeds the maximum number of jobs that can be processed by the flinkznode node.
• By default, each znode can store a maximum of 20000 files. The high availability (HA) policy of Flink allows files to be stored in the /flink_xxx directory on ZooKeeper. If the number of files exceeds the upper limit, the jobs fail to be submitted.

Solution

• Solution 1: Delete the root directory of flink_base.

  By default, Flink submits jobs in flink_base node. You can change the root directory of flink_base to avoid the problem.

  

• Solution 2: Modify the quota limit of the root directory that reports the error.
  1. Log in to any ZooKeeper node.

     Go to the ${BIGDATA_HOME}/FusionInsight_Current/X_X_quorumpeer/install/zookeeper/conf directory, find the zookeeper.keytab file, and upload the ticket to the client node where the command is executed.

     

  2. Run the following command to query the ticket information:

     klist -k

     

  3. Run the following command to connect to ZooKeeper:

     zkCli.sh zkCli.sh -server IP address of the ZooKeeper instance:Port number

     The default port number is 24002. You can obtain the value from the ZooKeeper parameter clientPort.

     

  4. Run the following command to change the total quota:

     setquota -n 250000 /flink_base

     If the jobs are not submitted to a znode in flink_base, determine the node on which the quota is to be modified based on the error messages in the log.