Why Does the Spark Streaming Application Fail to Be Submitted After the Token Validity Period Expires?

Updated on 2022-09-22 GMT+08:00

View PDF

Question

Change the validity period of the Kerberos ticket and HDFS token to 5 minutes, set dfs.namenode.delegation.token.renew-interval to a value less than 60 seconds, and submit the Spark Streaming application. If the token expires, the error message below is displayed, and the application exits. Why?

token (HDFS_DELEGATION_TOKEN token 17410 for spark2x) is expired

Answer

Possible causes:
The credential refresh thread of the ApplicationMaster process uploads the updated credential file to the HDFS based on the token renew period multiplied by 0.75.

In the executor process, the credential refresh thread obtains the updated credential file from the HDFS based on the time ratio of the token renewal period multiplied by 0.8 to update the token in UserGroupInformation, preventing the token from being invalid.

When the credential refresh thread of the executor process detects that the current time is later than the credential file update time (token renew period x 0.8), it waits for 1 minute and then obtains the latest credential file from the HDFS to ensure that the AM has stored the updated credential file in the HDFS.

When the value of dfs.namenode.delegation.token.renew-interval is less than 60 seconds, the started executor detects that the current time is later than the time when the credential file is updated. One minute later, the executor obtains the latest credential file from the HDFS. However, the token is already invalid, and the task fails to be executed. Then, other executor processes retry within 1 minute. The task also fails to run on other executors. As a result, the executors that fail to run are added to the blacklist. If no executors are available, the application exits.

Solution:

In the Spark application scenario, set dfs.namenode.delegation.token.renew-interval to a value greater than 80 seconds. For details about the dfs.namenode.delegation.token.renew-interval parameter, see Table 1.

**Table 1** Parameter description
Parameter	Description	Default Value
dfs.namenode.delegation.token.renew-interval	This parameter is a server parameter. It specifies the maximum lifetime to renew a token. Unit: milliseconds.	86400000

Parent topic: Spark Streaming

Previous topic: What Should I Pay Attention to When Optimizing Spark Streaming Task Parameters?

Next topic: Why Does the Spark Streaming Application Fail to Be Started from the Checkpoint When the Input Stream Has No Output Logic?

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot