Why the Job Fails with the "HDFS_DELEGATION_TOKEN Expired" Exception?

Question

Why is the "HDFS_DELEGATION_TOKEN expired" exception reported when a job fails in security mode?

Answer

HDFS_DELEGATION_TOKEN expires because the token is not updated or it is accessed after max. lifetime.

You can perform the following operations to adjust the token's maximum lifetime and ensure that it exceeds the job's expected runtime.

1. Log in to FusionInsight Manager.

   For details about how to log in to FusionInsight Manager, see Accessing MRS FusionInsight Manager.

2. Choose Cluster > Services > HDFS > Configurations > All Configurations.
3. Search for dfs.namenode.delegation.token.max-lifetime and change its values as required.

   This parameter specifies the maximum lifetime of a token. It is recommended that you set this parameter to a value several hours longer than the expected job runtime.

   • Unit: millisecond
   • Default value: 604800000, which indicates one week
   • Value range: 10,000 to 10,000,000,000,000

4. Save the settings. Restart the expired service or instance for the configuration to take effect.