Topology Submission Fails and the Message "Failed to check principle for keytab" Is Displayed

Symptom

An MRS streaming cluster in security mode is installed, and ZooKeeper, Storm, and Kafka are installed in the cluster.

When a topology is defined to access components such as HDFS and HBase and the topology fails to be submitted using client commands.

Possible Causes

• The submitted topology does not contain the keytab file of the user.
• The keytab file contained in the submitted topology is inconsistent with the user who submits the topology.
• The user.keytab file exists in the /tmp directory on the client, and the owner is not the running user.

Cause Analysis

1. Check the logs. Error information "Can not found user.keytab in storm.jar" is found. Details are as follows:

   [main] INFO  b.s.StormSubmitter - Get principle for stream@HADOOP.COM success
   [main] ERROR b.s.StormSubmitter - Can not found user.keytab in storm.jar.
   Exception in thread "main" java.lang.RuntimeException: Failed to check principle for keytab
   at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:219)
   at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:292)
   at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:176)
   at com.xxx.streaming.storm.example.hbase.SimpleHBaseTopology.main(SimpleHBaseTopology.java:77)

   Check the JAR file of the submitted topology. It is found that the keytab file is not contained.

2. Check the logs. Error information "The submit user is invalid,the principle is" is found. Details are as follows:

   [main] INFO  b.s.StormSubmitter - Get principle for stream@HADOOP.COM success
   [main] WARN  b.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the  client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock.
   [main] ERROR b.s.StormSubmitter - The submit user is invalid,the principle is : stream@HADOOP.COM
   Exception in thread "main" java.lang.RuntimeException: Failed to check principle for keytab
   at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:219)
   at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:292)
   at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:176)
   at com.xxx.streaming.storm.example.hbase.SimpleHBaseTopology.main(SimpleHBaseTopology.java:77)

   The authenticated user used to submit the topology is stream. However, the system displays a message indicating that the submit user is invalid during topology submission, indicating that the internal verification fails.

3. Check the JAR file of the submitted topology. It is found that the keytab file is contained.

   The principal parameter is set to zmk_kafka in the user.keytab file.

   

   It is found that the authenticated user does not match the principal in the user.keytab file.

4. Check the logs and find the error information "Delete the tmp keytab file failed, the keytab file is:/tmp/user.keytab". The detailed information is as follows:

   [main] WARN b.s.StormSubmitter - Delete the tmp keytab file failed, the keytab file is : /tmp/user.keytab
   [main] ERROR b.s.StormSubmitter - The submit user is invalid,the principle is : hbase1@HADOOP.COM
   Exception in thread "main" java.lang.RuntimeException: Failed to check principle for keytab
   at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:213)
   at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:286)
   at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:170)
   at com.touchstone.storm.cmcc.CmccDataHbaseTopology.main(CmccDataHbaseTopology.java:183)

   Check the /tmp directory. It is found that the user.keytab file exists and the file owner is not the running user.

Solution

• Ensure that the user.keytab file is carried when the topology is submitted.
• Ensure that the user for submitting the topology is the same as that of the user.keytab file.
• Delete the user.keytab file from the /tmp directory.