When a Node Outside a Cluster Is Connected to a Cluster with Kerberos Authentication Enabled, HTTP Cannot Find the Corresponding Record in the Kerberos Database

Question

The presto-examples-1.0-SNAPSHOT-jar-with-dependencies.jar file is running properly on nodes in the cluster. However, when PrestoJDBCExample running on a node outside the cluster connect to the cluster with Kerberos authentication enabled, the following error messages is displayed:

Error 1:

java.sql.SQLException:
Kerberos error for [HTTP@10.33.11.138]: No valid credentials provided
(Mechanism level: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - UNKNOWN_SERVER))
at
io.prestosql.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:281)
at
io.prestosql.jdbc.PrestoStatement.execute(PrestoStatement.java:229)
at
io.prestosql.jdbc.PrestoStatement.executeQuery(PrestoStatement.java:78)
at PrestoJDBCExample.main(PrestoJDBCExample.java:68)
Caused by:
io.prestosql.jdbc.$internal.client.ClientException: Kerberos error for
[HTTP@10.33.11.138]: No valid credentials provided (Mechanism level: No valid
credentials provided (Mechanism level: Server not found in Kerberos database
(7) - UNKNOWN_SERVER))
at
io.prestosql.jdbc.$internal.client.SpnegoHandler.generateToken(SpnegoHandler.java:174)
at
io.prestosql.jdbc.$internal.client.SpnegoHandler.authenticate(SpnegoHandler.java:140)
at
io.prestosql.jdbc.$internal.client.SpnegoHandler.authenticate(SpnegoHandler.java:128)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RetryAndFollowUpInterceptor.followUpRequest(RetryAndFollowUpInterceptor.java:289)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:157)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
io.prestosql.jdbc.$internal.client.SpnegoHandler.intercept(SpnegoHandler.java:115)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at io.prestosql.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
io.prestosql.jdbc.$internal.client.OkHttpUtil.lambda$userAgent$0(OkHttpUtil.java:64)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at
io.prestosql.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at
io.prestosql.jdbc.$internal.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at
io.prestosql.jdbc.$internal.okhttp3.RealCall.execute(RealCall.java:77)
at
io.prestosql.jdbc.$internal.client.JsonResponse.execute(JsonResponse.java:131)
at
io.prestosql.jdbc.$internal.client.StatementClientV1.<init>(StatementClientV1.java:132)
at
io.prestosql.jdbc.$internal.client.StatementClientFactory.newStatementClient(StatementClientFactory.java:24)
at
io.prestosql.jdbc.QueryExecutor.startQuery(QueryExecutor.java:46)
at io.prestosql.jdbc.PrestoConnection.startQuery(PrestoConnection.java:714)
at
io.prestosql.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:241)
... 3 more
Caused by: GSSException:
No valid credentials provided (Mechanism level: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER))
at
sun.security.jgss.spnego.SpNegoContext.initSecContext(SpNegoContext.java:454)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
io.prestosql.jdbc.$internal.client.SpnegoHandler.generateToken(SpnegoHandler.java:167)
... 23 more
Caused by: GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos database
(7) - UNKNOWN_SERVER)
at
sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
sun.security.jgss.spnego.SpNegoContext.GSS_initSecContext(SpNegoContext.java:882)
at
sun.security.jgss.spnego.SpNegoContext.initSecContext(SpNegoContext.java:317)
... 26 more
Caused by: KrbException:
Server not found in Kerberos database (7) - UNKNOWN_SERVER
at
sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
at
sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
at
sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
at
sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:466)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
... 30 more
Caused by: KrbException:
Identifier doesn't match expected value (906)
at
sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at
sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at
sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
at
sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 36 more

Error 2:

java.sql.SQLException:
Authentication failed: Authentication failed for token:
YIIC1wYGKwYBBQUCoIICyzCCAsegDTALBgkqhkiG9xIBAgKhBAMCAXaiggKuBIICqmCCAqYGg0MjBfNzI0Q180QUFCX0FBOEJfNzIwN0E3OEUwOEU1LkNPTaIfMB2gAwIBAKEWMBQbBEhUVFAbDDE5Mi4xNjguMC45MaOCASYwggEioAMCARKhAwIBAaKCARQEggEQpYLaTplwpMc0EjXgU/+bAn9Evk1UHysyhTPajzFpHtxzTAZCPm85ROufJ+cLlIoGEcp2JBH9Le8bSL2Y1cE0mdK2MIw2+S7J9G0mZKQYugMUXoIqS14XbnA8tvngPEXFa6e15lnqEUoRNb4yHt7Rr/zRvsPbWKCU4HQNkBtI8HSKAid2K2JpTuVvXkOQpa+kgfMwgfCgAwIBEqKB6ASB5U3I4qvcqbfPhy+0o97agfW9xBfeuZUjfKUUQC165Z8nbY7RJmM+3v6mrqsYRcGG2Agepd9F+neeL7Ljcf7ASzfSb4otvPDTWsvo7/TS097xdi+ixkyjDP4EDfbwXaYwASU9zXZZ75FJoO+lY3DLsyX68rgwSJioIIjK4jDRqVVoXTNRbeWipkIOoLsXHf9q+qUBQvbd3xhUIkPkng6pOCRV9gr6E=
        at
com.facebook.presto.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:271)
        at
com.facebook.presto.jdbc.PrestoStatement.execute(PrestoStatement.java:227)
        at
com.facebook.presto.jdbc.PrestoStatement.executeQuery(PrestoStatement.java:76)
        at
PrestoJDBCExample.main(PrestoJDBCExample.java:65)
Caused by:
com.facebook.presto.jdbc.internal.client.ClientException: Authentication failed:
Authentication failed for token:
YIIC1wYGKwYBBQUCoIICyzCCAsogcDBQAgAAAAo4IBhGGCAYAwggF8oAMCAQWhKhsoNDI1Nzg0MjBfNzI0Q180QUFCX0FBOEJfNzIwN0E3OEUwOEU1LkNPTaIfMB2gAwIBAKEWMBQbBEhUVFAbDDE5Mi4xNjguMC45MaOCASYwggEioAMtRrGdWOQlMggfUPbendaKESx1QtuEpJuoGtyPJ9QzKI4rbk1UHysyhTPajzFpHtxzTAZCPm85ROufJ+cLlIoGEcp2JBH9Le8bSL2Y1cE0mdK2MIw2+S7J9G0mZKQYugMUXoIqS14XbnA8tvngPEXFa6xEICV2sE02w1fNdOMilfeqZffUMfNBSanRkHQFW1xlyb3EK2JpTuVvXkOQpa+kgfMwgfCgAwIBEqKB6ASB5U3I4qvcqbfPhy+0o97agfW9xBfeuZUjfKUUQC165Z8nbY7RJmM+3v6mrqsYRcGG2AgepV2NKDSySvpgzPEfGspJPkVeBbi5YJSVL3G/fOFNkpLDjDpjDP4EDfbwXaYwASU9zXZZ75FJoO+lY3DLsyX68rgwSJioIIjK4jDRqVVoXTNRbeWipkIOoLsXHf9q+qUBQvbd3xhUIkPkng6pOCRV9gr6
        at
com.facebook.presto.jdbc.internal.client.StatementClientV1.requestFailedException(StatementClientV1.java:432)
        at
com.facebook.presto.jdbc.internal.client.StatementClientV1.<init>(StatementClientV1.java:132)
        at
com.facebook.presto.jdbc.internal.client.StatementClientFactory.newStatementClient(StatementClientFactory.java:24)
        at
com.facebook.presto.jdbc.QueryExecutor.startQuery(QueryExecutor.java:46)
        at
com.facebook.presto.jdbc.PrestoConnection.startQuery(PrestoConnection.java:683)
        at
com.facebook.presto.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:239)
        ... 3 more

Answer

The principal of HTTP concatenated by the client is inconsistent with that in the Kerberos database (Error 1) or the obtained token cannot be connected to Presto.

Run the cat /etc/hosts command in the cluster to add the IP address and host name of the Presto coordinator to the /etc/hosts file of the current node.