"Failed to execute session hooks: over max connections" Reported by Beeline

Symptom

The default maximum connections to HiveServer are 200. When the number of connections exceeds 200, Beeline reports error "Failed to execute session hooks: over max connections."

beeline> [root@172-27-16-38 c70client]# beeline
Connecting to jdbc:hive2://129.188.82.38:24002,129.188.82.36:24002,129.188.82.35:24002/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;sasl.qop=auth-conf;auth=KERBEROS;principal=hive/hadoop.hadoop.com@HADOOP.COM
Debug is  true storeKey false useTicketCache true useKeyTab false doNotPrompt false ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is false principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false
Acquire TGT from Cache
Principal is xxx@HADOOP.COM
Commit Succeeded 

Error: Failed to execute session hooks: over max connections. (state=,code=0)
Beeline version 1.2.1 by Apache Hive

The HiveServer log (/var/log/Bigdata/hive/hiveserver/hive.log) shows that error "over max connections" is reported.

2018-05-03 04:31:56,728 | WARN  | HiveServer2-Handler-Pool: Thread-137 | Error opening session:  | org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:542)
org.apache.hive.service.cli.HiveSQLException: Failed to execute session hooks: over max connections.
	at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:322)
	at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:663)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:527)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1257)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1242)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:710)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hive.service.cli.HiveSQLException: over max connections.
	at org.apache.hadoop.hive.transporthook.SessionControllerTsaslTransportHook.checkTotalSessionNumber(SessionControllerTsaslTransportHook.java:208)
	at org.apache.hadoop.hive.transporthook.SessionControllerTsaslTransportHook.postOpen(SessionControllerTsaslTransportHook.java:163)
	at org.apache.hadoop.hive.transporthook.SessionControllerTsaslTransportHook.run(SessionControllerTsaslTransportHook.java:134)
	at org.apache.hive.service.cli.session.SessionManager.executeSessionHooks(SessionManager.java:432)
	at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:314)
	... 12 more

Cause Analysis

Heavy service traffic causes the number of connections to one HiveServer node to exceed 200, and the maximum number of connections to HiveServer needs to be increased.

Solution

1. Go to the Hive configuration page.
   • For versions earlier than MRS 2.0.1: Log in to MRS Manager, choose Services > Hive > Service Configuration, and select All from the Basic drop-down list.
   • For MRS 2.0.1 or later: Click the cluster name on the MRS console, choose Components > Hive > Service Configuration, and select All from the Basic drop-down list.
     

     If the Components tab is unavailable, complete IAM user synchronization first. (On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.)

   • For MRS 3.x or later: Log in to FusionInsight Manager and choose Cluster. Click the name of the target cluster and choose Services > Hive > Configurations > All Configurations.

2. Search for hive.server.session.control.maxconnections and set it to a proper value not greater than 1000.
3. Save the configuration and restart the affected services or instances.