Task Execution Fails Because of Stack Memory Overflow

Symptom

When Hive performs a query operation, error "Error running child: java.lang.StackOverflowError" is reported. The error details are as follows:

FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.StackOverflowError
at org.apache.hive.come.esotericsoftware.kryo.io.Input.readVarInt(Input.java:355)
at org.apache.hive.come.esotericsoftware.kryo.util.DefautClassResolver.readName(DefautClassResolver.java:127)
at org.apache.hive.come.esotericsoftware.kryo.util.DefautClassResolver.readClass(DefautClassResolver.java:115)
at org.apache.hive.come.esotericsoftware.kryo.Kryo.readClass(Kryo.java.656)
at org.apache.hive.come.esotericsoftware.kryo.kryo.readClassAnd0bject(Kryo.java:767)
at org.apache.hive.come.esotericsoftware.kryo.serializers.collectionSerializer.read(CollectionSerializer.java:112)

Cause Analysis

Error "java.lang.StackOverflowError" indicates the memory overflow of the thread stack. It may occur if there are multiple levels of calls (for example, infinite recursive calls) or the thread stack is too small.

Solution

Adjust the stack memory in the JVM parameters of the Map and Reduce stages during execution of a MapReduce job, that is, mapreduce.map.java.opts (adjusting the stack memory of Map) and mapreduce.reduce.java.opts (adjusting the stack memory of Reduce). The following uses the mapreduce.map.java.opts parameter as an example.

• To increase the Map memory temporarily (only valid for Beeline):

  Run the set mapreduce.map.java.opts=-Xss8G; command on the Beeline client. (Change the value as required.)

• To permanently increase the Map memory specified by the mapreduce.map.memory.mb and mapreduce.map.java.opts parameters:
  1. Go to the Hive configuration page.
     • For versions earlier than MRS 2.0.1: Log in to MRS Manager, choose Services > Hive > Service Configuration, and select All from the Basic drop-down list.
     • For MRS 2.0.1 or later: Click the cluster name on the MRS console, choose Components > Hive > Service Configuration, and select All from the Basic drop-down list.
       

       If the Components tab is unavailable, complete IAM user synchronization first. (On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.)

     • For MRS 3.x or later: Log in to FusionInsight Manager and choose Cluster. Click the name of the target cluster and choose Services > Hive > Configurations > All Configurations.
  2. Add custom parameter mapreduce.map.java.opts and set it to a proper value.
  3. Save the configuration and restart the affected services or instances.

     Note that the modification takes effect after a service restart. During the restart, the Hive service is unavailable.