更新时间:2022-09-30 GMT+08:00

Hive配置类问题

  • Hive SQL执行报错:java.lang.OutOfMemoryError: Java heap space.

    解决方案:

    • 对于MapReduce任务,增大下列参数:

      set mapreduce.map.memory.mb=8192;

      set mapreduce.map.java.opts=-Xmx6554M;

      set mapreduce.reduce.memory.mb=8192;

      set mapreduce.reduce.java.opts=-Xmx6554M;

    • 对于Tez任务,增大下列参数:

      set hive.tez.container.size=8192;

  • Hive SQL对列名as为新列名后,使用原列名编译报错:Invalid table alias or column reference 'xxx'.

    解决方案:set hive.cbo.enable=true;

  • Hive SQL子查询编译报错:Unsupported SubQuery Expression 'xxx': Only SubQuery expressions that are top level conjuncts are allowed.

    解决方案:set hive.cbo.enable=true;

  • Hive SQL子查询编译报错:CalciteSubquerySemanticException [Error 10249]: Unsupported SubQuery Expression Currently SubQuery expressions are only allowed as Where and Having Clause predicates.

    解决方案:set hive.cbo.enable=true;

  • Hive SQL编译报错:Error running query: java.lang.AssertionError: Cannot add expression of different type to set.

    解决方案:set hive.cbo.enable=false;

  • Hive SQL执行报错:java.lang.NullPointerException at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.init.

    解决方案:set hive.map.aggr=false;

  • Hive SQL设置hive.auto.convert.join = true(默认开启)和hive.optimize.skewjoin=true执行报错:ClassCastException org.apache.hadoop.hive.ql.plan.ConditionalWork cannot be cast to org.apache.hadoop.hive.ql.plan.MapredWork.

    解决方案:set hive.optimize.skewjoin=false;

  • Hive SQL设置hive.auto.convert.join=true(默认开启)、hive.optimize.skewjoin=true和hive.exec.parallel=true执行报错:java.io.FileNotFoundException: File does not exist:xxx/reduce.xml.

    解决方案:

    • 方法一:切换执行引擎为Tez,详情请参考切换Hive执行引擎为Tez
    • 方法二:set hive.exec.parallel=false;
    • 方法三:set hive.auto.convert.join=false;
  • Hive on Tez执行Bucket表Join报错:NullPointerException at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys

    解决方案:set tez.am.container.reuse.enabled=false;