文档首页/ MapReduce服务 MRS/ 用户指南(巴黎区域)/ 故障排除/ 使用Hive/ drop partition操作,有大量分区时操作失败
更新时间:2024-10-11 GMT+08:00

drop partition操作,有大量分区时操作失败

问题背景与现象

执行drop partitions 操作,执行异常:

MetaStoreClient lost connection. Attempting to reconnect. | org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:187)
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.xxx(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
...

查看对应metaStore日志,有StackOverFlow异常

2017-04-22 01:00:58,834 | ERROR | pool-6-thread-208 | java.lang.StackOverflowError
at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330)
at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)

原因分析

drop partition的处理逻辑是将找到所有满足条件的分区,将其拼接起来,最后统一删除。由于分区数过多,拼删元数据堆栈较深,出现StackOverFlow异常。

解决办法

分批次删除分区。