HBase Failed to Start with "FileNotFoundException" in RegionServer Logs

Symptom

HBase fails to start, and the RegionServer stays in the Restoring state.

Cause Analysis

1. Check the RegionServer log (/var/log/Bigdata/hbase/rs/hbase-omm-XXX.out). It is found that the following information is printed:

    | ERROR | RS_OPEN_REGION-ab-dn01:21302-2 | ABORTING region server ab-dn01,21302,1487663269375: The coprocessor org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService threw java.io.FileNotFoundException: File does not exist: hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar | org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2123)
   java.io.FileNotFoundException: File does not exist: hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar
   at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1399)
   at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
   at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
   at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1391)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2038)
   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2007)
   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1983)
   at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:168)
   at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
   at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:224)
   at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:365)
   at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:227)
   at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:783)
   at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:689)
   at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6312)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6622)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6594)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6550)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6501)
   at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
   at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
   at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)

2. Run the hdfs command on the client. It is found that the following file does not exist:

   hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar

3. When configuring the coprocessor for HBase, make sure that the path of the corresponding JAR package is correct. Otherwise, HBase cannot be started.

Solution

Use the Apache Kylin engine to interconnect with MRS and make sure that the JAR file of the Kylin engine exists.