Storm-HBase Development Guideline

Scenario

This topic applies only to the interaction between Storm and HBase. Determine the versions of the jar packages described in this chapter based on the actual situation.

Procedure for Developing an Application

1. Verify that the Storm and HBase components have been installed and are running properly.
2. Import storm-examples to the IntelliJ IDEA development environment. For details, see Environment Preparation.
3. Download and install the HBase client.
4. Obtain the related configuration files using the following method.

   Go to the /opt/clientHbase/HBase/hbase/conf directory on the installed HBase client, and obtain configuration files core-site.xml, hdfs-site.xml, and hbase-site.xml.

5. Obtain the related JAR packages.
   • Go to the HBase/hbase/lib directory on the installed HBase client, and obtain the following JAR packages:
     • hbase-*.jar
     • hadoop-*.jar
     • jackson-core-asl-<version>.jar
     • jackson-mapper-asl-<version>.jar
     • commons-cli-<version>.jar
     • commons-io-<version>.jar
     • commons-lang-<version>.jar
     • commons-lang3-<version>.jar
     • commons-collections-<version>.jar
     • commons-configuration2-<version>.jar
     • guava-<version>.jar
     • protobuf-java-<version>.jar
     • netty-all-<version>.jar
     • zookeeper-<version>.jar
     • zookeeper-<version>.jar
     • zookeeper-jute-<version>.jar
     • metrics-core-<version>.jar
     • commons-validator-<version>.jar
   • Go to the HBase/hbase/lib/client-facing-thirdparty directory in the HBase client installation directory, and obtain the commons-logging-<version>.jar package.
   • Go to the HBase/hbase/lib/jdbc directory in the HBase client installation directory, and obtain the htrace-core-<version>-incubating.jar package and the htrace-core4-<version>-incubating.jar package.
   • Obtain the following JAR packages from the sample project /src/storm-examples/storm-examples/lib:
     • storm-hdfs-<version>.jar
     • storm-autocreds-<version>.jar

IntelliJ IDEA Code Sample

Create a topology.

 public static void main(String[] args) throws Exception  
     { 
         Config conf = new Config(); 
         //Add the plugin required for kerberos authentication to the list. The security mode is mandatory.
         setSecurityConf(conf,AuthenticationType.KEYTAB);

         if(args.length >= 2) 
         { 
         //The default keytab file name is changed by the user. Specify a new keytab file name as a parameter.
         conf.put(Config.STORM_CLIENT_KEYTAB_FILE, args[1]); 
         } 
         //hbase client configuration. Only ¡°hbase.rootdir¡± configuration item is provided, which is optional. 
         Map<String, Object> hbConf = new HashMap<String, Object>(); 
         if(args.length >= 3) 
         { 
         hbConf.put("hbase.rootdir", args[2]); 
         } 
         //Mandatory parameter. If it is not set, it is left blank. 
         conf.put("hbase.conf", hbConf); 

         //Spout generates a random word. 
         WordSpout spout = new WordSpout(); 
         WordCounter bolt = new WordCounter(); 

         //HbaseMapper, which is used for parsing tuple content.
         SimpleHBaseMapper mapper = new SimpleHBaseMapper() 
                 .withRowKeyField("word") 
                 .withColumnFields(new Fields("word")) 
                 .withCounterFields(new Fields("count")) 
                 .withColumnFamily("cf"); 

         //HBaseBolt, the first parameter is a table name. 
         //withConfigKey("hbase.conf")Transfer the hbase client configuration to HBaseBolt.
         HBaseBolt hbase = new HBaseBolt("WordCount", mapper).withConfigKey("hbase.conf"); 


         // wordSpout ==> countBolt ==> HBaseBolt 
         TopologyBuilder builder = new TopologyBuilder(); 

         builder.setSpout(WORD_SPOUT, spout, 1); 
         builder.setBolt(COUNT_BOLT, bolt, 1).shuffleGrouping(WORD_SPOUT); 
         builder.setBolt(HBASE_BOLT, hbase, 1).fieldsGrouping(COUNT_BOLT, new Fields("word")); 
         //Run a command to submit the topology. 
         StormSubmitter.submitTopology(args[0], conf, builder.createTopology()); 
 }

Running the Application and Viewing Results

1. Export the local JAR package. For details, see Packaging IntelliJ IDEA Code.
2. Combine the configuration files and JAR packages obtained respectively in 4 and 5, and export a complete service JAR package. For details, see Packaging Services.
3. Run a command to submit the topology.

   storm jar /opt/jartarget/source.jar com.huawei.storm.example.hbase.SimpleHBaseTopology hbase-test

   

   HBaseBolt in the preceding example does not provide the function for creating tables. Therefore, you must verify that necessary tables exist in HBase. If the tables do not exist, run the create 'WordCount', 'cf' statement to manually create HBase shell tables.

4. After the topology is submitted successfully, log in to the HBase cluster to view the topology.