Storm-HBase Development Guideline

Updated on 2022-08-16 GMT+08:00

View PDF

Scenario

This topic applies only to the interaction between Storm and HBase of MRS. Determine the versions of the jar packages described in this chapter based on the actual situation.

Security login modes include ticket login and keytab file login, and the procedures for these two login modes are the same. The ticket login mode is an open-source capability and requires manual ticket uploading to prevent ticket expiration, which may cause reliability and usability problems. Therefore, the keytab file login mode is recommended.

Procedure for Developing an Application

Verify that the Storm and HBase components have been installed and are running properly.
Import storm-examples to the IntelliJ IDEA development environment. For details, see Environment Preparation.
If security services are enabled in the cluster, perform the related configuration based on the login mode.
- keytab mode: Obtain a Human-Machine user from the administrator for login to the FusionInsight Manager platform and authentication, and obtain the keytab file of the user.
- Ticket mode: Obtain a Human-Machine user from the administrator for subsequent secure login, enable the renewable and forwardable functions of the Kerberos service, set the ticket update interval, and restart Kerberos and related components.
NOTE:
- The obtained user must belong to the storm group.
- The default validity period of a user password is 90 days. Therefore, the validity period of the obtained keytab file is 90 days. To prolong the validity period of the keytab file, modify the user password policy and obtain the keytab file again.
- The parameters for enabling the renewable and forwardable functions and setting the ticket update interval are on the System tab of the Kerberos service configuration page. The ticket update interval can be set to kdc_renew_lifetime or kdc_max_renewable_life based on the actual situation.
Download and install the HBase client.
Obtain the related configuration files using the following method.

Go to the /opt/clientHbase/HBase/hbase/conf directory on the installed HBase client, and obtain configuration files core-site.xml, hdfs-site.xml, and hbase-site.xml.

In keytab mode, obtain the keytab file by following 3. In ticket mode, no extra configuration file is required.

NOTE:

The obtained keytab file is named as user.keytab by default. A user can directory change the file name as required. However, the user must upload the changed file name as a parameter when submitting a task.
Obtain the related JAR packages.
- Go to the HBase/hbase/lib directory on the installed HBase client, and obtain the following JAR packages:
  - hbase-*.jar
  - hadoop-*.jar
  - jackson-core-asl-<version>.jar
  - jackson-mapper-asl-<version>.jar
  - commons-cli-<version>.jar
  - commons-io-<version>.jar
  - commons-lang-<version>.jar
  - commons-lang3-<version>.jar
  - commons-collections-<version>.jar
  - commons-configuration2-<version>.jar
  - guava-<version>.jar
  - protobuf-java-<version>.jar
  - netty-all-<version>.jar
  - zookeeper-<version>.jar
  - zookeeper-jute-<version>.jar
  - metrics-core-<version>.jar
  - commons-validator-<version>.jar
- Go to the HBase/hbase/lib/client-facing-thirdparty directory in the HBase client installation directory, and obtain the commons-logging-<version>.jar package.
- Go to the HBase/hbase/lib/jdbc directory in the HBase client installation directory, and obtain the htrace-core-<version>-incubating.jar package and the htrace-core4-<version>-incubating.jar package.
- Obtain the following JAR packages from the sample project /src/storm-examples/storm-examples/lib:
  - storm-hdfs-<version>.jar
  - storm-autocreds-<version>.jar

IntelliJ IDEA Code Sample

Create a topology.

 public static void main(String[] args) throws Exception  
     { 
         Config conf = new Config(); 
         //Add the plugin required for kerberos authentication to the list. The security mode is mandatory.
         setSecurityConf(conf,AuthenticationType.KEYTAB);

         if(args.length >= 2) 
         { 
         //The default keytab file name is changed by the user. Specify a new keytab file name as a parameter.
         conf.put(Config.STORM_CLIENT_KEYTAB_FILE, args[1]); 
         } 
         //hbase client configuration. Only ¡°hbase.rootdir¡± configuration item is provided, which is optional. 
         Map<String, Object> hbConf = new HashMap<String, Object>(); 
         if(args.length >= 3) 
         { 
         hbConf.put("hbase.rootdir", args[2]); 
         } 
         //Mandatory parameter. If it is not set, it is left blank. 
         conf.put("hbase.conf", hbConf); 

         //Spout generates a random word. 
         WordSpout spout = new WordSpout(); 
         WordCounter bolt = new WordCounter(); 

         //HbaseMapper, which is used for parsing tuple content.
         SimpleHBaseMapper mapper = new SimpleHBaseMapper() 
                 .withRowKeyField("word") 
                 .withColumnFields(new Fields("word")) 
                 .withCounterFields(new Fields("count")) 
                 .withColumnFamily("cf"); 

         //HBaseBolt, the first parameter is a table name. 
         //withConfigKey("hbase.conf")Transfer the hbase client configuration to HBaseBolt.
         HBaseBolt hbase = new HBaseBolt("WordCount", mapper).withConfigKey("hbase.conf"); 


         // wordSpout ==> countBolt ==> HBaseBolt 
         TopologyBuilder builder = new TopologyBuilder(); 

         builder.setSpout(WORD_SPOUT, spout, 1); 
         builder.setBolt(COUNT_BOLT, bolt, 1).shuffleGrouping(WORD_SPOUT); 
         builder.setBolt(HBASE_BOLT, hbase, 1).fieldsGrouping(COUNT_BOLT, new Fields("word")); 
         //Run a command to submit the topology. 
         StormSubmitter.submitTopology(args[0], conf, builder.createTopology()); 
 }

Running the Application and Viewing Results

Export the local JAR package. For details, see Packaging IntelliJ IDEA Code.
Combine the configuration files and JAR packages obtained respectively in 5 and 6, and export a complete service JAR package. For details, see Packaging Services.
Run a command to submit the topology.

In keytab mode, if the user changes the keytab file name, for example, huawei.keytab, the changed keytab file name must be added to the command as a parameter for description. The submission command example is as follows (the topology name is hbase-test):

storm jar /opt/jartarget/source.jar com.huawei.storm.example.hbase.SimpleHBaseTopology hbase-test huawei.keytab
NOTE:
- Before submitting source.jar in security mode, ensure that the Kerberos security login is implemented, and the login user and the user of the uploaded keytab file are the same user in keytab mode.
- HBaseBolt in the preceding example does not provide the function for creating tables. Therefore, you must verify that necessary tables exist in HBase. If the tables do not exist, run the create 'WordCount', 'cf' statement to manually create HBase shell tables.
- In HBase security mode, a user must have the permission to access related tables, column families, and columns. Therefore, the user must log in to the HBase cluster as an HBase administrator, run the grant command in HBase shell to apply for table access permission, such as WordCount, for the user, and submit the topology as the user.
After the topology is submitted successfully, log in to the HBase cluster to view the topology.
To log in with ticket mode, perform the following operations to regularly upload a ticket. The interval for uploading the ticket depends on the deadline for updating the ticket.
1. Add the following content to a new line at the end of the Storm/storm-1.2.1/conf/storm.yaml file in the Storm client installation directory:
  topology.auto-credentials:
  
  - org.apache.storm.security.auth.kerberos.AutoTGT
2. Run the ./storm upload-credentials hbase-test command.

Parent topic: More Information

Previous topic: Storm-HDFS Development Guideline

Next topic: Flux Development Guideline

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot