Storm-HBase Development Guideline

Updated on 2024-12-10 GMT+08:00

View PDF

Scenario

This topic applies only to the interaction between Storm and HBase. Determine the versions of the JAR files described in this section based on the actual situation.

Login in security mode is classified into ticket login and keytab file login, and the procedures for these two login modes are the same. The ticket login mode is an open-source capability and requires manual ticket uploading, which may cause reliability and usability problems. Therefore, the keytab file login mode is recommended.

Procedure for Developing an Application

Verify that the Storm and HBase components have been installed and are running correctly.
Import storm-examples to the Eclipse development environment. For details, see Importing and Configuring Storm Sample Projects.
If security services are enabled in the cluster, perform the related configuration based on the login mode.
- Keytab mode: You need to obtain a human-machine user from the administrator for authentication and obtain the keytab file of the user.
- Ticket mode: Obtain a human-machine user from the administrator for subsequent secure login, enable the renewable and forwardable functions of the Kerberos service, set the ticket update period, and restart Kerberos and related components.
NOTE:
- The obtained user must belong to the storm group.
- The parameters for enabling the renewable and forwardable functions and setting the ticket update interval are on the System tab of the Kerberos service configuration page. The ticket update interval can be set to kdc_renew_lifetime or kdc_max_renewable_life based on the actual situation.
Download and install the HBase client program.
Obtain the related configuration files by performing the following operations:

Go to the /opt/client/HBase/hbase/conf directory on the installed HBase client, and obtain configuration files core-site.xml, hdfs-site.xml, and hbase-site.xml. Copy the obtained files to the src/main/resources directory of the sample project.

In keytab mode, obtain the keytab file by following 3. In ticket mode, no extra configuration file is required.

NOTE:

The obtained keytab file is named as user.keytab by default. A user can directly change the file name as required. However, the user must upload the changed file name as a parameter when submitting a task.

Eclipse Sample Code

Create a topology.

 public static void main(String[] args) throws Exception   
      {  
          Config conf = new Config(); 

      //Add the plugin required for Kerberos authentication to the list. This operation is mandatory in security mode.  
          setSecurityConf(conf,AuthenticationType.KEYTAB); 

          if(args.length >= 2)  
          {  
          //The default keytab file name is changed by the user. Specify the new keytab file name as a parameter.  
          conf.put(Config.STORM_CLIENT_KEYTAB_FILE, args[1]);  
          }  
          //HBase client configuration. Only the hbase.rootdir configuration item is provided, which is optional.  
          Map<String, Object> hbConf = new HashMap<String, Object>();  
          if(args.length >= 3)  
          {  
              hbConf.put("hbase.rootdir", args[2]);  
          }  
          //Mandatory parameter. If it is not set, it is left blank.  
          conf.put("hbase.conf", hbConf);  

          //spout is a random word.  
          WordSpout spout = new WordSpout();  
          WordCounter bolt = new WordCounter();  

          //HbaseMapper, which is used for parsing tuple content.  
          SimpleHBaseMapper mapper = new SimpleHBaseMapper()  
                  .withRowKeyField("word")  
                  .withColumnFields(new Fields("word"))  
                  .withCounterFields(new Fields("count"))  
                  .withColumnFamily("cf");  

          //HBaseBolt. The first parameter is a table name.  
          //withConfigKey("hbase.conf") Transfer the HBase client configuration to HBaseBolt.  
          HBaseBolt hbase = new HBaseBolt("WordCount", mapper).withConfigKey("hbase.conf");  


          // wordSpout ==> countBolt ==> HBaseBolt  
          TopologyBuilder builder = new TopologyBuilder();  

          builder.setSpout(WORD_SPOUT, spout, 1);  
          builder.setBolt(COUNT_BOLT, bolt, 1).shuffleGrouping(WORD_SPOUT);  
          builder.setBolt(HBASE_BOLT, hbase, 1).fieldsGrouping(COUNT_BOLT, new Fields("word"));  
          //Run the related command to submit the topology.  
          StormSubmitter.submitTopology(args[0], conf, builder.createTopology());  
  }

Running the Application and Viewing Results

In the root directory of Storm sample code, run the mvn package command. After the command is executed successfully, the storm-examples-1.0.jar file is generated in the target directory.
Run the related command to submit the topology.

In keytab mode, if the user changes the keytab file name, for example, huawei.keytab, the changed keytab file name must be added to the command as a parameter for description. The submission command example is as follows (the topology name is hbase-test):

storm jar /opt/jartarget/storm-examples-1.0.jar com.huawei.storm.example.hbase.SimpleHBaseTopology hbase-test huawei.keytab

NOTE:

In security mode, ensure that Kerberos security login has been performed before the source.jar file is submitted. In keytab mode, the login user and the user to whom the uploaded keytab file belongs must be the same user.

HBaseBolt in the preceding example does not provide the function for creating tables. Therefore, you must verify that desired tables exist in HBase. If the tables do not exist, run the create 'WordCount', 'cf' statement to manually create HBase shell tables.

In HBase security mode, a user must have the permission to access related tables, column families, and columns. Therefore, the user must log in to the HBase cluster as an HBase administrator, run the grant command in HBase shell to apply for table access permission, such as WordCount, for the user, and submit the topology as the user.
After the topology is submitted successfully, log in to the HBase cluster to view the topology.
To perform login in ticket mode, perform the following operations to regularly upload a ticket. The interval for uploading the ticket depends on the deadline for updating the ticket.
1. Add the following content to a new line at the end of the Storm/storm-0.10.0/conf/storm.yaml file in the Storm client installation directory.
  topology.auto-credentials:
  
  - backtype.storm.security.auth.kerberos.AutoTGT
2. Run the ./storm upload-credentials hbase-test command.

Parent topic: FAQs About Storm Application Development

Previous topic: Storm-OBS Development Guideline

Next topic: Flux Development Guideline

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot