Help Center/ MapReduce Service/ Developer Guide (LTS)/ Rapidly Develop MRS Component Applications/ HBase Application Development
Updated on 2024-08-10 GMT+08:00
View PDF
Share

HBase Application Development

HBase is a column-based distributed storage system that features high reliability, performance, and scalability. It is designed to eliminate the limitations of relational databases in processing massive amounts of data.

Application scenarios of HBase have the following features:

  • Massive data processing (higher than the TB or PB level)
  • High throughput
  • Highly efficient random read of massive data
  • Excellent scalability
  • Concurrent processing of structured and non-structured data

MRS provides sample application development projects based on HBase. This practice provides guidance for you to obtain and import a sample project after creating an MRS cluster and then conduct building and commissioning locally. In this sample project, you can create HBase tables, insert data, create indexes, and delete tables in the MRS cluster.

Creating an MRS HBase Cluster

  1. Create and purchase an MRS cluster that contains HBase. For details, see Buying a Custom Cluster.

    In this practice, an MRS 3.1.0 cluster, with Hadoop and HBase installed and with Kerberos authentication enabled, is used as an example.

  2. Click Buy Now and wait until the MRS cluster is created.
    Figure 1 Cluster purchased

Preparing the Application Development Configuration File

  1. After the cluster is created, log in to FusionInsight Manager and create a cluster user for security authentication of the sample project.

    1. Choose System > Permission > User. In the right pane, click Create. On the displayed page, create a human-machine user, for example, developuser.

      Add the hadoop user group to User Group.

      After the user is created, log in to FusionInsight Manager as developuser and change the initial password as prompted.

    2. Log in to the Ranger web UI as the Ranger administrator rangeradmin.

      The default password of user rangeradmin is Rangeradmin@123. For details, see User Account List.

    3. On the Ranger homepage, click the component plug-in name in the HBASE area, for example, HBase.
    4. Click in the Action column of the row containing the all - table, column-family, column policy.
    5. In the Allow Conditions area, add an allow condition. Select the created user for Select User, and select Select/Deselect All for Permissions.
    6. Click Save.

  2. Log in to FusionInsight Manager as user admin and choose System > Permission > User. In the Operation column of developuser, choose More > Download Authentication Credential. Save the file and decompress it to obtain the user.keytab and krb5.conf files of the user.
  3. Choose Cluster. On the Dashboard tab, click More and select Download Client. In the dialog box that is displayed, set Select Client Type to Configuration Files Only and click OK. After the client package is generated, download the package as prompted and decompress it.

    For example, if the client configuration file package is FusionInsight_Cluster_1_Services_Client.tar, decompress it to obtain FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar. Then, continue to decompress this file.

    1. Go to the FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles\HBase\config directory and obtain the configuration files listed in Table 1.
      Table 1 Configuration files

      Configuration File

      Description

      core-site.xml

      Configures Hadoop Core parameters.

      hbase-site.xml

      Configures HBase parameters.

      hdfs-site.xml

      Configures HDFS parameters.
    2. Copy all content from the hosts file in the decompression directory to your local hosts file. Ensure that the local PC can communicate with the hosts listed in the hosts file in the decompression directory.
      • In this practice, ensure that the local environment can communicate with the network plane where the MRS cluster resides. Generally, you can access the MRS cluster via an EIP..
      • If the local environment cannot communicate with nodes in the MRS cluster, you can build the sample project first and upload the JAR package to the cluster to run. .
      • C:\WINDOWS\system32\drivers\etc\hosts is an example directory in a Windows environment for storing the local hosts file.

Obtaining the Sample Project

  1. Obtain the sample project from Huawei Mirrors.

    Download the Maven project source code and configuration files of the sample project, and configure related development tools on your local PC. For details, see Obtaining Sample Projects from Huawei Mirrors.

    Select a branch based on the cluster version and download the required MRS sample project.

    For example, the sample project suitable for this practice is hbase-example, which can be obtained at https://github.com/huaweicloud/huaweicloud-mrs-example/tree/mrs-3.1.0/src/hbase-examples/hbase-example.

  2. Use IDEA to import the sample project and wait for the Maven project to download the dependency packages. For details, see Configuring and Importing Sample Projects.

    Figure 2 HBase sample project

    After you configure Maven and SDK parameters on the local PC, the sample project automatically loads related dependency packages.

  3. Place the cluster configuration files and user authentication credentials obtained in Preparing the Application Development Configuration File into the ../src/main/resources/conf directory of the sample project.
  4. In the TestMain class of the com.huawei.bigdata.hbase.examples package, change userName to the actual username, for example, developuser. 

    private static void login() throws IOException {
        if (User.isHBaseSecurityEnabled(conf)) {
            userName = "developuser";
           
            //In Windows environment
            String userdir = TestMain.class.getClassLoader().getResource("conf").getPath() + File.separator;
            //In Linux environment
            //String userdir = System.getProperty("user.dir") + File.separator + "conf" + File.separator;

            LoginUtil.setJaasConf(ZOOKEEPER_DEFAULT_LOGIN_CONTEXT_NAME, userName, userKeytabFile);
            LoginUtil.login(userName, userKeytabFile, krb5File, conf);
        }
}

    Assume that you are developing an application to manage information about users of service A in an enterprise. The operation process is as follows.

    No.

    Step

    1

    Create a table based on existing information.

    2

    Import user data.

    3

    Add the Education Information column family and add the education backgrounds and titles of users to the user information table.

    4

    Query usernames and addresses by user ID.

    5

    Execute queries by username.

    6

    To improve query performance, create or delete secondary indexes.

    7

    Deregister users and delete user data from the user information table.

    8

    Delete the user information table after service A ends.

    For example, the following code snippet executes the testCreateTable method in the HBaseSample class of the com.huawei.bigdata.hbase.examples package to create a user information table..

    public void testCreateTable() {
             LOG.info("Entering testCreateTable.");
             TableDescriptorBuilder htd = TableDescriptorBuilder.newBuilder(tableName); //Create a table descriptor.
             ColumnFamilyDescriptorBuilder hcd = ColumnFamilyDescriptorBuilder.newBuilder(Bytes.toBytes("info")); //Create a column family descriptor.
             hcd.setDataBlockEncoding(DataBlockEncoding.FAST_DIFF); //Set the encoding algorithm. HBase provides DIFF, FAST_DIFF, and PREFIX encoding algorithms.
             hcd.setCompressionType(Compression.Algorithm.SNAPPY);
             htd.setColumnFamily(hcd.build());  //Add the column family descriptor to the table descriptor.
             Admin admin = null; 
             try {
               admin = conn.getAdmin(); //Obtain the Admin object, which allows you to create a table, create a column family, check whether the table exists, change the table structure and column family structure, and delete the table.
               if (!admin.tableExists(tableName)) {
                 LOG.info("Creating table...");
                 admin.createTable(htd.build());//Call the createTable method of Admin.
                 LOG.info(admin.getClusterMetrics().toString());
                 LOG.info(admin.listNamespaceDescriptors().toString());
                 LOG.info("Table created successfully.");
               } else {
                 LOG.warn("table already exists");
               }
             } catch (IOException e) {
                 LOG.error("Create table failed " ,e);
             } finally {
               if (admin != null) {
                 try {
                   admin.close();
                 } catch (IOException e) {
                   LOG.error("Failed to close admin " ,e);
                 }
               }
             }
             LOG.info("Exiting testCreateTable.");
  }

Building and Running the Application

  1. Click Reimport All Maven Projects in the Maven window on the right of IDEA to load the Maven project dependencies.

    Figure 3 Loading a sample project

  2. Build the application.

    1. Choose Maven, locate the target project name, and double-click clean under Lifecycle to run the clean command of Maven.
    2. Choose Maven, locate the target project name, and double-click compile under Lifecycle to run the compile command of Maven.
      Figure 4 clean and compile of Maven

    After the building is complete, message "Build Success" is displayed and the target directory is generated.

    [INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21.276 s
[INFO] Finished at: 2023-05-05T14:36:39+08:00
[INFO] ------------------------------------------------------------------------

  3. Run the application.

    Right-click the TestMain.java file and choose Run 'TestMain.main().

    Figure 5 Running the application

  4. Check the output information after running the hbase-example sample. The following information indicates that related table operations are successfully executed: 

    ...
2023-05-05 15:05:27,050 INFO  [main] examples.HBaseSample: Table created successfully.
2023-05-05 15:05:27,050 INFO  [main] examples.HBaseSample: Exiting testCreateTable.
2023-05-05 15:05:27,050 INFO  [main] examples.HBaseSample: Entering testMultiSplit.
2023-05-05 15:05:31,171 INFO  [main] client.HBaseAdmin: Operation: MULTI_SPLIT_REGION, Table Name: default:hbase_sample_table, procId: 21 completed
2023-05-05 15:05:31,171 INFO  [main] examples.HBaseSample: MultiSplit successfully.
2023-05-05 15:05:31,172 INFO  [main] examples.HBaseSample: Exiting testMultiSplit.
2023-05-05 15:05:31,172 INFO  [main] examples.HBaseSample: Entering testPut.
2023-05-05 15:05:32,862 INFO  [main] examples.HBaseSample: Put successfully.
2023-05-05 15:05:32,862 INFO  [main] examples.HBaseSample: Exiting testPut.
2023-05-05 15:05:32,862 INFO  [main] examples.HBaseSample: Entering createIndex.
2023-05-05 15:05:36,627 INFO  [main] examples.HBaseSample: Create index successfully.
2023-05-05 15:05:36,627 INFO  [main] examples.HBaseSample: Exiting createIndex.
2023-05-05 15:05:36,627 INFO  [main] examples.HBaseSample: Entering createIndex.
2023-05-05 15:05:37,912 INFO  [main] examples.HBaseSample: Successfully enable indices [index_name]  of the table hbase_sample_table
2023-05-05 15:05:37,912 INFO  [main] examples.HBaseSample: Entering testScanDataByIndex.
2023-05-05 15:05:37,915 INFO  [main] examples.HBaseSample: Scan indexed data.
2023-05-05 15:05:39,939 INFO  [main] examples.HBaseSample: Scan data by index successfully.
2023-05-05 15:05:39,939 INFO  [main] examples.HBaseSample: Exiting testScanDataByIndex.
2023-05-05 15:05:39,941 INFO  [main] examples.HBaseSample: Entering testModifyTable.
2023-05-05 15:05:40,191 INFO  [main] client.HBaseAdmin: Started disable of hbase_sample_table
2023-05-05 15:05:41,322 INFO  [main] client.HBaseAdmin: Operation: DISABLE, Table Name: default:hbase_sample_table, procId: 53 completed
2023-05-05 15:05:42,230 INFO  [main] client.HBaseAdmin: Started enable of hbase_sample_table
2023-05-05 15:05:43,187 INFO  [main] client.HBaseAdmin: Operation: ENABLE, Table Name: default:hbase_sample_table, procId: 65 completed
2023-05-05 15:05:43,187 INFO  [main] examples.HBaseSample: Modify table successfully.
2023-05-05 15:05:43,187 INFO  [main] examples.HBaseSample: Exiting testModifyTable.
2023-05-05 15:05:43,187 INFO  [main] examples.HBaseSample: Entering testGet.
2023-05-05 15:05:43,278 INFO  [main] examples.HBaseSample: 012005000201:info,address,Shenzhen, Guangdong
2023-05-05 15:05:43,279 INFO  [main] examples.HBaseSample: 012005000201:info,name,Zhang San
2023-05-05 15:05:43,279 INFO  [main] examples.HBaseSample: Get data successfully.
2023-05-05 15:05:43,279 INFO  [main] examples.HBaseSample: Exiting testGet.
2023-05-05 15:05:43,279 INFO  [main] examples.HBaseSample: Entering testScanData.
2023-05-05 15:05:43,576 INFO  [main] examples.HBaseSample: 012005000201:info,name,Zhang San
2023-05-05 15:05:43,576 INFO  [main] examples.HBaseSample: 012005000202:info,name,Li Wanting
2023-05-05 15:05:43,577 INFO  [main] examples.HBaseSample: 012005000203:info,name,Wang Ming
2023-05-05 15:05:43,577 INFO  [main] examples.HBaseSample: 012005000204:info,name,Li Gang
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: 012005000205:info,name,Zhao Enru
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: 012005000206:info,name,Chen Long
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: 012005000207:info,name,Zhou Wei
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: 012005000208:info,name,Yang Yiwen
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: 012005000209:info,name,Xu Bing
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: 012005000210:info,name,Xiao Kai
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: Scan data successfully.
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: Exiting testScanData.
2023-05-05 15:05:43,578 INFO  [main] examples.HBaseSample: Entering testSingleColumnValueFilter.
2023-05-05 15:05:43,883 INFO  [main] examples.HBaseSample: Single column value filter successfully.
2023-05-05 15:05:43,883 INFO  [main] examples.HBaseSample: Exiting testSingleColumnValueFilter.
2023-05-05 15:05:43,884 INFO  [main] examples.HBaseSample: Entering testFilterList.
2023-05-05 15:05:44,388 INFO  [main] examples.HBaseSample: 012005000201:info,name,Zhang San
2023-05-05 15:05:44,388 INFO  [main] examples.HBaseSample: 012005000202:info,name,Li Wanting
2023-05-05 15:05:44,388 INFO  [main] examples.HBaseSample: 012005000203:info,name,Wang Ming
2023-05-05 15:05:44,388 INFO  [main] examples.HBaseSample: 012005000204:info,name,Li Gang
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: 012005000205:info,name,Zhao Enru
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: 012005000206:info,name,Chen Long
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: 012005000207:info,name,Zhou Wei
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: 012005000208:info,name,Yang Yiwen
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: 012005000209:info,name,Xu Bing
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: 012005000210:info,name,Xiao Kai
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: Filter list successfully.
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: Exiting testFilterList.
2023-05-05 15:05:44,389 INFO  [main] examples.HBaseSample: Entering testDelete.
2023-05-05 15:05:44,586 INFO  [main] examples.HBaseSample: Delete table successfully.
2023-05-05 15:05:44,586 INFO  [main] examples.HBaseSample: Exiting testDelete.
2023-05-05 15:05:44,586 INFO  [main] examples.HBaseSample: Entering disableIndex.
2023-05-05 15:05:45,819 INFO  [main] examples.HBaseSample: Successfully disable indices [index_name]  of the table hbase_sample_table
2023-05-05 15:05:45,819 INFO  [main] examples.HBaseSample: Entering dropIndex.
2023-05-05 15:05:48,084 INFO  [main] examples.HBaseSample: Drop index successfully.
2023-05-05 15:05:48,084 INFO  [main] examples.HBaseSample: Exiting dropIndex.
2023-05-05 15:05:48,084 INFO  [main] examples.HBaseSample: Entering dropTable.
2023-05-05 15:05:48,237 INFO  [main] client.HBaseAdmin: Started disable of hbase_sample_table
2023-05-05 15:05:49,543 INFO  [main] client.HBaseAdmin: Operation: DISABLE, Table Name: default:hbase_sample_table, procId: 95 completed
2023-05-05 15:05:50,645 INFO  [main] client.HBaseAdmin: Operation: DELETE, Table Name: default:hbase_sample_table, procId: 106 completed
2023-05-05 15:05:50,645 INFO  [main] examples.HBaseSample: Drop table successfully.
2023-05-05 15:05:50,645 INFO  [main] examples.HBaseSample: Exiting dropTable.
2023-05-05 15:05:50,646 INFO  [main] client.ConnectionImplementation: Closing master protocol: MasterService
2023-05-05 15:05:50,652 INFO  [main] client.ConnectionImplementation: Connection has been closed by main.
2023-05-05 15:05:50,654 INFO  [main] hbase.ChoreService: Chore service for: AsyncConn Chore Service had [[ScheduledChore: Name: RefreshCredentials Period: 30000 Unit: MILLISECONDS]] on shutdown
2023-05-05 15:05:50,655 INFO  [main] examples.TestMain: -----------finish HBase -------------------
...