Help Center/ MapReduce Service/ Developer Guide (LTS)/ Rapidly Develop MRS Component Applications/ Hive HCatalog Application Development
Updated on 2024-08-10 GMT+08:00
View PDF
Share

Hive HCatalog Application Development

Hive is an open-source data warehouse framework built on Hadoop. You can use it to store structured data and analyze data with the Hive query language (HiveQL) statements. Hive converts HiveQL statements to MapReduce or Spark jobs to query and analyze massive amounts of data stored in Hadoop clusters.

You can use Hive to:

  • Extract, transform, and load (ETL) data with HiveQL.
  • Analyze massive amounts of structured data with HiveQL.
  • Process data in a wide range of formats, such as JSON, CSV, TEXTFILE, RCFILE, ORCFILE, and SEQUENCEFILE, and customize extensions.
  • Connect the client flexibly and call JDBC APIs.

Hive is good for offline massive data analysis (such as log and cluster status analysis), large-scale data mining (such as user behavior analysis, interest region analysis, and region display), and other scenarios.

MRS provides sample application development projects based on Hive. This practice provides guidance for you to obtain and import a sample project after creating an MRS cluster and then compile and debug the code locally. In this sample project, you can create Hive tables, insert data, and read data.

Creating an MRS Hive Cluster

  1. Create and purchase an MRS cluster that contains Hive. For details, see Buying a Custom Cluster.

    In this practice, an MRS 3.1.5 cluster, with Hadoop and Hive installed and with Kerberos authentication enabled, is used as an example.

  2. Click Buy Now and wait until the MRS cluster is created.

Preparing the Application Development Configuration File

  1. Log in to FusionInsight Manager to create a cluster user for creating Hive data tables and submitting the HCatalog program.

    Choose System > Permission > User. On the displayed page, click Create. On the displayed page, create a machine-machine user, for example, hiveuser.

    Add hive and supergroup to User Group.

  2. Download and install the cluster client to run the HCatalog program. For example, the installation directory is /opt/client.

Obtaining the Sample Project

  1. Obtain the sample project from Huawei Mirrors.

    Download the source code and configuration files of the sample project, and configure related development tools on your local PC. For details, see Obtaining Sample Projects from Huawei Mirrors.

    Select a branch based on the cluster version and download the required MRS sample project.

    For example, the sample project suitable for this practice is hcatalog-example, which can be obtained at https://github.com/huaweicloud/huaweicloud-mrs-example/tree/mrs-3.1.5/src/hive-examples/hcatalog-example.

  2. Use IDEA to import the sample project and wait for the Maven project to download the dependency packages. For details, see Configuring the JDBC Sample Project.

    Figure 1 Hive HCatalog sample project

    After you configure Maven and SDK parameters on the local PC, the sample project automatically loads related dependency packages.

Building and Running the Application

  1. Compile the HCatalog sample program.

    1. In the Maven tool window, select clean from Lifecycle to execute the Maven building process.
    2. Selecting package from Lifecycle and executing the Maven build process
      Figure 2 Packaging the sample program

      If "BUILD SUCCESS" is displayed, the compilation is successful.

      The hcatalog-example-XXX.jar package is generated in the target directory of the sample project.

      [INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:30 min
[INFO] Finished at: 2023-05-17T20:22:44+08:00
[INFO] ------------------------------------------------------------------------

  2. Log in to the Hive Beeline CLI and create source tables and data tables for HCatalog analysis.

    source /opt/client/bigdata_env

    kinit hiveuser

    beeline

    create table t1(col1 int);

    create table t2(col1 int,col2 int);

    Insert test data into the source data table t1.

    insert into table t1 select 1 union all select 1 union all select 2 union all select 2 union all select 3;

    select * from t1;

    +----------+
| t1.col1  |
+----------+
| 1        |
| 1        |
| 2        |
| 2        |
| 3        |
+----------+

  3. Upload the exported JAR package to the specified path of the Linux node where the cluster client is deployed, for example, /opt/hive_demo.
  4. To facilitate subsequent operations, configure the sample program directory and client component directory as public variables.

    Exit the Beeline CLI and run the following commands:

    export HCAT_CLIENT=/opt/hive_demo

    export HADOOP_HOME=/opt/client/HDFS/hadoop

    export HIVE_HOME=/opt/client/Hive/Beeline

    export HCAT_HOME=$HIVE_HOME/../HCatalog

    export LIB_JARS=$HCAT_HOME/lib/hive-hcatalog-core-XXX.jar,$HCAT_HOME/lib/hive-metastore-XXX.jar,$HCAT_HOME/lib/hive-standalone-metastore-XXX.jar,$HIVE_HOME/lib/hive-exec-XXX.jar,$HCAT_HOME/lib/libfb303-XXX.jar,$HCAT_HOME/lib/slf4j-api-XXX.jar,$HCAT_HOME/lib/jdo-api-XXX.jar,$HCAT_HOME/lib/antlr-runtime-XXX.jar,$HCAT_HOME/lib/datanucleus-api-jdo-XXX.jar,$HCAT_HOME/lib/datanucleus-core-XXX.jar,$HCAT_HOME/lib/datanucleus-rdbms-fi-XXX.jar,$HCAT_HOME/lib/log4j-api-XXX.jar,$HCAT_HOME/lib/log4j-core-XXX.jar,$HIVE_HOME/lib/commons-lang-XXX.jar,$HIVE_HOME/lib/hive-exec-XXX.jar

    export HADOOP_CLASSPATH=$HCAT_HOME/lib/hive-hcatalog-core-XXX.jar:$HCAT_HOME/lib/hive-metastore-XXX.jar:$HCAT_HOME/lib/hive-standalone-metastore-XXX.jar:$HIVE_HOME/lib/hive-exec-XXX.jar:$HCAT_HOME/lib/libfb303-XXX.jar:$HADOOP_HOME/etc/hadoop:$HCAT_HOME/conf:$HCAT_HOME/lib/slf4j-api-XXX.jar:$HCAT_HOME/lib/jdo-api-XXX.jar:$HCAT_HOME/lib/antlr-runtime-XXX.jar:$HCAT_HOME/lib/datanucleus-api-jdo-XXX.jar:$HCAT_HOME/lib/datanucleus-core-XXX.jar:$HCAT_HOME/lib/datanucleus-rdbms-fi-XXX.jar:$HCAT_HOME/lib/log4j-api-XXX.jar:$HCAT_HOME/lib/log4j-core-XXX.jar:$HIVE_HOME/lib/commons-lang-XXX.jar:$HIVE_HOME/lib/hive-exec-XXX.jar

    The version number XXX of the JAR package specified in LIB_JARS and HADOOP_CLASSPATH must be changed to the version you are using.

  5. Use the Yarn client to submit a task.

    yarn --config $HADOOP_HOME/etc/hadoop jar $HCAT_CLIENT/hcatalog-example-XXX.jar com.huawei.bigdata.HCatalogExample -libjars $LIB_JARS t1 t2

    ...
2023-05-18 20:05:56,691 INFO mapreduce.Job: The url to track the job: https://host-192-168-64-122:26001/proxy/application_1683438782910_0008/
2023-05-18 20:05:56,692 INFO mapreduce.Job: Running job: job_1683438782910_0008
2023-05-18 20:06:07,250 INFO mapreduce.Job: Job job_1683438782910_0008 running in uber mode : false
2023-05-18 20:06:07,253 INFO mapreduce.Job:  map 0% reduce 0%
2023-05-18 20:06:15,362 INFO mapreduce.Job:  map 25% reduce 0%
2023-05-18 20:06:16,386 INFO mapreduce.Job:  map 50% reduce 0%
2023-05-18 20:06:35,999 INFO mapreduce.Job:  map 100% reduce 0%
2023-05-18 20:06:42,048 INFO mapreduce.Job:  map 100% reduce 100%
2023-05-18 20:06:43,136 INFO mapreduce.Job: Job job_1683438782910_0008 completed successfully
2023-05-18 20:06:44,118 INFO mapreduce.Job: Counters: 54
...

  6. After the job is complete, go to the Hive Beeline CLI, query the data in the t2 table, and view the data analysis result.

    select * from t2;

    +----------+----------+
| t2.col1  | t2.col2  |
+----------+----------+
| 1        | 2        |
| 2        | 2        |
| 3        | 1        |
+----------+----------+