Hive HCatalog Application Development

Updated on 2024-08-10 GMT+08:00

View PDF

Hive is an open-source data warehouse framework built on Hadoop. You can use it to store structured data and analyze data with the Hive query language (HiveQL) statements. Hive converts HiveQL statements to MapReduce or Spark jobs to query and analyze massive amounts of data stored in Hadoop clusters.

You can use Hive to:

Extract, transform, and load (ETL) data with HiveQL.
Analyze massive amounts of structured data with HiveQL.
Process data in a wide range of formats, such as JSON, CSV, TEXTFILE, RCFILE, ORCFILE, and SEQUENCEFILE, and customize extensions.
Connect the client flexibly and call JDBC APIs.

Hive is good for offline massive data analysis (such as log and cluster status analysis), large-scale data mining (such as user behavior analysis, interest region analysis, and region display), and other scenarios.

MRS provides sample application development projects based on Hive. This practice provides guidance for you to obtain and import a sample project after creating an MRS cluster and then compile and debug the code locally. In this sample project, you can create Hive tables, insert data, and read data.

Creating an MRS Hive Cluster

Create and purchase an MRS cluster that contains Hive. For details, see Buying a Custom Cluster.
NOTE:

In this practice, an MRS 3.1.5 cluster, with Hadoop and Hive installed and with Kerberos authentication enabled, is used as an example.
Click Buy Now and wait until the MRS cluster is created.

Preparing the Application Development Configuration File

Log in to FusionInsight Manager to create a cluster user for creating Hive data tables and submitting the HCatalog program.

Choose System > Permission > User. On the displayed page, click Create. On the displayed page, create a machine-machine user, for example, hiveuser.

Add hive and supergroup to User Group.
Download and install the cluster client to run the HCatalog program. For example, the installation directory is /opt/client.

Obtaining the Sample Project

Obtain the sample project from Huawei Mirrors.

Download the source code and configuration files of the sample project, and configure related development tools on your local PC. For details, see Obtaining Sample Projects from Huawei Mirrors.

Select a branch based on the cluster version and download the required MRS sample project.

For example, the sample project suitable for this practice is hcatalog-example, which can be obtained at https://github.com/huaweicloud/huaweicloud-mrs-example/tree/mrs-3.1.5/src/hive-examples/hcatalog-example.
Use IDEA to import the sample project and wait for the Maven project to download the dependency packages. For details, see Configuring the JDBC Sample Project.

Figure 1 Hive HCatalog sample project

After you configure Maven and SDK parameters on the local PC, the sample project automatically loads related dependency packages.

Building and Running the Application

Compile the HCatalog sample program.
1. In the Maven tool window, select clean from Lifecycle to execute the Maven building process.
2. Selecting package from Lifecycle and executing the Maven build process
  Figure 2 Packaging the sample program
  
  If "BUILD SUCCESS" is displayed, the compilation is successful.
  
  The hcatalog-example-XXX.jar package is generated in the target directory of the sample project.
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:30 min
[INFO] Finished at: 2023-05-17T20:22:44+08:00
[INFO] ------------------------------------------------------------------------
```
Log in to the Hive Beeline CLI and create source tables and data tables for HCatalog analysis.

source /opt/client/bigdata_env

kinit hiveuser

beeline

create table t1(col1 int);

create table t2(col1 int,col2 int);

Insert test data into the source data table t1.

insert into table t1 select 1 union all select 1 union all select 2 union all select 2 union all select 3;

select * from t1;
```
+----------+
| t1.col1  |
+----------+
| 1        |
| 1        |
| 2        |
| 2        |
| 3        |
+----------+
```
Upload the exported JAR package to the specified path of the Linux node where the cluster client is deployed, for example, /opt/hive_demo.
To facilitate subsequent operations, configure the sample program directory and client component directory as public variables.

Exit the Beeline CLI and run the following commands:

export HCAT_CLIENT=/opt/hive_demo

export HADOOP_HOME=/opt/client/HDFS/hadoop

export HIVE_HOME=/opt/client/Hive/Beeline

export HCAT_HOME=$HIVE_HOME/../HCatalog

export LIB_JARS=$HCAT_HOME/lib/hive-hcatalog-core-XXX.jar,$HCAT_HOME/lib/hive-metastore-XXX.jar,$HCAT_HOME/lib/hive-standalone-metastore-XXX.jar,$HIVE_HOME/lib/hive-exec-XXX.jar,$HCAT_HOME/lib/libfb303-XXX.jar,$HCAT_HOME/lib/slf4j-api-XXX.jar,$HCAT_HOME/lib/jdo-api-XXX.jar,$HCAT_HOME/lib/antlr-runtime-XXX.jar,$HCAT_HOME/lib/datanucleus-api-jdo-XXX.jar,$HCAT_HOME/lib/datanucleus-core-XXX.jar,$HCAT_HOME/lib/datanucleus-rdbms-fi-XXX.jar,$HCAT_HOME/lib/log4j-api-XXX.jar,$HCAT_HOME/lib/log4j-core-XXX.jar,$HIVE_HOME/lib/commons-lang-XXX.jar,$HIVE_HOME/lib/hive-exec-XXX.jar

export HADOOP_CLASSPATH=$HCAT_HOME/lib/hive-hcatalog-core-XXX.jar:$HCAT_HOME/lib/hive-metastore-XXX.jar:$HCAT_HOME/lib/hive-standalone-metastore-XXX.jar:$HIVE_HOME/lib/hive-exec-XXX.jar:$HCAT_HOME/lib/libfb303-XXX.jar:$HADOOP_HOME/etc/hadoop:$HCAT_HOME/conf:$HCAT_HOME/lib/slf4j-api-XXX.jar:$HCAT_HOME/lib/jdo-api-XXX.jar:$HCAT_HOME/lib/antlr-runtime-XXX.jar:$HCAT_HOME/lib/datanucleus-api-jdo-XXX.jar:$HCAT_HOME/lib/datanucleus-core-XXX.jar:$HCAT_HOME/lib/datanucleus-rdbms-fi-XXX.jar:$HCAT_HOME/lib/log4j-api-XXX.jar:$HCAT_HOME/lib/log4j-core-XXX.jar:$HIVE_HOME/lib/commons-lang-XXX.jar:$HIVE_HOME/lib/hive-exec-XXX.jar

NOTE:

The version number XXX of the JAR package specified in LIB_JARS and HADOOP_CLASSPATH must be changed to the version you are using.

Use the Yarn client to submit a task.

yarn --config $HADOOP_HOME/etc/hadoop jar $HCAT_CLIENT/hcatalog-example-XXX.jar com.huawei.bigdata.HCatalogExample -libjars $LIB_JARS t1 t2

...
2023-05-18 20:05:56,691 INFO mapreduce.Job: The url to track the job: https://host-192-168-64-122:26001/proxy/application_1683438782910_0008/
2023-05-18 20:05:56,692 INFO mapreduce.Job: Running job: job_1683438782910_0008
2023-05-18 20:06:07,250 INFO mapreduce.Job: Job job_1683438782910_0008 running in uber mode : false
2023-05-18 20:06:07,253 INFO mapreduce.Job:  map 0% reduce 0%
2023-05-18 20:06:15,362 INFO mapreduce.Job:  map 25% reduce 0%
2023-05-18 20:06:16,386 INFO mapreduce.Job:  map 50% reduce 0%
2023-05-18 20:06:35,999 INFO mapreduce.Job:  map 100% reduce 0%
2023-05-18 20:06:42,048 INFO mapreduce.Job:  map 100% reduce 100%
2023-05-18 20:06:43,136 INFO mapreduce.Job: Job job_1683438782910_0008 completed successfully
2023-05-18 20:06:44,118 INFO mapreduce.Job: Counters: 54
...

After the job is complete, go to the Hive Beeline CLI, query the data in the t2 table, and view the data analysis result.

select * from t2;

+----------+----------+
| t2.col1  | t2.col2  |
+----------+----------+
| 1        | 2        |
| 2        | 2        |
| 3        | 1        |
+----------+----------+

Parent topic: Rapidly Develop MRS Component Applications

Previous topic: Hive JDBC Application Development

Next topic: Kafka Application Development

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot