Instance

Updated on 2024-10-23 GMT+08:00

View PDF

Scenario

The sample project illustrates how to compile MapReduce jobs to visit multiple service components in HDFS, HBase, and Hive, helping users to understand key actions such as certificating and configuration loading.

The logic of the sample project is as follows:

The input data is HDFS text file and the input file is log1.txt.

YuanJing,male,10
GuoYijun,male,5

Map:

Obtain one row of the input data and extract the user name.
Query one piece of data from HBase.
Query one piece of data from Hive.
Combine the data queried from HBase and that from Hive as the output of Map as the output of Map.

Reduce:

Obtain the last piece of data from Map output.
Import the data to HBase.
Save the data to HDFS.

Data Planning

Create an HDFS data file.
1. Create a text file named data.txt in the Linux-based HDFS and copy the content of log1.txt to data.txt.
2. Run the following commands to create a directory /tmp/examples/multi-components/mapreduce/input/ and copy the data.txt to the directory:
  1. Logging In to the HDFS Client.
    cd Client installation directory
    
    source bigdata_env
  2. hdfs dfs -mkdir -p /tmp/examples/multi-components/mapreduce/input/
  3. hdfs dfs -put data.txt /tmp/examples/multi-components/mapreduce/input/
Create a HBase table and insert data into it.
1. Run the source bigdata_env command on a Linux-based HBase client and run the hbase shell command.
2. Run the create 'table1', 'cf' command in the HBase shell to create table1 with column family cf.
3. Run the put 'table1', '1', 'cf:cid', '123' command to insert data whose rowkey is 1, column name is cid, and data value is 123.
4. Run the quit command to exit the table.
Create a Hive table and load data to it.
1. Run the beeline command on a Linux-based Hive client.
2. In the Hive beeline interaction window, run the CREATE TABLE person(name STRING, gender STRING, stayTime INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' stored as textfile; command to create data table person with three fields.
3. In the Hive beeline interaction window, run the LOAD DATA INPATH '/tmp/examples/multi-components/mapreduce/input/' OVERWRITE INTO TABLE person; command to load data files to the person table.
4. Run !q to exit the table.
The data of HDFS is cleared in the preceding step. Therefore, perform 1 again.

Parent topic: MapReduce Accessing Multi-Component Example Project

Previous topic: MapReduce Accessing Multi-Component Example Project

Next topic: Example Code

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot