Using Hadoop from Scratch

Procedure

1. Prepare the wordcount program.

   Multiple open source Hadoop sample programs are provided, including wordcount. You can download the Hadoop sample program from https://dist.apache.org/repos/dist/release/hadoop/common/.

   For example, select hadoop-2.10.x, download hadoop-2.10.x.tar.gz, decompress it, and obtain hadoop-2.10.x\share\hadoop\mapreduce (the Hadoop sample program) from hadoop-mapreduce-examples-2.10.x.jar. The hadoop-mapreduce-examples-2.10.x.jar sample program contains the wordcount program.

   

   hadoop-2.10.x indicates the Hadoop version.

2. Prepare data files.

   There is no format requirement for data files. Prepare one or more .txt files. The following are examples of the .txt file:

   qwsdfhoedfrffrofhuncckgktpmhutopmma
   jjpsffjfjorgjgtyiuyjmhombmbogohoyhm
   jhheyeombdhuaqqiquyebchdhmamdhdemmj
   doeyhjwedcrfvtgbmojiyhhqssddddddfkf
   kjhhjkehdeiyrudjhfhfhffooqweopuyyyy

3. Upload data to OBS.
   1. Log in to OBS Console.
   2. Click Parallel File System and choose Create Parallel File System to create a file system named wordcount01.

      wordcount01 is only an example. The file system name must be globally unique. Otherwise, the parallel file system fails to be created.

   3. In the OBS file system list, click wordcount01 and choose Files > Create Folder to create the program and input folders.
      • program: stores user programs.
      • input: stores user data files.
   4. Go to the program folder, choose Upload File > add file, select the program package downloaded in 1 from the local host, and click Upload.
   5. Go to the input folder and upload the data file prepared in 2 to the input folder.

4. Log in to the MRS console. In the navigation pane on the left, click Clusters and choose Active Clusters. Click the cluster name. The cluster must contain Hadoop components.
5. Submit the wordcount job.

   On the MRS console, click the Jobs tab and click Create. The Create Job page is displayed.

   • Set Type to MapReduce.
   • Set Name to mr_01.
   • Set the path of the executable program to the address of the program stored on the OBS. For example: obs://wordcount01/program/hadoop-mapreduce-examples-2.10.x.jar
   • Enter wordcount obs://wordcount01/input/ obs://wordcount01/output/ in the Parameter pane.
     

     • Replace the OBS file system name in obs://wordcount01/input/ with the actual name of the file system created in the environment.
     • Replace the OBS file system name in obs://wordcount01/output/ with the actual name of the file system created in the environment. Enter a directory that does not exist in the output directory.
   • Service Parameter can be left blank.

   A job can be submitted only when the cluster is in the Running state.

   After a job is submitted successfully, it is in the Accepted state by default. You do not need to manually execute the job.

6. View the job execution result.
   1. Go to the Jobs tab page and check whether the job is successfully executed.

      It takes some time to run the job. After the job is complete, refresh the job list

      Once a job has succeeded or failed, you cannot execute it again. However, you can add or copy a job, and set job parameters to submit a job again.

   2. Log in to the OBS console, go to the OBS path, and view the job output information.

      You can view output files in the output directory created in 5. You need to download the file to the local host and open it in text format.