Using Hadoop from Scratch

Procedure

1. Prepare the wordcount program.

   Multiple open source Hadoop sample programs are provided, including wordcount. You can download the Hadoop sample program from https://dist.apache.org/repos/dist/release/hadoop/common/.

   For example, choose hadoop-x.x.x. On the page that is displayed, click hadoop-x.x.x.tar.gz to download it. Then, decompress it to obtain hadoop-mapreduce-examples-x.x.x.jar (the Hadoop sample program) from hadoop-x.x.x\share\hadoop\mapreduce. The hadoop-mapreduce-examples-x.x.x.jar package contains the wordcount program.

   

   hadoop-x.x.x indicates the Hadoop version. Choose a version based on your requirements.

2. Prepare data files.

   There is no format requirement for data files. Prepare one or more .txt files. The following are examples of the .txt file:

   qwsdfhoedfrffrofhuncckgktpmhutopmma
   jjpsffjfjorgjgtyiuyjmhombmbogohoyhm
   jhheyeombdhuaqqiquyebchdhmamdhdemmj
   doeyhjwedcrfvtgbmojiyhhqssddddddfkf
   kjhhjkehdeiyrudjhfhfhffooqweopuyyyy

3. Upload data to OBS.
   1. Log in to OBS Console.
   2. Click Parallel File System and choose Create Parallel File System to create a file system named wordcount01.

      wordcount01 is only an example. The file system name must be globally unique. Otherwise, the parallel file system fails to be created.

      

   3. In the OBS file system list, click wordcount01 and choose Files > Create Folder to create the program and input folders, as shown in Figure 1.
      Figure 1 Folder list of the wordcount01 file system
      
      • program: stores user programs.
      • input: stores user data files.
   4. Go to the program folder, choose Upload File > add file, select the program package downloaded in 1 from the local host, and click Upload. After the upload is complete, the page shown in Figure 2 is displayed.
      Figure 2 Program list
      
   5. Go to the input folder and upload the data file prepared in 2 to the input folder. After the upload is complete, the page shown in Figure 3 is displayed.
      Figure 3 Data file list
      

4. Log in to the MRS console. In the navigation pane on the left, choose Clusters > Active Clusters. Click the cluster name. The cluster must contain Hadoop components and has been bound to an IAM agency with the OBS file system operation permission.

   To view or bind an agency, perform the following steps:

   1. In the Dashboard tab of the cluster, check whether the agency parameter has a value and whether the bound agency has the permission to operate an OBS file system.

      

      • If there is the agency name, the cluster has been bound to an agency.
      • If there is no agency name, go to 4.b.
   2. Click Manage Agency to bind an agency that has the permission to operate the OBS file system to the cluster.

      You can select the default MRS_ECS_DEFAULT_AGENCY agency or create an agency that has the permission to operate the OBS file system.

5. Submit the wordcount job.

   On the MRS console, click the Jobs tab and click Create. The Create Job page is displayed. For details, see Running a MapReduce Job.

   Figure 4 wordcount job
   
   • Set Type to MapReduce.
   • Set Name to mr_01.
   • Set the path of the executable program to the address of the program stored on the OBS, for example, obs://wordcount01/program/hadoop-mapreduce-examples-x.x.x.jar.
   • Enter wordcount obs://wordcount01/input/ obs://wordcount01/output/ in the Parameter pane.
     

     • Replace the OBS file system name in obs://wordcount01/input/ with the actual name of the file system created in the environment.
     • Replace the OBS file system name in obs://wordcount01/output/ with the name of the file system created in the actual environment. Replace output with a directory that does not exist based on site requirements.
   • Service Parameter can be left blank.

   A job can be submitted only when the cluster is in the Running state.

   After a job is submitted successfully, it is in the Accepted state by default. You do not need to manually execute the job.

6. View the job execution result.
   1. Go to the Jobs tab page and check whether the job is successfully executed.

      It takes some time to run the job. After the job is complete, refresh the job list to view the job execution, as shown in Figure 5.

      Figure 5 Job list
      

      Once a job has succeeded or failed, you cannot execute it again. However, you can add or copy a job, and set job parameters to submit a job again.

   2. Log in to the OBS console, go to the OBS path, and view the job output information.

      You can view output files in the output directory created in 5. You need to download the file to the local host and open it in text format, as shown in Figure 6.

      Figure 6 Output file list