Updated on 2022-09-14 GMT+08:00

Checking the Commissioning Result

Scenario

After a MapReduce application is run, you can check the running result through one of the following methods:
  • Viewing the program running status in IntelliJ IDEA.
  • Viewing MapReduce logs.
  • Logging in to the MapReduce WebUI.
  • Logging in to the Yarn WebUI.

You must have permission to access WebUI. If not, contact the admin to obtain an account and password.

Procedure

  • Check the running result by obtaining the MapReduce log.

    As shown below, the console outputs the application running result.

    3614 [main] INFO  org.apache.hadoop.hdfs.PeerCache  - SocketCache disabled.
    10159 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input files to process : 2
    11378 [main] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:2
    12707 [main] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - Submitting tokens for job: job_1468241424339_0002
    16434 [main] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl  - Submitted application application_1468241424339_0002
    16656 [main] INFO  org.apache.hadoop.mapreduce.Job  - The url to track the job: http://10-120-180-170:8088/proxy/application_1468241424339_0002/
    16657 [main] INFO  org.apache.hadoop.mapreduce.Job  - Running job: job_1468241424339_0002
    31177 [main] INFO  org.apache.hadoop.mapreduce.Job  - Job job_1468241424339_0002 running in uber mode : false
    31200 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
    45893 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 100% reduce 0%
    57172 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 100% reduce 100%
    58554 [main] INFO  org.apache.hadoop.mapreduce.Job  - Job job_1468241424339_0002 completed successfully
    58908 [main] INFO  org.apache.hadoop.mapreduce.Job  - Counters: 49
    File System Counters
    FILE: Number of bytes read=75
    FILE: Number of bytes written=436979
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=674
    HDFS: Number of bytes written=23
    HDFS: Number of read operations=9
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=2
    Job Counters 
    Launched map tasks=2
    Launched reduce tasks=1
    Data-local map tasks=2
    Total time spent by all maps in occupied slots (ms)=206088
    Total time spent by all reduces in occupied slots (ms)=73824
    Total time spent by all map tasks (ms)=25761
    Total time spent by all reduce tasks (ms)=9228
    Total vcore-seconds taken by all map tasks=25761
    Total vcore-seconds taken by all reduce tasks=9228
    Total megabyte-seconds taken by all map tasks=105517056
    Total megabyte-seconds taken by all reduce tasks=37797888
    Map-Reduce Framework
    Map input records=26
    Map output records=16
    Map output bytes=186
    Map output materialized bytes=114
    Input split bytes=230
    Combine input records=16
    Combine output records=6
    Reduce input groups=3
    Reduce shuffle bytes=114
    Reduce input records=6
    Reduce output records=2
    Spilled Records=12
    Shuffled Maps =2
    Failed Shuffles=0
    Merged Map outputs=2
    GC time elapsed (ms)=356
    CPU time spent (ms)=2860
    Physical memory (bytes) snapshot=1601576960
    Virtual memory (bytes) snapshot=12999819264
    Total committed heap usage (bytes)=2403336192
    Shuffle Errors
    BAD_ID=0
    CONNECTION=0
    IO_ERROR=0
    WRONG_LENGTH=0
    WRONG_MAP=0
    WRONG_REDUCE=0
    File Input Format Counters 
    Bytes Read=444
    File Output Format Counters 
    Bytes Written=23

    In the Windows environment, the following exception occurs but does not affect services.

    java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

  • Check the running result by using MapReduce WebUI.

    LLog in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Mapreduce > JobHistoryServer. On the web page that is displayed, view the task execution status.

    Figure 1 JobHistory Web UI
  • Check the running result by using YARN WebUI.
    Log in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active). On the web page that is displayed, view the task execution status.
    Figure 2 ResourceManager Web UI
  • View MapReduce logs to learn application running conditions.

    MapReduce logs offers immediate visibility into application running conditions. You can adjust application programs based on the logs.