Checking the Commissioning Result

Updated on 2022-09-14 GMT+08:00

View PDF

Scenario

After a MapReduce application is run, you can check the running result through one of the following methods:

Viewing the program running status in IntelliJ IDEA.
Viewing MapReduce logs.
Logging in to the MapReduce WebUI.
Logging in to the Yarn WebUI.

NOTE:

You must have permission to access WebUI. If not, contact the admin to obtain an account and password.

Procedure

Check the running result by obtaining the MapReduce log.

As shown below, the console outputs the application running result.

3614 [main] INFO  org.apache.hadoop.hdfs.PeerCache  - SocketCache disabled.
10159 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input files to process : 2
11378 [main] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:2
12707 [main] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - Submitting tokens for job: job_1468241424339_0002
16434 [main] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl  - Submitted application application_1468241424339_0002
16656 [main] INFO  org.apache.hadoop.mapreduce.Job  - The url to track the job: http://10-120-180-170:8088/proxy/application_1468241424339_0002/
16657 [main] INFO  org.apache.hadoop.mapreduce.Job  - Running job: job_1468241424339_0002
31177 [main] INFO  org.apache.hadoop.mapreduce.Job  - Job job_1468241424339_0002 running in uber mode : false
31200 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
45893 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 100% reduce 0%
57172 [main] INFO  org.apache.hadoop.mapreduce.Job  -  map 100% reduce 100%
58554 [main] INFO  org.apache.hadoop.mapreduce.Job  - Job job_1468241424339_0002 completed successfully
58908 [main] INFO  org.apache.hadoop.mapreduce.Job  - Counters: 49
File System Counters
FILE: Number of bytes read=75
FILE: Number of bytes written=436979
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=674
HDFS: Number of bytes written=23
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters 
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=206088
Total time spent by all reduces in occupied slots (ms)=73824
Total time spent by all map tasks (ms)=25761
Total time spent by all reduce tasks (ms)=9228
Total vcore-seconds taken by all map tasks=25761
Total vcore-seconds taken by all reduce tasks=9228
Total megabyte-seconds taken by all map tasks=105517056
Total megabyte-seconds taken by all reduce tasks=37797888
Map-Reduce Framework
Map input records=26
Map output records=16
Map output bytes=186
Map output materialized bytes=114
Input split bytes=230
Combine input records=16
Combine output records=6
Reduce input groups=3
Reduce shuffle bytes=114
Reduce input records=6
Reduce output records=2
Spilled Records=12
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=356
CPU time spent (ms)=2860
Physical memory (bytes) snapshot=1601576960
Virtual memory (bytes) snapshot=12999819264
Total committed heap usage (bytes)=2403336192
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters 
Bytes Read=444
File Output Format Counters 
Bytes Written=23

NOTE:

In the Windows environment, the following exception occurs but does not affect services.

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

Check the running result by using MapReduce WebUI.
LLog in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Mapreduce > JobHistoryServer. On the web page that is displayed, view the task execution status.

Figure 1 JobHistory Web UI
Check the running result by using YARN WebUI.
Log in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active). On the web page that is displayed, view the task execution status.
Figure 2 ResourceManager Web UI
View MapReduce logs to learn application running conditions.
MapReduce logs offers immediate visibility into application running conditions. You can adjust application programs based on the logs.