Checking the Commissioning Result

Updated on 2022-09-14 GMT+08:00

View PDF

Scenario

After a MapReduce application is run, you can check the running result through one of the following methods:

Viewing the command output.
Logging in to the MapReduce WebUI.
Logging in to the Yarn WebUI.
Viewing MapReduce logs.

NOTE:

You must have permission to access WebUI. If not, contact the admin to obtain an account and password.

Procedure

Check the running result by using MapReduce WebUI.
Log in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Mapreduce > JobHistoryServer. On the web page that is displayed, view the task execution status.

Figure 1 JobHistory Web UI
Check the running result by using YARN WebUI.
Log in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active). On the web page that is displayed, view the task execution status.
Figure 2 ResourceManager Web UI

Check the running result of the MapReduce application.

After running the yarn jar MRTest-XXX.jar command in the Linux environment, you can check the running status of the application by the returned information about the command.

yarn jar MRTest-XXX.jar /tmp/mapred/example/input/ /tmp/root/output/1
16/07/12 17:07:16 INFO hdfs.PeerCache: SocketCache disabled.
16/07/12 17:07:17 INFO input.FileInputFormat: Total input files to process : 2
16/07/12 17:07:18 INFO mapreduce.JobSubmitter: number of splits:2
16/07/12 17:07:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468241424339_0006
16/07/12 17:07:18 INFO impl.YarnClientImpl: Submitted application application_1468241424339_0006
16/07/12 17:07:18 INFO mapreduce.Job: The url to track the job: http://10-120-180-170:8088/proxy/application_1468241424339_0006/
16/07/12 17:07:18 INFO mapreduce.Job: Running job: job_1468241424339_0006
16/07/12 17:07:31 INFO mapreduce.Job: Job job_1468241424339_0006 running in uber mode : false
16/07/12 17:07:31 INFO mapreduce.Job:  map 0% reduce 0%
16/07/12 17:07:41 INFO mapreduce.Job:  map 50% reduce 0%
16/07/12 17:07:43 INFO mapreduce.Job:  map 100% reduce 0%
16/07/12 17:07:51 INFO mapreduce.Job:  map 100% reduce 100%
16/07/12 17:07:51 INFO mapreduce.Job: Job job_1468241424339_0006 completed successfully
16/07/12 17:07:51 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=75
                FILE: Number of bytes written=435659
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=674
                HDFS: Number of bytes written=23
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=144984
                Total time spent by all reduces in occupied slots (ms)=56280
                Total time spent by all map tasks (ms)=18123
                Total time spent by all reduce tasks (ms)=7035
                Total vcore-milliseconds taken by all map tasks=18123
                Total vcore-milliseconds taken by all reduce tasks=7035
                Total megabyte-milliseconds taken by all map tasks=74231808
                Total megabyte-milliseconds taken by all reduce tasks=28815360
        Map-Reduce Framework
                Map input records=26
                Map output records=16
                Map output bytes=186
                Map output materialized bytes=114
                Input split bytes=230
                Combine input records=16
                Combine output records=6
                Reduce input groups=3
                Reduce shuffle bytes=114
                Reduce input records=6
                Reduce output records=2
                Spilled Records=12
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=202
                CPU time spent (ms)=2720
                Physical memory (bytes) snapshot=1595645952
                Virtual memory (bytes) snapshot=12967759872
                Total committed heap usage (bytes)=2403860480
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=444
        File Output Format Counters 
                Bytes Written=23

In the Linux environment, run the yarn application -status <ApplicationID> command to check the running result of the current application. Example:

yarn application -status application_1468241424339_0006
Application Report : 
        Application-Id : application_1468241424339_0006
        Application-Name : Collect Female Info
        Application-Type : MAPREDUCE
        User : root
        Queue : default
        Start-Time : 1468314438442
        Finish-Time : 1468314470080
        Progress : 100%
        State : FINISHED
        Final-State : SUCCEEDED
        Tracking-URL : http://10-120-180-170:19888/jobhistory/job/job_1468241424339_0006
        RPC Port : 27100
        AM Host : 10-120-169-46
        Aggregate Resource Allocation : 172153 MB-seconds, 64 vcore-seconds
        Log Aggregation Status : SUCCEEDED
        Diagnostics : Application finished execution. 
        Application Node Label Expression : <Not set>
        AM container Node Label Expression : <DEFAULT_PARTITION>

View MapReduce logs to learn application running conditions.
MapReduce logs offers immediate visibility into application running conditions. You can adjust application programs based on the logs.