Viewing the MapReduce Application Commissioning Result

Updated on 2024-08-16 GMT+08:00

View PDF

After a MapReduce application is run, you can view the running result by using WebUI or MapReduce logs.

Viewing job execution status by using the MapReduce Web UI
Log in to MRS Manager, choose Service > MapReduce > JobHistoryServer, and view the job execution status on the web UI.

Figure 1 JobHistory web UI
Viewing job execution status by using Yarn web UI
Log in to MRS Manager, choose Service > Yarn > ResourceManager(Active), and view the job execution status on the web UI.
Figure 2 ResourceManager web UI

Viewing the running result of a MapReduce application

After you run the yarn jar mapreduce-example.jar command in the Linux environment, view application running status in the command output. Example:

yarn jar mapreduce-example.jar /tmp/mapred/example/input/ /tmp/root/output/1
16/07/12 17:07:16 INFO hdfs.PeerCache: SocketCache disabled.
16/07/12 17:07:17 INFO input.FileInputFormat: Total input files to process : 2
16/07/12 17:07:18 INFO mapreduce.JobSubmitter: number of splits:2
16/07/12 17:07:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468241424339_0006
16/07/12 17:07:18 INFO impl.YarnClientImpl: Submitted application application_1468241424339_0006
16/07/12 17:07:18 INFO mapreduce.Job: The url to track the job: http://10-120-180-170:26000/proxy/application_1468241424339_0006/
16/07/12 17:07:18 INFO mapreduce.Job: Running job: job_1468241424339_0006
16/07/12 17:07:31 INFO mapreduce.Job: Job job_1468241424339_0006 running in uber mode : false
16/07/12 17:07:31 INFO mapreduce.Job:  map 0% reduce 0%
16/07/12 17:07:41 INFO mapreduce.Job:  map 50% reduce 0%
16/07/12 17:07:43 INFO mapreduce.Job:  map 100% reduce 0%
16/07/12 17:07:51 INFO mapreduce.Job:  map 100% reduce 100%
16/07/12 17:07:51 INFO mapreduce.Job: Job job_1468241424339_0006 completed successfully
16/07/12 17:07:51 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=75
                FILE: Number of bytes written=435659
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=674
                HDFS: Number of bytes written=23
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=144984
                Total time spent by all reduces in occupied slots (ms)=56280
                Total time spent by all map tasks (ms)=18123
                Total time spent by all reduce tasks (ms)=7035
                Total vcore-milliseconds taken by all map tasks=18123
                Total vcore-milliseconds taken by all reduce tasks=7035
                Total megabyte-milliseconds taken by all map tasks=74231808
                Total megabyte-milliseconds taken by all reduce tasks=28815360
        Map-Reduce Framework
                Map input records=26
                Map output records=16
                Map output bytes=186
                Map output materialized bytes=114
                Input split bytes=230
                Combine input records=16
                Combine output records=6
                Reduce input groups=3
                Reduce shuffle bytes=114
                Reduce input records=6
                Reduce output records=2
                Spilled Records=12
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=202
                CPU time spent (ms)=2720
                Physical memory (bytes) snapshot=1595645952
                Virtual memory (bytes) snapshot=12967759872
                Total committed heap usage (bytes)=2403860480
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=444
        File Output Format Counters 
                Bytes Written=23

After you run the yarn application -status <ApplicationId> command in the Linux environment, view the application running status in the command output. Example:

yarn application -status application_1468241424339_0006
Application Report : 
        Application-Id : application_1468241424339_0006
        Application-Name : Collect Female Info
        Application-Type : MAPREDUCE
        User : root
        Queue : default
        Start-Time : 1468314438442
        Finish-Time : 1468314470080
        Progress : 100%
        State : FINISHED
        Final-State : SUCCEEDED
        Tracking-URL : http://10-120-180-170:26012/jobhistory/job/job_1468241424339_0006
        RPC Port : 27100
        AM Host : 10-120-169-46
        Aggregate Resource Allocation : 172153 MB-seconds, 64 vcore-seconds
        Log Aggregation Status : SUCCEEDED
        Diagnostics : Application finished execution. 
        Application Node Label Expression : <Not set>
        AM container Node Label Expression : <DEFAULT_PARTITION>

Viewing MapReduce logs to learn application running status
View MapReduce logs to learn application running status, and adjust applications based on log information.

Parent topic: Commissioning a MapReduce Application

Previous topic: Compiling and Running a MapReduce Application

Next topic: FAQs About MapReduce Application Development

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot