Updated on 2022-09-14 GMT+08:00

Checking the Commissioning Result

Scenario

After a MapReduce application is run, you can check the running result through one of the following methods:

  • Viewing the command output.
  • Logging in to the MapReduce WebUI.
  • Logging in to the Yarn WebUI.
  • Viewing MapReduce logs.

You must have permission to access WebUI. If not, contact the admin to obtain an account and password.

Procedure

  • Check the running result by using MapReduce WebUI.

    Log in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Mapreduce > JobHistoryServer. On the web page that is displayed, view the task execution status.

    Figure 1 JobHistory Web UI
  • Check the running result by using YARN WebUI.
    Log in to FusionInsight Manager as a user who has the permission to view task and choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active). On the web page that is displayed, view the task execution status.
    Figure 2 ResourceManager Web UI
  • Check the running result of the MapReduce application.
    • After running the yarn jar MRTest-XXX.jar command in the Linux environment, you can check the running status of the application by the returned information about the command.
      yarn jar MRTest-XXX.jar /tmp/mapred/example/input/ /tmp/root/output/1
      16/07/12 17:07:16 INFO hdfs.PeerCache: SocketCache disabled.
      16/07/12 17:07:17 INFO input.FileInputFormat: Total input files to process : 2
      16/07/12 17:07:18 INFO mapreduce.JobSubmitter: number of splits:2
      16/07/12 17:07:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468241424339_0006
      16/07/12 17:07:18 INFO impl.YarnClientImpl: Submitted application application_1468241424339_0006
      16/07/12 17:07:18 INFO mapreduce.Job: The url to track the job: http://10-120-180-170:8088/proxy/application_1468241424339_0006/
      16/07/12 17:07:18 INFO mapreduce.Job: Running job: job_1468241424339_0006
      16/07/12 17:07:31 INFO mapreduce.Job: Job job_1468241424339_0006 running in uber mode : false
      16/07/12 17:07:31 INFO mapreduce.Job:  map 0% reduce 0%
      16/07/12 17:07:41 INFO mapreduce.Job:  map 50% reduce 0%
      16/07/12 17:07:43 INFO mapreduce.Job:  map 100% reduce 0%
      16/07/12 17:07:51 INFO mapreduce.Job:  map 100% reduce 100%
      16/07/12 17:07:51 INFO mapreduce.Job: Job job_1468241424339_0006 completed successfully
      16/07/12 17:07:51 INFO mapreduce.Job: Counters: 49
              File System Counters
                      FILE: Number of bytes read=75
                      FILE: Number of bytes written=435659
                      FILE: Number of read operations=0
                      FILE: Number of large read operations=0
                      FILE: Number of write operations=0
                      HDFS: Number of bytes read=674
                      HDFS: Number of bytes written=23
                      HDFS: Number of read operations=9
                      HDFS: Number of large read operations=0
                      HDFS: Number of write operations=2
              Job Counters 
                      Launched map tasks=2
                      Launched reduce tasks=1
                      Data-local map tasks=2
                      Total time spent by all maps in occupied slots (ms)=144984
                      Total time spent by all reduces in occupied slots (ms)=56280
                      Total time spent by all map tasks (ms)=18123
                      Total time spent by all reduce tasks (ms)=7035
                      Total vcore-milliseconds taken by all map tasks=18123
                      Total vcore-milliseconds taken by all reduce tasks=7035
                      Total megabyte-milliseconds taken by all map tasks=74231808
                      Total megabyte-milliseconds taken by all reduce tasks=28815360
              Map-Reduce Framework
                      Map input records=26
                      Map output records=16
                      Map output bytes=186
                      Map output materialized bytes=114
                      Input split bytes=230
                      Combine input records=16
                      Combine output records=6
                      Reduce input groups=3
                      Reduce shuffle bytes=114
                      Reduce input records=6
                      Reduce output records=2
                      Spilled Records=12
                      Shuffled Maps =2
                      Failed Shuffles=0
                      Merged Map outputs=2
                      GC time elapsed (ms)=202
                      CPU time spent (ms)=2720
                      Physical memory (bytes) snapshot=1595645952
                      Virtual memory (bytes) snapshot=12967759872
                      Total committed heap usage (bytes)=2403860480
              Shuffle Errors
                      BAD_ID=0
                      CONNECTION=0
                      IO_ERROR=0
                      WRONG_LENGTH=0
                      WRONG_MAP=0
                      WRONG_REDUCE=0
              File Input Format Counters 
                      Bytes Read=444
              File Output Format Counters 
                      Bytes Written=23
    • In the Linux environment, run the yarn application -status <ApplicationID> command to check the running result of the current application. Example:
      yarn application -status application_1468241424339_0006
      Application Report : 
              Application-Id : application_1468241424339_0006
              Application-Name : Collect Female Info
              Application-Type : MAPREDUCE
              User : root
              Queue : default
              Start-Time : 1468314438442
              Finish-Time : 1468314470080
              Progress : 100%
              State : FINISHED
              Final-State : SUCCEEDED
              Tracking-URL : http://10-120-180-170:19888/jobhistory/job/job_1468241424339_0006
              RPC Port : 27100
              AM Host : 10-120-169-46
              Aggregate Resource Allocation : 172153 MB-seconds, 64 vcore-seconds
              Log Aggregation Status : SUCCEEDED
              Diagnostics : Application finished execution. 
              Application Node Label Expression : <Not set>
              AM container Node Label Expression : <DEFAULT_PARTITION>
  • View MapReduce logs to learn application running conditions.

    MapReduce logs offers immediate visibility into application running conditions. You can adjust application programs based on the logs.