查看MapReduce应用调测结果

MapReduce应用程序运行完成后，可以通过WebUI查看应用程序运行情况，也可以通过MapReduce日志获取应用运行情况。

• 通过MapReduce服务的WebUI进行查看

  登录MRS Manager，单击“服务管理 > MapReduce > JobHistoryServer”进入Web界面后查看任务执行状态。

  图1 JobHistory Web UI界面
  
• 通过YARN服务的WebUI进行查看
  登录MRS Manager，单击“服务管理 > Yarn > ResourceManager(主)”进入Web界面后查看任务执行状态。

  图2 ResourceManager Web UI页面
  

• 查看MapReduce应用运行结果数据。
  • 当用户在Linux环境下执行yarn jar mapreduce-example.jar命令后，可以通过执行结果显示正在执行的应用的运行情况。例如：

    yarn jar mapreduce-example.jar /tmp/mapred/example/input/ /tmp/root/output/1
    16/07/12 17:07:16 INFO hdfs.PeerCache: SocketCache disabled.
    16/07/12 17:07:17 INFO input.FileInputFormat: Total input files to process : 2
    16/07/12 17:07:18 INFO mapreduce.JobSubmitter: number of splits:2
    16/07/12 17:07:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468241424339_0006
    16/07/12 17:07:18 INFO impl.YarnClientImpl: Submitted application application_1468241424339_0006
    16/07/12 17:07:18 INFO mapreduce.Job: The url to track the job: http://10-120-180-170:26000/proxy/application_1468241424339_0006/
    16/07/12 17:07:18 INFO mapreduce.Job: Running job: job_1468241424339_0006
    16/07/12 17:07:31 INFO mapreduce.Job: Job job_1468241424339_0006 running in uber mode : false
    16/07/12 17:07:31 INFO mapreduce.Job:  map 0% reduce 0%
    16/07/12 17:07:41 INFO mapreduce.Job:  map 50% reduce 0%
    16/07/12 17:07:43 INFO mapreduce.Job:  map 100% reduce 0%
    16/07/12 17:07:51 INFO mapreduce.Job:  map 100% reduce 100%
    16/07/12 17:07:51 INFO mapreduce.Job: Job job_1468241424339_0006 completed successfully
    16/07/12 17:07:51 INFO mapreduce.Job: Counters: 49
            File System Counters
                    FILE: Number of bytes read=75
                    FILE: Number of bytes written=435659
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=674
                    HDFS: Number of bytes written=23
                    HDFS: Number of read operations=9
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=2
            Job Counters 
                    Launched map tasks=2
                    Launched reduce tasks=1
                    Data-local map tasks=2
                    Total time spent by all maps in occupied slots (ms)=144984
                    Total time spent by all reduces in occupied slots (ms)=56280
                    Total time spent by all map tasks (ms)=18123
                    Total time spent by all reduce tasks (ms)=7035
                    Total vcore-milliseconds taken by all map tasks=18123
                    Total vcore-milliseconds taken by all reduce tasks=7035
                    Total megabyte-milliseconds taken by all map tasks=74231808
                    Total megabyte-milliseconds taken by all reduce tasks=28815360
            Map-Reduce Framework
                    Map input records=26
                    Map output records=16
                    Map output bytes=186
                    Map output materialized bytes=114
                    Input split bytes=230
                    Combine input records=16
                    Combine output records=6
                    Reduce input groups=3
                    Reduce shuffle bytes=114
                    Reduce input records=6
                    Reduce output records=2
                    Spilled Records=12
                    Shuffled Maps =2
                    Failed Shuffles=0
                    Merged Map outputs=2
                    GC time elapsed (ms)=202
                    CPU time spent (ms)=2720
                    Physical memory (bytes) snapshot=1595645952
                    Virtual memory (bytes) snapshot=12967759872
                    Total committed heap usage (bytes)=2403860480
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters 
                    Bytes Read=444
            File Output Format Counters 
                    Bytes Written=23

  • 在Linux环境下执行yarn application -status <ApplicationId> ，可以通过执行结果显示正在执行的应用的运行情况。例如：

    yarn application -status application_1468241424339_0006
    Application Report : 
            Application-Id : application_1468241424339_0006
            Application-Name : Collect Female Info
            Application-Type : MAPREDUCE
            User : root
            Queue : default
            Start-Time : 1468314438442
            Finish-Time : 1468314470080
            Progress : 100%
            State : FINISHED
            Final-State : SUCCEEDED
            Tracking-URL : http://10-120-180-170:26012/jobhistory/job/job_1468241424339_0006
            RPC Port : 27100
            AM Host : 10-120-169-46
            Aggregate Resource Allocation : 172153 MB-seconds, 64 vcore-seconds
            Log Aggregation Status : SUCCEEDED
            Diagnostics : Application finished execution. 
            Application Node Label Expression : <Not set>
            AM container Node Label Expression : <DEFAULT_PARTITION>

• 查看MapReduce日志获取应用运行情况。

  您可以查看MapReduce日志了解应用运行情况，并根据日志信息调整应用程序。