Viewing the Debugging Result

Scenarios

After a Flink application completes running, you can view the running result, or use Apache Flink Dashboard to view application running status.

Procedure

• View the running result of the Flink application.

  If you want to check the execution result, view the Stdout log of TaskManager on the Apache Flink Dashboard.

  If the execution result is exported to a file or a location specified by Flink, view the result from the exported file or the location. The checkpoint, pipeline, and join between configuration tables and streams are used as examples.

  • View checkpoint results and files
    • The results are stored in the taskmanager.out file of Flink. User can log in to the WEB UI of the Yarn. Choose Jobs > Checkpoints to view the submitted jobs as shown inFigure 1. Choose Task Managers > Stdout to view the running result, as shown inFigure 2.
      Figure 1 submitted jobs
      
      Figure 2 execution result
      
    • Either the following methods can be used to view the checkpoint file:
      • If the checkpoint snapshot information is saved in the HDFS, run the hdfs dfs -ls hdfs://hacluster/flink/checkpoint/ command to view checkpoint files.
      • If the checkpoint snapshot information is saved to a local file, log in to each node to view checkpoint files.
  • View pipeline results
    • The results are stored in the taskmanager.out file of Flink. User can log in to the WEB UI of the Yarn. Choose Jobs > Running Jobs to view the running jobs as shown inFigure 3. Choose Task Managers. You can see two tasks, as shown in Figure 4. Click any task, choose Stdout to view the output of the task, as shown in Figure 5 and Figure 6.
      Figure 3 running jobs
      
      Figure 4 submitted Tasks
      
      Figure 5 output of task1
      
      Figure 6 output of task2
      
  • View the JOIN result of configuration table and steams
    • The results are stored in the taskmanager.out file of Flink. User can log in to the WEB UI of the Yarn. Choose Jobs > Completed Jobs to view the completed job as shown in Figure 7. Choose Task Managers, you can see submitted task, as shown in Figure 8. Choose Stdout to view the running result, as shown in Figure 9.
      Figure 7 completed job
      
      Figure 8 submitted task
      
      Figure 9 execution result
      
  • View the result of DataStream
    • The results are stored in the taskmanager.out file of Flink. User can log in to the WEB UI of the Yarn. Choose Jobs > Completed Jobs to view the completed job as shown in Figure 10. Choose Task Managers, you can see submitted task, as shown in Figure 11. Choose Stdout to view the running result, as shown in Figure 12.
      Figure 10 completed job
      
      Figure 11 submitted task
      
      Figure 12 execution result
      
  • View the result of stream sql join
    • The results are stored in the taskmanager.out file of Flink. User can log in to the WEB UI of the Yarn. Choose Jobs > Running Jobs to view the running jobs as shown in Figure 13. Choose Task Managers, you can see submitted task, as shown in Figure 14. Choose Stdout to view the running result, as shown in Figure 15.
      Figure 13 running jobs
      
      Figure 14 submitted task
      
      Figure 15 execution result
      
  • View the result of produce and consume data in Kafka
    • The results are stored in the taskmanager.out file of Flink. User can log in to the WEB UI of the Yarn. Choose Jobs > Running Jobs to view the running jobs as shown in Figure 16. Choose Task Managers, you can see submitted task, as shown in Figure 17. Choose Stdout to view the running result, as shown in Figure 18.
      Figure 16 runnning jobs
      
      Figure 17 submitted task
      
      Figure 18 execution result
      
• Use Apache Flink Dashboard to view the running status of the Flink application.

  The Apache Flink Dashboard mainly includes Overview, Running Jobs, Completed Jobs, Task Managers, Job Manager and Logout and so on.

  On In the YARN web UI, find the desired Flink application. Click the ApplicationMaster at the last column of the application to switch to the Apache Flink Dashboard.

  View the print results of the program execution: find the corresponding Task Manager to see the corresponding Stdout tag log information.

• View Flink logs.

  Three methods can be used to obtain Flink logs:

  • Log in to the Apache Flink Dashboard and view logs of TaskManagers and JobManager.
  • Log in to the YARN web UI to view logs about JobManager and GC.

    On the YARN web UI wind, find the desired Flink application. Click the ID of the application. On the switched page, click Logs in the Logs column.

  • On the Yarn client, obtain or view logs of Task Managers and Job Manager.
    1. Download and install the Yarn client, for example, in the /opt/client directory.
    2. Log in to the node where the client is installed as the client installation user.
    3. Run the following command to switch to the client installation directory:

       cd /opt/client

    4. Run the following command to configure environment variables:

       source bigdata_env

    5. If the cluster employs the security mode, run the following command to authenticate the user. If the normal mode is used, skip this step.

       kinit component service user

    6. Run the following commands to obtain container information of the Flink cluster:

       yarn logs -applicationId application_* -show_application_log_info

       

    7. Run the following command to obtain run logs of the specified container. Generally, container_*_000001 is the container where the Job Manager is running.

       yarn logs -applicationId application_* --containerId container_1547547065745_0001_01_000004 -out logdir/

       

       After the command is executed, container run logs, including run logs of the Task Manager and Job Manager and GC logs, are downloaded to the local host.

    8. You can also run a command to obtain the log with the specified name:

       The following command is used to obtain the container log list.

       yarn logs -applicationId application_* -show_container_log_info --containerId container_1547547065745_0001_01_000004

       

       Download taskmanager.log to the local host.

       yarn logs -applicationId application_* --containerId container_1547547065745_0001_01_000004 -log_files taskmanager.log -out localpath