Obtaining Container Logs of a Running Spark Application
Scenario
During the running of a Spark application, container logs are important for fault locating and application status monitoring. Container logs of running Spark applications are distributed on multiple nodes. This section describes how to quickly obtain container logs.
Obtaining Logs
You can run the yarn logs command to obtain the logs of applications running on YARN. In different scenarios, you can run the following commands to obtain required logs:
- Run the following command to obtain the complete logs of the application:
yarn logs --applicationId <appId> -out <outputDir>
<appId> indicates the ID of the Spark application, and <outputDir> indicates the log output path. For example:
yarn logs --applicationId application_1574856994802_0016 -out /opt/test
The following figure shows the command output.
- If the application is running, container logs in the dead state cannot be obtained.
- If the application is stopped, all archived container logs can be obtained.
- Run the following command to obtain the logs of a specified container:
yarn logs -applicationId <appId> -containerId <containerId>
<containerId> indicates the target container ID. For example:
yarn logs -applicationId application_1574856994802_0018 -containerId container_e01_1574856994802_0018_01_000003
The following figure shows the command output.
- If the application is running, container logs in the dead state cannot be obtained.
- If the application is stopped, you can obtain logs of any container.
- Run the following command to obtain container logs in any state:
yarn logs -applicationId <appId> -containerId <containerId> -nodeAddress <nodeAddress>
<nodeAddress> indicates the node IP address and port number. You can obtain them by running the following command:
yarn node -list -all
For example:
yarn logs -applicationId application_1574856994802_0019 -containerId container_e01_1574856994802_0019_01_000003 -nodeAddress 192-168-1-1:8041
Execution result: Logs of any container can be obtained.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot