Failed to Switch to the Log Page from stderr and stdout on the Native Spark Web UI

Symptom

When Spark tasks are executed in an MRS 3.x cluster, stderr and stdout on the native Spark2x web UI cannot redirect to the log page.

Cause Analysis

The spark.httpdProxy.enable configuration item is not enabled.

Procedure (Modifying Configuration Items)

1. On the client, set spark.httpdProxy.enable of the spark-defaults.conf configuration item to true. The default value is false.
2. Log in to Manager, choose Cluster > Services > Spark2x. On the Spark2x page, choose Configurations > All Configurations > JobHistory2x > Customization. Locate the custom parameter in the right pane, add configuration item spark.httpdProxy.enable, and set the value to true.
   Figure 1 Customizing a configuration item
   

3. By referring to step 2, add configuration item spark.httpdProxy.enable and set the value to true respectively for JDBCServer2x and SparkResource2x.
4. Save the configuration. In the upper right corner of the Spark2x page, click More and select Service Rolling Restart.
5. Log in to the native Spark2x page and click stdout or stderr to access the executor log page.

Procedure (Without Modifying Configuration Items)

• View the logs of completed tasks.

  On the native Spark2x page, click AggregatedLogs to view the aggregated task logs.

  Figure 2 AggregatedLogs
  
• View the logs of running tasks.

  On the native Spark2x page, choose Jobs > Stage > Task to locate the target executor and then click stdout or stderr to view logs of the executor.

  Figure 3 Jobs
  
  Figure 4 Stage
  
  Figure 5 Task
  

Related Information

After you click stdout or stderr to go to the log page, only stdout or stderr logs are displayed. To view complete logs, you can delete stdout or stderr from the URL and then access all logs of the executor.

For example:

https://<EIP>:9022/component/Yarn/NodeManager/15/node/containerlogs/container_e04_1627976981617_0002_01_000002/root/stderr?start=-4096

You can modify the URL as follows:

https://<EIP>:9022/component/Yarn/NodeManager/15/node/containerlogs/container_e04_1627976981617_0002_01_000002/root/?start=-4096

Then you can view all logs of the executor.

Figure 6 All log files