The SparkUI Page Cannot Be Opened During the Running of a Spark Job

Symptom

When a Spark job is running in an MRS cluster, the SparkUI page cannot be accessed from the Yarn web UI. The page is loading for a long time.

After the job is complete, the SparkUI page can be accessed.

Cause Analysis

This problem is usually caused by a driver exception. Check the GC logs of the driver first. You may find that full GC logs are frequently generated.

Procedure

1. If the SparkUI page cannot be opened due to insufficient driver memory and frequent full GC, you are advised to increase the driver memory by adjusting the spark.driver.memory parameter.
2. Check whether the upper-layer service code is improper. For example:
   1. Operators such as collect() and show() are called. These operators extract all data in the cluster to the driver memory.
   2. When a large table or partition is selected, no restriction is added. As a result, the data volume is too large.