Driver Displays a Message Indicating That the Running Memory Exceeds the Threshold When a Spark Task Is Submitted

Symptom

A Spark task fails to be submitted because the running memory exceeds the threshold.

Cause Analysis

... INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (6144 MB per container)
... ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (10240+1024 MB) is above the max threshold (6144 MB) of this cluster!

Spark tasks are submitted to Yarn and the resources used by the Executor to run tasks are managed by Yarn. From the error message, you can see that when a user starts the Executor, 10 GB memory is specified, which exceeds the upper memory limit of each Container set by Yarn. As a result, the task cannot be started.

Solution

Modify the Yarn configuration to increase the restriction on containers. For example, you can adjust parameter yarn.scheduler.maximum-allocation-mb to control the resources for starting the Executor. Restart the Yarn service after the modification.

MRS Manager:

1. Log in to MRS Manager.
2. Choose Services > Yarn > Service Configuration and set Type to All.
3. In Search, enter yarn.scheduler.maximum-allocation-mb to modify the parameter, save the configuration, and then restart the service. See the following figure.
   Figure 1 Modifying Yarn service parameters
   

FusionInsight Manager:

1. Log in to FusionInsight Manager.
2. Choose Cluster > Service > Yarn. Click Configurations and select All Configurations.
3. In Search, enter yarn.scheduler.maximum-allocation-mb to modify the parameter, save the configuration, and then restart the service.