What Are the Differences Between the Client Mode and Cluster Mode of Spark Jobs?
You need to understand the concept ApplicationMaster before understanding the essential differences between Yarn-client and Yarn-cluster.
In Yarn, each application instance has an ApplicationMaster process, which is the first container started by the application. It interacts with ResourceManager and requests resources. After obtaining resources, it instructs NodeManager to start containers. The essential difference between the Yarn-cluster and Yarn-client modes lies in the ApplicationMaster process.
In Yarn-cluster mode, Driver runs in ApplicationMaster, which requests resources from Yarn and monitors the running status of a job. After a user submits a job, the client can be stopped and the job continues running on Yarn. Therefore, the Yarn-cluster mode is not suitable for running interactive jobs.
In Yarn-client mode, ApplicationMaster requests only Executor from Yarn. The client communicates with the requested containers to schedule tasks. Therefore, the client cannot be stopped.
Job Development FAQs
- How Do I Get My Data into OBS or HDFS?
- What Types of Spark Jobs Can Be Submitted in a Cluster?
- Can I Run Multiple Spark Tasks at the Same Time After the Minimum Tenant Resources of an MRS Cluster Is Changed to 0?
- What Should I Do If Job Parameters Separated By Spaces Cannot Be Identified?
- What Are the Differences Between the Client Mode and Cluster Mode of Spark Jobs?
- How Do I View MRS Job Logs?
- What Can I Do If the System Displays a Message Indicating that the Current User Does Not Exist on Manager When I Submit a Job?
- What Can I Do If LauncherJob Fails to Be Executed and the Error Message "jobPropertiesMap is null" Is Displayed?
- What Should I Do If the Flink Job Status on the MRS Console Is Inconsistent with That on Yarn?
- What Can I Do If a SparkStreaming Job Fails After Running for Dozens of Hours and Error 403 Is Reported for OBS Access?
- What Should I Do If an Alarm Is Reported Indicating that the Memory Is Insufficient When I Execute a SQL Statement on the ClickHouse Client?
- What Should I Do If Error Message "java.io.IOException: Connection reset by peer" Is Displayed During the Execution of a Spark Job?
- What Should I Do If the Error Message "requestId=XXX" Is Displayed When a Spark Job Accesses OBS?
- What Should I Do If the Error Message "UnknownScannerExeception" Is Displayed for Spark Jobs?
- What Can I Do If DataArts Studio Occasionally Fails to Schedule Spark Jobs?
- What Should I Do If a Flink Job Fails to Execute and the Error Message "java.lang.NoSuchFieldError: SECURITY_SSL_ENCRYPT_ENABLED" Is Displayed?
- What Should I Do If Submitted Yarn Jobs Cannot Be Viewed on the Web UI?
- How Do I Modify the HDFS fs.defaultFS of an Existing Cluster?
- What Can I Do If launcher-job Is Terminated by Yarn When a Flink Task Is Submitted?
- What Should I Do If the Error Message "slot request timeout" Is Displayed When I Submit a Flink Job?
- FAQs About Importing and Exporting Data Using DistCP Jobs
- How Do I View SQL Statements of Hive Jobs on the Yarn Web UI?
- How Do I View Logs of a Specified Yarn Task?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore