Configure YARN Task Preemption
Scenarios
- Preemption principles of the Capacity Scheduler:
The Capacity Scheduler in the ResourceManager implements job preemption to balance resource allocation across queues and maximize overall cluster utilization. The process is as follows:
- Assume that there are two queues (Queue A and Queue B). The capacity of Queue A is 25%, and the capacity of Queue B is 75%.
- In the initial state, Task 1 is distributed to Queue A for processing, requiring 75% cluster resources. Task 2 is distributed to Queue B for processing, requiring 50% cluster resources.
- Task 1 uses 25% cluster resources provided by Queue A and 50% resources from Queue B. Queue B reserves 25% cluster resources.
- If task preemption is enabled, the resources of Task 1 will be preempted. Queue B preempts 25% cluster resources from Queue A for Task 2.
- Task 1 executes once Task 2 is complete and the cluster has sufficient resources.
- Preemption Principles of the Superior Scheduler
When cluster resources are abundant, the Superior Scheduler allows queues to utilize resources beyond their minimum and shared allocations, maximizing overall resource utilization. Conversely, when resources are scarce, preemption ensures that each queue receives its minimum guaranteed resources as per the queue policy, maintaining fairness. The following are examples: The queue resource information is as follows (minimum indicates the minimum resource, and share indicates the weight):
- User A submits Job 1 to Queue 1. The ApplicationMaster requests <1 vCPU and 2 GB memory>, and the job consists of eight tasks. Each task requires <2 vCPUs and 4 GB memory>. Because cluster resources are sufficient, Job 1 executes successfully. Queue 1 uses <17 vCPUs and 34 GB memory>. At this point, the cluster has <23 vCPUs and 6 GB memory> remaining.
- User B submits Job 2 to Queue 2. The ApplicationMaster requests <1 vCPU and 2 GB memory>, and the job consists of five tasks. Each task requires <2 vCPUs and 4 GB memory>. Before preemption is triggered, the ApplicationMaster and one task of Job 2 can execute successfully. Queue 2 uses <3 vCPUs and 6 GB memory>. At this point, the remaining cluster capacity is <20 vCPUs and 0 GB memory>.
- Since Queue 2 has not yet reached its minimum guaranteed allocation and has pending tasks, the Capacity Scheduler triggers preemption to reclaim resources from a task in Job 1. This action allows Job 2 to run an additional task using the preempted resources. After preemption, Queue 2 uses <5 vCPUs and 10 GB memory>.
If either the CPU or memory allocations of a queue fall at or below its minimum guaranteed threshold, those resources cannot be preempted by other queues.
Procedure
Navigation path for setting parameters:
Go to the All Configurations page of Yarn and enter a parameter name in the search box by referring to Modifying Cluster Service Configuration Parameters.
| Parameter | Description | Default Value |
|---|---|---|
| yarn.resourcemanager.scheduler.monitor.enable | Whether to start scheduler monitoring according to yarn.resourcemanager.scheduler.monitor.policies. If this parameter is set to true, scheduler monitoring is enabled based on policies specified by yarn.resourcemanager.scheduler.monitor.policies and task resource preemption is enabled based on the scheduler information. If this parameter is set to false, scheduler monitoring is disabled. | false |
| yarn.resourcemanager.scheduler.monitor.policies | List of the SchedulingEditPolicy class to be used with the scheduler | org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy |
| yarn.resourcemanager.monitor.capacity.preemption.observe_only |
| false |
| yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval | Monitoring interval, in millisecond. If this parameter is set to a larger value, capacity detection will not be performed frequently. | 3000 |
| yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill | Interval between the time when a resource preemption request is sent and the time when the container is stopped (resources are released), in millisecond. The value must be greater than or equal to 0. By default, if the ApplicationMaster does not gracefully stop the container within 15 seconds, the ResourceManager will forcibly terminate it. | 15000 |
| yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round | Maximum resource preemption ratio in a period. This value can be used to limit the speed at which containers are reclaimed from the cluster. After the expected total preemption value is calculated, the policy scales the preemption ratio back to this limit. | 0.1 |
| yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity | Resource preemption dead zone = Total number of resources in the cluster x Value of this configuration item + Original resources of a queue (for example, Queue A). When resources actually used by a task in Queue A exceeds the preemption dead zone, the resource beyond the preemption dead zone is preempted. The value range is 0 to 1. NOTE: A smaller value is recommended for effective preemption. | 0 |
| yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor | Preemption percentage. Containers preempt only this percentage of the resources. For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 times of yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill, even in the absence of natural termination. That is, 5 consecutive preemptions will be performed and each time half of the target resources will be preempted. The trend is geometric convergence. The interval of each preemption is yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill. The value range is 0 to 1. | 1 |
| ss.engine.scheduler.preemption-enable | Whether to enable Superior Scheduler preemption. | false |
| ss.engine.scheduler.preemption-interval-ms | Minimum interval for triggering preemption (also referred to as the preemption period), measured in milliseconds. | 3000 |
| ss.engine.scheduler.preemption-max-per-interval | Maximum number of YARN containers that can be preempted during a single preemption period. The default value is -1, indicating that no restriction is placed on the maximum number of YARN containers. | -1 |
| ss.engine.scheduler.preemption-warn-period-ms | Warning period provided to the ApplicationMaster before resource preemption occurs. If the ApplicationMaster fails to release the borrowed resources within the specified timeframe, the YARN container using those resources will be terminated by the ResourceManager. The unit is milliseconds. | 10000 |
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
