Preempting a Task

Updated on 2024-12-13 GMT+08:00

View PDF

Scenario

Preemption principle of the Capacity scheduler:
The capacity scheduler of ResourceManager implements job preemption to simplify job running in queues and improve resource utilization. The process is as follows:
1. Assume that there are two queues (Queue A and Queue B). The capacity of Queue A is 25%, and the capacity of Queue B is 75%.
2. In the initial state, Task 1 is distributed to Queue A for processing, requiring 75% cluster resources. Task 2 is distributed to Queue B for processing, requiring 50% cluster resources.
3. Task 1 uses 25% cluster resources provided by Queue A and 50% resources from Queue B. Queue B reserves 25% cluster resources.
4. If task preemption is enabled, the resources of Task 1 will be preempted. Queue B preempts 25% cluster resources from Queue A for Task 2.
5. Task 1 will be executed when Task 2 is complete and the cluster has sufficient resources.
Preemption Principle of the Superior Scheduler
When cluster resources are abundant, the Superior Scheduler permits queues to utilize resources beyond their minimum and shared allocations, improving overall resource utilization. Conversely, when resources are scarce, preemption ensures that each queue receives its minimum guaranteed resources as per the queue policy, maintaining fairness. The following are examples: The queue resource information is as follows (minimum indicates the minimum resource, and share indicates the weight):
1. User A submits Job 1 to Queue 1. The Application Master (AM) allocates 1 vCore and 2 GB memory for itself and initiates 8 tasks, each consuming 2 vCores and 4 GB memory. Given that resources are sufficient, Job 1 executes smoothly. Queue1, in total, utilizes 17 vCores and 34 GB memory. Subsequently, the cluster's remaining resources tally at 23 vCores and 6 GB memory.
2. User B submits Job 2 to Queue 2. The AM allocates 1 vCore and 2 GB memory to itself and initiates 5 tasks, each task consuming 2 vCores and 4 GB memory. Prior to any preemption, both the AM of Job 2 and one of its tasks operate seamlessly. Queue 2 consumes 3 vCores and 6 GB memory, leaving the cluster with no available memory and merely 20 vCores.
3. Since Queue2 has yet to utilize its minimum guaranteed resources and has pending tasks, the Scheduler initiates preemption to reclaim resources from a task of Job 1. This action allows Job 2 to deploy an additional task using the recaptured resources. After preemption, Queue 2's resource usage increases to 5 vCores and 10 GB memory.
NOTE:

If either CPU resources or memory resources of a queue are its minimum guaranteed resources, the queue resources will not be preempted by other queues.

Procedure

Navigation path for setting parameters:

Go to the All Configurations page of Yarn and enter a parameter name in the search box by referring to Modifying Cluster Service Configuration Parameters.

**Table 1** Parameter description
Parameter	Description	Default Value
yarn.resourcemanager.scheduler.monitor.enable	Whether to start scheduler monitoring according to yarn.resourcemanager.scheduler.monitor.policies. If this parameter is set to true, scheduler monitoring is enabled based on policies specified by yarn.resourcemanager.scheduler.monitor.policies and task resource preemption is enabled based on the scheduler information. If this parameter is set to false, scheduler monitoring is disabled.	false
yarn.resourcemanager.scheduler.monitor.policies	List of the SchedulingEditPolicy class to be used with the scheduler	org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
yarn.resourcemanager.monitor.capacity.preemption.observe_only	If this parameter is set to true, policies will be applied but task resource preemption will not be performed. If this parameter is set to false, policies will be applied and task resource preemption will be performed based on the policies.	false
yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval	Monitoring interval, in millisecond. If this parameter is set to a larger value, capacity detection will not be performed frequently.	3000
yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill	Interval between the time when a resource preemption request is sent and the time when the container is stopped (resources are released), in millisecond. The value must be greater than or equal to 0. By default, if ApplicationMaster does not stop the container within 15 seconds, ResourceManager will forcibly stop the container after 15 seconds.	15000
yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round	Maximum resource preemption ratio in a period. This value can be used to limit the speed at which containers are reclaimed from the cluster. After the expected total preemption value is calculated, the policy scales the preemption ratio back to this limit.	0.1
yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity	Resource preemption dead zone = Total number of resources in the cluster x Value of this configuration item + Original resources of a queue (for example, Queue A). When resources actually used by a task in Queue A exceeds the preemption dead zone, the resource beyond the preemption dead zone is preempted. The value range is 0 to 1. NOTE: A smaller value is recommended for effective preemption.	0
yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor	Preemption percentage. Containers preempt only this percentage of the resources. For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 times of yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill, even in the absence of natural termination. That is, 5 consecutive preemptions will be performed and each time half of the target resources will be preempted. The trend is geometric convergence. The interval of each preemption is yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill. The value range is 0 to 1.	1
ss.engine.scheduler.preemption-enable	Whether to enable Superior scheduler preemption.	false
ss.engine.scheduler.preemption-interval-ms	Minimum interval for triggering preemption (also called preemption period). The unit is millisecond.	3000
ss.engine.scheduler.preemption-max-per-interval	Maximum number of YARN containers that can be preempted within a preemption period. The default value is -1, indicating that there is no restriction on the maximum number of YARN containers.	-1
ss.engine.scheduler.preemption-warn-period-ms	Preemption duration to be notified to AM. Should the AM fail to relinquish the borrowed resources within the stipulated timeframe, the YARN container employing those resources will be forcibly terminated by the Resource Manager. The unit is millisecond.	10000

Parent topic: Yarn Performance Tuning

Previous topic: Yarn Performance Tuning

Next topic: Setting the Task Priority

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot