[Feature] [Alpha] Application Priority and Preemption

[RainbowMango]

Summary
Currently, the karmada-scheduler operates based on a First-In-First-Out scheduling order. In some real-world scenarios, like AI training, some jobs are more critical than others and require preferential treatment in terms of scheduling and resource allocation.

We need the feature to enable users to specify the importance of an application. Applications with higher priority are scheduled ahead of other applications with lower priority. When member clusters do not have enough capacity for running a high priority application, the scheduler preempts and removes lower priority applications in order to make room for the high priority application.

Proposal:

• Application Priority and Preemption

Author:

• @LeonZh0u
• @seanlaii
• @wengyao04
• @whitewindmills
• @zclyne (Yifan Zhang)

References:

• [Scheduler] about priority scheduling #4938
• Proposal in Google Doc: https://docs.google.com/document/d/1MixmgLwnmiRrukyFP25JvE2hqiqB3f_JKT12tXiOsbc/edit#

Iteration Tasks -- Part-1: Scheduling based on application priority:

• API change: Introduce priority class to PropagationPolicy (@seanlaii, Add priorityClassName in PropagationPolicy & ClusterPropagationPolicy #5962)
• API change: Introduce priority to ResourceBinding (@seanlaii, Add Priority and PreemptionPolicy fields in ResourceBinding #5963)
• Feature gate: Introduce a feature gate for this feature. (@LeonZh0u)
• Controller: Build application priority when creating ResourceBinding (@LeonZh0u )
• webhook: Introduce validation to restrict PriorityClass modification
• scheduler: Introduce PriorityQueue to replace the current RateLimitedQueue. (@whitewindmills)
• E2E tests
• Documentation

Iteration Tasks -- Part-2: Preemption in case of member clusters lack of capacity:

• API change: Introduce preemptionPolicy to ResourceBinding
• TBD

Welcome Join us
If anyone is interested in this feature or would like to participate in its development, please leave a comment in this issue to claim the task you are interested in.

Note that this feature is still under discussion, and the listed iteration tasks might be subject to change.

[seanlaii]

Hi @RainbowMango , I can take the first two tasks first. Thank you so much for this!
Also, for the second task, we might need to add preemptionPolicy in the resourceBinding as well.

[RainbowMango]

Sure thanks. Assigned and updated.
By the way, do you know Yifan's GitHub handle? He is the co-author of this proposal, right?

[whitewindmills]

@RainbowMango
in the part-I, we don't need any feature gates.

[whitewindmills]

I have already implemented a priority queue demo, so I will continue to work on it. please assign it to me.
/assign
/assign @seanlaii

[RainbowMango]

@RainbowMango in the part-I, we don't need any feature gates.

Not sure about it yet. We probably need a feature gate to isolate the changes. Let's keep it on the list for now.

[seanlaii]

Sure thanks. Assigned and updated.

By the way, do you know Yifan's GitHub handle? He is the co-author of this proposal, right?

Yes, this is his github: https://github.com/zclyne. Thank you!

[zhzhuang-zju]

webhook: Introduce validation to restrict PriorityClass modification

I'm quite interested in this task. If no one else claims it, I can be assigned it.

[LeonZh0u]

I can take on these two tasks:

• Feature gate: Introduce a feature gate for this feature. (if needed)
• Controller: Build application priority when creating ResourceBinding

[RainbowMango]

OK Leon. Assigned them to you.
But I think we are still waiting for @kevin-wangzefeng's comments about the feature, so no rush~ I will talk to him.

[RainbowMango]

Hi all, @seanlaii @LeonZh0u @whitewindmills Do you guys want to bring this feature into the coming release, which is v1.13.0 scheduled at the end of this month? I know the time is tight, but there is still a chance to have a minimal version of this feature.

[seanlaii]

Hi all, @seanlaii @LeonZh0u @whitewindmills Do you guys want to bring this feature into the coming release, which is v1.13.0 scheduled at the end of this month? I know the time is tight, but there is still a chance to have a minimal version of this feature.

Yes, I really hope we can include this feature in the v1.13.0 release. I understand the timeline is tight, but I’m willing to push for a minimal version within the schedule. Thank you!

[RainbowMango]

OK. Let's start with the API parts which are the prerequisites of this feature. I left some comments on your PR, by the way.

[whitewindmills]

+1

[RainbowMango]

FYI. The API part has been merged, we can proceed with the implementations now. Thank you!

[seanlaii]

Hi @zhzhuang-zju , are you planning to work on the Webhook? I can take the E2E tests. Thank you!

[zhzhuang-zju]

Hi @zhzhuang-zju , are you planning to work on the Webhook? I can take the E2E tests. Thank you!

@seanlaii If you're interested in this task, feel free to take it. I can spend more time on the review.