feat: add startup taint removal feature

[andyzhangx]

What type of PR is this?
/kind feature

What this PR does / why we need it:
feat: add startup taint removal feature

This PR is for solving the race condition between CSI driver initialization and disk volume scheduling, you could find details here: kubernetes/kubernetes#95911

user should add taint(disk.csi.azure.com/agent-not-ready=value:NoExecute) to node pool before CSI driver starts up

https://learn.microsoft.com/en-us/azure/aks/use-node-taints#create-a-node-pool-with-a-node-taint

az aks nodepool add \
    --resource-group $RESOURCE_GROUP_NAME \
    --cluster-name $CLUSTER_NAME \
    --name $NODE_POOL_NAME \
    --node-count 1 \
    --node-taints "disk.csi.azure.com/agent-not-ready=value:NoExecute" \
    --no-wait

• testing example

kubectl taint nodes aks-user-34333166-vmss000000 disk.csi.azure.com/agent-not-ready:NoExecute

• related logs

I0429 13:40:02.781428       1 azuredisk.go:754] CSINode Allocatable value is set for driver on node aks-user-34333166-vmss000000, count 4
I0429 13:40:02.781458       1 azuredisk.go:703] removing taint with key disk.csi.azure.com/agent-not-ready from local node aks-user-34333166-vmss000000
I0429 13:40:02.781480       1 azuredisk.go:706] checking taint key disk.csi.azure.com/agent-not-ready, value , effect NoExecute
I0429 13:40:02.781492       1 azuredisk.go:710] queued taint for removal with key disk.csi.azure.com/agent-not-ready, effect NoExecute
I0429 13:40:02.801729       1 azuredisk.go:741] removed taint with key disk.csi.azure.com/agent-not-ready from local node aks-user-34333166-vmss000000 successfully

Which issue(s) this PR fixes:

Fixes #

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Special notes for your reviewer:
There is a race condition between CSI driver initialization and kube-scheduler, we have added a taint removal feature (when CSI driver is started up) from aks 1.30, so cx could mitigate this issue, add taint disk.csi.azure.com/agent-not-ready:NoExecute in node pool creation and then after CSI driver is started, the taint would be removed by the CSI driver, that won’t be any race condition between CSI driver initialization and kube-scheduler any more.

Use node taints in an Azure Kubernetes Service (AKS) cluster - Azure Kubernetes Service | Microsoft Learn
az aks nodepool add
--resource-group $RESOURCE_GROUP_NAME
--cluster-name $CLUSTER_NAME
--name $NODE_POOL_NAME
--node-count 1
--node-taints "disk.csi.azure.com/agent-not-ready:NoExecute"
--no-wait

Release note:

feat: add startup taint removal feature

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andyzhangx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andyzhangx]

/retest

[andyzhangx]

/retest

[andyzhangx]

/retest

[andyzhangx]

/retest

[andyzhangx]

/retest

[JoeyC-Dev]

Since this is a temporary taint which is subject to be removed, shouldn't we use node initialization taints (--node-init-taints) instead of normal taint?

[andyzhangx]

If the initialization taint is still present on existing nodes, you can permanently remove it by performing a node image upgrade operation.

this does not apply for azure disk csi driver, we only want to remove the taint after azure disk csi driver is started to avoid race condition between kube-scheduler and azure disk csi driver.

[JoeyC-Dev]

Good to know. Many thanks for your hard work❤️