Solution for one-way downscaling of node groups

[criscola]

Which component are you using?:

/area cluster-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

We would like to have a solution for one-way scaling that scales down a node group to 0 replicas as its nodes become candidates for deletion (e.g. not used anymore). We could define this feature as one-way downscaling, or graceful downscaling. We use the cluster-api provider.

Describe the solution you'd like.:

We don't have a particular preference for a specific solution. For example, we thought about designing a new annotation that would trigger downscaling.
We also evaluated using the existing cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size annotation, see below.

Describe any alternative solutions you've considered.:

We tried to set the annotation cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size to 0, but we noticed that the behavior of the autoscaler depends on the provider implementation and can vary:

• When max-size = 0, don't scale up or down anymore like for the cluster-api cloudprovider, see relevant code.
• Disallow max-size = 0 entirely like for the GCE cloudprovider, see relevant code.
• Other behaviors can be observed in other providers probably, by reading from this FAQ.

Additional context.:

Consider the following thread in Kubernetes Slack for some more information: Message in Slack

cc @mweibel