[Request] Upgrade channel - be able to have patch AND node images

[lgmorand]

Currently, the auto-upgrade channel makes you to choose between either an upgrade of k8s or an update of the node image.

It seems they are handled separatly which means you have to choose between having:

• the cluster up to date
• the node + aks layer up to date

A cluster upgrade (version) may create new nodes and thus provide the last version of node image (if I'm not wrong) but if there is no new patch for k8s version, then the node-image will never be updated, which is not good because node-image is the best way to ensure node security and "aks" layer being up to date

Request : having a channel mode to have patch + node-image and having both of them triggering an update when available.

ps: I didn't update the existing features as they are GA already #1486 and #1303

[ghost]

Hi lgmorand, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[qpetraroia]

Hello!

Thanks for this feature request, our team will look into it.

[siegenthalerroger]

I can concur that I would prefer to have the ability to have a patch auto-upgrade of k8s, but having the node-image updated as soon as possible. Currently this is not possible (despite some of the documentation's wording making it seem like it). This is definitely required functionality IMO.

[palma21]

So you would like to have your node image updated every week and your patch version + node image every month correct? (keeping in might they are cumulative)

[lgmorand]

Hi Jorge, yes that's it.

Basically, customer just want their cluster be updated as soon as possible, whether on the OS part or the kubernetes version. On the second one, customers seem only confident with automated patch version. BTW I never heard for automated minor version upgrade need yet

[vrdse]

+1
The requested described in the issue is exactly what I'm looking for and was expecting when I read about the new feature.

[asubmani]

SCENARIO: Currently running 1.20.3

1. AKS releases 1.20.7 in my region that has "Patch" updates enabled for auto-node image updates.
   1.1) the cluster automatically gets updated from 1.20.3 --> 1.20.7 (how long since it 1.20.7 shows in az aks get-versions --location ??)
   1.2 EXPECTATION : Get the latest OS Image available when the patch version was upgraded e.g. AKSUbuntu-1804gen2containerd-2022.01.19 (Not clear how long we need to wait for auto-update to kick in)
2. AKS has a new release and a new Node Image e.g AKSUbuntu-1804gen2containerd-2022.01.24. No updates to Kubernetes minor versions. (i.e. latest version for 1.20 series is still 1.20.7)
   2.1) EXPECTATION: The cluster will get the new image AKSUbuntu-1804gen2containerd-2022.01.24
   2.1.1) If above is true, how long should one wait before expecting "Auto-update" to kick in? How often does nodeImage feature check and initate action? HINT: az aks nodepool get-upgrades does show that AKSUbuntu-1804gen2containerd-2022.01.24 is available for my cluster.
   2.2.2) Based on this issue, it seems node image for same AKS version WON'T be updated if one chooses "patch" version auto-upgrade channel.

Would prefer to choose between:

• K8s versions only (any one auto upgrade channel)
• K8 version and node image
• Node Image only.

[asubmani]

UPDATE: I changed my "upgradeChannel" to "node-image" and I haven't recieved the new AKSUbuntu-1804gen2containerd-2022.01.24 images yet.

1. Does the node-image work on new images that are release AFTER you configure "upgradeChannel": "node-image"
2. If the node-images will pick up latest available, what is the polling interval? Is there an API call to forcibly trigeer an evaluation? (I know I can upgrade manually, but trying to understand when node-image will auto update and when it won't)

[asubmani]

After changing to "upgradeChannel": "patch"' TO "upgradeChannel": "node-image"`, it took me <48 hrs to get the 2022.01.24 images. previous image was the 01.19

As of now I will keep it as "node-image" updates only, and change to patch when ever there is a new patch version released.

[lgmorand]

My fellas, news are good. PG is already working on it so we should see something displayed in the roadmap very soon.

[ondrejhlavacek]

Adding this to my wishlist. I see no reason to delay any kind of Kubernetes patch version upgrades and node image upgrades. We handle Kubernetes minor version upgrades manually, but everything else should be upgraded automatically.

The Kubernetes patch versions are pretty quick and a cluster after a few weeks is running an unsupported Kubernetes versions. We had some incidents in the past when instead of solving the actual root cause we had to upgrade Kubernetes patch version first and then deal with the incident. Pretty annoying if you ask me.

[timown]

My fellas, news are good. PG is already working on it so we should see something displayed in the roadmap very soon.

hey @lgmorand, was it added to the roadmap? couldn't find anything

[kaarthis]

@lgmorand @timown we are working on a Security channel (For linux) that can be chosen in parallel to an Auto upgrade channel like patch or stable. This security channel will allow you to seamlessly get security updates (i.e Unattended upgrades) in your defined maintenance window and instead of maintaining a KURED to reboot , AKS will ensure you get a fully patched up image (eliminating need to keep KURED schedule) - and this is not a new VHD but on the very same VHD(image) the nodepool chose , so in effect there wont be a need to upgrade the Node's VHD (to a new node image) unless necessary. This security channel can be chosen separately and put in its own maintenance schedule orthogonal to the Auto upgrade channels (K8s).

[lgmorand]

@kaarthis awesome, that'd perfectly match the need of my customers. any card in the roadmap board ? I didn't find it.

this way I could watch updates during the monthly call :)

[kaarthis]

#2181

[lgmorand]

Hi @kaarthis

That's great because we can now update the OS image before a new image is available but regarding the first need, it does not seem the need is filled. How can we ensure to get:

• a node image upgrade automatically if a component needs to be updated (i.e. ContainerD)
• a kubernetes patch upgrade as soon as patch is available ?

From what I understand, we still have the same issue to choose between "k8s" & "AKS" components upgrade. Am I right ?

[ArunSuryaPrakash]

Do we have any update on this issue ?

[amirschw]

I saw in the release notes from today that it's now in preview:

Node OS auto-upgrade channel for automatically applying OS security patches promptly

[kaarthis]

With nodeOSUpgrade Channel release you can do this. https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-image