Cannot configure hugepages after bottlerocket 1.27

[le-duane]

Image I'm using:

Bottlerocket 1.28+ via AWS provided AMI e.g. bottlerocket-aws-k8s-1.30-x86_64-v1.32.0-cacc4ce9

What I expected to happen:

With the following configuration

[settings.boot.kernel-parameters]
"transparent_hugepage" = ["never"]
"hugepages"            = ["6550"]
"hugepagesz"           = ["2M"]

We expect 6550 huge pages are configured

What actually happened:

0 huge pages are configured

How to reproduce the problem:

Use version 1.28-1.32 of bottlerocket AMI

[le-duane]

Possibly related to this PR? https://github.com/bottlerocket-os/bottlerocket/pull/4290/files

[ytsssun]

Hi @le-duane , can you try using

[settings.kernel.sysctl]
"vm/nr_hugepages" = "6550"

[le-duane]

We were considering that change, but were unsure if that was the correct direction, given there are huge-page specific settings in the 'settings.boot.kernel-parameters' section.

[ytsssun]

kernel-parameter theoretically can work. However, a reboot is required to apply this kernel parameter (need to set reboot-to-reconcile = true). The reason is the userdata were applied after the kernel boot (which is by design) and we will only be able to apply the kernel parameter at the second boot. This will end up with a long boot time.

At minimal you would need

[settings.boot]
reboot-to-reconcile = true

[settings.boot.kernel-parameters]
"hugepages" = [ "3521" ]

Note that, both kernel-parameters and the sysctl path would work and the end result would be the same:
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages will be updated. The sysctl path /proc/sys/vm/nr_hugepages will also be updated.

I would recommend using sysctl over kernel-parameters for hugepages since sysctl does not require a reboot and you can overwrite hugepage settings even after boot.

We are also aware of the issue that the setting generator we introduced recently for sysctl hugepages setting for EFA support. If you only use "vm.nr_hugepages" in the settings, and the instance does not have EFA card attached. The settings generator would try to generate a setting for "vm/nr_hugepages" with value "0". And it could clobber the hugepages value set via "vm.nr_hugepages". As a workaround, setting hugepages via "vm/nr_hugepages" is guaranteed to work.

[settings.kernel.sysctl]
"vm/nr_hugepages" = "6550"

We are working on the fix for this so that we honor the "vm.nr_hugepages" entry as well.

[le-duane]

Thank you so much for the detailed feedback!