Unable to override cilium-agent config with CiliumNodeConfigs

[admun]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

kopsVersion: 1.30.3

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

v1.30.7

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

From a 1.30 k8s cluster bootstrapped by kOps with cilium networking add-on:

1. deploy CiliumNodeConfigs CRD, i.e. this yaml (see here for how this CR works)
2. create a CiliumNodeConfig object (see below)
3. restart all cilium pods
4. run kubectl exec <cilium pod> -n <ns> -- cilium config | grep -i policyaudit

The test object:

apiVersion: cilium.io/v2alpha1
kind: CiliumNodeConfig
metadata:
  name: policy-audit-mode-override
  namespace: kube-system
spec:
  defaults:
    policy-audit-mode: "true"
  nodeSelector:
    matchLabels: {}

5. What happened after the commands executed?

PolicyAuditMode is Disabled

6. What did you expect to happen?

PolicyAuditMode should be Enabled

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

After looking into kOps template that deploy cilium, we found that

1. There is a config initContainer that read cilium-config, override values and write configs to /tmp/cilium/config-map under a /tmp volume mount
2. In cilium-agent container, it’s configured to use config from /tmp/cilium/config-map as expected
3. However, that container also mount the cilium-config ConfigMap on /tmp/cilium/config-map, that overwritten the node level configs generated by the initContainer, effectively rollback the override values

The solution is to remove the unneeded configmap volume mount in the cilium-agent container at https://github.com/kubernetes/kops/blob/release-1.31/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template#L1198-L1200