(Experimental) bare-metal with IPv6

[justinsb]

IPv6 brings some new complexities, particularly around IPAM.

We create a test and then fix a few things:

• We need to assign the podCIDR for IPv6, so we add support to kops-controller. The source of this information is the host CRD.
• Because we are assigning the podCIDR from the Host CRD, we need Host records for the control plane nodes. However, there are bootstrapping problems around creating a CRD during enrollment of the control-plane nodes. So instead, we can now generate a Host Object in yaml, and can apply it separately. A high-security workflow would probably create the host records separately anyway, because they are how we validate nodes.
• Previously we were always setting the kubelet cloud-provider=external flag. But this assumes we are running a CCM. If we are not running a CCM (like metal), then we should not set the flag. If we do set the flag, kubelet sets the node.kops.k8s.io/uninitialized taint for CCM to clear, and nobody clears it.
• We need to make sure there is an IPv6 default route so that kubelet can discover its node ip correctly. We could put this into the Host CRD, but it does seem like most nodes will have a default route.

[justinsb]

I am trying to upload this and then I can rebasing as I/we fix each problem.

Current problem is from nodeup:

vm0 nodeup[703]: W1111 17:07:00.322041     703 main.go:133] got error running nodeup (will retry in 30s): error building loader: building *model.PrefixBuilder: kOps IPAM controller not supported on cloud "metal"

So we need to decide how the podCIDR is assigned!

[justinsb]

/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from justinsb. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

/retest

[justinsb]

/retest

[k8s-ci-robot]

@justinsb: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kops-e2e-cni-flannel	4ff49a7	link	true	/test pull-kops-e2e-cni-flannel
pull-kops-e2e-gce-cni-calico	4ff49a7	link	true	/test pull-kops-e2e-gce-cni-calico
pull-kops-e2e-gce-cni-kindnet	4ff49a7	link	true	/test pull-kops-e2e-gce-cni-kindnet
pull-kops-e2e-k8s-aws-amazonvpc-u2404	4ff49a7	link	true	/test pull-kops-e2e-k8s-aws-amazonvpc-u2404
pull-kops-e2e-k8s-gce-cilium	ebd96f9	link	true	/test pull-kops-e2e-k8s-gce-cilium
pull-kops-kubernetes-e2e-ubuntu-gce-build	ebd96f9	link	false	/test pull-kops-kubernetes-e2e-ubuntu-gce-build

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.