Merge branch 'main' into breaking/port-default-lb

README.md

@@ -104,6 +104,7 @@ This repository is based on [@zeerorg](https://github.com/zeerorg/)'s [zeerorg/k
 - [vscode-k3d](https://github.com/inercia/vscode-k3d): vscode plugin for k3d
 - [AbsaOSS/k3d-action](https://github.com/AbsaOSS/k3d-action): fully customizable GitHub Action to run lightweight Kubernetes clusters.
 - [AutoK3s](https://github.com/cnrancher/autok3s): a lightweight tool to help run K3s everywhere including k3d provider.
+- [nolar/setup-k3d-k3s](https://github.com/nolar/setup-k3d-k3s): setup K3d/K3s for GitHub Actions.
 
 ## Contributing
 

cmd/image/imageImport.go

@@ -41,9 +41,21 @@ func NewCmdImageImport() *cobra.Command {
 
 	// create new command
 	cmd := &cobra.Command{
-		Use:     "import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]]",
-		Short:   "Import image(s) from docker into k3d cluster(s).",
-		Long:    `Import image(s) from docker into k3d cluster(s).`,
+		Use:   "import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]]",
+		Short: "Import image(s) from docker into k3d cluster(s).",
+		Long: `Import image(s) from docker into k3d cluster(s).
+
+If an IMAGE starts with the prefix 'docker.io/', then this prefix is stripped internally.
+That is, 'docker.io/rancher/k3d-tools:latest' is treated as 'rancher/k3d-tools:latest'.
+
+If an IMAGE starts with the prefix 'library/' (or 'docker.io/library/'), then this prefix is stripped internally.
+That is, 'library/busybox:latest' (or 'docker.io/library/busybox:latest') are treated as 'busybox:latest'.
+
+If an IMAGE does not have a version tag, then ':latest' is assumed.
+That is, 'rancher/k3d-tools' is treated as 'rancher/k3d-tools:latest'.
+
+A file ARCHIVE always takes precedence.
+So if a file './rancher/k3d-tools' exists, k3d will try to import it instead of the IMAGE of the same name.`,
 		Aliases: []string{"images"},
 		Args:    cobra.MinimumNArgs(1),
 		Run: func(cmd *cobra.Command, args []string) {

docs/index.md

@@ -110,3 +110,4 @@ kubectl get nodes
 - [k3x](https://github.com/inercia/k3x): a graphics interface (for Linux) to k3d.
 - [AbsaOSS/k3d-action](https://github.com/AbsaOSS/k3d-action): fully customizable GitHub Action to run lightweight Kubernetes clusters.
 - [AutoK3s](https://github.com/cnrancher/autok3s): a lightweight tool to help run K3s everywhere including k3d provider.
+- [nolar/setup-k3d-k3s](https://github.com/nolar/setup-k3d-k3s): setup K3d/K3s for GitHub Actions.

docs/requirements.txt

@@ -3,4 +3,5 @@ mkdocs-material
 pymdown-extensions
 mkdocs-git-revision-date-localized-plugin
 mkdocs-awesome-pages-plugin
-mdx_truly_sane_lists
+mdx_truly_sane_lists
+mkdocs-include-markdown-plugin # https://github.com/mondeja/mkdocs-include-markdown-plugin

docs/usage/commands/k3d_image_import.md

@@ -6,6 +6,15 @@ Import image(s) from docker into k3d cluster(s).
 
 Import image(s) from docker into k3d cluster(s).
 
+If an IMAGE starts with the prefix 'docker.io/', then this prefix is stripped internally.
+That is, 'docker.io/rancher/k3d-tools:latest' is treated as 'rancher/k3d-tools:latest'.
+
+If an IMAGE does not have a version tag, then ':latest' is assumed.
+That is, 'rancher/k3d-tools' is treated as 'rancher/k3d-tools:latest'.
+
+A file ARCHIVE always takes precedence.
+So if a file './rancher/k3d-tools' exists, k3d will try to import it instead of the IMAGE of the same name.
+
 ```
 k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]
 ```

docs/usage/guides/cuda.md

@@ -1,63 +1,35 @@
 # Running CUDA workloads
 
-If you want to run CUDA workloads on the K3S container you need to customize the container.  
+If you want to run CUDA workloads on the K3s container you need to customize the container.  
 CUDA workloads require the NVIDIA Container Runtime, so containerd needs to be configured to use this runtime.  
-The K3S container itself also needs to run with this runtime.  
+The K3s container itself also needs to run with this runtime.  
 If you are using Docker you can install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).
 
-## Building a customized K3S image
+## Building a customized K3s image
 
-To get the NVIDIA container runtime in the K3S image you need to build your own K3S image.  
-The native K3S image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet.  
+To get the NVIDIA container runtime in the K3s image you need to build your own K3s image.  
+The native K3s image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet.  
 To get around this we need to build the image with a supported base image.
 
-### Adapt the Dockerfile
+### Dockerfiles
+
+[Dockerfile.base](cuda/Dockerfile.base):
 
 ```Dockerfile
-FROM ubuntu:18.04 as base
-RUN apt-get update -y && apt-get install -y ca-certificates
-ADD k3s/build/out/data.tar.gz /image
-RUN mkdir -p /image/etc/ssl/certs /image/run /image/var/run /image/tmp /image/lib/modules /image/lib/firmware && \
-    cp /etc/ssl/certs/ca-certificates.crt /image/etc/ssl/certs/ca-certificates.crt
-RUN cd image/bin && \
-    rm -f k3s && \
-    ln -s k3s-server k3s
-
-FROM ubuntu:18.04
-RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
-RUN apt-get update -y && apt-get -y install gnupg2 curl
-
-# Install the NVIDIA CUDA drivers and Container Runtime
-RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
-RUN sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
-RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | apt-key add -
-RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/nvidia-container-runtime.list | tee /etc/apt/sources.list.d/nvidia-container-runtime.list
-RUN apt-get update -y
-RUN apt-get -y install cuda-drivers nvidia-container-runtime
-
-COPY --from=base /image /
-RUN mkdir -p /etc && \
-    echo 'hosts: files dns' > /etc/nsswitch.conf
-RUN chmod 1777 /tmp
-# Provide custom containerd configuration to configure the nvidia-container-runtime
-RUN mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/
-COPY config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
-# Deploy the nvidia driver plugin on startup
-RUN mkdir -p /var/lib/rancher/k3s/server/manifests
-COPY gpu.yaml /var/lib/rancher/k3s/server/manifests/gpu.yaml
-VOLUME /var/lib/kubelet
-VOLUME /var/lib/rancher/k3s
-VOLUME /var/lib/cni
-VOLUME /var/log
-ENV PATH="$PATH:/bin/aux"
-ENTRYPOINT ["/bin/k3s"]
-CMD ["agent"]
+{% include "cuda/Dockerfile.base" %}
+
+```  
+
+[Dockerfile.k3d-gpu](cuda/Dockerfile.k3d-gpu):  
+
+```Dockerfile
+{% include "cuda/Dockerfile.k3d-gpu" %}
 ```
 
-This [Dockerfile](cuda/Dockerfile) is based on the [K3s Dockerfile](https://github.com/rancher/k3s/blob/master/package/Dockerfile).
+These Dockerfiles are based on the [K3s Dockerfile](https://github.com/rancher/k3s/blob/master/package/Dockerfile)
 The following changes are applied:
 
-1. Change the base images to Ubuntu 18.04 so the NVIDIA Container Runtime can be installed
+1. Change the base images to nvidia/cuda:11.2.0-base-ubuntu18.04 so the NVIDIA Container Runtime can be installed. The version of `cuda:xx.x.x` must match the one you're planning to use.
 2. Add a custom containerd `config.toml` template to add the NVIDIA Container Runtime. This replaces the default `runc` runtime
 3. Add a manifest for the NVIDIA driver plugin for Kubernetes
 
@@ -66,61 +38,7 @@ The following changes are applied:
 We need to configure containerd to use the NVIDIA Container Runtime. We need to customize the config.toml that is used at startup. K3s provides a way to do this using a [config.toml.tmpl](cuda/config.toml.tmpl) file. More information can be found on the [K3s site](https://rancher.com/docs/k3s/latest/en/advanced/#configuring-containerd).
 
 ```go
-[plugins.opt]
-  path = "{{ .NodeConfig.Containerd.Opt }}"
-
-[plugins.cri]
-  stream_server_address = "127.0.0.1"
-  stream_server_port = "10010"
-
-{{- if .IsRunningInUserNS }}
-  disable_cgroup = true
-  disable_apparmor = true
-  restrict_oom_score_adj = true
-{{end}}
-
-{{- if .NodeConfig.AgentConfig.PauseImage }}
-  sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
-{{end}}
-
-{{- if not .NodeConfig.NoFlannel }}
-[plugins.cri.cni]
-  bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
-  conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
-{{end}}
-
-[plugins.cri.containerd.runtimes.runc]
-  # ---- changed from 'io.containerd.runc.v2' for GPU support
-  runtime_type = "io.containerd.runtime.v1.linux"
-
-# ---- added for GPU support
-[plugins.linux]
-  runtime = "nvidia-container-runtime"
-
-{{ if .PrivateRegistryConfig }}
-{{ if .PrivateRegistryConfig.Mirrors }}
-[plugins.cri.registry.mirrors]{{end}}
-{{range $k, $v := .PrivateRegistryConfig.Mirrors }}
-[plugins.cri.registry.mirrors."{{$k}}"]
-  endpoint = [{{range $i, $j := $v.Endpoints}}{{if $i}}, {{end}}{{printf "%q" .}}{{end}}]
-{{end}}
-
-{{range $k, $v := .PrivateRegistryConfig.Configs }}
-{{ if $v.Auth }}
-[plugins.cri.registry.configs."{{$k}}".auth]
-  {{ if $v.Auth.Username }}username = "{{ $v.Auth.Username }}"{{end}}
-  {{ if $v.Auth.Password }}password = "{{ $v.Auth.Password }}"{{end}}
-  {{ if $v.Auth.Auth }}auth = "{{ $v.Auth.Auth }}"{{end}}
-  {{ if $v.Auth.IdentityToken }}identitytoken = "{{ $v.Auth.IdentityToken }}"{{end}}
-{{end}}
-{{ if $v.TLS }}
-[plugins.cri.registry.configs."{{$k}}".tls]
-  {{ if $v.TLS.CAFile }}ca_file = "{{ $v.TLS.CAFile }}"{{end}}
-  {{ if $v.TLS.CertFile }}cert_file = "{{ $v.TLS.CertFile }}"{{end}}
-  {{ if $v.TLS.KeyFile }}key_file = "{{ $v.TLS.KeyFile }}"{{end}}
-{{end}}
-{{end}}
-{{end}}
+{% include "cuda/config.toml.tmpl" %}
 ```
 
 ### The NVIDIA device plugin
@@ -132,95 +50,42 @@ To enable NVIDIA GPU support on Kubernetes you also need to install the [NVIDIA
 * Run GPU enabled containers in your Kubernetes cluster.
 
 ```yaml
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: nvidia-device-plugin-daemonset
-  namespace: kube-system
-spec:
-  selector:
-    matchLabels:
-      name: nvidia-device-plugin-ds
-  template:
-    metadata:
-      # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler
-      # reserves resources for critical add-on pods so that they can be rescheduled after
-      # a failure.  This annotation works in tandem with the toleration below.
-      annotations:
-        scheduler.alpha.kubernetes.io/critical-pod: ""
-      labels:
-        name: nvidia-device-plugin-ds
-    spec:
-      tolerations:
-      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.
-      # This, along with the annotation above marks this pod as a critical add-on.
-      - key: CriticalAddonsOnly
-        operator: Exists
-      containers:
-      - env:
-        - name: DP_DISABLE_HEALTHCHECKS
-          value: xids
-        image: nvidia/k8s-device-plugin:1.11
-        name: nvidia-device-plugin-ctr
-        securityContext:
-          allowPrivilegeEscalation: true
-          capabilities:
-            drop: ["ALL"]
-        volumeMounts:
-          - name: device-plugin
-            mountPath: /var/lib/kubelet/device-plugins
-      volumes:
-        - name: device-plugin
-          hostPath:
-            path: /var/lib/kubelet/device-plugins
+{% include "cuda/gpu.yaml" %}
 ```
 
-### Build the K3S image
+### Build the K3s image
 
-To build the custom image we need to build K3S because we need the generated output.
+To build the custom image we need to build K3s because we need the generated output.
 
 Put the following files in a directory:
 
-* [Dockerfile](cuda/Dockerfile)
+* [Dockerfile.base](cuda/Dockerfile.base)
+* [Dockerfile.k3d-gpu](cuda/Dockerfile.k3d-gpu)
 * [config.toml.tmpl](cuda/config.toml.tmpl)
 * [gpu.yaml](cuda/gpu.yaml)
 * [build.sh](cuda/build.sh)
 * [cuda-vector-add.yaml](cuda/cuda-vector-add.yaml)
 
-The `build.sh` files takes the K3S git tag as argument, it defaults to `v1.18.10+k3s1`. The script performs the following steps:
+The `build.sh` script is configured using exports & defaults to `v1.21.2+k3s1`. Please set your CI_REGISTRY_IMAGE! The script performs the following steps:
 
-* pulls K3S
-* builds K3S
-* build the custom K3S Docker image
+* pulls K3s
+* builds K3s
+* build the custom K3D Docker image
 
 The resulting image is tagged as k3s-gpu:<version tag>. The version tag is the git tag but the '+' sign is replaced with a '-'.
 
 [build.sh](cuda/build.sh):
 
 ```bash
-#!/bin/bash
-set -e
-cd $(dirname $0)
- 
-K3S_TAG="${1:-v1.18.10+k3s1}"
-IMAGE_TAG="${K3S_TAG/+/-}"
-
-if [ -d k3s ]; then
-    rm -rf k3s
-fi
-git clone --depth 1 https://github.com/rancher/k3s.git -b $K3S_TAG
-cd k3s
-make
-cd ..
-docker build -t k3s-gpu:$IMAGE_TAG .
+{% include "cuda/build.sh" %}
 ```
 
 ## Run and test the custom image with Docker
 
 You can run a container based on the new image with Docker:
 
 ```bash
-docker run --name k3s-gpu -d --privileged --gpus all k3s-gpu:v1.18.10-k3s1
+docker run --name k3s-gpu -d --privileged --gpus all $CI_REGISTRY_IMAGE:$IMAGE_TAG
 ```
 
 Deploy a [test pod](cuda/cuda-vector-add.yaml):
@@ -236,7 +101,7 @@ docker exec k3s-gpu kubectl logs cuda-vector-add
 Tou can use the image with k3d:
 
 ```bash
-k3d cluster create --no-lb --image k3s-gpu:v1.18.10-k3s1 --gpus all
+k3d cluster create local --image=$CI_REGISTRY_IMAGE:$IMAGE_TAG --gpus=1
 ```
 
 Deploy a [test pod](cuda/cuda-vector-add.yaml):
@@ -256,4 +121,11 @@ Most of the information in this article was obtained from various sources:
 
 * [Add NVIDIA GPU support to k3s with containerd](https://dev.to/mweibel/add-nvidia-gpu-support-to-k3s-with-containerd-4j17)
 * [microk8s](https://github.com/ubuntu/microk8s)
-* [K3S](https://github.com/rancher/k3s)
+* [K3s](https://github.com/rancher/k3s)
+* [k3s-gpu](https://gitlab.com/vainkop1/k3s-gpu)
+
+## Authors
+
+* [@markrexwinkel](https://github.com/markrexwinkel)
+* [@vainkop](https://github.com/vainkop)
+* [@iwilltry42](https://github.com/iwilltry42)