Unable to deploy QHub on Minikube on Mac

[iameskild]

Has anyone deployed qhub from main (on minikube or on the cloud) after any of the recent PRs? I seem to be encountering the same error (see below) each time I try. This issue only seems to go away if I deploy from commit 55d5b2a or earlier…

[terraform]: │ Error: error initializing keycloak provider
[terraform]: │ 
[terraform]: │   with provider["registry.terraform.io/mrparkers/keycloak"],
[terraform]: │   on kubernetes.tf line 130, in provider "keycloak":
[terraform]: │  130: provider "keycloak" {
[terraform]: │ 
[terraform]: │ Post
[terraform]: │ "https://github-actions.qhub.dev/auth/realms/master/protocol/openid-connect/token":
[terraform]: │ context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I have read through the docs to verify I’m still deploying correctly but perhaps I’m missing some important keycloak env var or something.

This issue was discussed on slack and this is the current state according to Dan:

The problem is that https://github-actions.qhub.dev/ doesn't resolve to anything useful on the Mac version, so the Keycloak operator can't access Keycloak. The problem has always been that there is no accessible load balancer on Mac Docker.
It might be possible to set /etc/hosts to include github-actions.qhub.dev like we do for the AWS minikube. And meanwhile use kubectl port-forward to actually forward the traffic (from port 443 to something similar?). But you'd have to start that forwarding at the right point in the deployment! (When Kubernetes is ready, but before terraform runs the Keycloak operator...)

[leej3]

I am struggling to deploy locally using Minikube too. This is on NixOS (Dell XPS) but it looks like a similar issue. Let me know if it is not an I can open a separate one.

With my previous attempts removed (via minikube delete) I run the following attempt at running through the docs to setup a deployment:

minikube start --cpus 2 --memory 4096 --driver=docker
minikube ssh "sudo apt update; sudo apt install nfs-common -y"
wget https://raw.githubusercontent.com/Quansight/qhub/main/tests/scripts/minikube-loadbalancer-ip.py
python minikube-loadbalancer-ip.py 
minikube addons enable metallb
mkdir data
pip install -e .  # 27e2d94e52f44a2e, env created from conda yaml a 2-4 weeks ago
cd data/

python -m qhub init local --project=thisisatest  --domain github-actions.qhub.dev --auth-provider=password --terraform-state=local
python -m qhub deploy --config qhub-config.yaml --disable-prompt

This results in the following error

Tail of error message

[terraform]: 
[terraform]: Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
[terraform]: 
[terraform]: Outputs:
[terraform]: 
[terraform]: ingress_jupyter = {
[terraform]:   "hostname" = ""
[terraform]:   "ip" = "192.168.49.100"
[terraform]: }
INFO:qhub.provider.terraform:terraform apply took 82.651 [s]
INFO:qhub.deploy:Running Terraform Stage: ['module.kubernetes-keycloak-config']
INFO:qhub.provider.terraform:terraform apply directory=infrastructure targets=['module.kubernetes-keycloak-config']
INFO:qhub.provider.terraform: terraform at /tmp/terraform/1.0.5/terraform
[terraform]: module.kubernetes-initialization.kubernetes_namespace.main: Refreshing state... [id=dev]
[terraform]: random_password.keycloak-qhub-bot-password: Refreshing state... [id=none]
[terraform]: module.kubernetes-ingress.kubernetes_service.traefik_internal: Refreshing state... [id=dev/qhub-traefik-internal]
[terraform]: module.kubernetes-ingress.kubernetes_manifest.tls_stores: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_manifest.serverstransports: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_manifest.ingress_route_udp: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_manifest.tls_option: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_service.main: Refreshing state... [id=dev/qhub-traefik-ingress]
[terraform]: module.kubernetes-ingress.kubernetes_manifest.ingress_route_tcp: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_cluster_role.main: Refreshing state... [id=qhub-traefik-ingress]
[terraform]: module.kubernetes-ingress.kubernetes_manifest.ingress_route: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_service_account.main: Refreshing state... [id=dev/qhub-traefik-ingress]
[terraform]: module.kubernetes-ingress.kubernetes_manifest.traefik_service: Refreshing state...
[terraform]: module.kubernetes-ingress.kubernetes_cluster_role_binding.main: Refreshing state... [id=qhub-traefik-ingress]
[terraform]: module.kubernetes-ingress.kubernetes_deployment.main: Refreshing state... [id=dev/qhub-traefik-ingress]
[terraform]: module.kubernetes-ingress.kubernetes_manifest.middleware: Refreshing state...
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Refreshing state... [id=keycloak]
[terraform]: module.kubernetes-keycloak-helm.kubernetes_manifest.keycloak-http: Refreshing state...
[terraform]: ╷
[terraform]: │ Warning: Resource targeting is in effect
[terraform]: │ 
[terraform]: │ You are creating a plan with the -target option, which means that the
[terraform]: │ result of this plan may not represent all of the changes requested by the
[terraform]: │ current configuration.
[terraform]: │ 		
[terraform]: │ The -target option is not for routine use, and is provided only for
[terraform]: │ exceptional situations such as recovering from errors or mistakes, or when
[terraform]: │ Terraform specifically suggests to use it as part of an error message.
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: error initializing keycloak provider
[terraform]: │ 
[terraform]: │   with provider["registry.terraform.io/mrparkers/keycloak"],
[terraform]: │   on kubernetes.tf line 137, in provider "keycloak":
[terraform]: │  137: provider "keycloak" {
[terraform]: │ 
[terraform]: │ Post
[terraform]: │ "https://github-actions.qhub.dev/auth/realms/master/protocol/openid-connect/token":
[terraform]: │ dial tcp: lookup github-actions.qhub.dev on 192.168.0.1:53: no such host

[viniciusdc]

HI @leej3, my current guess is that the terraform provider for Keyclok might be initialized before the Keyclok server pod reaches a running stated

@iameskild in the error above, were you deploying on local? if so, I think this error might be related to MInikube... as I was able to deploy main on GCP recently. We might need to use a null_resource to redirect this POST (keylock token) request until the Keyclok pod reaches a running state.

[iameskild]

Hi @leej3 and @viniciusdc, I was indeed deploying on local and I think you're correct that this is an issue with deploying on Minikube.

[danlester]

@iameskild Please could you make sure our docs say this doesn't work, and perhaps give other ideas (using an AWS VM, or maybe Parallels with an Ubunutu VM).