[bug] qhub upgrade to 0.3.12 fails

[iameskild]

Describe the bug

A clear and concise description of what the problem is.

I am trying to update the qhub version from 0.3.11 to 0.3.12 but keep running into the same error:

[terraform]: Error: namespaces "dev" is forbidden: User "system:anonymous" cannot get resource "namespaces" in API group "" in the namespace "dev"

I have double checked:

• AWS/Prefect environment variables are properly set
• ~/.aws profile is properly configured
• AWS ARN has been added to the configmap/aws-auth

I followed the directions outlined in the qhub docs and made sure to update all the relevant image tags.

All of my recent deployments have been performed locally (not using CI). I performed the same deployment once using qhub version 0.3.12 and it failed (see error message above) and then I ran it using qhub version 0.3.11 and it successfully deployed.

Your environment

Describe the environment in which you are experiencing the bug.

Include your conda version (use conda --version), k8s and any other relevant details.

conda: 4.10.1
k8s: 1.19
kubectl (client): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}

[iameskild]

I also recently tried installing from the latest qhub commit fab0586 and I got the same error with some additional information:

[terraform]: │ Error: namespaces "dev" is forbidden: User "system:anonymous" cannot get resource "namespaces" in API group "" in the namespace "dev"
[terraform]: │ 
[terraform]: │   with module.kubernetes-initialization.kubernetes_namespace.main,
[terraform]: │   on modules/kubernetes/initialization/main.tf line 1, in resource "kubernetes_namespace" "main":
[terraform]: │    1: resource "kubernetes_namespace" "main" {

[iameskild]

I tried installing from qhub commit 2770c97 (PR #685) and I got the same error again. However installing from the commit just prior, 426d327, I was able to successfully deploy.

[iameskild]

I undid all of the changes in PR #685 in a local branch of qhub (branching from main commit fab0586). I then installed this modified v0.3.12 qhub and was able to redeploy successfully.

[aktech]

It seems the issue is with expiring token, while connecting to EKS. They are quite short lived.
See this:

Some cloud providers have short-lived authentication tokens that can expire relatively quickly. To ensure the Kubernetes provider is receiving valid credentials, an exec-based plugin can be used to fetch a new token before initializing the provider. For example, on EKS, the command eks get-token can be used

I have a fix in here: https://github.com/Quansight/qhub/tree/eks-connection-issue will create a PR soon.