Name Resolution Failed when deployed with Ingress on an IPv6 EKS Cluster

[thisisayush]

What happened?

I deployed the latest version with Helm as mention in README, and modified certain values to enable ingress and metrics server. The deployment is done on a IPv6 EKS Cluster on AWS with Public and Private Subnet with a NLB in front of the cluster. When I tried to access the dashboard at deployed domain, myexampledomain.com I got the following in browser:

Text:

Error
name resolution failed.

request_id: 66c36627c4e2411556b770167ed0af26

Here are the logs from the kong pod:

2024/04/03 04:12:48 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
2024/04/03 04:12:49 [notice] 1#0: [lua] init.lua:776: init(): [request-debug] token for request debugging: 491180da-4a9a-45ea-93b3-dc9cd03c16bd
2024/04/03 04:12:49 [notice] 1#0: using the "epoll" event method
2024/04/03 04:12:49 [notice] 1#0: openresty/1.25.3.1
2024/04/03 04:12:49 [notice] 1#0: OS: Linux 5.10.210-201.855.amzn2.x86_64
2024/04/03 04:12:49 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/04/03 04:12:49 [notice] 1#0: start worker processes
2024/04/03 04:12:49 [notice] 1#0: start worker process 1322
2024/04/03 04:12:49 [notice] 1#0: start worker process 1323
2024/04/03 04:12:49 [notice] 1322#0: *1 [lua] globalpatches.lua:75: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2024/04/03 04:12:49 [notice] 1323#0: *2 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: init_worker_by_lua*
2024/04/03 04:12:49 [notice] 1323#0: *2 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: init_worker_by_lua*
2024/04/03 04:12:49 [notice] 1323#0: *2 [kong] init.lua:589 declarative config loaded from /kong_dbless/kong.yml, context: init_worker_by_lua*
2024/04/03 04:13:26 [error] 1322#0: *1297 [lua] init.lua:371: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)kubernetes-dashboard-web:(na) - cache-miss","kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local:1 - cache-miss/querying/dns client error: 101 empty record received","kubernetes-dashboard-web.svc.cluster.local:1 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.cluster.local:1 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.ec2.internal:1 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web:1 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local:5 - cache-miss/querying/dns client error: 101 empty record received","kubernetes-dashboard-web.svc.cluster.local:5 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.cluster.local:5 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.ec2.internal:5 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web:5 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local:33 - cache-miss/querying/28:kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local removed/recursion detected","kubernetes-dashboard-web.svc.cluster.local:33 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.cluster.local:33 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web.ec2.internal:33 - cache-miss/querying/dns server error: 3 name error","kubernetes-dashboard-web:33 - cache-miss/querying/dns server error: 3 name error"], client: 2600:1f10:41ad:dd03:9223::6, server: kong, request: "GET / HTTP/1.1", host: "myexampledomain.com", request_id: "6162f53c86955a448bf48e58a3a7b23d"
2600:1f10:41ad:dd03:9223::6 - - [03/Apr/2024:04:13:26 +0000] "GET / HTTP/1.1" 503 231 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0" kong_request_id: "6162f53c86955a448bf48e58a3a7b23d"
2024/04/03 04:13:27 [error] 1322#0: *1297 [lua] init.lua:371: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)kubernetes-dashboard-web:(na) - cache-miss","kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local:1 - cache-hit/dns client error: 101 empty record received","kubernetes-dashboard-web.svc.cluster.local:1 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.cluster.local:1 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.ec2.internal:1 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web:1 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local:5 - cache-hit/dns client error: 101 empty record received","kubernetes-dashboard-web.svc.cluster.local:5 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.cluster.local:5 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.ec2.internal:5 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web:5 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.kubernetes-dashboard.svc.cluster.local:33 - cache-hit/recursion detected","kubernetes-dashboard-web.svc.cluster.local:33 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.cluster.local:33 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web.ec2.internal:33 - cache-hit/dns server error: 3 name error","kubernetes-dashboard-web:33 - cache-hit/dns server error: 3 name error"], client: 2600:1f10:41ad:dd03:9223::6, server: kong, request: "GET /favicon.ico HTTP/1.1", host: "myexampledomain.com", referrer: "https://myexampledomain.com/", request_id: "027d7b619be4416246f9101f5bef5a73"
2600:1f10:41ad:dd03:9223::6 - - [03/Apr/2024:04:13:27 +0000] "GET /favicon.ico HTTP/1.1" 503 91 "https://myexampledomain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0" kong_request_id: "027d7b619be4416246f9101f5bef5a73"

What did you expect to happen?

Expected Kubernetes Dashboard Login Screen

How can we reproduce it (as minimally and precisely as possible)?

Script Used for deployment

#!/bin/bash

helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --values override-values.yaml --create-namespace --namespace kubernetes-dashboard

kubectl create secret tls my-ssl --key certs/key.pem --cert certs/cert.crt -n kubernetes-dashboard

kubectl apply -f user.yaml

echo "Token: "

kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath={".data.token"} | base64 -d

File: override-values.yaml

app:
  ingress:
    enabled: true
    hosts:
      - myexampledomain.com
    ingressClassName: nginx
    useDefaultIngressClass: false
    useDefaultAnnotations: true
    pathType: ImplementationSpecific
    path: /
    issuer:
      name: selfsigned
      scope: disabled
    tls:
      secretName: "my-ssl"
metrics-server:
  enabled: true
  args:
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls

File: user.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/service-account.name: "admin-user"   
type: kubernetes.io/service-account-token 

Anything else we need to know?

No response

What browsers are you seeing the problem on?

No response

Kubernetes Dashboard version

7.1.3

Kubernetes version

1.29

Dev environment

No response

[floreks]

You could try updating kong.env.dns_order value to AAAA,A,CNAME,LAST,SRV.

[thisisayush]

Thanks, that solved the issue!