[k8s-keystone-auth] TokenReview raises exception

[gustavofaganello]

Hello, folks.

I've encountered an issue while trying to request a TokenReview with the k8s-keystone-auth image.

All I've done until now was following these docs.

This is my deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-keystone-auth
  namespace: kube-system
  labels:
    app: k8s-keystone-auth
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-keystone-auth
  template:
    metadata:
      labels:
        app: k8s-keystone-auth
    spec:
      serviceAccountName: k8s-keystone
      containers:
        - name: k8s-keystone-auth
          image: registry.k8s.io/provider-os/k8s-keystone-auth:v1.31.2
          args:
            - ./bin/k8s-keystone-auth
            - --tls-cert-file
            - /etc/pki/tls.crt
            - --tls-private-key-file
            - /etc/pki/tls.key
            - --policy-configmap-name
            - k8s-auth-policy
            - --keystone-url
            - https://my-keystone-url:5000/v3
            - --keystone-ca-file
            - /etc/pki/keystoneca.crt
            - --v
            - "100"
          volumeMounts:
            - mountPath: /etc/pki
              name: certs
              readOnly: true
          ports:
            - containerPort: 8443
      volumes:
      - name: certs
        secret:
          secretName: keystone-auth-certs

And this is my keystone-auth-certs secret:

apiVersion: v1
kind: Secret
type: kubernetes.io/tls
data:
  keystoneca.crt: ...
  tls.crt: ...
  tls.key: ...

I realize this is not the best way to put the custom CA file inside the pods, but I just want to accomplish a PoC. I have also tried using a configmap to mount the custom CA certificate file inside the pods and got the same results.

After the pod is running, trying a curl from another pod inside the same cluster, as the docs suggest, gives me the following error:

I0115 18:25:10.765068      10 log.go:245] http2: panic serving 10.0.4.79:38610: runtime error: invalid memory address or nil pointer dereference
goroutine 54 [running]:
net/http.(*http2serverConn).runHandler.func1()
	net/http/h2_bundle.go:6177 +0x145
panic({0x1ba0320?, 0x31ca510?})
	runtime/panic.go:770 +0x132
k8s.io/cloud-provider-openstack/pkg/identity/keystone.(*Keystoner).GetTokenInfo(0xc00007c3f0, {0xc0001db800, 0xf7})
	k8s.io/cloud-provider-openstack/pkg/identity/keystone/authenticator.go:83 +0x5aa
k8s.io/cloud-provider-openstack/pkg/identity/keystone.(*Authenticator).AuthenticateToken(0xc000266c00, {0xc0001db800, 0xf7})
	k8s.io/cloud-provider-openstack/pkg/identity/keystone/authenticator.go:120 +0x42
k8s.io/cloud-provider-openstack/pkg/identity/keystone.(*Auth).authenticateToken(0xc0000bd180, {0x218aef0, 0xc00007c050}, 0x5?, {0xc0001db800, 0xf7}, 0xc0004c4de0)
	k8s.io/cloud-provider-openstack/pkg/identity/keystone/keystone.go:287 +0x5e
k8s.io/cloud-provider-openstack/pkg/identity/keystone.(*Auth).Handler(0xc0000bd180, {0x218aef0, 0xc00007c050}, 0xc000272480)
	k8s.io/cloud-provider-openstack/pkg/identity/keystone/keystone.go:269 +0x365
net/http.HandlerFunc.ServeHTTP(0x1b859e0?, {0x218aef0?, 0xc00007c050?}, 0xc000012d60?)
	net/http/server.go:2166 +0x29
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc0000ff620, {0x218aef0, 0xc00007c050}, 0xc000272480)
	github.com/go-chi/chi/[email protected]/mux.go:444 +0x207
net/http.HandlerFunc.ServeHTTP(0x219a048?, {0x218aef0?, 0xc00007c050?}, 0x31cb4c0?)
	net/http/server.go:2166 +0x29
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc0000ff620, {0x218aef0, 0xc00007c050}, 0xc000272360)
	github.com/go-chi/chi/[email protected]/mux.go:90 +0x2ee
net/http.serverHandler.ServeHTTP({0xc000100b40?}, {0x218aef0?, 0xc00007c050?}, 0xc0001b62a0?)
	net/http/server.go:3137 +0x8e
net/http.initALPNRequest.ServeHTTP({{0x219a010?, 0xc0000b00f0?}, 0xc0001c7888?, {0xc0001dc5a0?}}, {0x218aef0, 0xc00007c050}, 0xc000272360)
	net/http/server.go:3745 +0x231
net/http.(*http2serverConn).runHandler(0x3260ec0?, 0x0?, 0x0?, 0x0?)
	net/http/h2_bundle.go:6184 +0xbb
created by net/http.(*http2serverConn).scheduleHandler in goroutine 41
	net/http/h2_bundle.go:6119 +0x21d

For the sake of clarity, this is the curl I'm running and its return:

$ curl -k -XPOST https://k8s-keystone-auth-service.kube-system:8443/webhook -d '
{
  "apiVersion": "authentication.k8s.io/v1beta1",
  "kind": "TokenReview",
  "metadata": {
    "creationTimestamp": null
  },
  "spec": {
    "token": "'$token'"
  }
}'

curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

Adding to this, I've noticed that providing a wrong auth token results in an authentication failure, so this exception is only raised when the token is valid.

I'm wondering if this is a mistake on my end or a bug, and would appreciate any help.

Thanks in advance!

[gustavofaganello]

Originally, I thought this was an issue with the --keystone-ca-file option, but I'm still getting the same error when trying a keystone endpoint signed with a public CA. I've adjusted the issue title and description accordingly.

[rkferreira]

It was a missing projectID+projectName field on test token.

Struct tokenInfo requires those.

Previous token was domain scoped.

  return &tokenInfo{
    userName:    tokenUser.Name,
    userID:      tokenUser.ID,
    projectName: project.Name,
    projectID:   project.ID,
    roles:       userRoles,
    domainID:    tokenUser.Domain.ID,
    domainName:  tokenUser.Domain.Name,
  }