'Failed to get /openapi/v2 and /swagger.json: Created Component, but require templated one'

[kferrone]

Here is the error I get when I installed v6.0.0 on aws eks

npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart: [email protected] info lifecycle [email protected]~start: [email protected]
> [email protected] start /app
> ./bin/daemon.js

{"level":30,"time":1607551661639,"pid":17,"hostname":"external-secrets-559966b9bf-jkck9","msg":"loading kube specs"}
Error: Failed to get /openapi/v2 and /swagger.json: Created Component, but require templated one. This is a bug. Please report: https://github.com/silasbw/fluent-openapi/issues    
    at /app/node_modules/kubernetes-client/lib/swagger-client.js:58:15
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async main (/app/bin/daemon.js:33:3)
npm info lifecycle [email protected]~start: Failed to exec start script
npm ERR! code ELIFECYCLEnpm ERR! errno 1npm ERR! [email protected] start: `./bin/daemon.js`npm ERR! Exit status 1npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm timing npm Completed in 1392msnpm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2020-12-09T22_07_41_808Z-debug.log

Here is the values.yaml I used to install:

    env:
      AWS_REGION: eu-west-2
    fullnameOverride: external-secrets
    nodeSelector:
      eks.amazonaws.com/nodegroup: Main
    securityContext:
      fsGroup: 65534
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: 'arn:aws:iam::123456789:role/system/external-secrets'
    serviceMonitor:
      enabled: true

[Flydiverny]

See #543 and #563

[kferrone]

Oh weird, for me it was prometheus-adapter from here: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter
It had to be uninstalled

[Flydiverny]

@silasbw any chance fluent-openapi could not hard fail on this? Seems to appear more and more :(

[jbilliau-rcd]

Hi, we are having this exact same issue; sure enough, backing out the prometheus-adapter fixed it. This issue, #563 , mentions that 0.8.2 changed the openapi custom.metrics.k8s.io endpoint and that's what broke external-secrets 6.0.0. Is there a plan to adjust external-secrets to accomodate whatever change occurred in prometheus-adapter 0.8.2?

[Flydiverny]

@jbilliau-rcd KES is using some dependencies which in turn is using https://github.com/silasbw/swagger-fluent which is the library that isn't able to handle this.
It would require quite a bit of work to swap the dependency that is used by KES for communicating with the kubernetes APIs.
There's ongoing work for a rewrite of KES in golang in https://github.com/external-secrets/external-secrets but it will be awhile before this reaches feature parity.
So, sadly, I haven't planned for any major refactoring like this in KES. Ideally the issue is fixed in our dependency and we can update that, but I deem that unlikely as well.

[jbilliau-rcd]

@Flydiverny thanks for the response; is the nutshell of this then that, in order to use external-secrets and prometheus adapter together, we need to stay on 0.7.0 of the adapter and not upgrade for the forseeable future? No biggie if that's the case, this controller has a been a big hit with our devs so if we have to make accommodations to keep using it, so be it.

[dudicoco]

@Flydiverny can you please elaborate on what exactly the swagger-fluent library isn't able to handle?

[Flydiverny]

@dudicoco not really. But the error originates from the library :)
There's something in the resulting swagger.json that it cannot parse. Which in the above cases seems to occur when having the prometheus-adapter installed. Suggesting that the apis added to the swagger.json or openapi spec are not compliant to either all required fields or at least fields expected by the swagger-fluent library.
One would need to debug the library with a spec that triggers the issue and figure out what the cause is.

[linxcat]

I got this error as well in an effort to upgrade to EKS 1.18 and take several charts along with it (with the stable repo now gone). Luckily this was an easy find. adapter v0.8.3 was released moments ago, so I'm going to give it a try to see if that fixed anything, but if not I hope the problem doesn't persist unsolved for too long.

[JihadMotii-REISys]

I'm running into similar issue and i'm not sure how to resolve it, please find below the details of the issue

  |   | 2021-01-29 18:40:05 | npm ERR!     /home/node/.npm/_logs/2021-01-29T17_40_05_244Z-debug.log
  |   | 2021-01-29 18:40:05 | npm ERR! A complete log of this run can be found in:
  |   | 2021-01-29 18:40:05 |  
  |   | 2021-01-29 18:40:05 | npm timing npm Completed in 262960ms
  |   | 2021-01-29 18:40:05 | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
  |   | 2021-01-29 18:40:05 | npm ERR! Failed at the [email protected] start script.
  |   | 2021-01-29 18:40:05 | npm ERR!
  |   | 2021-01-29 18:40:05 | npm ERR! Exit status 1
  |   | 2021-01-29 18:40:05 | npm ERR! [email protected] start: `./bin/daemon.js`
  |   | 2021-01-29 18:40:05 | npm ERR! errno 1
  |   | 2021-01-29 18:40:05 | npm ERR! code ELIFECYCLE
  |   | 2021-01-29 18:40:05 | npm info lifecycle [email protected]~start: Failed to exec start script
  |   | 2021-01-29 18:40:05 | at async main (/app/bin/daemon.js:35:3)
  |   | 2021-01-29 18:40:05 | at processTicksAndRejections (internal/process/task_queues.js:97:5)
  |   | 2021-01-29 18:40:05 | at /app/node_modules/kubernetes-client/lib/swagger-client.js:58:15
  |   | 2021-01-29 18:40:05 | Error: Failed to get /openapi/v2 and /swagger.json: connect ETIMEDOUT 172.20.0.1:443
  |   | 2021-01-29 18:35:43 | {"level":30,"time":1611941743291,"pid":18,"hostname":"kubernetes-external-secrets-8968c6fd5-lj85z","msg":"loading kube specs"}
  |   | 2021-01-29 18:35:42 |  
  |   | 2021-01-29 18:35:42 | > ./bin/daemon.js
  |   | 2021-01-29 18:35:42 | > [email protected] start /app

as shown above, the main issue is Error: Failed to get /openapi/v2 and /swagger.json: connect ETIMEDOUT 172.20.0.1:443

This is the command line that I run to install this plugin

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
helm install kubernetes-external-secrets external-secrets/kubernetes-external-secrets --skip-crds \
        --set env.DISABLE_POLLING=true --set env.AWS_REGION="us-east-1" \
        --set securityContext.fsGroup=65534 --set serviceAccount.name="devsecops-service-account" --set serviceAccount.create=false \
        --namespace build

does anyone have an idea on how to resolve it?

Thanks in advance!

[JihadMotii-REISys]

I've found the root cause of my problem above. (Not sure if it applicable to you as well)

in my EKS, i'm using security groups for pods, each pod will be associated to an ENI and this ENI will be attached to security group. When I removed this feature and redeploy this plugin, everything went well. This means, could be a missing security group rules to allow specific inbound/outbound to EKS control plane (where the API Server returns the requested json call made by this plugin as mentioned in the title above). For the time being, I will remove the security group from the namespace where this plugin is deployed until someone can provide a solution or knows how to fix it using the security group feature). Hope this help!

[dudicoco]

@Flydiverny it seems that this library is now affecting a few different components (prometheus-adapter, kube-metrics-adapter, aws-node eni) and is preventing from upgrading or implementing new features (aws sgs for pods is a more critical one).
Do you think we can raise the priority here?
Thanks

[Flydiverny]

@dudicoco Its not so much about priority as finding someone willing/able to dig into it :)

[alx13]

Digged a little bit.

So, it's failing on this endpoint:

/apis/custom.metrics.k8s.io/v1beta1/{resource}/{name}/{subresource}

Without custom.metrics you don't have any endpoints that have combinations like {path}/{path}/{path} and even {path}/{path}.

I don't really understand the logic there, but it seems wrong:

    while (nextSplits.length) {
      const split = nextSplits.shift()
      splits.push(split)

      let template = null
      if (nextSplits.length && nextSplits[0].startsWith('{')) {
        template = nextSplits.shift().slice(1, -1)
      }

[ib-ak]

@Flydiverny anyone looking into this issue? It seems to be a big one that conflicts custom metrics api and does not seem to be going way, I dont think there is a workaround to make everything work.
I can help and dig deep. Just need to know where to dig 😄

[davidquarles]

@Flydiverny anyone looking into this issue? It seems to be a big one that conflicts custom metrics api and does not seem to be going way, I dont think there is a workaround to make everything work.
I can help and dig deep. Just need to know where to dig 😄

kubernetes-sigs/prometheus-adapter#341 (comment)

IIUC the rewrite is nowhere near ready for prime-time and the fix that needs to be applied is upstream, here, with pretty much no traction on that PR since it was submitted in May. Perhaps we can all 👍 that PR and try to kickstart inertia over there? Or is there another openapi client we can swap in here to negate the issue in the meantime?

I certainly agree this is a bummer – we are now eternally pinned to 0.7.0 of the prometheus-adapter pending the v0 release of an untested rewrite. I intend to contribute on that front and am not trying to be negative here, at all – our setup is functional and I am quite grateful for this software – but this feels like gridlock, as it stands.

[diranged]

So we just noticed that we were bit by this as well. However, going back to the the prometheus-adapter 0.7.0 didn't help us. Deleting the APIService does, but then that makes the prometheus-adapter useless, right? For now we are disabling the prometheus-adapter.. but that seems like the wrong fix here.

What can be done to help move a fix forward here?

[jbilliau-rcd]

@diranged We got this working. Using 6.0.0 of the external-secrets chart, and 2.8.1 of the prometheus-adapter chart; have it installed on 50+ clusters, working fine.

chart:
repository: https://prometheus-community.github.io/helm-charts
name: prometheus-adapter
version: 2.8.1

[atsai1220]

@diranged We got this working. Using 6.0.0 of the external-secrets chart, and 2.8.1 of the prometheus-adapter chart; have it installed on 50+ clusters, working fine.

chart:
repository: https://prometheus-community.github.io/helm-charts
name: prometheus-adapter
version: 2.8.1

If anyone was curious chart version 2.8.1 for prometheus-adapter is app version v0.7.0

  - apiVersion: v1
    appVersion: v0.7.0
    created: "2020-12-05T10:58:46.558955576Z"
    description: A Helm chart for k8s prometheus adapter
    digest: 85e2ebab3925c7f0ae0fb481210e5716993840ca63855a793ded1dd4113753d1
    home: https://github.com/DirectXMan12/k8s-prometheus-adapter
    keywords:
    - hpa
    - metrics
    - prometheus
    - adapter
    maintainers:
    - email: [email protected]
      name: mattiasgees
    - name: steven-sheehy
    - email: [email protected]
      name: hectorj2f
    name: prometheus-adapter
    sources:
    - https://github.com/kubernetes/charts
    - https://github.com/DirectXMan12/k8s-prometheus-adapter
    urls:
    - https://github.com/prometheus-community/helm-charts/releases/download/prometheus-adapter-2.8.1/prometheus-adapter-2.8.1.tgz
    version: 2.8.1

[rmendal]

@diranged We got this working. Using 6.0.0 of the external-secrets chart, and 2.8.1 of the prometheus-adapter chart; have it installed on 50+ clusters, working fine.

chart:
repository: https://prometheus-community.github.io/helm-charts
name: prometheus-adapter
version: 2.8.1

This did work for us but I had to completely rip out all remnants of prometheus adapter and then install v2.8.1 of the chart then install v6 of external-secrets. Then it worked.

[travis-sobeck]

I'm a bit confused on Why a prometheus adapter is a dependency for this app and what they have to do with each other. I installed it on a barebones cluster just fine so it seems not needed. Can the offending call just be skipped or let the failure just be ignored?

[linxcat]

It isn't a dependancy, they share one.

[keramblock]

My workaround: switching from prometheus-adapter to keda.sh
Hope it helps someone.

[diranged]

My recommendation is to switch to the GoLang-based https://github.com/external-secrets/external-secrets controller that is being more actively developed. It works well and is a fairly easy switch.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[presidenten]

Any updates to this?

[Flydiverny]

@presidenten

• KES is deprecated, migrate to ESO! #864

[vikas027]

I too faced this today, migrating to ESO is the way to go :)