trivy-db fails to download with UNAUTHORIZED: The client does not have permission for manifest aquasecurity/trivy-db/2/manifest.json #695

seb-835 · 2022-11-16T08:19:23Z

What steps did you take and what happened:

trivy-operator pod failed to update db with error :

init error: DB error: failed to download vulnerability DB:
 OCI artifact error: OCI artifact error: OCI repository error: GET https://******/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: The client does not have permission for manifest; map[manifest:aquasecurity/trivy-db/2/manifest.json]\n"
,"stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:551\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:376\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}

What did you expect to happen:

I expected the trivy-db to be downloaded fine.

Anything else you would like to add:

What permission do i have to look for and authorized to fix this issue ?
Thanks a lot for advice.

Environment:

Trivy-Operator version : 0.33.0
Helm-chart install : trivy-operator-0.6.0
Kubernetes version : 1;24.4
OS : Redhat 7.9

The text was updated successfully, but these errors were encountered:

chen-keinan · 2022-11-16T09:22:45Z

@seb-835 you should not set up a permission for it , is this happen intermittently? or happen on every scan job?

seb-835 · 2022-11-16T09:45:13Z

@chen-keinan it happens for every scan job. None Start at all.

chen-keinan · 2022-11-16T09:51:17Z

@seb-835 we have not changed anything about it in trivy-operator v0.6.0 can you check if you get the same issue with trivy-operator v0.5.0 ?

seb-835 · 2022-11-16T10:54:29Z

@chen-keinan i install trivy-operator v0.5.0, and got same behaviour.

chen-keinan · 2022-11-16T11:10:13Z

@chen-keinan i install trivy-operator v0.5.0, and got same behaviour.

thanks for checking , I think it might be env. / config issue.
can you please share more info on env./ special config you are using ?

seb-835 · 2022-11-16T13:11:07Z

i agree @chen-keinan ,
here is more details,
i use a private registry to get/store image , this registry is only able to query docker.io, so image path is aquasec (docker.io) not aquasecurity (ghcr.io)
The registry credential are set through imagePullSecrets in both "default", "trivy-system" service account
so i have change helm value file to :

image:
  repository:  ***/aquasec/trivy-operator"
serviceAccount:
  name: "trivy-system"

trivy:
  repository: ***/aquasec/trivy"
  dbRepository: ***/aquasec/trivy-db
  dbRepositoryInsecure: "true"

My cluster is deployed with rke2 and cis-compliance enabled, means there is some restriction.
But default there is a restricted PSP policy apply to the cluster.

NAME                         PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
global-restricted-psp        false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

But I have assign to the namespace : trivy-system + serviceaccount trivy-system a rolebinding using an unrestricted PSP :

NAME                         PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
system-unrestricted-psp      true    *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *

it have solve and allow the trivy-operator container to run with root user,

but it does not solve the trivy-db update UNAUTHORIZE , i don't know what can block it and where to search.

Thanks for your help

chen-keinan · 2022-11-16T13:21:23Z

if you do not have an access to (ghcr.io) , then this is your problem as the trivy-db now is hosted on (ghcr.io) only
it use to be also in docker.io in the past but it is no longer maintained for quite a while

you can download it and publish it your self to private repo and overwrite the trivy.dbRepository param via helm or trivy-operator-trivy-config configmap

seb-835 · 2022-11-16T13:26:15Z

ok, i will fix that on the registry to get the image from ghcr.io
and get back to you if i got the same behaviour

seb-835 · 2022-11-16T13:51:58Z

@chen-keinan : is it expected ?

docker pull ghcr.io/aquasecurity/trivy-db:2
2: Pulling from /aquasecurity/trivy-db:2
b4f49697b38B : Pulling fs Layer
unexpected end of Json Input

crictl  --config=/var/lib/rancher/rke2/agent/etc/crictl.yaml pull ghcr.io/aquasecurity/trivy-db:2

E1116 14:45:30.116981   75414 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = InvalidArgument desc = failed to pull and unpack image \"ghcr.io/aquasecurity/trivy-db:2\": unable to fetch descriptor (sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) which reports content size of zero: invalid argument" image="ghcr.io/aquasecurity/trivy-db:2"

FATA[0001] pulling image: rpc error: code = InvalidArgument desc = failed to pull and unpack image "ghcr.io/aquasecurity/trivy-db:2": unable to fetch descriptor (sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) 
which reports content size of zero: invalid argument

chen-keinan · 2022-11-16T13:54:47Z

@chen-keinan : is it expected ?

docker pull ghcr.io/aquasecurity/trivy-db:2
2: Pulling from /aquasecurity/trivy-db:2
b4f49697b38B : Pulling fs Layer
unexpected end of Json Input

you need to use oras see docs

seb-835 · 2022-11-16T15:03:08Z

Thanks for help @chen-keinan
now, all images are pull from ghcr.io, but still having the UNAUTHORIZED issue.

chen-keinan · 2022-11-16T15:05:30Z

Thanks for help @chen-keinan
now, all images are pull from ghcr.io, but still having the UNAUTHORIZED issue.

Is it the same error or different, can you share the log ?

seb-835 · 2022-11-16T15:16:26Z

Same error

{"level":"error","ts":1668610817.513167,"logger":"reconciler.vulnerabilityreport","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-7759bd949","container":"79ffd7da-935c-43f3-9c1b-0c0c6fdf9fc3","status.reason":"Error","status.message":"2022-11-16T15:00:14.258Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n2022-11-16T15:00:14.258Z\t\u001b[34mINFO\u001b[0m\tDB Repository: registry.xxxx/aquasecurity/trivy-db\n2022-11-16T15:00:14.258Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n2022-11-16T15:00:14.631Z\t\u001b[31mFATAL\u001b[0m\tinit error: DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: GET https://registry.xxxx/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: The client does not have permission for manifest; map[manifest:aquasecurity/trivy-db/2/manifest.json]\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:551\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:376\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}

Does the job use the same ServiceAccount as trivy-operator deployment to get the imagePullSecret to pull the trivy-db ?

chen-keinan · 2022-11-16T15:24:57Z

Should have , checking and getting back to you

seb-835 · 2022-11-16T15:33:33Z

hum, even if my artifactory registry pull the image from ghcr.io as puvlic , my cluster need to authenticate against the artifactory registry to get the image.

anyway, if you do not specify a service account in the job manifest, it may use the "default" one of the trivy-system namespace,
it put the imagePullSecret in the default too, but still the same issue :(

chen-keinan · 2022-11-16T15:37:57Z

Download db is done in init container need to check if we can pass TRIVY_USER/TRIVY_PASSWORD then it will work

chen-keinan · 2022-11-16T16:42:27Z

@seb-835 I will try to find a away to make it work with current release if I will not succeed I will create PR for it

seb-835 · 2022-11-16T16:46:17Z

@chen-keinan thanks,
i have add a look to the pullSecrets
but this value is only use by trivy-operator deployment,
it is not create as an entry in the trivy-operator-trivy-config configmap used to run the job.

chen-keinan · 2022-11-16T16:55:03Z

@seb-835 yes it do create entry in trivy-operator-trivy-config cm.

still not sure it will solve your problem , I need to review it deeply and get back to you.

chen-keinan · 2022-11-16T17:41:43Z

@seb-835 is the images that you want to scan and trivy db are located on the same private registry ?

seb-835 · 2022-11-16T21:09:56Z

Sorry for late, yes, they are.

chen-keinan · 2022-11-20T10:08:33Z

Sorry for late, yes, they are.

@seb-835 we might need to do a change in trivy to support download db from private registry , I will get back to you with final answer shortly

mfilotto · 2023-03-14T00:25:39Z

Any update on this one please ?

chen-keinan · 2023-03-14T06:33:13Z

Any update on this one please ?

I'll open an issue on trivy to support download db (Server) from private registry

sathieu · 2023-03-16T11:01:42Z

Any update on this one please ?

I'll open an issue on trivy to support download db (Server) from private registry

This is already possible:

trivy-operator/deploy/helm/values.yaml

Line 339 in 487faa0

dbRepository: "ghcr.io/aquasecurity/trivy-db"

chen-keinan · 2023-03-16T11:04:20Z

This is already possible:

trivy-operator/deploy/helm/values.yaml

Line 339 in 487faa0

dbRepository: "ghcr.io/aquasecurity/trivy-db"

this is for public repo not private , if it require credential it will now work

sathieu · 2023-03-16T11:11:08Z

Thanks for the clarification.

mfilotto · 2023-03-23T07:30:53Z

@chen-keinan what is the number of the issue you opened on trivy ?

chen-keinan · 2023-03-23T09:45:03Z

@chen-keinan what is the number of the issue you opened on trivy ?

thanks for reminder , here is the issue

mfilotto · 2023-03-30T06:58:44Z

@chen-keinan the PR for this issue on trivy has just been merged, and the issue is closed.

chen-keinan · 2023-03-30T08:45:47Z

@chen-keinan the PR for this issue on trivy has just been merged, and the issue is closed.

@mfilotto great. once trivy v0.39.0 will be released I'll update trivy-operator

mfilotto · 2023-04-04T20:47:28Z

@chen-keinan: how can I configure authentication in helm chart ?

chen-keinan · 2023-04-05T06:19:43Z

@chen-keinan: how can I configure authentication in helm chart ?

User and Password

mfilotto · 2023-04-05T07:57:11Z

@chen-keinan I created a secret named trivy-operator-trivy-config with values TRIVY_USERNAME and TRIVY_PASSWORD, but I still have the error UNAUTHORIZED in trivy operator logs.
This secret seems to be used only in builtInTrivyServer mode, should I actiavte that mode ?
I'm not sure to understand what this mode is for though

chen-keinan · 2023-04-05T09:52:10Z

@chen-keinan I created a secret named trivy-operator-trivy-config with values TRIVY_USERNAME and TRIVY_PASSWORD, but I still have the error UNAUTHORIZED in trivy operator logs. This secret seems to be used only in builtInTrivyServer mode, should I actiavte that mode ? I'm not sure to understand what this mode is for though

yes , what I have specified is for builtInTrivyServer.
builtInTrivyServer param is simply spin up in cluster server, it remove the server concern from the user

mfilotto · 2023-04-05T23:20:38Z

Now I have this error in trivy server logs
TRIVY_USERNAME and TRIVY_PASSWORD are correctly set in secret trivy-operator-trivy-config

$ kubectl logs -f trivy-server-0
2023-04-05T23:17:27.933Z WARN 'TRIVY_SKIP_UPDATE' is deprecated. Use 'TRIVY_SKIP_DB_UPDATE' instead.
2023-04-05T23:17:27.934Z INFO Need to update DB
2023-04-05T23:17:27.934Z INFO DB Repository: docker-nexus.sln.nc/aquasecurity/trivy-db
2023-04-05T23:17:27.934Z INFO Downloading DB...
2023-04-05T23:17:27.988Z FATAL failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:
* GET https://my-private-registry/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: access to the requested resource is not authorized

chen-keinan · 2023-04-06T10:21:49Z

Now I have this error in trivy server logs TRIVY_USERNAME and TRIVY_PASSWORD are correctly set in secret trivy-operator-trivy-config

$ kubectl logs -f trivy-server-0 2023-04-05T23:17:27.933Z WARN 'TRIVY_SKIP_UPDATE' is deprecated. Use 'TRIVY_SKIP_DB_UPDATE' instead. 2023-04-05T23:17:27.934Z INFO Need to update DB 2023-04-05T23:17:27.934Z INFO DB Repository: docker-nexus.sln.nc/aquasecurity/trivy-db 2023-04-05T23:17:27.934Z INFO Downloading DB... 2023-04-05T23:17:27.988Z FATAL failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred: * GET https://my-private-registry/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: access to the requested resource is not authorized

@mfilotto Thanks for feedback, I see now that the PR done on trivy support only trivy image ... sub command and not trivy server... example:

$ TRIVY_USERNAME=foo TRIVY_PASSWORD=bar trivy image --db-repository ghcr.io/your/private-db --download-db-only

I'm Checking it with trivy project and getting back to you , opening this issue for now

chen-keinan · 2023-04-09T08:57:39Z

@mfilotto @seb-835 the latest PR in trivy is not supported for server command.
I have created another trivy/4008 PR to support server side , trivy 0.39.2 will be out next week with the fix

chen-keinan · 2023-04-18T07:48:59Z

@mfilotto now adding trivy v0.40.0 which include credential server support for private repo, will be available with trivy-operator v0.13.1

seb-835 added the kind/bug Categorizes issue or PR as related to a bug. label Nov 16, 2022

chen-keinan added target/kubernetes Issues relating to kubernetes cluster scanning priority/backlog Higher priority than priority/awaiting-more-evidence. labels Nov 16, 2022

chen-keinan mentioned this issue Apr 2, 2023

chore: bump trivy-version v0.39.0 #1128

Merged

3 tasks

chen-keinan closed this as completed in #1128 Apr 2, 2023

chen-keinan reopened this Apr 6, 2023

chen-keinan mentioned this issue Apr 18, 2023

chore: bump trivy v0.40.0 #1166

Merged

3 tasks

chen-keinan closed this as completed in #1166 Apr 18, 2023

trivy-db fails to download with UNAUTHORIZED: The client does not have permission for manifest aquasecurity/trivy-db/2/manifest.json #695

trivy-db fails to download with UNAUTHORIZED: The client does not have permission for manifest aquasecurity/trivy-db/2/manifest.json #695

Comments

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022 • edited Loading

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022

seb-835 commented Nov 16, 2022 • edited Loading

chen-keinan commented Nov 16, 2022 • edited Loading

seb-835 commented Nov 16, 2022

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022 • edited Loading

seb-835 commented Nov 16, 2022 • edited Loading

chen-keinan commented Nov 16, 2022 • edited Loading

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022

chen-keinan commented Nov 16, 2022

seb-835 commented Nov 16, 2022

chen-keinan commented Nov 16, 2022 • edited Loading

chen-keinan commented Nov 16, 2022

seb-835 commented Nov 16, 2022 via email • edited Loading

chen-keinan commented Nov 20, 2022 • edited Loading

mfilotto commented Mar 14, 2023

chen-keinan commented Mar 14, 2023 • edited Loading

sathieu commented Mar 16, 2023

chen-keinan commented Mar 16, 2023

sathieu commented Mar 16, 2023

mfilotto commented Mar 23, 2023

chen-keinan commented Mar 23, 2023

mfilotto commented Mar 30, 2023

chen-keinan commented Mar 30, 2023

mfilotto commented Apr 4, 2023

chen-keinan commented Apr 5, 2023

mfilotto commented Apr 5, 2023

chen-keinan commented Apr 5, 2023

mfilotto commented Apr 5, 2023 • edited Loading

chen-keinan commented Apr 6, 2023

chen-keinan commented Apr 9, 2023 • edited Loading

chen-keinan commented Apr 18, 2023

chen-keinan commented Nov 16, 2022 •

edited

Loading

seb-835 commented Nov 16, 2022 •

edited

Loading

chen-keinan commented Nov 16, 2022 •

edited

Loading

chen-keinan commented Nov 16, 2022 •

edited

Loading

seb-835 commented Nov 16, 2022 •

edited

Loading

chen-keinan commented Nov 16, 2022 •

edited

Loading

chen-keinan commented Nov 16, 2022 •

edited

Loading

seb-835 commented Nov 16, 2022 via email •

edited

Loading

chen-keinan commented Nov 20, 2022 •

edited

Loading

chen-keinan commented Mar 14, 2023 •

edited

Loading

mfilotto commented Apr 5, 2023 •

edited

Loading

chen-keinan commented Apr 9, 2023 •

edited

Loading