Bug: Mimir 2.15.0 unable to access block storage

[eric-engberg]

What is the bug?

I was upgrading from chart version 5.5.0-weekly.304 to 5.6.0 and after upgrading mimir was unable to access the configured s3 buckets. Downgrading fixed the problem so it shouldn't be a permissions issue. The error message doesn't really give any details, just "Access Denied".

All services that use block storage are affected. None of them work. When comparing the manifest changes that would be deployed there's nothing that would cause this. No changes to the config and no changes to the service account which uses IRSA

How to reproduce it?

Deploy chart 5.5.0-weekly.304 using s3. Upgrade to chart 5.6.0

What did you think would happen?

Mimir would work and be able to access block storage

What was your environment?

Kubernetes 1.30, deployed using kustomize and helm via ArgoCD. Mimir version 2.15.0 chart 5.6.0

Any additional context to share?

No response

[narqo]

upgrading from chart version 5.5.0-weekly.304 to 5.6.0

There have been lots of internal changes between these two, but I don't recall anything that was intentional on the mimir's side.

Could you provide more details on how you configure the s3 access — do you maybe have an IAM policy with a strict list of API methods in the allow-list? Under the hood, Mimir uses objstore: I do recall there was some back and worth there about using one s3 API over the other. Maybe there is something to explore there in details.

[eric-engberg]

The policy for the role only contains the permissions below. I didn't see any errors reporting in CloudTrail. I didn't see any usage at all really. I saw stuff assuming the role but that could have been existing pods that hadn't updated yet. It didn't seem that CloudTrail could identify it.

Someone in my comment in #mimir on the grafana slack said they had this same issue and they switched to Pod Identity instead of IRSA and it worked for them. I'm assuming that it's not assuming the role but not sure. The error message did not include any error info from AWS.. just said it couldn't access block storage with an "Access Denied"

"Action": [
  "s3:PutObject",
  "s3:GetObject",
  "s3:ListBucket",
  "s3:DeleteObject"
],

[narqo]

So far, I cannot spot anything obvious in the recent changes, that would require extra IAM permissions, neither can I find any recent related reports in the internal tracker.

Initially, I thought if maybe this one thanos-io/objstore#138 could be the issue. But 1) it was reverted long time ago; 2) the ListBucket and GetObject permissions cover the "stat" API from minio-go.

All services that use block storage are affected. None of them work.

Could you share the full log from one of the services? I wonder if there could be any hints there.

[eric-engberg]

I'll have to try the update again. Maybe I can do it next week.