Error: SignatureDoesNotMatch when running behind a cloudflare tunnel

[abasu0713]

Summary

So I have a multii-node Ceph Cluster (Squid Release) deployed using microceph that is running on multiple Arm64 based boards (Orange Pis and Raspberry Pis)

services:
  microceph.daemon:     simple, enabled, active
  microceph.mds:        simple, enabled, active
  microceph.mgr:        simple, enabled, active
  microceph.mon:        simple, enabled, active
  microceph.osd:        simple, enabled, active
  microceph.rbd-mirror: simple, disabled, inactive
  microceph.rgw:        simple, enabled, active
snap-id:      ct1DgtIGBaljdOwomQGMLr8EJcL5pOP1
tracking:     squid/edge
refresh-date: 37 days ago, at 16:19 CST
hold:         forever

It's been working great. I have 2 RGW gateways exposed to internet using Cloudflare Tunnels. I am able to use AWS CLI and SDKs (both dart and python) to interact with it using a Cloudflare DNS.. the setup is simple. I have a cloudflare tunnel running on a Kubernetes cluster (separate from the nodes actually running the RGW Gateways) -> Offloads to a Envoy Proxy -> that load balances between my RGW Gateway instances.

cat ~/.aws/config
[profile Terraform]
endpoint_url = https://<redacted-record>.arkobasu.space
region = default
output = json
alpha@Arkos-MacBook-Pro workspace 

I can confirm that Accounts and IAM API are also functional. So this profile for example is created under a new Account, by the root user of the account -> that then created this user.

Issue

The problem is: when I use this with a Terraform s3 backend it errors out only when using the Cloudflare DNS but not when I am using it with IP based endpoint:

terraform {
  backend "s3" {
    bucket  = "<redacted-bucket-name>"
    key     = "ceph-s3-gateway/terraform.tfstate"
    region  = "default"
    profile = "Terraform"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
    use_path_style              = true
    skip_s3_checksum = true
    endpoints = {
        # iam = "https://<redacted-record>.arkobasu.space"
        # s3 = "https://<redacted-record>.arkobasu.space/" <- THIS DOES NOT WORK
        s3 = "http://192.168.5.81:80" <- THIS WORKS
    }
  }
  required_providers {
    random = {
      source = "hashicorp/random"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.35.1"
    }
  }
}

2025-02-03T00:37:25.557-0600 [DEBUG] backend-s3: HTTP Request Sent: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<redacted>/20250203/default/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=<redacted> http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250203T063725Z http.url="https://<redacted-record>.arkobasu.space/<redacted-bucket-name>?list-type=2&max-keys=1000&prefix=env%3A%2F" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.10.4 (+https://www.terraform.io) m/C aws-sdk-go-v2/1.31.0 os/macos lang/go#1.23.3 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.63.0" http.request.header.amz_sdk_invocation_id=8c5bcb89-da72-4cab-8227-74a9974c963d http.request.header.accept_encoding=identity http.request.body="" http.method=GET net.peer.name=<redacted-record>.arkobasu.space
2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: HTTP Response Received: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/ http.response.header.cf_cache_status=DYNAMIC http.response.header.x_amz_request_id=tx000004599d10cd4712119-0067a06425-1192799-default http.response.header.server=cloudflare http.duration=178 http.status_code=403 http.response.header.nel="{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}" http.response.header.alt_svc="h3=\":443\"; ma=86400" http.response_content_length=219 http.response.header.cf_ray=90c0698bee2d124e-ORD http.response.header.report_to="{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v4?s=bkYoWKv7FpD9Lr8Vj8bmE8KPocsIWMjJegF5%2FeUrO8Tn5GsnSAba%2B4w9sGeZcVC4n85JSWq0EHbRCydcGWdct9N7bGDqLQ8qnQkjmhEEFDWCiCnWyYi7cPmgyOe%2Bz0BqJIxpVKiEBhSPsbWj1k%2FRtw%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}" http.response.header.date="Mon, 03 Feb 2025 06:37:25 GMT"
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message></Message><RequestId>tx000004599d10cd4712119-0067a06425-1192799-default</RequestId><HostId>1192799-default-default</HostId></Error>
   http.response.header.content_type=application/xml http.response.header.accept_ranges=bytes http.response.header.x_envoy_upstream_service_time=34 http.response.header.server_timing="cfL4;desc=\"?proto=TCP&rtt=25278&min_rtt=17126&rtt_var=8980&sent=8&recv=12&lost=0&retrans=0&sent_bytes=4263&recv_bytes=2255&delivery_rate=169645&cwnd=254&unsent_bytes=0&cid=6d94aa0b9071c6de&ts=116&x=0\""
2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/
╷
│ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "<redacted-bucket-name>" with prefix "env:/": operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError
│ 
│ 

I have been stuck on this for a bit. I have tried everything. The Signature validation does work when I am using something like Presign for objects and accessing them over the browser. I have investigated in Hashicorp's github.. and they say it's a support issue for the vendor (of the S3 provider). Hence writing here.

I would appreciate it bery much if you could give me some direction.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1150.

This message was autogenerated