Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] [Remote Store] S3 throttling during translog upload #7390

Closed
linuxpi opened this issue May 3, 2023 · 0 comments
Closed

[BUG] [Remote Store] S3 throttling during translog upload #7390

linuxpi opened this issue May 3, 2023 · 0 comments
Labels
bug Something isn't working Storage:Durability Issues and PRs related to the durability framework v2.8.0 'Issues and PRs related to version v2.8.0'

Comments

@linuxpi
Copy link
Collaborator

linuxpi commented May 3, 2023

Describe the bug
Saw s3 throttling during upload during translog upload flow.

[2023-05-02T08:02:41,067][ERROR][o.o.i.t.t.TranslogTransferManager] [node-1] Exception during transfer for file translog-3169.ckp
org.opensearch.index.translog.transfer.FileTransferException: java.io.IOException: Unable to upload object [cluster-4/MYce-mgETOGu23bJrxMY7w/17/1/translog-3169.ckp] using a single upload
        at org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobAsync$1(BlobStoreTransferService.java:59) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: java.io.IOException: Unable to upload object [cluster-4/MYce-mgETOGu23bJrxMY7w/17/1/translog-3169.ckp] using a single upload
        at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:425) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:153) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:151) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:205) ~[?:?]
        at org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobAsync$1(BlobStoreTransferService.java:55) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        ... 6 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: 74DNE2P17XNSRJNA; S3 Extended Request ID: PQQVETekOFNX9uZ6XjrCTahx45e/zryA9iLJf7t2xecmmIhySJ8RkZyq9DBig4fRt4PQ7CY6wxQ=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:423) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:79) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:78) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:423) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:153) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:151) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:205) ~[?:?]
        at org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobAsync$1(BlobStoreTransferService.java:55) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        ... 6 more
[2023-05-02T08:02:41,068][ERROR][o.o.i.t.t.TranslogTransferManager] [node-1] Transfer failed for snapshot TranslogTransferSnapshot [ primary term = 1, generation = 3169 ]
java.io.IOException: Failed to upload 1 files during transfer
        at org.opensearch.index.translog.transfer.TranslogTransferManager.transferSnapshot(TranslogTransferManager.java:125) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.translog.RemoteFsTranslog.upload(RemoteFsTranslog.java:240) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.translog.RemoteFsTranslog.prepareAndUpload(RemoteFsTranslog.java:213) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.translog.RemoteFsTranslog.ensureSynced(RemoteFsTranslog.java:163) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.translog.Translog.ensureSynced(Translog.java:834) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.translog.InternalTranslogManager.ensureTranslogSynced(InternalTranslogManager.java:179) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.shard.IndexShard.lambda$createTranslogSyncProcessor$43(IndexShard.java:3955) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.shard.IndexShard$6.write(IndexShard.java:3968) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:129) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:117) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.BufferedAsyncIOProcessor.process(BufferedAsyncIOProcessor.java:79) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.lang.Thread.run(Thread.java:1589) [?:?]
        Suppressed: org.opensearch.index.translog.transfer.FileTransferException: java.io.IOException: Unable to upload object [cluster-4/MYce-mgETOGu23bJrxMY7w/17/1/translog-3169.ckp] using a single upload
                at org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobAsync$1(BlobStoreTransferService.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
                at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
                at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
                at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
                at java.lang.Thread.run(Thread.java:1589) [?:?]
        Caused by: java.io.IOException: Unable to upload object [cluster-4/MYce-mgETOGu23bJrxMY7w/17/1/translog-3169.ckp] using a single upload
                at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:425) ~[?:?]
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:153) ~[?:?]
                at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
                at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
                at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:151) ~[?:?]
                at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:205) ~[?:?]
                at org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobAsync$1(BlobStoreTransferService.java:55) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
                ... 6 more
        Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: 74DNE2P17XNSRJNA; S3 Extended Request ID: PQQVETekOFNX9uZ6XjrCTahx45e/zryA9iLJf7t2xecmmIhySJ8RkZyq9DBig4fRt4PQ7CY6wxQ=; Proxy: null)
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
                at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
                at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
                at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@linuxpi linuxpi added bug Something isn't working untriaged labels May 3, 2023
@sachinpkale sachinpkale added Storage:Durability Issues and PRs related to the durability framework v2.8.0 'Issues and PRs related to version v2.8.0' and removed untriaged labels May 4, 2023
@linuxpi linuxpi closed this as completed Aug 22, 2024
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Storage:Durability Issues and PRs related to the durability framework v2.8.0 'Issues and PRs related to version v2.8.0'
Projects
Status: ✅ Done
Development

No branches or pull requests

2 participants