Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase retry backoff for Storage API batch to survive AppendRows quota refill #31837

Merged
merged 7 commits into from
Jul 15, 2024
Merged

Increase retry backoff for Storage API batch to survive AppendRows quota refill #31837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
49f0ba3
Increase retry backoff for Storage API batch
ahmedabu98 Jul 10, 2024
cbbe3f0
longer waits for quota error only
ahmedabu98 Jul 11, 2024
8019b8d
cleanup
ahmedabu98 Jul 11, 2024
9207f94
Merge branch 'master' of https://github.com/ahmedabu98/beam into larg…
ahmedabu98 Jul 12, 2024
347ade2
add to CHANGES.md
ahmedabu98 Jul 12, 2024
72c8968
no need for quota backoff. just increase allowed retries
ahmedabu98 Jul 15, 2024
a99b776
cleanup
ahmedabu98 Jul 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@

* Multiple RunInference instances can now share the same model instance by setting the model_identifier parameter (Python) ([#31665](https://github.com/apache/beam/issues/31665)).
* Added options to control the number of Storage API multiplexing connections ([#31721](https://github.com/apache/beam/pull/31721))
* [BigQueryIO] Better handling for batch Storage Write API when it hits AppendRows throughput quota ([#31837](https://github.com/apache/beam/pull/31837))
* [IcebergIO] All specified catalog properties are passed through to the connector ([#31726](https://github.com/apache/beam/pull/31726))
* Removed a 3rd party LGPL dependency from the Go SDK ([#31765](https://github.com/apache/beam/issues/31765)).
* Support for MapState and SetState when using Dataflow Runner v1 with Streaming Engine (Java) ([[#18200](https://github.com/apache/beam/issues/18200)])
Expand All @@ -83,7 +84,7 @@

## Bugfixes

* Fixed a bug in BigQueryIO batch Storage Write API that frequently exhausted concurrent connections quota ([#31710](https://github.com/apache/beam/pull/31710))
* [BigQueryIO] Fixed a bug in batch Storage Write API that frequently exhausted concurrent connections quota ([#31710](https://github.com/apache/beam/pull/31710))
* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## Security Fixes
Expand Down
2 changes: 1 addition & 1 deletion ...rm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -771,7 +771,7 @@ long flush(
invalidateWriteStream();
allowedRetry = 5;
} else {
allowedRetry = 10;
allowedRetry = 35;
}

// Maximum number of times we retry before we fail the work item.
Expand Down
Loading