AWS s3 stalled stream upload errors

[fmassot]

On airmail, we have errors like this one:

Aug 1 06:45:11 quickwit-indexer-17 quickwit-indexer ERROR exit activating-kill-switch actor=SourceActor-icy-zjHo exit_status=DownstreamClosed
Aug 1 06:45:20 quickwit-indexer-16 quickwit-indexer 
Caused by:
    0: storage error(kind=Timeout, source=dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })))
    1: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })) split_id="01J46ARVF1D5HMTZTSSJ91QVQY"
Aug 1 06:45:22 quickwit-indexer-16 quickwit-indexer ERROR actor-exit-without-success actor="MergePlanner-bitter-91dX"
Aug 1 06:45:22 quickwit-indexer-16 quickwit-indexer ERROR actor-exit-without-success actor="MergeUploader-winter-cS3G"
Aug 1 06:45:22 quickwit-indexer-16 quickwit-indexer ERROR actor-exit-without-success actor="MergePublisher-ancient-OTCC"
Aug 1 06:45:22 quickwit-indexer-16 quickwit-indexer ERROR merge pipeline failed index_uid=metric.ba0c51bd17.2024-08-01.quickwit:01J45KSS12B0YDPTG09TK4QCGM source_id=_ingest-source generation=1 healthy_actors=["MergeSplitDownloader-twilight-MJaK", "MergeExecutor-white-5PvG", "MergePackager-holy-rZQ0"] failed_or_unhealthy_actors=["MergePlanner-bitter-91dX", "MergeUploader-winter-cS3G", "MergePublisher-ancient-OTCC"] success_actors=[]
Aug 1 06:46:55 quickwit-indexer-19 quickwit-indexer 
Caused by:
    0: storage error(kind=Timeout, source=dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })))
    1: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })) split_id="01J46ARPKEMSJH8NKTNSCFJ15Z"

Related issue that was opened on the AWS rust SDK project: awslabs/aws-sdk-rust#1148

[trinity-1686a]

this should be configurable through StalledStreamProtectionConfig.
The default behavior was changed at some point earlier this year