fix: customizable cloudwatch batch size when querying aws

[powersj]

Fixes: #10842

[Hipska]

Why is this approach used rather than just configuring metric_batch_size for the plugin?

[powersj]

Why is this approach used rather than just configuring metric_batch_size for the plugin?

This is an input, not an output. This batch size is specific to how the plugin gathers data to avoid a limit in AWS.

[Hipska]

Oh, didn’t notice that.

[reimda]

Is there a way to determine a good batch size automatically? Maybe we calculate how big a request will be, and ask cloudwatch what the max batch is, then have our code determine how many requests can fit in a batch?

Adding a setting gives the user a way to fix the problem, but requires them to think more than maybe they should have to.

[powersj]

Is there a way to determine a good batch size automatically?

AWS's GetMetricData docs reference a limit of 500 metrics in a single request. This explains the currently hard-coded value of 500. Presumably, this is why we have not seen this issue before.

The error the user received was a 413: RequestEntityTooLarge: Request size 639468 exceeded 614400 bytes. The message was 25,068 bytes too large.

I do not see any payload limit on the CloudWatch service quotas page. I also did not see any other reference to a 413 or too large error in that document or their docs on common errors.

In the code, during Gather we split the queries into batches and then launch goroutines for each batch. It is not clear to me how we could check that we would avoid the 413 error before the goroutines.

@reimda, thoughts?

[Hipska]

I'm wondering if this error actually comes from AWS or some in between proxy?

[reimda]

If there's no way to know the byte limit, maybe we should watch for 413 errors and handle them afterward by splitting the batch in half and trying again?

[reimda]

Maybe there is a documented api limit, just not a quota limit. The cloudwatch docs are not the easiest to sift through, but I would expect there to be something about this kind of limit there somewhere.

[powersj]

I have continued to go through the web docs, as well as CloudWatch API Reference and CloudWatch User Guide with no references to limits.

I have pushed another commit that tries the splitting method, but I think I prefer the user setting over this. The original bug report shows this occurring on every single request. This means the user will always be making 3x the requests. This does seem to be a one-off issue, and having a setting that tunes this makes more sense over something that introduces more complexity that is not easily tested either.

@reimda, that said, I'd like to close this one way or another, so please let me know which way you want to go with this.

[reimda]

I have continued to go through the web docs, as well as CloudWatch API Reference and CloudWatch User Guide with no references to limits.

I have pushed another commit that tries the splitting method, but I think I prefer the user setting over this. The original bug report shows this occurring on every single request. This means the user will always be making 3x the requests. This does seem to be a one-off issue, and having a setting that tunes this makes more sense over something that introduces more complexity that is not easily tested either.

@reimda, that said, I'd like to close this one way or another, so please let me know which way you want to go with this.

It looks like there's no way to tune the batch size automatically so a setting is the next best thing. Would it be useful to have both the splitting and a user setting? That way if the setting is too low they will get splitting instead of failures, but if/when they notice the 3x requests they can increase the setting?

It does seem like this is a relatively rare situation. I'm also ok with a setting and no splitting if that's what you prefer.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

Looks good to me.