Call GetMetricData api per region instead of per instance

[kaiyan-sheng]

In current ec2 metricset code, we are constructing metricDataQueries per region per instance for GetMetricData cloudwatch api call. Because we have some metadata from DescribeInstances api call for each instance to add into each event. It will be more efficient/cheaper to improve ec2 metricset to construct metricDataQueries per region but not per instance.

closes #11820

[exekias]

Left some minor comments. From what I understand you launch a bunch of metric queries (for every EC2 instance) with the same call.

I'm wondering if it's possible to launch a query for all instance ids, without setting dimensions in it?

[kaiyan-sheng]

Left some minor comments. From what I understand you launch a bunch of metric queries (for every EC2 instance) with the same call.

I'm wondering if it's possible to launch a query for all instance ids, without setting dimensions in it?

@exekias Sorry I definitely need to put in more explanation here. So the goal for this PR is to reduce the number of svc.GetMetricDataRequest api call, which is hidden in getMetricDataPerRegion function in x-pack/metricbeat/module/aws/utils.go. Without the changes in this PR, svc.GetMetricDataRequest gets called per region per EC2 instance. With the changes, svc.GetMetricDataRequest gets called per region. Since we still want to keep the metadata for each EC2 instance, describeInstances api is called per region. The result from describeInstances is stored into a map<instanceID><describeInstanceOutput>. Then this map is passed into createCloudWatchEvents function with the output from svc.GetMetricDataRequest to construct events for each EC2 instance.

I guess one more thing we can do is to remove the for loop for regions for svc.GetMetricDataRequest. But that loop will still exist for describeInstances because this api call can only be made per region.

[exekias]

This is already an improvement, I'm ok to getting this in and experiment more later. I want to play a little bit with the cloudwatch metricset for better understanding.

Left a comment, let's tackle that and this is ready to go

[kaiyan-sheng]

I spent some more time testing if we can make cloudwatch api calls without specifying ec2 instanceID. Unfortunately the answer is no. I also checked if we can make cloudwatch api calls without region specified, the answer is no too.

[exekias]

Thank you for doing the additional testing!

[exekias]

I think we can skip backport for this one, as it's not fixing a critical bug but improving the behavior