Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recommender use promethues sa historyProvider,multicore cpu will only be loaded with one core usage #7814

Open
SadShark0328 opened this issue Feb 8, 2025 · 1 comment
Labels
area/vertical-pod-autoscaler kind/bug Categorizes issue or PR as related to a bug.

Comments

@SadShark0328
Copy link

Which component are you using?:
vpa recommender 0.9.0

vertical-pod-autoscaler

What version of the component are you using?:
0.9.0

Component version:

What k8s version are you using (kubectl version)?:
k8s 1.20

kubectl version Output
$ kubectl version

What environment is this in?:

What did you expect to happen?:

What happened instead?:
In the prometheus and cadvisor versions we use, the reported container_cpu_usage_seconds_total is reported at a single-core granularity

However, recommender loads historical data from prometheus without aggregating these indicators. As a result, only the usage of single-core is loaded and the usage of other cores is discarded

In my case, the physical machine has 8 cores
Therefore, each container pair should have 8 indicators at the same time
· Add debug logs in recommender. It can be seen that 7 indicators are discarded because the time is less than the latest sample time

I0208 14:36:33.018051 1837887 cluster_feeder.go:254] Adding pod {default vpa-admission-plugin-sf-8db91-0} with labels map[] W0208 14:36:33.018083 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018246 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018256 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018261 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018265 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018269 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018293 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018300 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018304 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018310 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018315 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018321 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018326 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018331 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018341 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 15:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 15:36:32.891 +0800 CST, value 5 W0208 14:36:33.018345 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018348 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 15:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 15:36:32.891 +0800 CST, value 4 W0208 14:36:33.018352 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order)

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

@SadShark0328 SadShark0328 added the kind/bug Categorizes issue or PR as related to a bug. label Feb 8, 2025
@adrianmoisey
Copy link
Member

/area vertical-pod-autoscaler

What version of the component are you using?: 0.9.0
What k8s version are you using (kubectl version)?: k8s 1.20

Both the VPA and Kubernetes versions that you are using are out of support. Is this bug reproducible on the latest of each?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/vertical-pod-autoscaler kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants