You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Which component are you using?:
vpa recommender 0.9.0
vertical-pod-autoscaler
What version of the component are you using?:
0.9.0
Component version:
What k8s version are you using (kubectl version)?:
k8s 1.20
kubectl version Output
$ kubectl version
What environment is this in?:
What did you expect to happen?:
What happened instead?:
In the prometheus and cadvisor versions we use, the reported container_cpu_usage_seconds_total is reported at a single-core granularity
However, recommender loads historical data from prometheus without aggregating these indicators. As a result, only the usage of single-core is loaded and the usage of other cores is discarded
In my case, the physical machine has 8 cores
Therefore, each container pair should have 8 indicators at the same time
· Add debug logs in recommender. It can be seen that 7 indicators are discarded because the time is less than the latest sample time
I0208 14:36:33.018051 1837887 cluster_feeder.go:254] Adding pod {default vpa-admission-plugin-sf-8db91-0} with labels map[] W0208 14:36:33.018083 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018246 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018256 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018261 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018265 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018269 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018293 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018300 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018304 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018310 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018315 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018321 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018326 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018331 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018341 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 15:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 15:36:32.891 +0800 CST, value 5 W0208 14:36:33.018345 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018348 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 15:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 15:36:32.891 +0800 CST, value 4 W0208 14:36:33.018352 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order)
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
The text was updated successfully, but these errors were encountered:
Which component are you using?:
vpa recommender 0.9.0
vertical-pod-autoscaler
What version of the component are you using?:
0.9.0
Component version:
What k8s version are you using (
kubectl version
)?:k8s 1.20
kubectl version
OutputWhat environment is this in?:
What did you expect to happen?:
What happened instead?:
In the prometheus and cadvisor versions we use, the reported container_cpu_usage_seconds_total is reported at a single-core granularity
However, recommender loads historical data from prometheus without aggregating these indicators. As a result, only the usage of single-core is loaded and the usage of other cores is discarded
In my case, the physical machine has 8 cores
Therefore, each container pair should have 8 indicators at the same time
· Add debug logs in recommender. It can be seen that 7 indicators are discarded because the time is less than the latest sample time
I0208 14:36:33.018051 1837887 cluster_feeder.go:254] Adding pod {default vpa-admission-plugin-sf-8db91-0} with labels map[] W0208 14:36:33.018083 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018246 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018256 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018261 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018265 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 4 W0208 14:36:33.018269 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018293 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018300 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018304 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018310 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018315 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018321 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018326 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 14:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 14:36:32.891 +0800 CST, value 5 W0208 14:36:33.018331 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018341 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 15:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 15:36:32.891 +0800 CST, value 5 W0208 14:36:33.018345 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order) W0208 14:36:33.018348 1837887 container.go:89] sample is valid true, expected: true, sample measure start 2025-02-07 15:36:32.891 +0800 CST. comtainer last CPU sampleStart 2025-02-07 15:36:32.891 +0800 CST, value 4 W0208 14:36:33.018352 1837887 cluster_feeder.go:274] Error adding metric sample for container {{default vpa-admission-plugin-sf-8db91-0} vpa-admission-plugin-phnc-lihulitest}: sample discarded (invalid or out of order)
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
The text was updated successfully, but these errors were encountered: