Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Umbrella issue] How we monitor k8s-infra ? #2588

Closed
ameukam opened this issue Aug 20, 2021 · 19 comments
Closed

[Umbrella issue] How we monitor k8s-infra ? #2588

ameukam opened this issue Aug 20, 2021 · 19 comments
Labels
area/infra Infrastructure management, infrastructure design, code in infra/ lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.

Comments

@ameukam
Copy link
Member

ameukam commented Aug 20, 2021 •
edited
Loading

We initially had this conversation in #401.

Also kubernetes/test-infra#23317 (comment):

FYI @ameukam we don't have this feature enabled in kubernetes.io at the moment but will want to take a look at it soon

Some questions from thockin:

Cluster monitoring
a) What should we use?
GKE Workload metrics : https://cloud.google.com/stackdriver/docs/solutions/gke/managing-metrics#workload-metrics
Managed service for Prometheus : https://cloud.google.com/stackdriver/docs/managed-prometheus
b) How do we set it up with git-ops?
- #1376
- #1624
c) What exactly are we concerned about (signals)?
d) How are alerts delivered to a group of people?
e) How do we manage that group?
f) Do we need an on-call rotation?

App monitoring
a) Same tool as cluster monitoring?
b) What is the minimum expectation for an app to be deployed into community space
c) How do we manage groups of alerts for each app (ggroups?)
d) How do we manage on-call for each app?

GCP quotas monitoring
How do we monitoring them ?

More questions can be added.

/milestone v1.23
/are infra
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Aug 20, 2021
@ameukam
Copy link
Member Author

ameukam commented Aug 20, 2021

For this milestone, I would like to focus on how to flew out methods and practices about how we should do monitoring for k8s-infra.

/area infra

@k8s-ci-robot k8s-ci-robot added the area/infra Infrastructure management, infrastructure design, code in infra/ label Aug 20, 2021
@spiffxp
Copy link
Member

spiffxp commented Sep 2, 2021

/priority important-longterm

@k8s-ci-robot k8s-ci-robot added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Sep 2, 2021
@jimdaga
Copy link

jimdaga commented Sep 16, 2021

If Prometheus is the tool picked I'm happy to jump in and help, I have a decent amount of experience.

@k8s-ci-robot k8s-ci-robot added sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. and removed wg/k8s-infra labels Sep 29, 2021
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 8, 2021
@ameukam
Add Terraform subrepo for monitoring
ee65c22 
Related to:
 - Ref: kubernetes#2588

Bootstrap a new suberepo that will host the Terraform resources
consuming the GCP monitoring API.

Signed-off-by: Arnaud Meukam <[email protected]>
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 8, 2021
@ameukam
Add Terraform subrepo for monitoring
14f9570 
Related to:
 - Ref: kubernetes#2588

Bootstrap a new suberepo that will host the Terraform resources
consuming the GCP monitoring API.
I also bumped the terraform provider for this subrepo and will the other
declarations of the provider in a followup PR.

Signed-off-by: Arnaud Meukam <[email protected]>
@ameukam ameukam mentioned this issue Oct 8, 2021
Merged
@spiffxp spiffxp mentioned this issue Oct 14, 2021
Merged
ameukam added a commit to ameukam/k8s.io that referenced this issue Oct 15, 2021
@ameukam
kubernetes: Fix workload identity binding
699f81c 
Related:
  - Part of: kubernetes#2588
  - Fixes: kubernetes#2942
  - followup of: kubernetes#2898

Ensure service account tf-monitoring-deployer can be used in build cluster prow-build-trusted

Signed-off-by: Arnaud Meukam <[email protected]>
@ameukam ameukam mentioned this issue Oct 15, 2021
Merged
@ameukam
Copy link
Member Author

ameukam commented Dec 6, 2021

/milestone v1.24

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.23, v1.24 Dec 6, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 6, 2022
@ameukam
Copy link
Member Author

ameukam commented Mar 6, 2022

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 6, 2022
@ameukam
Copy link
Member Author

ameukam commented May 12, 2022

/milestone clear

@k8s-ci-robot k8s-ci-robot removed this from the v1.24 milestone May 12, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 10, 2022
@ameukam
Copy link
Member Author

ameukam commented Aug 10, 2022

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 10, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 8, 2022
@riaankleinhans
Copy link
Contributor

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 8, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 6, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 8, 2023
@riaankleinhans
Copy link
Contributor

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Mar 12, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 10, 2023
@riaankleinhans
Copy link
Contributor

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 11, 2023
@ameukam
Copy link
Member Author

ameukam commented Jul 27, 2023

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Jul 27, 2023
@ameukam
Copy link
Member Author

ameukam commented Dec 6, 2024

there is an effort to deploy an unified stack for monitoring.

See: #7377

/close

@k8s-ci-robot
Copy link
Contributor

@ameukam: Closing this issue.

In response to this:

there is an effort to deploy an unified stack for monitoring.

See: #7377

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/infra Infrastructure management, infrastructure design, code in infra/ lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Projects
SIG K8S Infra
Status: Done
Milestone
No milestone
Development

No branches or pull requests
6 participants
@spiffxp @ameukam @k8s-ci-robot @jimdaga @k8s-triage-robot @riaankleinhans