AlertManagerConfig: Add support for the new msteamsv2_configs receiver

[jonerer]

Component(s)

AlertManagerConfig

What is missing? Please describe.

I'm running the prometheus-operator v0.72.0, but the latest has the same issue. I have manually set the version of alertmanager to v0.28.0-rc.0 in the "Alertmanager" object. I did this because alertmanager has added support for a new receiver: prometheus/alertmanager#4024 which is highly anticipated. But when I try to use it by adding it to my config (following the steps outlaid in prometheus/alertmanager#3920 (comment) ), it doesn't actually get synced into the alertmanager Pod.

It gets blocked by the prometheus operator which doesn't recognize the field:

level=error ts=2024-10-29T09:50:34.557049663Z caller=klog.go:126 component=k8s_client_runtime func=ErrorDepth msg="sync "prometheus-s ││ tack/prometheus-stack-prometheu-alertmanager" failed: provision alertmanager configuration: failed to initialize from secret: yaml: u ││ nmarshal errors:\n line 27: field msteamsv2_configs not found in type config.plain"

The problem is that prometheus-operator tries to marshal it, and it doesn't exist in the object here:

prometheus-operator/pkg/alertmanager/types.go

Line 99 in d2599cf

type receiver struct {

But it does here:
https://github.com/prometheus/alertmanager/blob/ff8c09d60174f27850830ceed6be38b3cf86974b/config/config.go#L988

Thanks in advance

Describe alternatives you've considered.

No response

Environment Information.

Environment

Kubernetes Version: 1.29.8
Prometheus-Operator Version: v0.72.0
Alertmanager version: v0.28.0-rc.0

[sebagarayco]

+1 to this. I'm having the same issue

[Zheer09]

+1

[slashpai]

We would need to wait till 0.28 is released 🙂

[sebagarayco]

We would need to wait till 0.28 is released 🙂

Do you know if this is happening before end of December ?

[NicoGartmann]

Hello, we are having the same issue. Is there any workaround until the 0.28 gets released?
E.g adding the msteamsv2_configs via a configmap or secret to the receiver struct?

[RuStyC0der]

Hi there, just got updated alertmanager to 0.28.0 but issue still persists. I use prometheus stack chart version 68.3.0 with alertmanager v0.28.0 and prometheus operator v0.79.2 and can confirm issue exists. I can see reconciliation error like this
field msteamsv2_configs not found in type config.plain

However if I will exec into pod, change config directly and then reload alertmanager manually with kill -HUP 1 it will load and validate new conf.

Can someone validate it works on their side or it still an issue. Thanks)

[shawmarsh]

I have the same issue with v68.3.0 kube prometheus stack chart. I also tried installing the alertmanager v0.28.0 chart by itself and encountered the same issue.

[kallaics]

+1

Same issue appears with msteamsv2_config on our side as well. Error message exactly the same. Please fix it. Thanks.

[slashpai]

+1

Same issue appears with msteamsv2_config on our side as well. Error message exactly the same. Please fix it. Thanks.

v0.80.0 should have fixed this

[slashpai]

It's yet to be supported in CRD level but is supported in config secret

[kallaics]

@slashpai Unfortunately, it is not working with version v0.80.0

We are using the kube.prometheus-stack Helm chart too, it is already added the version v0.80.0 of Prometheus Operator and the error message "field msteamsv2_configs not found in type config.plain" still there in the logs and I needed to remove the msteamsv2_config entries.

The configuration not comes form the CRD yet, it would be the goal. It was configured in the kube-prometheus-stack Helm chart values file.
Exactly in this section:
https://github.com/prometheus-community/helm-charts/blob/2bd9a92d87e7e7c007424eb1644d686149cf4232/charts/kube-prometheus-stack/values.yaml#L468

Example from the config (focused to the receiver configuration only):

alertmanager:
  config:
    receivers:
      - name: Test
        msteamsv2_configs:
          - webhook_url: https://<redacted MS Teams link, that was created with Workflows>
            title: Alert
            summary: Alert summary

Kube-prometheus-stack: 69.3.1 (current latest)
Prometheus Operator version: v0.80.0 (image: quay.io/prometheus-operator/prometheus-operator:v0.80.0)

[soerennielsen]

@slashpai Unfortunately, it is not working with version v0.80.0

We are using the kube.prometheus-stack Helm chart too, it is already added the version v0.80.0 of Prometheus Operator and the error message "field msteamsv2_configs not found in type config.plain" still there in the logs and I needed to remove the msteamsv2_config entries.

The configuration not comes form the CRD yet, it would be the goal. It was configured in the kube-prometheus-stack Helm chart values file. Exactly in this section: https://github.com/prometheus-community/helm-charts/blob/2bd9a92d87e7e7c007424eb1644d686149cf4232/charts/kube-prometheus-stack/values.yaml#L468

Example from the config (focused to the receiver configuration only):

alertmanager:
config:
receivers:
- name: Test
msteamsv2_configs:
- webhook_url: https://<redacted MS Teams link, that was created with Workflows>
title: Alert
summary: Alert summary

Kube-prometheus-stack: 69.3.1 (current latest) Prometheus Operator version: v0.80.0 (image: quay.io/prometheus-operator/prometheus-operator:v0.80.0)

Very strange!

I have it working here with the exact same chart version(!). I have another problem with it that I'll create another issue for, but this part is running.

My containers (so you might compare):

k get pods -n monitoring -o json | ConvertFrom-Json | Select -ExpandProperty items |% { $_.metadata.name + ":" + ($_.spec.containers |% {"\n - " + $_.name + ":" +$_.image})}
alertmanager-monitoring-prometheus-alertmanager-0:
 - alertmanager:quay.io/prometheus/alertmanager:v0.28.0
 - config-reloader:quay.io/prometheus-operator/prometheus-config-reloader:v0.80.0
alertmanager-monitoring-prometheus-alertmanager-1:
 - alertmanager:quay.io/prometheus/alertmanager:v0.28.0
 - config-reloader:quay.io/prometheus-operator/prometheus-config-reloader:v0.80.0
monitoring-grafana-8595fd78b6-5xxs4:
 - grafana-sc-dashboard:quay.io/kiwigrid/k8s-sidecar:1.30.0
 - grafana-sc-datasources:quay.io/kiwigrid/k8s-sidecar:1.30.0
 - grafana:docker.io/grafana/grafana:11.5.1
monitoring-kube-state-metrics-76f6f58dd5-wkngx:
 - kube-state-metrics:registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.14.0
monitoring-prometheus-node-exporter-f25s8:
 - node-exporter:quay.io/prometheus/node-exporter:v1.8.2
monitoring-prometheus-node-exporter-tqp29:
 - node-exporter:quay.io/prometheus/node-exporter:v1.8.2
monitoring-prometheus-operator-75fd665cdc-4wnzb:
 - prometheus:quay.io/prometheus-operator/prometheus-operator:v0.80.0
prometheus-monitoring-prometheus-prometheus-0:
 - prometheus:quay.io/prometheus/prometheus:v3.1.0
 - config-reloader:quay.io/prometheus-operator/prometheus-config-reloader:v0.80.0
prometheus-monitoring-prometheus-prometheus-1:
 - prometheus:quay.io/prometheus/prometheus:v3.1.0
 - config-reloader:quay.io/prometheus-operator/prometheus-config-reloader:v0.80.0

Config is similar:

prometheus:
  alertmanager:
      receivers:
        - name: "teams"
          msteamsv2_configs:
            - webhook_url_file: /etc/alertmanager/secrets/monitoring-secrets/teams-webhook-url
              title: '{{ template "slack.title" . }}'
              text: '{{ template "slack.text" . }}'

[kallaics]

The difference, you tested with webhook_url_file option, I tried with webhook_url.
Maybe the webhook_url definition is not 100%?

Thanks for your efforts and test the issue.

[kallaics]

HI all,

I had a bit more time to investigate my issue more deeper. I found the/my mistake. I used the summary field instead of text field.

Sorry about the wrong way. Thanks for your time @soerennielsen !