{"id":142,"date":"2019-03-05T06:46:00","date_gmt":"2019-03-05T06:46:00","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/2019\/03\/05\/alertmanager-integration-with-prometheus\/"},"modified":"2025-11-24T14:21:31","modified_gmt":"2025-11-24T08:51:31","slug":"alertmanager-integration-with-prometheus","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2019\/03\/05\/alertmanager-integration-with-prometheus\/","title":{"rendered":"AlertManager Integration with Prometheus"},"content":{"rendered":"<div dir=\"ltr\" style=\"text-align: left;\">\n<p>One day I got a call from one of my friend and he\u00a0said to me that he is facing difficulties while setting up AlertManager with Prometheus. Then, I observed that most of the people face such issues while establishing a connection between AlertManager and receiver such as E-mail, Slack etc.<\/p>\n<p>From there, I got motivation for writing this blog so AlertManager setup with Prometheus will be a piece of cake for everyone.<\/p>\n<p>If you are new to AlertManager I would suggest you go through with our\u00a0<b>Prometheus <\/b>blog.<\/p>\n<p style=\"text-align: left;\">What Actually AlertManager Is?<\/p>\n<p>AlertManager is used to handle alerts for client applications (like <b>Prometheus<\/b>). It also takes care of alerts deduplicating, grouping and then routes them to different receivers such as E-mail, Slack, Pager Duty.<\/p>\n<p>In this blog, we will only discuss on Slack and E-mail receivers.<\/p>\n<p>AlertManager can be configured via command-line flags and configuration file. While command line flags configure system parameters for AlertManager,\u00a0\u00a0the configuration file defines inhibition rules, notification routing, and notification receivers.<\/p>\n<p style=\"text-align: left;\">Architecture<\/p>\n<p>Here is a basic architecture of AlertManager with Prometheus.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*Zp5xEhpasFynSMP1\" width=\"640\" height=\"400\" \/><\/p>\n<p>This is how Prometheus architecture works:-<\/p>\n<ul style=\"text-align: left;\">\n<li>If you see in the above picture Prometheus is scraping the metrics from its client application(exporters).<\/li>\n<li>When the alert is generated then it pushes it to the AlertManager, later AlertManager validates the alerts groups on the basis of labels.<\/li>\n<li>and then forward it to the receivers\u00a0like Email or Slack.<\/li>\n<\/ul>\n<p>If you want to use a single AlertManager for multiple Prometheus server you can also do that. Then architecture will look like this:-<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*vsKAKG-v4tbk1H6O\" width=\"640\" height=\"326\" \/><\/p>\n<p style=\"text-align: left;\">Installation<\/p>\n<p>Installation part of AlertManager is not a fancy thing, we just simply\u00a0need to download the latest binary of AlertManager from <a href=\"https:\/\/github.com\/prometheus\/alertmanager\/releases\" target=\"_blank\" rel=\"noopener\">here<\/a>.<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">$ cd \/opt\/\r\n$ wget https:\/\/github.com\/prometheus\/alertmanager\/releases\/download\/v0.11.0\/alertmanager-0.11.0.linux-amd64.tar.gz<\/span><\/pre>\n<p>After downloading, let&#8217;s extract the files.<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">$ tar -xvzf alertmanager-0.11.0.linux-amd64.tar.gz<\/span><\/pre>\n<p>So we can start AlertManager from here as well but it is always a good practice to follow Linux directory structure.<\/p>\n<pre><code>$ mv alertmanager-0.11.0.linux-amd64\/alertmanager \/usr\/local\/bin\/<\/code><\/pre>\n<p style=\"text-align: left;\">\u00a0Configuration<\/p>\n<p>Once the tar file is extracted and binary file is placed at the right location then the configuration part will come. Although AlertManager extracted directory contains the configuration file as well but it is not of our use. So we will create our own configuration. Let&#8217;s start by creating a directory for configuration.<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">$ mkdir \/etc\/alertmanager\/<\/span><\/pre>\n<p>Then the configuration file will take place.<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">$ vim \/etc\/alertmanager\/alertmanager.yml<\/span><\/pre>\n<p>The configuration file for <b>Slack <\/b>will look like this:-<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">global:\r\n\r\n\r\n# The directory from which notification templates are read.\r\ntemplates:\r\n- '\/etc\/alertmanager\/template\/*.tmpl'\r\n\r\n# The root route on which each incoming alert enters.\r\nroute:\r\n  # The labels by which incoming alerts are grouped together. For example,\r\n  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would\r\n  # be batched into a single group.\r\n  group_by: ['alertname', 'cluster', 'service']\r\n\r\n  # When a new group of alerts is created by an incoming alert, wait at\r\n  # least 'group_wait' to send the initial notification.\r\n  # This way ensures that you get multiple alerts for the same group that start\r\n  # firing shortly after another are batched together on the first\r\n  # notification.\r\n  group_wait: 3s\r\n\r\n  # When the first notification was sent, wait 'group_interval' to send a batch\r\n  # of new alerts that started firing for that group.\r\n  group_interval: 5s\r\n\r\n  # If an alert has successfully been sent, wait 'repeat_interval' to\r\n  # resend them.\r\n  repeat_interval: 1m\r\n\r\n  # A default receiver\r\n  receiver: mail-receiver\r\n\r\n  # All the above attributes are inherited by all child routes and can\r\n  # overwritten on each.\r\n\r\n  # The child route trees.\r\n  routes:\r\n  - match:\r\n      service: node\r\n    receiver: mail-receiver\r\n\r\n    routes:\r\n    - match:\r\n        severity: critical\r\n      receiver: critical-mail-receiver\r\n\r\n  # This route handles all alerts coming from a database service. If there's\r\n  # no team to handle it, it defaults to the DB team.\r\n  - match:\r\n      service: database\r\n    receiver: mail-receiver\r\n    routes:\r\n    - match:\r\n        severity: critical\r\n      receiver: critical-mail-receiver\r\n\r\nreceivers:\r\n- name: 'mail-receiver'\r\n  slack_configs:\r\n  - api_url:  https:\/\/hooks.slack.com\/services\/T2AGPFQ9X\/B94D2LHHD\/jskljaganauheajao2\r\n<\/span><span style=\"font-size: 13.6px;\">    channel: '#prom-alert'<\/span>\r\n<span style=\"font-size: 13.6px;\">\r\n   - name: 'critical-mail-receiver'\r\n  slack_configs: \r\n<\/span>  \r\n  - api_url: <span style=\"font-size: 13.6px;\">  https:\/\/hooks.slack.com\/services\/T2AGPFQ9X\/B94D2LHHD\/abhajkaKajKaALALOPaaaJk<\/span> <span style=\"font-size: 13.6px;\"> channel: '#prom-alert'<\/span><\/pre>\n<p>You just have to replace the channel name and api_url of the Slack with your information.<\/p>\n<p>The configuration file for <b>E-mail <\/b>will look something like this:-<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">global:\r\n\r\ntemplates:\r\n- '\/etc\/alertmanager\/*.tmpl'\r\n# The root route on which each incoming alert enters.\r\nroute:\r\n  # default route if none match\r\n  receiver: alert-emailer\r\n\r\n  # The labels by which incoming alerts are grouped together. For example,\r\n  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would\r\n  # be batched into a single group.\r\n  # TODO:\r\n  group_by: ['alertname', 'priority']\r\n\r\n  # All the above attributes are inherited by all child routes and can\r\n  # overwritten on each.\r\n\r\nreceivers:\r\n- name: alert-emailer\r\n  email_configs:\r\n  - to: 'receiver@example.com'\r\n    send_resolved: false\r\n    from: 'sender@example.com'\r\n    smarthost: 'smtp.example.com:587'\r\n    auth_username: 'sender@example.com'\r\n    auth_password: 'IamPassword'\r\n    auth_secret: 'sender@example.com'\r\n    auth_identity: 'sender@example.com'<\/span>\r\n<\/pre>\n<p>In this configuration file, you need to update the sender and receiver mail details and the authorization password of the sender.<\/p>\n<p>Once the configuration part is done we just have to create a storage directory where AlertManger will store its data.<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">$ mkdir \/var\/lib\/alertmanager<\/span><\/pre>\n<p>Then only last piece which will be remaining is my favorite part i.e creating service \ud83d\ude42<\/p>\n<pre><code>$ vi \/etc\/systemd\/system\/alertmanager.service<\/code><\/pre>\n<p>The service file will look like this:-<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">[Unit]\r\nDescription=AlertManager Server Service\r\nWants=network-online.target\r\nAfter=network-online.target\r\n\r\n[Service]\r\nUser=root\r\nGroup=root\r\nType=Simple\r\nExecStart=\/usr\/local\/bin\/alertmanager \\\r\n    --config.file \/etc\/alertmanager\/alertmanager.yml \\<\/span>\r\n<span style=\"font-size: 13.6px;\">    --storage.tsdb.path \/var\/lib\/alertmanager<\/span><span style=\"font-size: 13.6px;\">\r\n\r\n[Install]\r\nWantedBy=multi-user.target<\/span><\/pre>\n<p>Then reload the daemon and start the service<\/p>\n<pre style=\"background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;\"><span style=\"font-size: 13.6px;\">$ systemctl daemon-reload\r\n$ systemctl start alertmanager\r\n$ systemctl enable alertmanager<\/span><\/pre>\n<p>Now you are all set to fire up your <b>monitoring<\/b> and <b>alerting<\/b>. So just take a beer and relax until Alert Manager notifies you for alerts. All the best!!!!<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>One day I got a call from one of my friend and he\u00a0said to me that he is facing difficulties while setting up AlertManager with Prometheus. Then, I observed that most of the people face such issues while establishing a connection between AlertManager and receiver such as E-mail, Slack etc. From there, I got motivation &hellip; <a href=\"https:\/\/opstree.com\/blog\/2019\/03\/05\/alertmanager-integration-with-prometheus\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;AlertManager Integration with Prometheus&#8221;<\/span><\/a><\/p>\n","protected":false},"author":89038429,"featured_media":29900,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[1],"tags":[237094284,768739308,768739293,676319245,768739287],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/11\/DevSecOps-1.jpg","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-2i","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/142"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/89038429"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=142"}],"version-history":[{"count":5,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/142\/revisions"}],"predecessor-version":[{"id":30039,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/142\/revisions\/30039"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29900"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=142"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=142"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=142"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}