{"id":10799,"date":"2022-05-31T18:08:14","date_gmt":"2022-05-31T12:38:14","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=10799"},"modified":"2026-02-18T15:16:19","modified_gmt":"2026-02-18T09:46:19","slug":"azure-ha-kubernetes-monitoring-using-prometheus-and-thanos","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2022\/05\/31\/azure-ha-kubernetes-monitoring-using-prometheus-and-thanos\/","title":{"rendered":"Azure HA Kubernetes Monitoring using Prometheus and Thanos"},"content":{"rendered":"\r\n
\"\"<\/figure>\r\n\r\n\r\n\r\n

Introduction<\/strong><\/h3>\r\n\r\n\r\n\r\n

Long since Prometheus took on the role of monitoring the systems, it has been the undisputed open-source leader for monitoring and alerting in Kubernetes systems, it has become a go-to solution. While Prometheus does some general instructions for achieving high availability but it has limitations when it comes to data retention, historical data retrieval, and multi-tenancy. This is where Thanos comes into play. In this blog post, we will discuss how to integrate Thanos with Prometheus in Kubernetes environments and why one should choose a particular approach. So let\u2019s get started.<\/p>\r\n\r\n\r\n\r\n

<\/p>\r\n\r\n\r\n\r\n

Why Thanos<\/strong><\/h3>\r\n\r\n\r\n\r\n

The deployments in Prometheus are based on persistent volumes, which are scaled using federated set-ups. Federated methodologies are not applicable to all types of data. You will often need a different tool to manage Prometheus configurations. In this section, we will be using Thanos to handle these issues. Prometheus can be run on multiple instances in Thanos, data can be deduplicated, and data can be archived in long-term storage via GCS, Azure account storage, or S3.<\/p>\r\n\r\n\r\n\r\n

Thanos Architecture<\/strong><\/h3>\r\n\r\n\r\n\r\n
\"\"\r\n
Thanos Architecture<\/figcaption>\r\n<\/figure>\r\n\r\n\r\n\r\n

Thanos Components<\/strong><\/h3>\r\n\r\n\r\n\r\n
    \r\n
  1. Thanos Sidecar<\/li>\r\n
  2. Thanos Store<\/li>\r\n
  3. Thanos Query<\/li>\r\n
  4. Thanos Compact<\/li>\r\n
  5. Thanos Ruler<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n

    Thanos Implementation<\/h3>\r\n\r\n\r\n\r\n

    We\u2019ll need an Azure Storage Account, you can create one<\/a> using the Azure Portal or the Azure CLI. You will also need the storage account access key<\/a> which can also be retrieved using the Azure CLI.<\/p>\r\n\r\n\r\n\r\n

      \r\n
    1. (a) Create a storage account.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
      az storage account create --name <name> --resource-group <resource-group><\/pre>\r\n\r\n\r\n\r\n
        \r\n
      1. (b) Create a storage container called metrics<\/code>.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
        az storage container create \u2014 name metrics \u2014 account-name <name><\/pre>\r\n\r\n\r\n\r\n
          \r\n
        1. (c) Retrieve the storage account access key for later use.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
          az storage account keys list --account-name <name> --resource-group <resource-group> -o tsv --query \"[0].value\"<\/pre>\r\n\r\n\r\n\r\n
          2. Implementing Ingress Controller and Ingress objects (We will use Nginx Ingress Controller)<\/p>\r\n\r\n\r\n\r\n
          3. Creating credentials to be used by Thanos components to access object store (in this case, Azure bucket)<\/p>\r\n\r\n\r\n\r\n
          a. Create a Kubernetes secret using the credentials, as you can see in the following snippet:<\/p>\r\n\r\n\r\n\r\n
          kubectl -n monitoring create secret generic thanos-objstore-config --from-file=thanos.yaml=thanos.yaml<\/pre>\r\n\r\n\r\n\r\n
          Deployment<\/h3>\r\n\r\n\r\n\r\n
          Create a monitoring namespace, service accounts, cluster role, and cluster role bindings for Prometheus using the following manifest.<\/p>\r\n\r\n\r\n\r\n
          kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/sa.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
          Creating the Prometheus Configuration configmap<\/strong><\/h3>\r\n\r\n\r\n\r\n
          This configmap creates the Prometheus configuration file template that is read by the Thanos sidecar component. The template will also generate the actual configuration file. The file will be consumed by the Prometheus container running in the same pod. It is important to include the external_labels section in the configuration file so that the querier can use it to deduplicate data.<\/p>\r\n\r\n\r\n\r\n
          kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/prometheus-configmap.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
          Creating Prometheus Rules configmap<\/strong><\/h3>\r\n\r\n\r\n\r\n
          This will create alert rules that will be relayed to Alertmanager for delivery.<\/p>\r\n\r\n\r\n\r\n
          kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/prometheus-rules.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
          Creating Prometheus Stateful\u00a0Set<\/strong><\/h3>\r\n\r\n\r\n\r\n
          kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/prometheus-deployment.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
          It is important to understand the following about the above manifest:<\/p>\r\n\r\n\r\n\r\n
            \r\n
          1. Prometheus is deployed as a stateful set with three replicas. Each replica provisions its own persistent volume dynamically.<\/li>\r\n
          2. Prometheus configuration is generated by the Thanos Sidecar container using the template file created above.<\/li>\r\n
          3. Thanos handles data compaction and therefore we need to set\u200a\u2014\u200astorage.tsdb.min-block-duration=2h<\/strong> and\u200a\u2014\u200astorage.tsdb.max-block-duration=2h<\/strong><\/li>\r\n
          4. Prometheus stateful set is labeled as thanos-store-api: \u201ctrue\u201d<\/strong> so that each pod gets discovered by the headless service. This headless service will be used by Thanos Query<\/strong> to query data across all the Prometheus instances.<\/li>\r\n
          5. We apply the same label to the Thanos Store<\/strong> and Thanos Ruler <\/strong>components so that they are also discovered by the querier and can be used for querying metrics.<\/li>\r\n
          6. The Azure bucket credentials path is provided using the AZURE_APPLICATION_CREDENTIALS <\/strong>environment variable. The configuration file is mounted to that from the secret created as a part of the prerequisites.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
            Creating Prometheus Services<\/strong><\/h3>\r\n\r\n\r\n\r\n
            kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/prometheus-service.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
            Creating Thanos\u00a0Query<\/strong><\/h3>\r\n\r\n\r\n\r\n
            One of the key components of Thanos deployment<\/p>\r\n\r\n\r\n\r\n
              \r\n
            1. The container argument\u200a\u2014\u200astore=dnssrv+thanos-store-gateway:10901 assists in discovering all the components from which metric data should be queried.<\/li>\r\n
            2. Thanos-querier provides a web-based interface for running PromQL queries.. It also has the option to deduplicate data across various Prometheus clusters.<\/li>\r\n
            3. Here we provide Grafana as a data source for all dashboards.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
              kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/thanos-querier.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Creating Thanos Store\u00a0Gateway<\/strong><\/h3>\r\n\r\n\r\n\r\n
              This will create the store component which serves metrics from the object storage to the querier.<\/p>\r\n\r\n\r\n\r\n
              kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/thanos-store-gateway.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Creating Thanos\u00a0Compact<\/strong><\/h3>\r\n\r\n\r\n\r\n
              kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/thanos-compact.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Creating Thanos\u00a0Ruler<\/strong><\/h3>\r\n\r\n\r\n\r\n
              kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/thanos-ruler.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Creating Alertmanager<\/strong><\/h3>\r\n\r\n\r\n\r\n
              kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/alertmanger.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Creating Kube state\u00a0Metrics<\/strong><\/h3>\r\n\r\n\r\n\r\n
              In order to relay some important container metrics, Kubestate metrics deployment is required. The metrics are not exposed natively by the kubelet and are not directly accessible via Prometheus.<\/p>\r\n\r\n\r\n\r\n
              https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/kube-state-metrics.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Creating Node-exporter Daemon set<\/strong><\/h3>\r\n\r\n\r\n\r\n
              On each node, a node-exporter pod runs the daemon set. It exposes important metrics that can be retrieved by Prometheus instances.<\/p>\r\n\r\n\r\n\r\n
              kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/node-exporter.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
              Deploying Grafana<\/strong><\/h3>\r\n\r\n\r\n\r\n
              We will create our Grafana deployment and Service that will be exposed by means of our ingress object. We will add thanos-querier as the datasource for our Grafana deployment. In order to do so:<\/p>\r\n\r\n\r\n\r\n
                \r\n
              1. Click on Add DataSource<\/strong><\/li>\r\n
              2. Set Name: DS_PROMETHEUS<\/strong><\/li>\r\n
              3. Set Type: Prometheus<\/strong><\/li>\r\n
              4. Set URL: <\/strong>http:\/\/thanos-querier:9090<\/strong><\/li>\r\n
              5. Save and Test. You can now build your custom dashboards or simply import dashboards from grafana.net. Dashboards #315 and #1471 are very good places to start.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
                kubectl apply -f https:\/\/raw.githubusercontent.com\/v5arcus\/azure-ha-prometheus-thanos\/main\/grafana.yaml<\/a><\/pre>\r\n\r\n\r\n\r\n
                Creating the Ingress\u00a0Object<\/strong><\/h3>\r\n\r\n\r\n\r\n
                Here is the final component of the journey. As a result, we will be able to access all of our services outside of the Kubernetes cluster.<\/p>\r\n\r\n\r\n\r\n
                Your domain name should be changed to <yourdomain>.<\/p>\r\n\r\n\r\n\r\n
                apiVersion: extensions\/v1beta1\r\nkind: Ingress\r\nmetadata:\r\n  name: monitoring-ingress\r\n  namespace: monitoring\r\n  annotations:\r\n    kubernetes.io\/ingress.class: \"nginx\"\r\nspec:\r\n  rules:\r\n  - host: grafana.<yourdomain>.com\r\n    http:\r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: grafana\r\n          servicePort: 3000\r\n  - host: prometheus-0.<yourdomain>.com\r\n    http:\r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: prometheus-0-service\r\n          servicePort: 8080\r\n  - host: prometheus-1.<yourdomain>.com\r\n    http:\r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: prometheus-1-service\r\n          servicePort: 8080\r\n  - host: prometheus-2.<yourdomain>.com\r\n    http:\r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: prometheus-2-service\r\n          servicePort: 8080\r\n  - host: alertmanager.<yourdomain>.com\r\n    http: \r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: alertmanager\r\n          servicePort: 9093\r\n  - host: thanos-querier.<yourdomain>.com\r\n    http:\r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: thanos-querier\r\n          servicePort: 9090\r\n  - host: thanos-ruler.<yourdomain>.com\r\n    http:\r\n      paths:\r\n      - path: \/\r\n        backend:\r\n          serviceName: thanos-ruler\r\n          servicePort: 9090<\/pre>\r\n\r\n\r\n\r\n
                Access Thanos Querier at http:\/\/thanos-querier.<yourdomain>.com<\/p>\r\n\r\n\r\n\r\n
                Make sure deduplication is selected.<\/p>\r\n\r\n\r\n\r\n
                \"\"<\/figure>\r\n\r\n\r\n\r\n
                If you click on Stores<\/strong>, you will be able to see all the active endpoints discovered by thanos-store-gateway<\/strong>.<\/p>\r\n\r\n\r\n\r\n
                Grafana Dashboards<\/h3>\r\n\r\n\r\n\r\n
                Thanos Querier is then added as the data source in Grafana, and you can start creating dashboards.<\/p>\r\n\r\n\r\n\r\n
                Kubernetes Cluster Monitoring Dashboard:<\/p>\r\n\r\n\r\n\r\n
                \"\"<\/figure>\r\n\r\n\r\n\r\n
                \"\"<\/figure>\r\n\r\n\r\n\r\n
                Conclusion<\/h3>\r\n\r\n\r\n\r\n
                So in this blog, we have seen the limitations of the Prometheus High availability and how we tried to come up with a solution to overcome those limitations by using Thanos.<\/p>\r\n\r\n\r\n\r\n
                If you guys have any other ideas or suggestions around the approach, please comment in the comment section. Thanks for reading, I\u2019d really appreciate your suggestions and feedback.<\/p>\r\n\r\n\r\n\r\n
                References:-\u00a0
                https:\/\/thanos.io\/<\/a><\/p>\r\n\r\n\r\n\r\n

                Blog Pundit:<\/strong>                 \u00a0Naveen Verma<\/strong><\/a>, Sanjeev Pandey<\/strong> and Sandeep Rawat<\/strong><\/a><\/p>\r\n\r\n\r\n\r\n
                Opstree<\/a><\/strong> is an End to End DevOps solution provider<\/p>\r\n\r\n\r\n\r\n
                \r\n
                CONTACT US<\/a><\/div>\r\n<\/div>\r\n\r\n\r\n\r\n
                Connect Us <\/strong><\/p>\r\n\r\n\r\n\r\n