{"id":19179,"date":"2024-10-08T16:20:50","date_gmt":"2024-10-08T10:50:50","guid":{"rendered":"https:\/\/opstree.com\/blog\/?p=19179"},"modified":"2024-10-08T16:20:50","modified_gmt":"2024-10-08T10:50:50","slug":"prometheus-the-prom-king-part-2","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2024\/10\/08\/prometheus-the-prom-king-part-2\/","title":{"rendered":"Prometheus \u2014 The Prom King (Part 2)"},"content":{"rendered":"<p id=\"a96f\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\">Hi Guys !! Welcome to my blog where concepts are taught in a fun way. This is continuing part of series. In this we will try to understand Prometheus Architecture , Metrics and Instrumentation.<\/p>\n<p data-selectable-paragraph=\"\"><!--more--><\/p>\n<h1 id=\"e3b4\" class=\"ain aio aeb bf aip kv aiq kw kx ky air kz la lb ais lc ld le ait lf lg lh aiu li lj aiv bk\" data-selectable-paragraph=\"\">Prometheus Architecture ( Funny Version \ud83d\ude02)<\/h1>\n<figure class=\"ajl ajm ajn ajo ajp adu ado adp paragraph-image\">\n<div class=\"adv adw gh adx bh ady\" role=\"button\">\n<div class=\"ado adp aos\"><img loading=\"lazy\" decoding=\"async\" class=\"bh sw adz c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:770\/1*9ry9ArApOvqxYB9_74OZkg.gif\" alt=\"\" width=\"700\" height=\"394\" \/><\/div>\n<\/div><figcaption class=\"aot jg aou ado adp aov aow bf b bg z gb\" data-selectable-paragraph=\"\">Picture is Beautiful But not mine<\/figcaption><\/figure>\n<h1 id=\"135f\" class=\"ain aio aeb bf aip kv aiq kw kx ky air kz la lb ais lc ld le ait lf lg lh aiu li lj aiv bk\" data-selectable-paragraph=\"\"><strong class=\"al\">Components of Prometheus ( Fun Version \ud83d\ude02)<\/strong><\/h1>\n<h2 id=\"8298\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Prometheus Server \u2014 Detective Prom\u2019s Headquarters<\/h2>\n<p id=\"eac9\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">In the middle of everything is the\u00a0<strong class=\"ahu nu\">Prometheus Server<\/strong>\u00a0\u2014 this is Detective Prom\u2019s headquarters. It\u2019s where all the data (clues) from different suspects (targets) is gathered, stored, and analyzed. Prom is constantly working here, retrieving metrics and storing them in the\u00a0<strong class=\"ahu nu\">TSDB<\/strong>\u00a0(Time-Series Database), which is like his extensive archive of case files.<\/p>\n<h2 id=\"a2aa\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Metrics Retrieval \u2014 Prom\u2019s Daily Patrol<\/h2>\n<p id=\"a57a\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Prom goes on his daily patrols, checking in on different parts of the city (your infrastructure). He does this by visiting the\u00a0<code class=\"cv akj akk akl akb b\">\/metrics<\/code>\u00a0endpoints of\u00a0<strong class=\"ahu nu\">exporters<\/strong>\u00a0(which are like Prom&#8217;s snitches), collecting data from them. This data includes all sorts of details about what\u2019s going on in the city\u2014CPU usage, memory stats, and more.<\/p>\n<h2 id=\"7f29\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Exporters \u2014 The Snitches<\/h2>\n<p id=\"fc7d\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">The\u00a0<strong class=\"ahu nu\">exporters<\/strong>\u00a0are the snitches who gather information from various parts of the system (like apps, servers, etc.). They know everything that\u2019s happening and report back to Prom via the\u00a0<code class=\"cv akj akk akl akb b\">\/metrics<\/code>\u00a0endpoint. Prom regularly visits these snitches to collect all the juicy details.<\/p>\n<h2 id=\"697f\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Push Gateway \u2014 The Drop Point<\/h2>\n<p id=\"a96b\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Sometimes, Prom can\u2019t always check on certain suspects because they only appear occasionally (like batch jobs). For those cases, Prom sets up a\u00a0<strong class=\"ahu nu\">Push Gateway<\/strong>, which acts like a drop point. These batch jobs push their metrics to the Push Gateway, and Prom collects them when he\u2019s available.<\/p>\n<h2 id=\"93ff\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Service Discovery \u2014 Prom\u2019s Network of Informants<\/h2>\n<p id=\"78b6\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Prom doesn\u2019t manually keep track of all the suspects in the city. Instead, he has a network of informants, which includes\u00a0<strong class=\"ahu nu\">Kubernetes<\/strong>,\u00a0<strong class=\"ahu nu\">Consul<\/strong>, and\u00a0<strong class=\"ahu nu\">File SD<\/strong>. These informants let him know when new suspects appear or when existing ones change locations, ensuring that Prom\u2019s target list is always up to date.<\/p>\n<h2 id=\"a85b\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Alert manager \u2014 Prom\u2019s Sidekick<\/h2>\n<p id=\"9ce4\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Prom isn\u2019t just about collecting data \u2014 he\u2019s also about taking action when something goes wrong. That\u2019s where\u00a0<strong class=\"ahu nu\">Alert manager<\/strong>\u00a0comes in. If Prom sees something suspicious (like a server going down), he tells Alert manager, who then notifies the right people via\u00a0<strong class=\"ahu nu\">Slack<\/strong>,\u00a0<strong class=\"ahu nu\">email<\/strong>, or\u00a0<strong class=\"ahu nu\">xmatters<\/strong>. These are the channels through which the SWAT team (DevOps engineers) gets alerted to solve the issue.<\/p>\n<h2 id=\"9bc4\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">PromQL \u2014 Detective Prom\u2019s Secret Code<\/h2>\n<p id=\"98e8\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">When Prom needs to solve a case, he dives into his archives using\u00a0<strong class=\"ahu nu\">PromQL<\/strong>, his secret code language. This helps him pull out exactly the information he needs from the tons of data stored in his TSDB.<\/p>\n<h2 id=\"53f1\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Grafana \u2014 The Forensic Artist<\/h2>\n<p id=\"2b8c\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Prom loves visual aids, so he works with\u00a0<strong class=\"ahu nu\">Grafana<\/strong>, the forensic artist, to create detailed graphs and dashboards. Grafana takes all of Prom\u2019s case files and turns them into visuals that are easy to understand. Prometheus\u2019s Web UI and Grafana are where these visualizations live, making it easier to see patterns and trends in the data.<\/p>\n<h2 id=\"2762\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Node Local Storage \u2014 Prom\u2019s Filing Cabinet<\/h2>\n<p id=\"c71b\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">All the case files Prom collects need to be stored somewhere safe, so he uses\u00a0<strong class=\"ahu nu\">Node Local Storage<\/strong>\u00a0\u2014 his sturdy filing cabinet where all the data is kept in order.<\/p>\n<h2 id=\"e376\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">API \u2014 Prom\u2019s Hotline<\/h2>\n<p id=\"b805\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Lastly, there\u2019s an\u00a0<strong class=\"ahu nu\">API<\/strong>\u00a0that acts like Prom\u2019s hotline. Other tools or services can call up Prom to get specific data whenever they need it, making it easy to integrate Prometheus with other systems.<\/p>\n<h1 id=\"2920\" class=\"ain aio aeb bf aip kv aiq kw kx ky air kz la lb ais lc ld le ait lf lg lh aiu li lj aiv bk\" data-selectable-paragraph=\"\">Data Model (Time Series Data and Metrics)<\/h1>\n<p id=\"7b4f\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Boring Version Definition \ud83d\udca4<br \/>\n<\/strong>In Prometheus, data is stored in the form of time series. A time series is essentially a sequence of data points, each associated with a specific timestamp. These data points are called\u00a0<strong class=\"ahu nu\">metrics<\/strong>, which represent the state or performance of a system at a particular time.<\/p>\n<p id=\"166e\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Fun Version \ud83d\ude02<br \/>\n<\/strong>Imagine Prometheus as a seasoned detective (Detective Prom), and time series data as his\u00a0<strong class=\"ahu nu\">case files<\/strong>. Here\u2019s how it works in his world:<\/p>\n<h2 id=\"61f0\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Key Concepts:<\/h2>\n<ol class=\"\">\n<li id=\"1422\" class=\"ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Metric Name \u2014 The Case Title:<\/strong>\u00a0Every time Detective Prom opens a new investigation, he gives the case a title \u2014 this is the\u00a0<strong class=\"ahu nu\">metric name<\/strong>. It tells Prom what he\u2019s investigating, like \u201cNumber of Burglaries\u201d (<code class=\"cv akj akk akl akb b\">http_requests_total<\/code>). This helps him quickly know what the case is about.<\/li>\n<li id=\"d761\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Labels \u2014 The Clues:<\/strong>\u00a0Every case has clues, right? In Prom\u2019s world, these clues are called\u00a0<strong class=\"ahu nu\">labels<\/strong>. They help him figure out the specifics, like who did what, when, and how. For instance, in the \u201cNumber of Burglaries\u201d case, a label could be\u00a0<code class=\"cv akj akk akl akb b\">method=\"Break-in\"<\/code>\u00a0or\u00a0<code class=\"cv akj akk akl akb b\">status=\"Solved\"<\/code>. These clues help Prom narrow down his investigation.<\/li>\n<li id=\"7d24\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Samples \u2014 Detective Notes:<\/strong>\u00a0As Prom gathers clues, he takes\u00a0<strong class=\"ahu nu\">notes<\/strong>. Each note (sample) has a\u00a0<strong class=\"ahu nu\">timestamp<\/strong>\u00a0(the exact time the clue was discovered) and a\u00a0<strong class=\"ahu nu\">value<\/strong>\u00a0(the actual clue itself, like the number of burglaries that happened at that time). These notes are crucial for solving the case.<\/li>\n<li id=\"6138\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Storage \u2014 The Case Archive:<\/strong>\u00a0Detective Prom keeps all his case files in a special archive (TSDB) that\u2019s designed for quick access and retrieval. Whenever Prom needs to revisit an old case, he can pull out the relevant files in no time.<\/li>\n<li id=\"dc7b\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Querying \u2014 Prom\u2019s Magnifying Glass:<\/strong>\u00a0When Prom needs to dig deeper into his notes, he uses his special\u00a0<strong class=\"ahu nu\">magnifying glass<\/strong>\u00a0\u2014 this is PromQL. With it, he can zoom in on specific details, find patterns, and even predict future crimes (or in the real world, system issues).<\/li>\n<\/ol>\n<h1 id=\"0ceb\" class=\"ain aio aeb bf aip kv aiq kw kx ky air kz la lb ais lc ld le ait lf lg lh aiu li lj aiv bk\" data-selectable-paragraph=\"\">Service Discovery in Prometheus<\/h1>\n<p id=\"39f1\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Boring Version \ud83d\udca4<br \/>\n<\/strong>Service Discovery in Prometheus is a mechanism used to automatically identify and monitor various targets (like servers, containers, or services) without needing manual configuration. Prometheus can dynamically discover targets based on their metadata and automatically start scraping metrics from them.<\/p>\n<h2 id=\"49b3\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Key Concepts:<\/h2>\n<ol class=\"\">\n<li id=\"9d24\" class=\"ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Static Configuration:<\/strong>\u00a0In simpler setups, targets can be defined statically in the configuration file. This is fine for a small number of fixed targets but doesn\u2019t scale well for dynamic environments.<\/li>\n<li id=\"bab5\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Dynamic Service Discovery:<\/strong>\u00a0In dynamic environments like Kubernetes, where targets can frequently change, Prometheus uses Service Discovery mechanisms to automatically find these targets. Supported methods include:<\/li>\n<\/ol>\n<ul class=\"\">\n<li id=\"ae9b\" class=\"ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim ajq ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Kubernetes:<\/strong>\u00a0Automatically discovers all pods, services, and nodes within a Kubernetes cluster.<\/li>\n<li id=\"4b1c\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim ajq ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Consul:<\/strong>\u00a0Discovers services registered in Consul.<\/li>\n<li id=\"ae0b\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim ajq ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">File-based Discovery:<\/strong>\u00a0Prometheus reads target information from files that can be dynamically updated.<\/li>\n<\/ul>\n<p id=\"9cdd\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">3. Relabeling:<\/strong>\u00a0After discovering targets, Prometheus can modify or filter the target list based on labels and other criteria through a process called\u00a0<strong class=\"ahu nu\">relabeling<\/strong>.<\/p>\n<p id=\"1e89\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">4. Integration:<\/strong>\u00a0This feature ensures that Prometheus can adapt to changes in infrastructure, maintaining up-to-date monitoring without manual intervention.<\/p>\n<h2 id=\"fe4c\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Now Comes My Version Fun Version\u00a0<strong class=\"al\">\ud83d\ude02<\/strong><\/h2>\n<h2 id=\"d24f\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Detective Prom and the City of Mischief: The Service Discovery Saga<\/h2>\n<p id=\"072a\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">In the bustling metropolis of Monitorville, Detective Prom was once the sharpest tool in the shed. Back in the day, he kept his eyes on a few notorious criminals (a.k.a targets) using his trusty old Rolodex \u2014 a dusty, creaky thing that he\u2019d flip through while sipping his coffee. But Monitorville was just a sleepy little town back then, and there were only a few bad apples to watch over.<\/p>\n<h2 id=\"97c0\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Static Configuration \u2014 Prom\u2019s Dusty Old Rolodex:<\/h2>\n<p id=\"6683\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Yup, Prom used to be old-school. He had every criminal\u2019s name, photo, and favorite snack recorded in that Rolodex. If you wanted to know who the bank robbers were, just ask Prom \u2014 he\u2019d flip through those cards, and boom, there\u2019s your guy. But the problem? The town grew into a city, and suddenly Prom couldn\u2019t keep up. Crooks were popping up faster than a whack-a-mole game at the carnival, and that Rolodex? It was smoking from all the flipping!<\/p>\n<h2 id=\"357d\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Dynamic Service Discovery \u2014 Prom\u2019s New Secret Agent Squad:<\/h2>\n<p id=\"9381\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Prom knew he needed to step up his game. So what did he do? He put together a squad of secret agents who would keep their ears to the ground and their eyes on every shady alley in the city. These agents are his\u00a0<strong class=\"ahu nu\">Service Discovery mechanisms<\/strong>\u00a0\u2014 and trust me, they\u2019re good!<\/p>\n<ol class=\"\">\n<li id=\"df30\" class=\"ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Kubernetes \u2014 Prom\u2019s Right-Hand Spy:<\/strong>\u00a0Kubernetes isn\u2019t just any agent; this guy\u2019s Prom\u2019s top spy. He knows everything that goes on in the city \u2014 every gang (service) forming, every shady character (pod) hanging around. Kubernetes reports back to Prom like, \u201cBoss, a new gang\u2019s in town, calling themselves \u2018The Microservices.\u2019 They\u2019re hanging out by the old docker warehouse.\u201d<\/li>\n<li id=\"bbce\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Consul \u2014 The City\u2019s Gossip Queen:<\/strong>\u00a0Then there\u2019s Consul. Consul doesn\u2019t miss a beat. She\u2019s always at the heart of the city\u2019s registry office, chatting up the clerks, eavesdropping on the water cooler talk. If a new business opens up or if an old one closes, Consul\u2019s the first to know. She whispers the details straight to Prom: \u201cHey Prom, there\u2019s a new service called \u2018DataCooks\u2019 that just registered. Looks like they\u2019re up to something fishy!\u201d<\/li>\n<li id=\"7e25\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">File-based Discovery \u2014 The Sneaky Snitch:<\/strong>\u00a0Last but not least, there\u2019s Prom\u2019s snitch \u2014 File-based Discovery. This guy doesn\u2019t talk much, but he leaves little notes for Prom in hidden files. Whenever a tip-off comes in, Prom knows where to look: \u201cPsst, Prom, check the file behind the fridge. Word on the street is, there\u2019s a new container hangout spot.\u201d<\/li>\n<\/ol>\n<h2 id=\"589e\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Relabeling \u2014 The Master of Aliases:<\/h2>\n<p id=\"be6c\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">Now, Prom isn\u2019t just chasing after every two-bit crook. He\u2019s a pro \u2014 he gives his targets codenames and aliases, only going after the ones who matter. He\u2019s got no time for small fry. This is\u00a0<strong class=\"ahu nu\">relabeling<\/strong>\u00a0\u2014 or as Prom likes to call it, \u201ctagging the real troublemakers.\u201d<\/p>\n<p id=\"de55\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\">\u201cWhy chase \u2019em all when you can just focus on the top dogs? Call this guy \u2018Fast Eddie\u2019 and that one \u2018Slick Rick.\u2019 Ignore those low-level goons \u2014 they\u2019re not worth the time.\u201d<\/p>\n<h2 id=\"278d\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Integration \u2014 Prom Never Sleeps:<\/h2>\n<p id=\"5eaa\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\">With his agents all over Monitorville, Prom\u2019s got eyes everywhere. Whether a new crook shows up or an old one skips town, Prom knows it all, without lifting a finger. He\u2019s got his feet up on the desk, sipping his coffee, but don\u2019t be fooled \u2014 Prom never sleeps. His agents keep him in the loop 24\/7, so his list of targets is always fresh and ready for action.<\/p>\n<p id=\"dea9\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\">\u201cAnother day, another criminal watched,\u201d Prom smirks, as his agents do all the hard work. Who said a detective\u2019s life wasn\u2019t cushy?<\/p>\n<p id=\"698b\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Note :-\u00a0<\/strong>And that, my friend, is how Detective Prom stays on top of the ever-growing city of Monitorville, never losing track of who\u2019s who in the world of monitoring. The Rolodex days are over, and now, it\u2019s all about working smart with his undercover crew!<\/p>\n<h1 id=\"9d96\" class=\"ain aio aeb bf aip kv aiq kw kx ky air kz la lb ais lc ld le ait lf lg lh aiu li lj aiv bk\" data-selectable-paragraph=\"\"><strong class=\"al\">Metrics and Instrumentation<\/strong><\/h1>\n<h2 id=\"70c4\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\"><strong class=\"al\">Understanding Metrics: Counters, Gauges, Histograms, and Summaries<\/strong><\/h2>\n<p id=\"528c\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Boring Version \ud83d\udca4<br \/>\n<\/strong>Metrics are essential in monitoring and understanding the behavior of your application. In Prometheus, metrics are divided into four types:<\/p>\n<ol class=\"\">\n<li id=\"397e\" class=\"ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Counters<\/strong>: A Counter is a metric that only ever goes up. It\u2019s used to count things like the number of requests served or errors encountered.<\/li>\n<li id=\"49bb\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Gauges<\/strong>: A Gauge represents a value that can go up and down. It\u2019s useful for tracking things like memory usage or the number of active users.<\/li>\n<li id=\"c517\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Histograms<\/strong>: A Histogram samples observations and counts them in configurable buckets. It\u2019s great for measuring things like request latency over a period.<\/li>\n<li id=\"eb54\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Summaries<\/strong>: A Summary is similar to a Histogram but provides quantiles like the median or the 90th percentile. It\u2019s useful for precise tracking of things like response times.<\/li>\n<\/ol>\n<p id=\"1af9\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Funny Version \ud83d\ude02<br \/>\n<\/strong>Imagine Prometheus as a coach of a quirky superhero team \u2014 each hero has their own unique power to keep the city of Monitorville safe and sound. Let\u2019s meet the squad:<\/p>\n<ol class=\"\">\n<li id=\"73ab\" class=\"ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Counter the Climber<\/strong>: Counter is that one guy in the gym who only ever increases the weights on his barbell \u2014 never lowering them. He\u2019s always counting upwards, like \u201cOne\u2026 Two\u2026 Three\u2026 A million!\u201d Whether it\u2019s counting how many bad guys got caught or how many donuts he\u2019s eaten, Counter\u2019s number always goes up and up!<\/li>\n<li id=\"e751\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Gauge the Yo-Yo Master<\/strong>: Gauge is a bit more unpredictable. One minute he\u2019s at the top of his game, and the next, he\u2019s dipping low like a yo-yo. He\u2019s perfect for keeping track of things that fluctuate, like the number of people at the gym or how much coffee is left in the pot. Up, down, up, down \u2014 Gauge is all about that balance!<\/li>\n<li id=\"bbaf\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Histogram the Bucketeer<\/strong>: Histogram loves buckets \u2014 seriously, this guy can\u2019t get enough of them! He collects observations and sorts them into buckets: \u201cThis many bad guys were slow, but ooh, look at how many were super fast!\u201d He\u2019s your go-to for figuring out how long things take, like how quickly the villain escapes or how long it takes to catch them.<\/li>\n<li id=\"38ca\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Summary the Stat Guru<\/strong>: Summary is Histogram\u2019s cousin, but she\u2019s all about those fancy stats. She doesn\u2019t just throw things into buckets \u2014 she calculates stuff like the median and the 90th percentile. \u201cLet me tell you, 90% of the time, we catch the crooks within 5 minutes,\u201d she boasts. She\u2019s got the details nailed down!<\/li>\n<\/ol>\n<h2 id=\"8e1c\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Instrumenting Your Application with Prometheus Client Libraries<\/h2>\n<p id=\"0d86\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Boring Version \ud83d\udca4<br \/>\n<\/strong>To collect these metrics, you need to instrument your application. Prometheus provides client libraries for various programming languages (e.g., Go, Java, Python, Ruby). You can use these libraries to add metric collection directly into your application code. This involves defining and registering the metrics you want to track, and then updating these metrics at appropriate points in your code.<\/p>\n<p id=\"cc65\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Funny Version \ud83d\ude02<br \/>\nProm\u2019s Gadget Shop<br \/>\n<\/strong>Now, Prometheus isn\u2019t just any old detective \u2014 he\u2019s got gadgets! To keep his superhero squad in check, he uses these cool tools called\u00a0<strong class=\"ahu nu\">client libraries<\/strong>. They\u2019re like the Bat-gadgets of the Prom world, allowing you to add superpowers directly into your app code. Imagine Prom using his gadget to track how many times Counter climbs a building or how often Gauge yo-yos.<\/p>\n<p id=\"9bdc\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\">You just have to tell these gadgets what to track \u2014 like counting how many times you\u2019ve rewatched \u201cFriends\u201d or checking how often your coffee machine is on. The gadgets do the rest, sending the data back to Prom\u2019s headquarters.<\/p>\n<h2 id=\"5cc1\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Best Practices for Instrumentation<\/h2>\n<p id=\"51a4\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Boring Version \ud83d\udca4<\/strong><\/p>\n<ol class=\"\">\n<li id=\"facf\" class=\"ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Instrument with a purpose<\/strong>: Only collect metrics that provide valuable insights.<\/li>\n<li id=\"67ba\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Keep it simple<\/strong>: Avoid complex instrumentation that could impact application performance.<\/li>\n<li id=\"9545\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Label carefully<\/strong>: Use labels to distinguish different dimensions of your metrics, but don\u2019t overuse them, as this can lead to high cardinality issues.<\/li>\n<li id=\"b94b\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Test your metrics<\/strong>: Ensure that your metrics provide the intended information and are accurate.<\/li>\n<\/ol>\n<h2 id=\"c454\" class=\"akm aio aeb bf aip tf akn io kx th ako ir la nz akp iu oc od akq ix og oh akr ja ok aks bk\" data-selectable-paragraph=\"\">Funny Version \ud83d\ude02: Prom\u2019s Super Squad Training Rules<\/h2>\n<ol class=\"\">\n<li id=\"af82\" class=\"ahs aht aeb ahu b ahv aiw ahx ahy ahz aix aib aic nz aiy aie aif od aiz aih aii oh aja aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Train with a purpose<\/strong>: Prom tells his squad, \u201cNo unnecessary fights \u2014 focus on the crooks that matter!\u201d Only track the important stuff.<\/li>\n<li id=\"22c2\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Keep it simple, superheroes<\/strong>: Don\u2019t overcomplicate things \u2014 if it takes too long to explain, it\u2019s too much.<\/li>\n<li id=\"b31c\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Label smartly, not excessively<\/strong>: Prom\u2019s rule \u2014 every crook gets a label, but don\u2019t label the entire city! Too many labels, and you\u2019ll lose track of who\u2019s who.<\/li>\n<li id=\"8e1f\" class=\"ahs aht aeb ahu b ahv ajt ahx ahy ahz aju aib aic nz ajv aie aif od ajw aih aii oh ajx aik ail aim akt ajr ajs bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">Test those gadgets<\/strong>: Before going into action, Prom makes sure his gadgets work perfectly \u2014 no one wants a grappling hook that doesn\u2019t hook!<\/li>\n<\/ol>\n<p id=\"661f\" class=\"pw-post-body-paragraph ahs aht aeb ahu b ahv ahw ahx ahy ahz aia aib aic nz aid aie aif od aig aih aii oh aij aik ail aim nl bk\" data-selectable-paragraph=\"\"><strong class=\"ahu nu\">So Guys !! This was all about Day 2, I hope this blog will enhance your learning. Please like, share and subscribe.<\/strong><\/p>\n<p><strong>Blog Pundit:<\/strong>\u00a0 <a href=\"https:\/\/opstree.com\/blog\/\/author\/sandeep7c51ad81ba\/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Sandeep Rawat<\/strong><\/a><\/p>\n<p><strong><a href=\"https:\/\/www.opstree.com\/contact-us?utm_source=wordpress&amp;utm_campaign=Learn-everything-about-Amazon-Cloudfront&amp;utm_id=Blog\">Opstree<\/a><\/strong> is an End to End DevOps solution provider<\/p>\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button is-style-fill\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.opstree.com\/contact-us\" target=\"_blank\" rel=\"noreferrer noopener\">CONTACT US<\/a><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Hi Guys !! Welcome to my blog where concepts are taught in a fun way. This is continuing part of series. In this we will try to understand Prometheus Architecture , Metrics and Instrumentation.<\/p>\n","protected":false},"author":244582682,"featured_media":19230,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474],"tags":[223796770,768739287],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/09\/Prometheus-\u2014-The-Prom-King-Part-2.png","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-4Zl","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/19179"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/244582682"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=19179"}],"version-history":[{"count":3,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/19179\/revisions"}],"predecessor-version":[{"id":19231,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/19179\/revisions\/19231"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/19230"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=19179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=19179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=19179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}