Hi Guys !! Welcome to my blog where concepts are taught in a fun way. This is continuing part of series. In this we will try to understand Prometheus Architecture , Metrics and Instrumentation.

Prometheus Architecture ( Funny Version 😂)

Components of Prometheus ( Fun Version 😂)

Prometheus Server — Detective Prom’s Headquarters

In the middle of everything is the Prometheus Server — this is Detective Prom’s headquarters. It’s where all the data (clues) from different suspects (targets) is gathered, stored, and analyzed. Prom is constantly working here, retrieving metrics and storing them in the TSDB (Time-Series Database), which is like his extensive archive of case files.

Metrics Retrieval — Prom’s Daily Patrol

Prom goes on his daily patrols, checking in on different parts of the city (your infrastructure). He does this by visiting the /metrics endpoints of exporters (which are like Prom’s snitches), collecting data from them. This data includes all sorts of details about what’s going on in the city—CPU usage, memory stats, and more.

Exporters — The Snitches

The exporters are the snitches who gather information from various parts of the system (like apps, servers, etc.). They know everything that’s happening and report back to Prom via the /metrics endpoint. Prom regularly visits these snitches to collect all the juicy details.

Push Gateway — The Drop Point

Sometimes, Prom can’t always check on certain suspects because they only appear occasionally (like batch jobs). For those cases, Prom sets up a Push Gateway, which acts like a drop point. These batch jobs push their metrics to the Push Gateway, and Prom collects them when he’s available.

Service Discovery — Prom’s Network of Informants

Prom doesn’t manually keep track of all the suspects in the city. Instead, he has a network of informants, which includes Kubernetes, Consul, and File SD. These informants let him know when new suspects appear or when existing ones change locations, ensuring that Prom’s target list is always up to date.

Alert manager — Prom’s Sidekick

Prom isn’t just about collecting data — he’s also about taking action when something goes wrong. That’s where Alert manager comes in. If Prom sees something suspicious (like a server going down), he tells Alert manager, who then notifies the right people via Slack, email, or xmatters. These are the channels through which the SWAT team (DevOps engineers) gets alerted to solve the issue.

PromQL — Detective Prom’s Secret Code

When Prom needs to solve a case, he dives into his archives using PromQL, his secret code language. This helps him pull out exactly the information he needs from the tons of data stored in his TSDB.

Grafana — The Forensic Artist

Prom loves visual aids, so he works with Grafana, the forensic artist, to create detailed graphs and dashboards. Grafana takes all of Prom’s case files and turns them into visuals that are easy to understand. Prometheus’s Web UI and Grafana are where these visualizations live, making it easier to see patterns and trends in the data.

Node Local Storage — Prom’s Filing Cabinet

All the case files Prom collects need to be stored somewhere safe, so he uses Node Local Storage — his sturdy filing cabinet where all the data is kept in order.

API — Prom’s Hotline

Lastly, there’s an API that acts like Prom’s hotline. Other tools or services can call up Prom to get specific data whenever they need it, making it easy to integrate Prometheus with other systems.

Data Model (Time Series Data and Metrics)

Boring Version Definition 💤
In Prometheus, data is stored in the form of time series. A time series is essentially a sequence of data points, each associated with a specific timestamp. These data points are called metrics, which represent the state or performance of a system at a particular time.

Fun Version 😂
Imagine Prometheus as a seasoned detective (Detective Prom), and time series data as his case files. Here’s how it works in his world:

Key Concepts:

1. Metric Name — The Case Title: Every time Detective Prom opens a new investigation, he gives the case a title — this is the metric name. It tells Prom what he’s investigating, like “Number of Burglaries” (http_requests_total). This helps him quickly know what the case is about.
2. Labels — The Clues: Every case has clues, right? In Prom’s world, these clues are called labels. They help him figure out the specifics, like who did what, when, and how. For instance, in the “Number of Burglaries” case, a label could be method="Break-in" or status="Solved". These clues help Prom narrow down his investigation.
3. Samples — Detective Notes: As Prom gathers clues, he takes notes. Each note (sample) has a timestamp (the exact time the clue was discovered) and a value (the actual clue itself, like the number of burglaries that happened at that time). These notes are crucial for solving the case.
4. Storage — The Case Archive: Detective Prom keeps all his case files in a special archive (TSDB) that’s designed for quick access and retrieval. Whenever Prom needs to revisit an old case, he can pull out the relevant files in no time.
5. Querying — Prom’s Magnifying Glass: When Prom needs to dig deeper into his notes, he uses his special magnifying glass — this is PromQL. With it, he can zoom in on specific details, find patterns, and even predict future crimes (or in the real world, system issues).

Service Discovery in Prometheus

Boring Version 💤
Service Discovery in Prometheus is a mechanism used to automatically identify and monitor various targets (like servers, containers, or services) without needing manual configuration. Prometheus can dynamically discover targets based on their metadata and automatically start scraping metrics from them.

Key Concepts:

1. Static Configuration: In simpler setups, targets can be defined statically in the configuration file. This is fine for a small number of fixed targets but doesn’t scale well for dynamic environments.
2. Dynamic Service Discovery: In dynamic environments like Kubernetes, where targets can frequently change, Prometheus uses Service Discovery mechanisms to automatically find these targets. Supported methods include:

• Kubernetes: Automatically discovers all pods, services, and nodes within a Kubernetes cluster.
• Consul: Discovers services registered in Consul.
• File-based Discovery: Prometheus reads target information from files that can be dynamically updated.

3. Relabeling: After discovering targets, Prometheus can modify or filter the target list based on labels and other criteria through a process called relabeling.

4. Integration: This feature ensures that Prometheus can adapt to changes in infrastructure, maintaining up-to-date monitoring without manual intervention.

Now Comes My Version Fun Version 😂

Detective Prom and the City of Mischief: The Service Discovery Saga

In the bustling metropolis of Monitorville, Detective Prom was once the sharpest tool in the shed. Back in the day, he kept his eyes on a few notorious criminals (a.k.a targets) using his trusty old Rolodex — a dusty, creaky thing that he’d flip through while sipping his coffee. But Monitorville was just a sleepy little town back then, and there were only a few bad apples to watch over.

Static Configuration — Prom’s Dusty Old Rolodex:

Yup, Prom used to be old-school. He had every criminal’s name, photo, and favorite snack recorded in that Rolodex. If you wanted to know who the bank robbers were, just ask Prom — he’d flip through those cards, and boom, there’s your guy. But the problem? The town grew into a city, and suddenly Prom couldn’t keep up. Crooks were popping up faster than a whack-a-mole game at the carnival, and that Rolodex? It was smoking from all the flipping!

Dynamic Service Discovery — Prom’s New Secret Agent Squad:

Prom knew he needed to step up his game. So what did he do? He put together a squad of secret agents who would keep their ears to the ground and their eyes on every shady alley in the city. These agents are his Service Discovery mechanisms — and trust me, they’re good!

1. Kubernetes — Prom’s Right-Hand Spy: Kubernetes isn’t just any agent; this guy’s Prom’s top spy. He knows everything that goes on in the city — every gang (service) forming, every shady character (pod) hanging around. Kubernetes reports back to Prom like, “Boss, a new gang’s in town, calling themselves ‘The Microservices.’ They’re hanging out by the old docker warehouse.”
2. Consul — The City’s Gossip Queen: Then there’s Consul. Consul doesn’t miss a beat. She’s always at the heart of the city’s registry office, chatting up the clerks, eavesdropping on the water cooler talk. If a new business opens up or if an old one closes, Consul’s the first to know. She whispers the details straight to Prom: “Hey Prom, there’s a new service called ‘DataCooks’ that just registered. Looks like they’re up to something fishy!”
3. File-based Discovery — The Sneaky Snitch: Last but not least, there’s Prom’s snitch — File-based Discovery. This guy doesn’t talk much, but he leaves little notes for Prom in hidden files. Whenever a tip-off comes in, Prom knows where to look: “Psst, Prom, check the file behind the fridge. Word on the street is, there’s a new container hangout spot.”

Relabeling — The Master of Aliases:

Now, Prom isn’t just chasing after every two-bit crook. He’s a pro — he gives his targets codenames and aliases, only going after the ones who matter. He’s got no time for small fry. This is relabeling — or as Prom likes to call it, “tagging the real troublemakers.”

“Why chase ’em all when you can just focus on the top dogs? Call this guy ‘Fast Eddie’ and that one ‘Slick Rick.’ Ignore those low-level goons — they’re not worth the time.”

Integration — Prom Never Sleeps:

With his agents all over Monitorville, Prom’s got eyes everywhere. Whether a new crook shows up or an old one skips town, Prom knows it all, without lifting a finger. He’s got his feet up on the desk, sipping his coffee, but don’t be fooled — Prom never sleeps. His agents keep him in the loop 24/7, so his list of targets is always fresh and ready for action.

“Another day, another criminal watched,” Prom smirks, as his agents do all the hard work. Who said a detective’s life wasn’t cushy?

Note :- And that, my friend, is how Detective Prom stays on top of the ever-growing city of Monitorville, never losing track of who’s who in the world of monitoring. The Rolodex days are over, and now, it’s all about working smart with his undercover crew!

Metrics and Instrumentation

Understanding Metrics: Counters, Gauges, Histograms, and Summaries

Boring Version 💤
Metrics are essential in monitoring and understanding the behavior of your application. In Prometheus, metrics are divided into four types:

1. Counters: A Counter is a metric that only ever goes up. It’s used to count things like the number of requests served or errors encountered.
2. Gauges: A Gauge represents a value that can go up and down. It’s useful for tracking things like memory usage or the number of active users.
3. Histograms: A Histogram samples observations and counts them in configurable buckets. It’s great for measuring things like request latency over a period.
4. Summaries: A Summary is similar to a Histogram but provides quantiles like the median or the 90th percentile. It’s useful for precise tracking of things like response times.

Funny Version 😂
Imagine Prometheus as a coach of a quirky superhero team — each hero has their own unique power to keep the city of Monitorville safe and sound. Let’s meet the squad:

1. Counter the Climber: Counter is that one guy in the gym who only ever increases the weights on his barbell — never lowering them. He’s always counting upwards, like “One… Two… Three… A million!” Whether it’s counting how many bad guys got caught or how many donuts he’s eaten, Counter’s number always goes up and up!
2. Gauge the Yo-Yo Master: Gauge is a bit more unpredictable. One minute he’s at the top of his game, and the next, he’s dipping low like a yo-yo. He’s perfect for keeping track of things that fluctuate, like the number of people at the gym or how much coffee is left in the pot. Up, down, up, down — Gauge is all about that balance!
3. Histogram the Bucketeer: Histogram loves buckets — seriously, this guy can’t get enough of them! He collects observations and sorts them into buckets: “This many bad guys were slow, but ooh, look at how many were super fast!” He’s your go-to for figuring out how long things take, like how quickly the villain escapes or how long it takes to catch them.
4. Summary the Stat Guru: Summary is Histogram’s cousin, but she’s all about those fancy stats. She doesn’t just throw things into buckets — she calculates stuff like the median and the 90th percentile. “Let me tell you, 90% of the time, we catch the crooks within 5 minutes,” she boasts. She’s got the details nailed down!

Instrumenting Your Application with Prometheus Client Libraries

Boring Version 💤
To collect these metrics, you need to instrument your application. Prometheus provides client libraries for various programming languages (e.g., Go, Java, Python, Ruby). You can use these libraries to add metric collection directly into your application code. This involves defining and registering the metrics you want to track, and then updating these metrics at appropriate points in your code.

Funny Version 😂
Prom’s Gadget Shop
Now, Prometheus isn’t just any old detective — he’s got gadgets! To keep his superhero squad in check, he uses these cool tools called client libraries. They’re like the Bat-gadgets of the Prom world, allowing you to add superpowers directly into your app code. Imagine Prom using his gadget to track how many times Counter climbs a building or how often Gauge yo-yos.

You just have to tell these gadgets what to track — like counting how many times you’ve rewatched “Friends” or checking how often your coffee machine is on. The gadgets do the rest, sending the data back to Prom’s headquarters.

Best Practices for Instrumentation

Boring Version 💤

1. Instrument with a purpose: Only collect metrics that provide valuable insights.
2. Keep it simple: Avoid complex instrumentation that could impact application performance.
3. Label carefully: Use labels to distinguish different dimensions of your metrics, but don’t overuse them, as this can lead to high cardinality issues.
4. Test your metrics: Ensure that your metrics provide the intended information and are accurate.

Funny Version 😂: Prom’s Super Squad Training Rules

1. Train with a purpose: Prom tells his squad, “No unnecessary fights — focus on the crooks that matter!” Only track the important stuff.
2. Keep it simple, superheroes: Don’t overcomplicate things — if it takes too long to explain, it’s too much.
3. Label smartly, not excessively: Prom’s rule — every crook gets a label, but don’t label the entire city! Too many labels, and you’ll lose track of who’s who.
4. Test those gadgets: Before going into action, Prom makes sure his gadgets work perfectly — no one wants a grappling hook that doesn’t hook!

So Guys !! This was all about Day 2, I hope this blog will enhance your learning. Please like, share and subscribe.

Blog Pundit:  Sandeep Rawat

Opstree is an End to End DevOps solution provider