When we talk about security, AWS IAM is one of the most fundamental & critical AWS service which needs suitable observation to design it because any careless & negligible exercise leads to huge complication & misshaping. AWS IAM is one of a kind which is ignored at the starting means not organise properly which leads to enormous complications while managing access to resources. Most of the time, a proper technique of managing AWS IAM access is ignored or doesn’t documented or configured properly due to which team has to change the IAM management format every time when there is a new requirement or modification.
Continue reading “AWS IAM: The challenge”Tag: DevOps
Event Monitoring Using AWS CloudTrail
Introduction
If you are using cloud based services, it is evident and paramount to track events that have happened. Isn’t it?
Monitoring events in the cloud is important.
If you are using AWS, let’s assume you find that one autoscaling group in your AWS account is deleted. What will be your response?
How will you know who did it?
Postgres – CIS Benchmark
We have seen many security incidents. Any breach in security cause concern among enterprises. To be honest it not only concern them, it also gives birth to their nightmare, distrust and scepticism as organisation. The root cause of this distrust is improper implementation and configuration.
Opstree Security has started a new initiative where we rigorously analyse and implement CIS Benchmark of every tools being used today.
In this CIS series, we will discuss the CIS Benchmarks of PostgreSQL. Continue reading “Postgres – CIS Benchmark”
DevSecOps Diary | HIPAA Compliance
HIPAA stands for Health Insurance Portability and Accountability Act. This act of 1996 is a United States federal statute enactment. It is a series of regulatory standards that outline the lawful use and disclosure of protected health information (PHI). HIPAA compliance is regulated by the Department of Health and Human Services (HHS) and enforced by the Office for Civil Rights (OCR).
But what this ACT has to do with DevOps..? Is it related to the Corona Virus..?
No, not at all..! Let me explain to you how I landed here.
Prometheus at Scale – Part 1
Prometheus has gained a lot of popularity because of its cloud-native approach for monitoring systems. Its popularity has reached a level that people are now giving native support to it, while developing software and applications such as Kubernetes, Envoy, etc. For other applications, there are already exporters(agent) available to monitor it.
Since I have been working on Prometheus for quite a long time and recently have started doing development on it, I was confident that I can handle any kind of scenario in it. Here, in this blog, I am going to discuss a scenario that was a very good learning experience for me.
One thing I love about working with a service-based organization is that it keeps you on your toes, so you have to learn constantly. The same is the case with the current organization I am associated with.
Recently I got an opportunity to work on a project in which the client had a requirement of implementing a Prometheus HA solution. Here is a brief information about the requirement:-
- They had a 100+ node Kubernetes cluster and they wanted to keep the data for a longer period. Moreover, the storage on the node was a blocker for them.
- In the case of Prometheus failure, they didn’t have a backup plan ready.
- They needed the scaling solution for Prometheus as well.
Our Solution
So, we started with our research for the best possible scenarios, for the HA part, we thought we can implement the Federated Prometheus concept and for long-term storage, we thought of implementing the Thanos project. But while doing the research, we came across one more interesting project called Cortex.
So, we did our comparison between Thanos and Cortex. Here are some interesting highlights:-
| Cortex | Thanos |
| Recent data stored in injestors | Recent data stored in Prometheus |
| Use Prometheus write API to write data at a remote location | Use sidecar approach to write data at a remote location |
| Supports Long Term storage | Supports Long term storage |
| HA is supported | HA is not supported |
| Single setup can be integrated with multiple Prometheus | Single setup can be associated with single Prometheus |
So after this metrics comparison, we decided to go with the Cortex solution as it was able to fulfill the above mentioned requirements of the client.
But the cortex solutions is not free of complications, there are some complications of cortex project as well:-
- As the architecture is a bit complex, it requires an in-depth understanding of Prometheus as TSDB.
- These projects require a decent amount of computing power in terms of memory and CPU.
- It can increase your remote storage costs like S3, GCS, Azure storage, etc.
Since all these complications were not blockers for us, so we moved ahead with the Cortex approach and implemented it in the project and it started working fine right from day one.
But in terms of scaling, we have to scale Prometheus vertically not horizontally because it is not designed to scale horizontally.
If we try to scale Prometheus horizontally, we will end up with scattered data that cannot be consolidated easily, so in terms of the scaling part, we would suggest you go with a vertical approach.
To automate the vertical scaling of Prometheus in Kubernetes we have used VPA(Vertical Pod Autoscaler). It can both down-scale pods that are over-requesting resources, and also up-scale pods that are under-requesting resources based on their usage over time.
Conclusion
So in this blog, we have seen that what approach we have taken for implementing the High Availability, Scalability, and Long Term storage in Prometheus. In the next part of the blog, we will see how we actually setup these things in our environment.
If you guys have any other ideas or suggestions around the approach, please comment in the comment section. Thanks for reading, I’d really appreciate your suggestions and feedback.