Author: Abhishek Dubey

I am Abhishek Dubey, a Senior Cloud Consultant at Opstree Solutions with a focus on cloud infrastructure, DevOps, and automation technologies. With extensive experience in cloud platforms like AWS, Azure, and Google Cloud, I specialize in building scalable and secure cloud environments that drive business efficiency. My expertise includes designing robust architectures, implementing CI/CD pipelines, and ensuring high availability and performance for enterprise applications.

Load Testing using AWS Distributed System – Part 1

In today’s fast-paced world, ensuring your application and business can handle growing users is very important. Whether you are building a Mobile application, Web application, or REST API it is important to understand how our system performs under pressure. Load testing is one of the ways we identify the bottlenecks, performance issues, and potential feature failures before the end user identifies it.

While consulting for an Edtech platform they highlighted an issue of having a dynamic and distributed load testing environment that can scale itself if we want to generate more load to the system and once the goal is achieved it should incur minimal cost to the organization since we are not leveraging a static provisioned environment.

AWS Distributed load testing system is a powerful automated setup provided by AWS that can be installed in our AWS cloud environment. This setup helps developers and testers generate real-world traffic at scale and test how well the system handles it. In this blog, we will talk about how the AWS Distributed Load Testing system works and how it can be leveraged to improve the reliability and performance of the system.

Continue reading “Load Testing using AWS Distributed System – Part 1”

Integration of Prometheus with Cortex

As we promised in our previous blog Prometheus as Scale – Part 1 that in our next blog we will be writing about the implementation part of Cortex with Prometheus, so here we are with our promise. But before going to the implementation part, we would suggest you guys go through our first blog to know the need for it.

Previously we talked that Prometheus is becoming a go-to option for people who want to implement event-based monitoring and alerting. The implementation and management of Prometheus are quite easy. But when we have a large infrastructure to monitor or the infrastructure has started to grow you require to scale monitoring solution as well.

A few days back we were also in a similar kind of situation where one of our client’s infrastructure was growing as per the need and they need a resilient, scalable, and reliable monitoring system. Since they were already using the Prometheus, so we explored our option and came across an interesting project called “Cortex“.

Continue reading “Integration of Prometheus with Cortex”

Cassandra to ScyllaDB Migration Without Any Downtime

Enterprises need multiple things to run their business successfully. One of the critical things is the data store they use for storing the data that applications and different analytics platforms will use. To ensure that business is healthy, companies need reliable databases, and according to their tech budget and expertise, they choose databases.

While working and consulting with an enterprise, they ran Cassandra to support their NoSQL data-store operations. Cassandra was working really well but now when the company grew, they needed help to support the NoSQL store. They experienced cascading latencies by Cassandra Hot’s partition as the traffic increased with the events and campaigns. Also, garbage collection was becoming the bottleneck because it heavily impacted the database performance, resulting in poor application performance. Also, one more significant reason was that they wanted to avoid managing the database and were looking for an expert company that could manage it for them without significant application changes.

Why ScyllaDB?

While exploring the different solutions for the database, ScyllaDB caught our attention. We were curious about the solution and did multiple proofs of concept on the ScyllaDB. We finally decided this would be the right choice for our environment and scale. A few primary reasons for our decision were:

Continuation Of Redis Throughput and Management

As promised in our previous blog on Redis Performance tunning and Best practices, we have explored more best practices and optimizations in Redis as a cache and database management system. This blog will share some new findings and optimizations we learned in our previous blog’s delta period.

We know that Redis is a high-speed and flexible data storage that can fulfill different cache and database requirements. But if a system is not configured and tested correctly, even a fast and reliable one can quickly become limited. Here we will talk about the different needs of Redis as a system and how we can optimize it further to fully use it.

So while consulting and collaborating with different Redis architects from Redis Labs, I learned different ways of designing a performance-grade, highly available, and secure Redis architecture. Based on my learning, I would like to categorize it into these dimensions:-

Right-sizing and deployment of Redis setup.
Proxy and connection pooling.
Use the correct data type for storing keys.
Sharding and replication strategy.

All Redis Setup Under 7 Minutes!

Redis is a popular open-source in-memory database that supports multiple data structures like strings, hashes, lists, and sets. But similar to other tools, we can scale standalone Redis to a particular extent, but not beyond that. That’s why we have a cluster mode setup in which we can scale Redis nodes horizontally and then distribute data among those nodes.

Generally, we categorize the Redis setup into three different types:

Standalone
Leader-Follower (Replication)
Leader-Leader(Sharding)

Standalone Setup

In a Standalone setup, the complexity is minimal, but we cannot scale the solution if the data increases. Also, the fail-over and high availability will not be supported inside it.