A blog site on our Real life experiences with various phases of DevOps starting from VCS, Build & Release, CI/CD, Cloud, Monitoring, Containerization.
Running a growing e-commerce platform like Opszilla is thrilling. You’re processing thousands of orders daily across the US and Canada, scaling infrastructure, and expanding into new markets. But amidst all that momentum, something starts to break: your data infrastructure and database performance.
At first, it’s subtle—slower queries, lagging reports, a few scaling hiccups. Then the real issue surfaces: you’re still running on MySQL 5.7, a version nearing its end-of-life in October 2023.
If you’re using AWS, you may have noticed data transfer fees being added to your expenses. These costs are often included in your cost and usage reports, but don’t be fooled, if they’re not monitored they can quickly add up and become a significant contributor to your AWS bills.
Many organizations may face unexpectedly high data transfer charges, which reach up to lakhs of rupees per year. To get a handle on these costs and potentially reduce them, it’s essential to get a clear picture of your data transfer costs and identify which resources are driving them.
Continue reading “How to Reduce AWS Data Transfer Costs: A CFO’s Guide to Cloud Savings”
Apache Hudi (Hadoop Upserts Deletes and Incrementals) is a powerful framework designed for managing large datasets on cloud storage systems, enabling efficient data ingestion, storage, and retrieval. One of the key features of Hudi is its support for two distinct storage types: Copy-On-Write (COW) and Merge-On-Read (MOR). Each of these storage strategies has unique characteristics and serves different use cases. In this blog, we will explore COW and MOR. Continue reading “Understanding COW and MOR in Apache Hudi: Choosing the Right Storage Strategy “
Uploading files using pre-signed URLs is a common technique for securely uploading files to cloud storage services like AWS S3. This approach allows you to generate a URL that includes authentication details and permissions, so users or applications can upload files directly to cloud storage without needing to have credentials for the storage service.
Here’s a step-by-step guide on how to implement file uploads using pre-signed URLs to a specific storage class, specifically with AWS S3. I’ll cover how to generate a pre-signed URL in Python and how to use it in Postman.
Continue reading “Uploading Files Using Pre-Signed URLs to a Specific Storage Class”
In this blog post, we’ll explore a real-world scenario where we need to migrate and replicate data between an EC2-hosted MySQL database and Amazon RDS (Relational Database Service) MySQL instance. This setup is critical for maintaining data integrity, ensuring high availability, and facilitating seamless scalability in a production environment. Continue reading “Use Case: Backup and Replication Setup Between EC2 MySQL and RDS MySQL”
