{"id":29443,"date":"2025-07-29T16:46:28","date_gmt":"2025-07-29T11:16:28","guid":{"rendered":"https:\/\/opstree.com\/blog\/?p=29443"},"modified":"2025-07-30T20:12:41","modified_gmt":"2025-07-30T14:42:41","slug":"zero-downtime-mysql-migration","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2025\/07\/29\/zero-downtime-mysql-migration\/","title":{"rendered":"How OpsZilla Achieved Zero-Downtime MySQL Migration with Scalable Data Engineering\u00a0Practices"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Running a growing e-commerce platform like Opszilla is thrilling. You\u2019re processing thousands of orders daily across the US and Canada, scaling infrastructure, and expanding into new markets. But amidst all that momentum, something\u00a0 starts to break: your <strong>data infrastructure and database performance<\/strong>.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">At first, it\u2019s subtle\u2014slower queries, lagging reports, a few scaling hiccups. Then the real issue surfaces: you\u2019re still running on MySQL 5.7, a version nearing its end-of-life in October 2023.<\/span><\/p>\n<p><!--more--><\/p>\n<p><span style=\"font-weight: 400;\">Suddenly, you\u2019re not just dealing with performance issues\u2014you\u2019re staring down a serious risk. No more updates. No more security patches. No future support. For a business built on real-time transactions, that\u2019s not just inconvenient\u2014it\u2019s unacceptable.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This blog dives into a real-world scenario at Opszilla to unpack why upgrading to MySQL 8.0 isn\u2019t just a technical decision\u2014it\u2019s a business-critical move. We\u2019ll explore how Opszilla upgraded their MySQL 5.7 databases to MySQL 8.0.<\/span><\/p>\n<p>This real-world case from OpsZilla highlights how effective <a href=\"https:\/\/opstree.com\/services\/middleware-database-and-data-engineering\/\"><strong>data engineering services<\/strong><\/a> and database modernization can mitigate risk and enhance system performance<\/p>\n<h2>Why MySQL 8.0 Migration Matters in Modern Data Engineering<\/h2>\n<h3><b>Migration Challenges (And How OpsZilla Solved Them)<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Upgrading a live database while maintaining uptime, performance, and <a href=\"https:\/\/opstree.com\/blog\/2024\/10\/08\/etl-vs-elt-which-data-integration-approach-is-right-for-you\/\">data integrity<\/a> isn\u2019t easy. OpsZilla faced a series of technical challenges\u2014each of which had to be solved before the upgrade could be safely executed.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Zero Downtime Was Non-Negotiable<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Even a few seconds of interruption could compromise transaction integrity, disrupt customer experience, or impact live analytics dashboards. The challenge was clear: migrate safely, quickly, and invisibly.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">OpsZilla&#8217;s infrastructure spanned multiple environments. Some production clusters ran on a standalone <a href=\"https:\/\/opstree.com\/blog\/2023\/10\/24\/my-learning-in-migration-of-mysql-from-5-7-to-8-0\/\">MySQL<\/a> server without replicas, while others had a primary node paired with one or more read replicas.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Each presented its own risks and required a tailored migration strategy:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Scenario 1: Primary database only (no existing replica)<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Scenario 2: Primary with existing secondary replica<\/span><\/li>\n<\/ul>\n<h4><b>Scenario 1: Primary Database Only (No Existing Replica)<\/b><\/h4>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A single MySQL 5.7 primary server with no replicas.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Directly upgrading in-place risked downtime and performance bottlenecks.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Large databases could cause mysqldump to strain the server if not optimized.<\/span><\/li>\n<\/ul>\n<p><b>Solution: Minimizing Contention with Chunked Backups:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">To avoid overloading the primary server during the backup process, we used a chunked mysqldump strategy with transaction consistency:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-29444 size-full\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master.jpg\" alt=\"MySQL 5.7 primary server\" width=\"800\" height=\"512\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master.jpg 800w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-300x192.jpg 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-768x492.jpg 768w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><b>Take a Consistent Logical Backup:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">mysqldump<\/span> <span style=\"color: #dcdcaa;\">&#8211;single-transaction<\/span> <span style=\"color: #dcdcaa;\">&#8211;max_allowed_packet=500M<\/span> <span style=\"color: #dcdcaa;\">&#8211;databases<\/span> <span style=\"color: #9cdcfe;\">opszilla_db<\/span> <span style=\"color: #d4d4d4;\">&gt;<\/span> <span style=\"color: #ce9178;\">opszilla_dump.sql<\/span><\/div>\n<div><\/div>\n<ul>\n<li><span style=\"font-weight: 400;\">single-transaction ensures a consistent snapshot without locking tables.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">max_allowed_packet=500M prevents large transactions from failing.<\/span><\/li>\n<\/ul>\n<p><b>Capture Binary Log Position for Replication Setup (If Needed):<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #9cdcfe;\">SHOW<\/span> <span style=\"color: #4ec9b0;\">MASTER<\/span> <span style=\"color: #9cdcfe;\">STATUS<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<ul>\n<li><span style=\"font-weight: 400;\">Record the File and Position to set up replication later.<\/span><\/li>\n<\/ul>\n<p><b>Restore on MySQL 8.0 Instance:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">mysql<\/span> <span style=\"color: #dcdcaa;\">-u<\/span> <span style=\"color: #9cdcfe;\">root<\/span> <span style=\"color: #dcdcaa;\">-p<\/span> <span style=\"color: #d4d4d4;\">&lt;<\/span> <span style=\"color: #ce9178;\">opszilla_dump.sql<\/span><\/div>\n<p>&nbsp;<\/p>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Avoided excessive load on the primary by controlling dump packet size.\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Ensured data consistency with transaction isolation.\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Enabled future replication setup if required.<\/span><\/li>\n<\/ul>\n<p>This aligns with best practices in <a href=\"https:\/\/opstree.com\/services\/middleware-database-and-data-engineering\/\"><strong>data engineering services<\/strong><\/a> for high-availability systems.<\/p>\n<h4><b>Scenario 2: Primary with Existing Secondary Replica<\/b><\/h4>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A primary-replica setup where the primary handled live traffic.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Running <a href=\"https:\/\/opstree.com\/blog\/2024\/07\/23\/comparison-between-mydumper-mysqldump-xtrabackup\/\">mysqldump<\/a> directly on the primary could impact performance.<\/span><\/li>\n<\/ul>\n<p><b>Solution: Offloading Dump to a Secondary Replica<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Instead of burdening the primary, we used an existing (or temporary) replica to perform the backup:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-29446 size-full\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-1.jpg\" alt=\"MySQL\" width=\"800\" height=\"512\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-1.jpg 800w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-1-300x192.jpg 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-1-768x492.jpg 768w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><b>Use a Read Replica for Backup:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">If no replica existed, we first set up a temporary one from the primary.\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Ensured the replica was in sync before proceeding.<\/span><\/li>\n<\/ul>\n<p><b>Run mysqldump on the Replica:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">mysqldump<\/span> <span style=\"color: #dcdcaa;\">&#8211;single-transaction<\/span> <span style=\"color: #dcdcaa;\">&#8211;max_allowed_packet=500M<\/span> <span style=\"color: #dcdcaa;\">&#8211;databases<\/span> <span style=\"color: #9cdcfe;\">opszilla_db<\/span> <span style=\"color: #d4d4d4;\">&gt;<\/span> <span style=\"color: #ce9178;\">opszilla_replica_dump.sql<\/span><\/div>\n<p><span style=\"font-weight: 400;\">No impact on primary server performance.<\/span><\/p>\n<p><b>Restore to MySQL 8.0 &amp; Reconfigure Replication:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Import the dump into the new MySQL 8.0 primary.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Set up new replicas under MySQL 8.0.<\/span><\/li>\n<\/ul>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Zero performance impact on the production primary.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Allowed parallel testing of MySQL 8.0 before cutover.\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Minimized downtime by keeping the old primary operational during migration.<\/span><\/li>\n<\/ul>\n<p>This approach reflects <a href=\"https:\/\/opstree.com\/blog\/2024\/08\/22\/data-interation-challenges\/\"><strong>scalable data engineering practices<\/strong><\/a> essential for modern database upgrades.<\/p>\n<p><a href=\"https:\/\/opstree.com\/services\/middleware-database-and-data-engineering\/\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-29452\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Searching-for-reliable-data-engineering-solutions.png\" alt=\"Searching for reliable data engineering solutions\" width=\"800\" height=\"190\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Searching-for-reliable-data-engineering-solutions.png 800w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Searching-for-reliable-data-engineering-solutions-300x71.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Searching-for-reliable-data-engineering-solutions-768x182.png 768w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/a><\/p>\n<h1><span style=\"font-weight: 400;\">Reserved Keywords Broke Schema Imports<\/span><\/h1>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-29454 size-full\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master.png\" alt=\"my sql\" width=\"800\" height=\"512\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master.png 800w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-300x192.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Master-768x492.png 768w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">MySQL 8.0 introduced new <\/span><b>reserved keywords<\/b><span style=\"font-weight: 400;\"> such as <\/span><span style=\"font-weight: 400;\">RANK<\/span><span style=\"font-weight: 400;\">, <\/span><span style=\"font-weight: 400;\">SYSTEM<\/span><span style=\"font-weight: 400;\">, and <\/span><span style=\"font-weight: 400;\">WINDOW<\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">These words were not reserved in MySQL 5.7, so it was common to use them as column names without issues<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In this scenario, the column rank was most commonly used in the following tables:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">copilot_authority<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">analysis_citations<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Both tables were tightly integrated with critical reporting pipelines and application services.<\/span><\/p>\n<p><b>Analysis citations table:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The <\/span><span style=\"font-weight: 400;\">analysis_citations<\/span><span style=\"font-weight: 400;\"> table stores external references or citations linked to analytical reports. The <\/span><span style=\"font-weight: 400;\">rank<\/span><span style=\"font-weight: 400;\"> column is used to define the order or relevance of each citation.<\/span><\/p>\n<p><b>Copilot authority<\/b><span style=\"font-weight: 400;\">: This table served as a recommendation engine to assign authority levels or trust scores to decisions, where <\/span><span style=\"font-weight: 400;\">rank<\/span><span style=\"font-weight: 400;\"> was used to prioritize recommendations or rules.<\/span><\/p>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">rank column broke queries due to MySQL 8.0 keyword conflict.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Table was large and critical, so renaming risked downtime.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Needed coordinated code changes across services using these tables.<\/span><\/li>\n<\/ul>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.9em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #9cdcfe;\">CREATE TABLE<\/span> <span style=\"color: #ce9178;\">`analysis_citations`<\/span> (<br \/>\n<span style=\"color: #ce9178;\">`id`<\/span> <span style=\"color: #569cd6;\">int(11)<\/span> <span style=\"color: #569cd6;\">NOT NULL AUTO_INCREMENT<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`report_id`<\/span> <span style=\"color: #569cd6;\">int(11)<\/span> <span style=\"color: #569cd6;\">DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`url`<\/span> <span style=\"color: #569cd6;\">varchar(100)<\/span> <span style=\"color: #569cd6;\">COLLATE utf8_unicode_ci DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`rank`<\/span> <span style=\"color: #569cd6;\">int(11)<\/span> <span style=\"color: #569cd6;\">DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`citation_link`<\/span> <span style=\"color: #569cd6;\">varchar(2500)<\/span> <span style=\"color: #569cd6;\">COLLATE utf8_unicode_ci DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`citation_title`<\/span> <span style=\"color: #569cd6;\">varchar(255)<\/span> <span style=\"color: #569cd6;\">COLLATE utf8_unicode_ci DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`url_host`<\/span> <span style=\"color: #569cd6;\">varchar(150)<\/span> <span style=\"color: #569cd6;\">COLLATE utf8_unicode_ci DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`client_id`<\/span> <span style=\"color: #569cd6;\">VARCHAR(255)<\/span> <span style=\"color: #569cd6;\">CHARACTER SET utf8 COLLATE utf8mb3_unicode_ci NOT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`created_at`<\/span> <span style=\"color: #569cd6;\">timestamp<\/span> <span style=\"color: #569cd6;\">NULL DEFAULT CURRENT_TIMESTAMP<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`updated_at`<\/span> <span style=\"color: #569cd6;\">TIMESTAMP<\/span> <span style=\"color: #569cd6;\">DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP<\/span>,<br \/>\n<span style=\"color: #569cd6;\">PRIMARY KEY<\/span> (<span style=\"color: #ce9178;\">`id`<\/span>),<br \/>\n<span style=\"color: #569cd6;\">KEY<\/span> <span style=\"color: #dcdcaa;\">`report id`<\/span> (<span style=\"color: #ce9178;\">`report_id`<\/span>),<br \/>\n<span style=\"color: #569cd6;\">KEY<\/span> <span style=\"color: #dcdcaa;\">`url host`<\/span> (<span style=\"color: #ce9178;\">`url_host`<\/span>)<br \/>\n) <span style=\"color: #569cd6;\">ENGINE=InnoDB AUTO_INCREMENT=81731627 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci<\/span>;<\/div>\n<p><span style=\"font-weight: 400;\">\u00a0 <img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-29448\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Screenshot-2025-07-29-132454.png\" alt=\"\" width=\"624\" height=\"512\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Screenshot-2025-07-29-132454.png 624w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/Screenshot-2025-07-29-132454-300x246.png 300w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/span><\/p>\n<p><b>Solution: Safe Column Rename Using Online Schema Change Tools<\/b><\/p>\n<p><b>Rename the Reserved Column\u00a0<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">pt-online-schema-change<\/span> <span style=\"color: #dcdcaa;\">&#8211;alter<\/span> <span style=\"color: #ce9178;\">&#8220;CHANGE \\`rank\\` \\`user_rank\\` TINYINT NOT NULL&#8221;<\/span> <span style=\"color: #dcdcaa;\">D=<\/span><span style=\"color: #9cdcfe;\">opszilla_db<\/span>,<span style=\"color: #dcdcaa;\">t=<\/span><span style=\"color: #9cdcfe;\">analysis_citations<\/span> <span style=\"color: #dcdcaa;\">&#8211;execute<\/span><\/div>\n<ul>\n<li aria-level=\"1\"><span style=\"font-weight: 400;\">Safely renames the <\/span><span style=\"font-weight: 400;\">rank<\/span><span style=\"font-weight: 400;\"> column to <\/span><span style=\"font-weight: 400;\">user_rank<\/span><span style=\"font-weight: 400;\"> without downtime using an online schema change.<\/span><\/li>\n<\/ul>\n<p><b>Using GitHub gh-ost:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">gh-ost<\/span> <span style=\"color: #dcdcaa;\">&#8211;alter=<\/span><span style=\"color: #ce9178;\">&#8220;CHANGE \\`rank\\` \\`user_rank\\` TINYINT NOT NULL&#8221;<\/span> \\<br \/>\n<span style=\"color: #dcdcaa;\">&#8211;database=<\/span><span style=\"color: #9cdcfe;\">opszilla_db<\/span> \\<br \/>\n<span style=\"color: #dcdcaa;\">&#8211;table=<\/span><span style=\"color: #9cdcfe;\">analysis_citations<\/span> \\<br \/>\n<span style=\"color: #dcdcaa;\">&#8211;user=<\/span><span style=\"color: #9cdcfe;\">root<\/span> \\<br \/>\n<span style=\"color: #dcdcaa;\">&#8211;password=<\/span><span style=\"color: #ce9178;\">your_password<\/span> \\<br \/>\n<span style=\"color: #dcdcaa;\">&#8211;host=<\/span><span style=\"color: #ce9178;\">your_db_host<\/span> \\<br \/>\n<span style=\"color: #dcdcaa;\">&#8211;execute<\/span><\/div>\n<ul>\n<li><span style=\"font-weight: 400;\">Renames the <\/span><span style=\"font-weight: 400;\">rank<\/span><span style=\"font-weight: 400;\"> column to <\/span><span style=\"font-weight: 400;\">user_rank<\/span><span style=\"font-weight: 400;\"> online using GitHub&#8217;s <\/span><span style=\"font-weight: 400;\">gh-ost<\/span><span style=\"font-weight: 400;\"> tool with zero downtime.<\/span><\/li>\n<\/ul>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Renamed the column <\/span><b>safely with zero downtime<\/b><span style=\"font-weight: 400;\">.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Avoided table locks and allowed live traffic during the change.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Enabled <\/span><b>safe rollback<\/b><span style=\"font-weight: 400;\"> if anything went wrong.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Fixed schema incompatibility and unblocked the MySQL 8.0 upgrade.<\/span><\/li>\n<\/ul>\n<p>This is a proven technique in enterprise-grade <a href=\"https:\/\/opstree.com\/blog\/2025\/05\/06\/technical-case-study-amazon-redshift-and-athena-as-data-warehousing-solutions\/\">data infrastructure upgrade<\/a><\/p>\n<p><strong>[Explore Our Case Study : <a href=\"https:\/\/opstree.com\/case-study\/27-aws-cost-reduction-through-database-optimization-for-a-fintech-platform\/\">AWS Cost Savings Through Database Optimization For Fintech<\/a>]<\/strong><\/p>\n<h2><strong>Timestamp Columns Introduced Data Drift<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">The order_delivery table used for tracking every stage of an order\u2019s delivery lifecycle\u2014from shipment to final handoff. It included several TIMESTAMP columns with auto-update behaviors critical for real-time delivery tracking.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Timestamp inaccuracy can confuse delivery status in customer-facing systems.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Real-time dashboards use these timestamps for order tracking, delivery efficiency metrics, etc.<\/span><\/li>\n<\/ul>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.9em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #9cdcfe;\">CREATE TABLE<\/span> <span style=\"color: #ce9178;\">`order_delivery`<\/span> (<br \/>\n<span style=\"color: #ce9178;\">`delivery_id`<\/span> <span style=\"color: #569cd6;\">int(11)<\/span> <span style=\"color: #569cd6;\">NOT NULL AUTO_INCREMENT<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`order_id`<\/span> <span style=\"color: #569cd6;\">int(11)<\/span> <span style=\"color: #569cd6;\">NOT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`customer_id`<\/span> <span style=\"color: #569cd6;\">int(11)<\/span> <span style=\"color: #569cd6;\">NOT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`status`<\/span> <span style=\"color: #569cd6;\">enum<\/span>(<span style=\"color: #ce9178;\">&#8216;PROCESSING&#8217;,&#8217;SHIPPED&#8217;,&#8217;IN_TRANSIT&#8217;,&#8217;OUT_FOR_DELIVERY&#8217;,&#8217;DELIVERED&#8217;<\/span>) <span style=\"color: #569cd6;\">NOT NULL DEFAULT &#8216;PROCESSING&#8217;<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`order_date`<\/span> <span style=\"color: #569cd6;\">timestamp<\/span> <span style=\"color: #569cd6;\">NOT NULL DEFAULT CURRENT_TIMESTAMP<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`ship_date`<\/span> <span style=\"color: #569cd6;\">timestamp<\/span> <span style=\"color: #569cd6;\">NULL DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`estimated_delivery`<\/span> <span style=\"color: #569cd6;\">timestamp<\/span> <span style=\"color: #569cd6;\">NULL DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`actual_delivery`<\/span> <span style=\"color: #569cd6;\">timestamp<\/span> <span style=\"color: #569cd6;\">NULL DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`carrier`<\/span> <span style=\"color: #569cd6;\">varchar(20)<\/span> <span style=\"color: #569cd6;\">DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`tracking_number`<\/span> <span style=\"color: #569cd6;\">varchar(50)<\/span> <span style=\"color: #569cd6;\">DEFAULT NULL<\/span>,<br \/>\n<span style=\"color: #ce9178;\">`last_updated`<\/span> <span style=\"color: #569cd6;\">timestamp<\/span> <span style=\"color: #569cd6;\">NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP<\/span>,<br \/>\n<span style=\"color: #569cd6;\">PRIMARY KEY<\/span> (<span style=\"color: #ce9178;\">`delivery_id`<\/span>)<br \/>\n) <span style=\"color: #569cd6;\">ENGINE=InnoDB DEFAULT CHARSET=utf8mb4<\/span>;<\/div>\n<p>&nbsp;<\/p>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">MySQL 8.0 broke auto-update <\/span><span style=\"font-weight: 400;\">TIMESTAMP<\/span><span style=\"font-weight: 400;\"> behavior due to stricter rules.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">TIMESTAMP<\/span><span style=\"font-weight: 400;\"> values were lost during dump\/import without correct flags.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Timezone mismatches between servers caused timestamp shifts.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">These issues risked breaking real-time order tracking and dashboards.<\/span><\/li>\n<\/ul>\n<p><b>Solution: Preserve Temporal Column Integrity During Migration<\/b><\/p>\n<p><b>Take a Consistent Logical Backup<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Used <\/span><span style=\"font-weight: 400;\">mysqldump<\/span><span style=\"font-weight: 400;\"> with the right flags to preserve timestamp values and behaviors<\/span><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">mysqldump<\/span> <span style=\"color: #dcdcaa;\">&#8211;single-transaction<\/span> <span style=\"color: #dcdcaa;\">&#8211;set-gtid-purged=<\/span><span style=\"color: #ce9178;\">OFF<\/span> <span style=\"color: #dcdcaa;\">&#8211;skip-tz-utc<\/span><span style=\"color: #d4d4d4;\"> \u00a0<\/span><span style=\"color: #dcdcaa;\">&#8211;databases<\/span> <span style=\"color: #9cdcfe;\">opszilla_db<\/span> <span style=\"color: #dcdcaa;\">&gt;<\/span> <span style=\"color: #ce9178;\">opszilla_dump.sql<\/span><\/div>\n<p><b>Restore the Dump to MySQL 8.0 Target:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Ensured all <\/span><span style=\"font-weight: 400;\">DEFAULT CURRENT_TIMESTAMP<\/span><span style=\"font-weight: 400;\"> and <\/span><span style=\"font-weight: 400;\">ON UPDATE<\/span><span style=\"font-weight: 400;\"> definitions were retained.Verified schema integrity using <\/span><span style=\"font-weight: 400;\">SHOW CREATE TABLE<\/span><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">mysql<\/span> <span style=\"color: #dcdcaa;\">-u<\/span> <span style=\"color: #9cdcfe;\">root<\/span> <span style=\"color: #dcdcaa;\">-p<\/span> <span style=\"color: #dcdcaa;\">&lt;<\/span> <span style=\"color: #ce9178;\">opszilla_dump.sql<\/span><\/div>\n<p><b>Use Row-Based Binary Logging<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Prevents subtle data drift that may occur in statement-based logging.<\/span><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">SET GLOBAL<\/span> <span style=\"color: #9cdcfe;\">binlog_format<\/span> <span style=\"color: #dcdcaa;\">=<\/span> <span style=\"color: #ce9178;\">&#8216;ROW&#8217;<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Preserved all <\/span><span style=\"font-weight: 400;\">TIMESTAMP<\/span><span style=\"font-weight: 400;\"> values and auto-update settings accurately.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Prevented data drift with row-based binary logging.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Avoided schema issues caused by MySQL 8.0&#8217;s stricter timestamp rules.<\/span><\/li>\n<\/ul>\n<p>A critical example of applying <a href=\"https:\/\/opstree.com\/blog\/2023\/06\/20\/database-migration-service-in-aws\/\">database modernization<\/a> in real-world production environments.<\/p>\n<h2><strong>Collation Mismatches Broke Unicode Handling<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">We faced issues with the <\/span><span style=\"font-weight: 400;\">customer_reviews<\/span><span style=\"font-weight: 400;\"> table, which had mixed character sets and deprecated collations due to years of inconsistent schema changes. This affected Unicode support and query performance.<\/span><\/p>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Emoji data got corrupted due to 3-byte <\/span><span style=\"font-weight: 400;\">utf8_unicode_ci<\/span><span style=\"font-weight: 400;\"> (couldn\u2019t store 4-byte emojis).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Mixed collations triggered <\/span><b>&#8220;Illegal mix of collations&#8221;<\/b><span style=\"font-weight: 400;\"> query failures.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">latin1<\/span><span style=\"font-weight: 400;\"> columns broke Unicode rendering after migration.<\/span><\/li>\n<\/ul>\n<p><b>Solution: Standardize to utf8mb4 and Modern Collation Pre-Migration<\/b><\/p>\n<p><span style=\"font-weight: 400;\">To avoid corruption and query errors, we normalized all text columns in MySQL 5.7 <\/span><i><span style=\"font-weight: 400;\">before migration<\/span><\/i><\/p>\n<p><b>Pre-Migration: Convert Entire Table to utf8mb4<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">ALTER TABLE<\/span> <span style=\"color: #9cdcfe;\">customer_reviews<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0<span style=\"color: #569cd6;\">CONVERT TO CHARACTER SET<\/span> <span style=\"color: #ce9178;\">utf8mb4<\/span> <span style=\"color: #569cd6;\">COLLATE<\/span> <span style=\"color: #ce9178;\">utf8mb4_0900_ai_ci<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><b>Validate Emojis and Character Accuracy<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">SELECT<\/span> <span style=\"color: #9cdcfe;\">emoji_feedback<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <span style=\"color: #569cd6;\">FROM<\/span> <span style=\"color: #9cdcfe;\">customer_reviews<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <span style=\"color: #569cd6;\">WHERE<\/span> <span style=\"color: #9cdcfe;\">emoji_feedback<\/span> <span style=\"color: #569cd6;\">LIKE<\/span> <span style=\"color: #ce9178;\">&#8216;%<code>\ud83d\ude0a<\/code>%&#8217;<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p>&nbsp;<\/p>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Fixed <\/span><span style=\"font-weight: 400;\">latin1<\/span><span style=\"font-weight: 400;\"> legacy data without corrupting product names.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Eliminated collation-related query errors.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Saved all emojis and special characters correctly.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Table is now fully compatible with MySQL 8.0 features.<\/span><\/li>\n<\/ul>\n<p><b>What are Collation and Charset<\/b><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Collation is like the &#8220;rule book&#8221; for how a database sorts and compares text\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>Whether letters are treated as uppercase or lowercase (case sensitivity)\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>How accented characters (like \u00e9, \u00f1) are handled\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>The order in which characters are sorted\u200b<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-weight: 400;\"> To prevent data corruption and ensure full compatibility with MySQL 8.0, we standardized the character\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">set and collation before performing the migration.<\/span><\/p>\n<p><b>Pre-Migration Character Set Conversion<\/b><b><\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">ALTER TABLE<\/span> <span style=\"color: #9cdcfe;\">customer_reviews<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <span style=\"color: #569cd6;\">CONVERT TO CHARACTER SET<\/span> <span style=\"color: #9cdcfe;\">utf8mb4<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <span style=\"color: #569cd6;\">COLLATE<\/span> <span style=\"color: #9cdcfe;\">utf8mb4_0900_ai_ci<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<h5><b>Collation Compatibility Matrix:<\/b><\/h5>\n<table>\n<tbody>\n<tr>\n<td><b>MySQL 5.7 Source\u200b<\/b><\/td>\n<td><b>MySQL 8.0 Replica\u200b<\/b><\/td>\n<td><b>Notes\u200b<\/b><\/td>\n<td><b>Recommendation\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>utf8mb4\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>utf8mb4\u200b<\/b><\/td>\n<td><b>Perfect\u200b<\/b><\/td>\n<td><b>Ideal for new deployments\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>utf8 (utf8mb3)\u200b<\/b><\/td>\n<td><b>utf8mb3\u200b<\/b><\/td>\n<td><b>Legacy\u200b<\/b><\/td>\n<td><b>Temporary migration state\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>utf8mb3\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>utf8mb4\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>Conditional\u200b<\/b><\/td>\n<td><b>Requires slave_type_conversions=ALL_NON_LOSSY\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>utf8mb4\u200b<\/b><\/td>\n<td><b>utf8mb3\u200b<\/b><\/td>\n<td><b>Dangerous\u200b<\/b><\/p>\n<p><b>\u200b<\/b><\/td>\n<td><b>Data loss guaranteed\u200b<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><strong>[ Don&#8217;t miss our latest eBook: <a href=\"https:\/\/opstree.com\/ebooks\/ebook-ultimate-guide-to-delivering-end-to-end-data-strategy\/\">Ultimate Guide to Delivering End-to-End Data Strategy<\/a> ]<\/strong><\/p>\n<h2><strong>Disk Space Exhaustion During Dump and Restore<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Migrate OpsZilla\u2019s 1TB <a href=\"https:\/\/opstree.com\/blog\/2024\/07\/30\/use-case-backup-and-replication-setup-between-ec2-mysql-and-rds-mysql\/\">MySQL database<\/a> without increasing disk capacity, due to budget and infrastructure constraints.<\/span><\/p>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">mysqldump<\/span><span style=\"font-weight: 400;\"> nearly doubled disk usage during export.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">MySQL 8.0 restore generated extra <\/span><b>temp files and binary logs<\/b><span style=\"font-weight: 400;\">.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Disk space exhausted mid-restore, causing crashes and risk of data loss.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Uncontrolled binary log growth led to \u201cDisk Full\u201d errors.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Leftover logs after migration increased storage costs.<\/span><\/li>\n<\/ul>\n<p><b>Solution: Optimizing Storage with Compression and Log Management<\/b><\/p>\n<p><b>Take a Compressed Logical Backup:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">mysqldump<\/span> <span style=\"color: #9cdcfe;\">&#8211;single-transaction<\/span> <span style=\"color: #9cdcfe;\">&#8211;quick<\/span> <span style=\"color: #9cdcfe;\">&#8211;databases<\/span> <span style=\"color: #ce9178;\">opszilla_db<\/span> <span style=\"color: #d4d4d4;\">|<\/span> <span style=\"color: #dcdcaa;\">gzip<\/span> <span style=\"color: #d4d4d4;\">&gt;<\/span> <span style=\"color: #ce9178;\">opszilla_dump.sql.gz<\/span><\/div>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Consistent snapshot without table locks<\/span><\/li>\n<\/ul>\n<p><b>Stream Restore with On-the-Fly Decompression:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">gunzip<\/span> &lt; <span style=\"color: #ce9178;\">opszilla_dump.sql.gz<\/span> <span style=\"color: #d4d4d4;\">|<\/span> <span style=\"color: #dcdcaa;\">mysql<\/span> <span style=\"color: #9cdcfe;\">-u<\/span> <span style=\"color: #ce9178;\">root<\/span> <span style=\"color: #9cdcfe;\">-p<\/span><\/div>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Avoids writing a large uncompressed <\/span><span style=\"font-weight: 400;\">.sql<\/span><span style=\"font-weight: 400;\"> file to disk.<\/span><\/li>\n<\/ul>\n<p><b>Temporarily Disable Logs During Restore<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">SET<\/span> <span style=\"color: #9cdcfe;\">GLOBAL<\/span> <span style=\"color: #4ec9b0;\">general_log<\/span> = <span style=\"color: #ce9178;\">&#8216;OFF&#8217;<\/span>;<br \/>\n<span style=\"color: #569cd6;\">SET<\/span> <span style=\"color: #9cdcfe;\">GLOBAL<\/span> <span style=\"color: #4ec9b0;\">slow_query_log<\/span> = <span style=\"color: #ce9178;\">&#8216;OFF&#8217;<\/span>;<\/div>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Reduces unnecessary disk writes and speeds up restore<\/span><\/li>\n<\/ul>\n<p><b>Purge Binary Logs Aggressively<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #569cd6;\">SET<\/span> <span style=\"color: #9cdcfe;\">GLOBAL<\/span> <span style=\"color: #4ec9b0;\">binlog_expire_logs_seconds<\/span> = <span style=\"color: #b5cea8;\">3600<\/span>;<br \/>\n<span style=\"color: #6a9955;\">&#8212; OR\u00a0<\/span><br \/>\n<span style=\"color: #569cd6;\">SET<\/span> <span style=\"color: #9cdcfe;\">GLOBAL<\/span> <span style=\"color: #4ec9b0;\">expire_logs_days<\/span> = <span style=\"color: #b5cea8;\">0<\/span>;<\/div>\n<ul>\n<li><span style=\"font-weight: 400;\">Automatically removes old logs to prevent storage bloat.<\/span><b><\/b><\/li>\n<\/ul>\n<p><b>Monitor Disk Space in Real-Time<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">watch<\/span> <span style=\"color: #b5cea8;\">-n<\/span> <span style=\"color: #b5cea8;\">5<\/span> <span style=\"color: #d4d4d4;\">df -h \/var\/lib\/mysql<\/span><\/div>\n<ul>\n<li aria-level=\"1\"><span style=\"font-weight: 400;\">Proactively tracks usage to avoid mid-operation crashes.<\/span><\/li>\n<\/ul>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Compressed backups minimized storage overhead.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Streamed restore reduced disk usage and I\/O strain.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Prevented downtime from full-disk crashes.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\">Reduced storage costs by auto-cleaning unused logs.<\/span><\/li>\n<\/ul>\n<h2><span style=\"font-weight: 400;\">I\/O Bottlenecks During Bulk Restore Operations<\/span><\/h2>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Binary logs from bulk imports filled disk rapidly..<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Restore failed midway due to lack of space.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Old binlogs weren\u2019t cleaned, bloating disk usage.<\/span><\/li>\n<\/ul>\n<p><b>Solution: Tuning InnoDB for Optimized Disk I\/O During Migration<\/b><\/p>\n<p><b>Adjust Transaction Log Flushing Behavior:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">SET<\/span> <span style=\"color: #d4d4d4;\">GLOBAL<\/span> <span style=\"color: #9cdcfe;\">innodb_flush_log_at_trx_commit<\/span> <span style=\"color: #d4d4d4;\">=<\/span> <span style=\"color: #b5cea8;\">2<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><b>Enable Direct Disk Writes for Data Files:<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">SET<\/span> <span style=\"color: #d4d4d4;\">GLOBAL<\/span> <span style=\"color: #9cdcfe;\">innodb_flush_method<\/span> <span style=\"color: #d4d4d4;\">=<\/span> <span style=\"color: #ce9178;\">&#8216;O_DIRECT&#8217;<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><span style=\"font-weight: 400;\">Bypasses OS cache to reduce double-buffering.<\/span><\/p>\n<p><b>Optimizing InnoDB I\/O Performance: innodb_flush_log_at_trx_commit<\/b><\/p>\n<p><span style=\"font-weight: 400;\">innodb_flush_log_at_trx_commit<\/span><span style=\"font-weight: 400;\"> is a critical InnoDB parameter that controls how frequently transaction logs are flushed to disk, directly impacting the balance between data durability and write performance in <a href=\"https:\/\/www.mysql.com\/\" target=\"_blank\" rel=\"noopener\">MySQL<\/a>\/InnoDB.<\/span><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Setting\u200b<\/b><\/td>\n<td><b>Flush Behavior\u200b<\/b><\/td>\n<td><b>Durability\u200b<\/b><\/td>\n<td><b>Performance\u200b<\/b><\/td>\n<td><b>Recommended Use Case\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>1 (Default)\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>Flush logs to disk on every transaction commit\u200b<\/b><\/td>\n<td><b>Highest durability\u200b<\/b><\/td>\n<td><b>Slowest\u200b<\/b><\/td>\n<td><b>Production environments requiring full ACID compliance\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>2\u200b<\/b><\/td>\n<td><b>Flush logs to OS cache on commit, disk once per second\u200b<\/b><\/td>\n<td><b>Moderate durability (survives OS crash but not power failure)\u200b<\/b><\/td>\n<td><b>Faster\u200b<\/b><\/td>\n<td><b>Data migrations\/restores where speed is prioritized\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>0\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>Write to OS cache, flush to disk once per second\u200b<\/b><\/p>\n<p><b>\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>Lowest durability\u200b<\/b><\/p>\n<p><b>\u200b<\/b><\/td>\n<td><b>Faster\u200b<\/b><\/td>\n<td><b>Temporary test environments only\u200b<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><b>Optimizing InnoDB I\/O Performance: innodb_flush_log_trx_commit<\/b><\/p>\n<p><b>innodb_flush_method<\/b><span style=\"font-weight: 400;\"> is a critical parameter that controls <\/span><b>how InnoDB interacts with the filesystem<\/b><span style=\"font-weight: 400;\"> for reading and writing data and log files. It has a significant impact on <\/span><b>I\/O performance<\/b><span style=\"font-weight: 400;\"> during both <\/span><b>migrations<\/b><span style=\"font-weight: 400;\"> and <\/span><b>normal operations<\/b><span style=\"font-weight: 400;\">, especially in disk-intensive environments.<\/span><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Method\u200b<\/b><\/td>\n<td><b>Description\u200b<\/b><\/td>\n<td><b>Durability\u200b<\/b><\/td>\n<td><b>Performance\u200b<\/b><\/td>\n<td><b>Recommended Use\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>fsync (default)\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>Uses fsync() for both data and log files\u200b<\/b><\/td>\n<td><b>Highest\u200b<\/b><\/td>\n<td><b>Slowest\u200b<\/b><\/td>\n<td><b>General purpose, most compatible\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>O_DSYNC\u200b<\/b><\/td>\n<td><b>Opens log files with O_SYNC (immediate sync)\u200b<\/b><\/td>\n<td><b>High\u200b<\/b><\/td>\n<td><b>Moderate\u200b<\/b><\/td>\n<td><b>Rarely better than fsync\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>O_DIRECT\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>Bypasses OS cache for data files\u200b<\/b><\/td>\n<td><b>\u200b<\/b><\/p>\n<p><b>High\u200b<\/b><\/td>\n<td><b>Fast for writes\u200b<\/b><\/td>\n<td><b>Recommended for most Linux systems\u200b<\/b><\/td>\n<\/tr>\n<tr>\n<td><b>O_DIRECT_NO_FSYNC\u200b<\/b><\/td>\n<td><b>Like O_DIRECT but skips final fsync\u200b<\/b><\/td>\n<td><b>Risky\u200b<\/b><\/td>\n<td><b>Fastest\u200b<\/b><\/td>\n<td><b>Not recommended for production\u200b<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Less I\/O load made imports faster.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Restore finished quicker without data loss.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Worked well for large, write-heavy tables.<\/span><\/li>\n<\/ul>\n<h1><span style=\"font-weight: 400;\">Authentication Plugin Mismatch Broke User Logins<\/span><\/h1>\n<p><span style=\"font-weight: 400;\">MySQL 8.0 introduced changes in the default authentication plugin, switching from <\/span><span style=\"font-weight: 400;\">mysql_native_password<\/span><span style=\"font-weight: 400;\"> to <\/span><span style=\"font-weight: 400;\">caching_sha2_password<\/span><span style=\"font-weight: 400;\">. This change caused compatibility issues for existing applications<\/span><\/p>\n<p><b>Challenge:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">App logins failed with <\/span><span style=\"font-weight: 400;\">ERROR 2059<\/span><span style=\"font-weight: 400;\">: plugin <\/span><span style=\"font-weight: 400;\">caching_sha2_password<\/span><span style=\"font-weight: 400;\"> not supported.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Legacy clients couldn\u2019t connect until their config was updated.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">mysql.user<\/span><span style=\"font-weight: 400;\"> schema changes (e.g., new <\/span><span style=\"font-weight: 400;\">authentication_string<\/span><span style=\"font-weight: 400;\"> column) broke user <a href=\"https:\/\/opstree.com\/blog\/2024\/04\/04\/migration-of-ms-sql-from-azure-vm-to-amazon-rds\/\">migration<\/a> via <\/span><span style=\"font-weight: 400;\">mysqldump<\/span><\/li>\n<\/ul>\n<p><b>Solution: Revert to Compatible Authentication Plugin During Migration<\/b><\/p>\n<p><b>Identify Users with Old Plugins in MySQL 5.7<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">SELECT<\/span> <span style=\"color: #9cdcfe;\">user<\/span><span style=\"color: #d4d4d4;\">,<\/span> <span style=\"color: #9cdcfe;\">host<\/span><span style=\"color: #d4d4d4;\">,<\/span> <span style=\"color: #9cdcfe;\">plugin<\/span> <span style=\"color: #dcdcaa;\">FROM<\/span> <span style=\"color: #9cdcfe;\">mysql<\/span><span style=\"color: #d4d4d4;\">.<\/span><span style=\"color: #9cdcfe;\">user<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><span style=\"font-weight: 400;\">Audit current auth plugins before upgrade.<\/span><\/p>\n<p><b>Pre-Migration: Force Legacy Plugin for Compatibility<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">ALTER USER<\/span> <span style=\"color: #ce9178;\">&#8216;app_user&#8217;<\/span><span style=\"color: #d4d4d4;\">@<\/span><span style=\"color: #ce9178;\">&#8216;%&#8217;<\/span> <span style=\"color: #dcdcaa;\">IDENTIFIED WITH<\/span> <span style=\"color: #4ec9b0;\">mysql_native_password<\/span> <span style=\"color: #dcdcaa;\">BY<\/span> <span style=\"color: #ce9178;\">&#8216;secure_pass&#8217;<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><span style=\"font-weight: 400;\">Ensures all new users default to legacy plugin.<\/span><\/p>\n<p><b>Post-Migration: Update Authentication Plugin in MySQL 8.0<\/b><\/p>\n<div style=\"background: #1e1e1e; color: #d4d4d4; padding: 1em; border-radius: 8px; font-family: monospace; font-size: 0.95em; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.2);\"><span style=\"color: #dcdcaa;\">ALTER USER<\/span> <span style=\"color: #ce9178;\">&#8216;app_user&#8217;<\/span><span style=\"color: #d4d4d4;\">@<\/span><span style=\"color: #ce9178;\">&#8216;%&#8217;<\/span><br \/>\n<span style=\"color: #dcdcaa;\">IDENTIFIED WITH<\/span> <span style=\"color: #4ec9b0;\">caching_sha2_password<\/span><br \/>\n<span style=\"color: #dcdcaa;\">BY<\/span> <span style=\"color: #ce9178;\">&#8216;secure_pass&#8217;<\/span><span style=\"color: #d4d4d4;\">;<\/span><\/div>\n<p><b>For services that support it, switch to the modern plugin<\/b><\/p>\n<p><b>Why This Worked:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Old apps could connect right after migration \u2014 no breakage.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Prevented downtime from unexpected login failures.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Allowed smooth, <\/span><b>controlled transition<\/b><span style=\"font-weight: 400;\"> to the secure <\/span><span style=\"font-weight: 400;\">caching_sha2_password<\/span><span style=\"font-weight: 400;\"> plugin later.<\/span><\/li>\n<\/ul>\n<h2><strong>Conclusion:<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">&#8220;By following these best practices in <strong>data engineering and <a href=\"https:\/\/opstree.com\/services\/cloud-engineering-modernisation-migrations\/\">cloud-native database modernization<\/a><\/strong> OpsZilla not only eliminates the risks of running an unsupported MySQL version but also unlocked performance improvements and modern features in MySQL 8.0\u2014future-proofing their database infrastructure for continued growth.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Running a growing e-commerce platform like Opszilla is thrilling. You\u2019re processing thousands of orders daily across the US and Canada, scaling infrastructure, and expanding into new markets. But amidst all that momentum, something\u00a0 starts to break: your data infrastructure and database performance. At first, it\u2019s subtle\u2014slower queries, lagging reports, a few scaling hiccups. Then the &hellip; <a href=\"https:\/\/opstree.com\/blog\/2025\/07\/29\/zero-downtime-mysql-migration\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;How OpsZilla Achieved Zero-Downtime MySQL Migration with Scalable Data Engineering\u00a0Practices&#8221;<\/span><\/a><\/p>\n","protected":false},"author":244582680,"featured_media":29450,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474,4419],"tags":[768739342,768739372,768739289,491749850,343865,768739407],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/07\/How-OpsZilla-Achieved-Zero-Downtime-MySQL-Migration-with-Scalable-Data-Engineering-Practices-1.jpg","jetpack_likes_enabled":false,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-7ET","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/29443"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/244582680"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=29443"}],"version-history":[{"count":7,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/29443\/revisions"}],"predecessor-version":[{"id":29457,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/29443\/revisions\/29457"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29450"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=29443"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=29443"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=29443"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}