{"id":3459,"date":"2020-06-23T13:49:56","date_gmt":"2020-06-23T08:19:56","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=3459"},"modified":"2020-07-21T17:34:13","modified_gmt":"2020-07-21T12:04:13","slug":"automatically-backup-restore-alibaba-mysql-using-grandfather-father-son-strategy","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2020\/06\/23\/automatically-backup-restore-alibaba-mysql-using-grandfather-father-son-strategy\/","title":{"rendered":"Automatically Backup Alibaba MySQL using Grandfather-Father-Son Strategy"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"431\" height=\"394\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2020\/06\/son-father-grandfather-backup-system.png?w=431\" alt=\"\" class=\"wp-image-3488\" \/><\/figure>\n\n\n\n<p>&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong><em>So, basically what is Grandfather-father-son or GFS?<\/em><\/strong><\/h2>\n\n\n\n<\/p>\r\n<p><strong>GFS <em>backup<\/em><\/strong><em> is a common rotation scheme for backup, in which there are three or more backup cycles, such as daily, weekly, and monthly. Typically, It consists of daily backups (son, at fixed intervals of hours in a day), a weekly full backup (father, once a week), and monthly full backup (Grandfather, once a month)<\/em>.<\/p>\r\n<p><!--more--><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p><em>It helps to restore the database from the most possible recovery points in case of any requirement or disaster.<\/em><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p><em>To achieve this, we are using the python script and running the script via Jenkins in a docker container for complete automation.<\/em><\/p>\r\n<p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong><em>Something About Aliyun MySQL &amp; Its features<\/em><\/strong><\/h2>\n\n\n\n<\/p>\r\n<p><em>Alibaba ApsaraDB for MySQL is an on-demand database hosting service for MySQL with automated monitoring, backup and disaster recovery capabilities, etc. <\/em><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p><strong><em>Features<\/em><\/strong>:<\/p>\r\n<p>\n\n\n\n<ol><li><em><strong>Secure<\/strong> &#8211; It is certified by more than 10 authorities such as ISO 20000, SOC, PCI DSS, and Grade III Protection of Information Security.<\/em> <em>It provides <strong>Access control<\/strong><\/em> <em>for IP address whitelisting<\/em>, <em><strong>Protection, and encryption<\/strong> against DDOS &amp; TDE to encrypt the data<\/em> and, <strong><em>SQL audit.<\/em><\/strong><\/li><li><em><strong>Highly available<\/strong> &#8211; &nbsp;It is the first database service that guarantees 99.99% availability in its&nbsp;Service Level Agreement (SLA).<\/em> It also<em>&nbsp;provides the Primary\/secondary architecture, Local disaster recovery &amp; Remote disaster recovery.<\/em><\/li><li><em><strong>Flexible and scalable<\/strong> &#8211; Users can select subscription or pay-as-you-go billing to better meet your business needs<\/em>. <em>Along with this provides the Creation of read-only instances, Automatic scaling, Allocation of dedicated resources, etc.<\/em><\/li><li><em>For more details, please click <a href=\"https:\/\/www.alibabacloud.com\/product\/apsaradb-for-rds-mysql\" target=\"_blank\" rel=\"noreferrer noopener\">here<\/a><\/em>.<\/li><\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><strong><em>How we are backing up &amp; restoring the MySQL Database?<\/em><\/strong><\/h2>\n\n\n\n<\/p>\r\n<p><em>For Backup, our <a href=\"https:\/\/github.com\/singhjaspreet291\/GFSBackup-AliyunRDS\/tree\/master\/rds_backup\/Docker-Setup\/Using-KMS\" target=\"_blank\" rel=\"noreferrer noopener\">script<\/a> will be doing below tasks using Alibaba python SDKs &amp; <a href=\"https:\/\/github.com\/aliyun\/aliyun-oss-python-sdk\" target=\"_blank\" rel=\"noreferrer noopener\">OSS2<\/a> module:<\/em><\/p>\r\n<p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<ul><li><em>List all the databases from RDS.<\/em><\/li><li><em>Create Directory Structure with respect to hourly, weekly &amp; monthly basis<\/em>.<\/li><li><em>Take the MySQL dump locally.<\/em><\/li><li><em>Push the dump to Alibaba OSS using the KMS method to encrypt the data on the fly.<\/em><\/li><li><em>Delete the dump from the local system.<\/em><\/li><\/ul>\n<\/div><\/div>\n\n\n\n<\/p>\r\n<p class=\"has-text-align-left\">Note<em>: Script will create the directory structure in the below format in OSS<\/em>.<\/p>\r\n<p>\n\n\n\n<pre class=\"wp-block-code\"><code>1. hourly --&gt; 06-November-2019 --&gt; DB_NAME --&gt; hourly.sql.gz\n2. weekly --&gt; November-19 --&gt; Week Number --&gt; DB_NAME --&gt; date.sql.gz\n3. monthly --&gt; 2019 --&gt; November --&gt; DB_NAME --&gt; date.sql.gz<\/code><\/pre>\n\n\n\n<\/p>\r\n<p><em>And for decryption <a href=\"https:\/\/github.com\/singhjaspreet291\/GFSBackup-AliyunRDS\/tree\/master\/rds_backup\/Docker-Setup\/Using-KMS\" target=\"_blank\" rel=\"noreferrer noopener\">script<\/a> performs the below tasks:<\/em><\/p>\r\n<p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<ul><li><em>Take user input for the absolute path of the dump which is located in Alibaba OSS.<\/em><\/li><li><em>Pull the data from Alibaba OSS.<\/em><\/li><li><em>Decrypt the data using the KMS method on the fly.<\/em><\/li><\/ul>\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><strong><em><u>I<\/u><span style=\"text-decoration:underline;\"><u>mplementa<\/u>tion<\/span><\/em><\/strong><\/h2>\n\n\n\n<\/p>\r\n<p><em>Before starting the implementation part make sure you meet the below <u><strong>Pre-requisites<\/strong><\/u><\/em><\/p>\r\n<p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<ul><li><em>Create Customer Managed Key(CMK) ID in Alibaba <a href=\"https:\/\/www.alibabacloud.com\/help\/doc-detail\/28935.htm\" target=\"_blank\" rel=\"noreferrer noopener\">KMS<\/a> service.<\/em><\/li><li><em><a href=\"https:\/\/www.alibabacloud.com\/help\/doc-detail\/74762.htm\" target=\"_blank\" rel=\"noopener\">Create OSS Bucket<\/a> for database backup in Alibaba.<\/em><\/li><li><em><a href=\"https:\/\/www.alibabacloud.com\/help\/doc-detail\/93720.htm\" target=\"_blank\" rel=\"noopener\">Create RAM user<\/a> with programmatic access for the OSS bucket &amp; KMS.<\/em><\/li><li>Jenkins system with a docker plugin installed.<\/li><li>And RDS MySQL DB on which wants to perform backup\/restore.<\/li><\/ul>\n<\/div><\/div>\n\n\n\n<\/p>\r\n<p><strong>Ste<\/strong><em><strong>p1:<\/strong> Create the Docker Image from <a href=\"https:\/\/github.com\/singhjaspreet291\/GFSBackup-AliyunRDS\/tree\/master\/rds_backup\/Docker-Setup\/Using-KMS\" target=\"_blank\" rel=\"noreferrer noopener\">Dockerfile<\/a>.<\/em><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p><em><strong>Step2:<\/strong> Push the Docker Image to the container registry.<\/em><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p><em><strong>Step3:<\/strong> Save the below sensitive information in base64 format in Jenkins credentials, which we will be using as an environmental variable inside the docker container to run the python scripts.<\/em><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p>Open the Jenkins &amp; navigate to credentials \u2192 Systems\u2192 Global credentials \u2192 Add Credentials and select kind as &#8216;Secret text&#8217;.<\/p>\r\n<p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<ul><li><em>Alibaba RDS MySQL User with tag &#8216;Rds-User&#8217;.<\/em><\/li><li><em>Alibaba RDS MySQL Password with tag &#8216;Rds-Pass&#8217;.<\/em><\/li><li><em>Alibaba RDS MySQL Endpoint with tag &#8216;Rds-Endpoint&#8217;.<\/em><\/li><li><em>Alibaba User Access ID with tag &#8216;Aliyun-Access-Id&#8217;.<\/em><\/li><li><em>Alibaba User Access Key with tag &#8216;Aliyun-Access-Key&#8217;.<\/em><\/li><li><em>Alibaba OSS Endpoint with tag &#8216;Bucket-Endpoint&#8217;.<\/em><\/li><li><em>Alibaba OSS Bucket Name with tag &#8216;OSS-Name&#8217;.<\/em><\/li><li><em>Alibaba KMS ID with tag &#8216;kms-id&#8217;.<\/em><\/li><li><em>Alibaba KMS Region with tag &#8216;Kms-Region&#8217;.<\/em><\/li><\/ul>\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"3330\" height=\"710\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2020\/06\/screenshot-2020-06-15-at-7.19.57-pm.png?w=840\" alt=\"\" class=\"wp-image-3520\" \/><\/figure><\/div>\n\n\n\n<\/p>\r\n<p><em><strong>Step4:<\/strong> Create the 3 Jobs as type &#8216;Pipeline&#8217; for automated backups namely &#8216;mysqlBackupHourly&#8217;, &#8216;mysqlBackupWeekly&#8217; &amp; &#8216;mysqlBackupMonthly&#8217; using the <a href=\"https:\/\/github.com\/singhjaspreet291\/GFSBackup-AliyunRDS\/tree\/master\/rds_backup\/Docker-Setup\/Jenkinsfile\" target=\"_blank\" rel=\"noreferrer noopener\">Jenkinsfiles<\/a><\/em>, as shown below.<em> Make sure you already have pushed these Jenkins files to your favorite Git repository<\/em><\/p>\r\n<p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"694\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2020\/06\/jenkins.png?w=1024\" alt=\"\" class=\"wp-image-3608\" \/><\/figure><\/div>\n\n\n\n<\/p>\r\n<p>&nbsp;<\/p>\r\n<p><em><strong>Step5:<\/strong> Similarly a Job can be created for the restoration.<\/em><\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p>So now you know how to configure the GFS Strategy for MySQL RDS using Jenkins Automation and I hope this will ease your problem for backing up &amp; restoration.<\/p>\r\n<p>\n\n\n\n<\/p>\r\n<p>Thank You &amp; Have a nice day \ud83d\ude42<\/p>\r\n<p><a href=\"https:\/\/i0.wp.com\/techpatio.com\/wp-content\/uploads\/2011\/05\/son-father-grandfather-backup-system.png\" target=\"_blank\" rel=\"noopener\">Image Source<\/a><\/p>\r\n<p>&nbsp;<\/p>\r\n<p>Opstree is an End to End DevOps solution provider<\/p>\r\n<p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link\" title=\"https:\/\/www.opstree.com\/contact-us\" href=\"https:\/\/www.opstree.com\/contact-us\" target=\"_blank\" rel=\"noopener\">contact us<\/a><\/div>\r\n<\/div>\r\n<p>\n<\/div>\n\n\n<p><\/p>","protected":false},"excerpt":{"rendered":"<p>&nbsp; So, basically what is Grandfather-father-son or GFS? GFS backup is a common rotation scheme for backup, in which there are three or more backup cycles, such as daily, weekly, and monthly. Typically, It consists of daily backups (son, at fixed intervals of hours in a day), a weekly full backup (father, once a week), &hellip; <a href=\"https:\/\/opstree.com\/blog\/2020\/06\/23\/automatically-backup-restore-alibaba-mysql-using-grandfather-father-son-strategy\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Automatically Backup Alibaba MySQL using Grandfather-Father-Son Strategy&#8221;<\/span><\/a><\/p>\n","protected":false},"author":173461938,"featured_media":29900,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474],"tags":[929443,495125218,44070,5469,218619,701536378,768739299,768739289,52697669,832,557092],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/11\/DevSecOps-1.jpg","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-TN","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/3459"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/173461938"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=3459"}],"version-history":[{"count":25,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/3459\/revisions"}],"predecessor-version":[{"id":3772,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/3459\/revisions\/3772"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29900"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=3459"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=3459"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=3459"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}