{"id":19923,"date":"2025-01-14T19:09:10","date_gmt":"2025-01-14T13:39:10","guid":{"rendered":"https:\/\/opstree.com\/blog\/?p=19923"},"modified":"2025-07-23T11:46:21","modified_gmt":"2025-07-23T06:16:21","slug":"part-2-automating-data-migration-with-apache-airflow","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2025\/01\/14\/part-2-automating-data-migration-with-apache-airflow\/","title":{"rendered":"Automating Data Migration Using Apache Airflow: A Step-by-Step Guide"},"content":{"rendered":"\r\n

In this second part of our blog, we\u2019ll walk through how we automated the migration process using Apache Airflow. We\u2019ll cover everything from unloading data from Amazon Redshift to S3, transferring it to Google Cloud Storage (GCS), and finally loading it into Google BigQuery. This comprehensive process was orchestrated with Airflow to make sure every step was executed smoothly, automatically, and without error.<\/p>\r\n

<\/p>\r\n

Step 1: Setting Up Apache Airflow<\/h2>\r\n

Before diving into the migration tasks, we first need to ensure that Apache Airflow is properly set up. Here\u2019s how I set it up for my project.<\/p>\r\n

1. Install Apache Airflow: If you don’t have Airflow installed, use the following command to install it via pip:<\/p>\r\n\r\n\r\n\r\n

pip                      install                              apache-airflow<\/code><\/pre>\r\n\r\n\r\n\r\n
2. Initialize the Airflow Database:<\/strong> Airflow requires a backend database to track tasks and maintain state. To initialize it, run:<\/p>\r\n\r\n\r\n\r\n
airflow db init<\/code><\/pre>\r\n\r\n\r\n\r\n
3. Start the Web Server and Scheduler:<\/strong> Once Airflow is initialized, start the web server (for monitoring) and scheduler (to run tasks):<\/p>\r\n\r\n\r\n\r\n
airflow webserver --port 8080 airflow scheduler<\/code><\/pre>\r\n\r\n\r\n\r\n
\"\"<\/figure>\r\n\r\n\r\n\r\n
    \r\n
  1. Set Up Connections in Airflow<\/strong>:\u00a0<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
      \r\n
    1. Redshift<\/strong>: Create a connection in the Airflow UI under Admin \u2192 Connections<\/strong> for Redshift (with your host, database, username, and password).\u00a0<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
        \r\n
      1. AWS<\/strong>: Set up an AWS connection<\/a> with your AWS Access Key<\/strong> and Secret Access Key<\/strong>.\u00a0<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
          \r\n
        1. Google Cloud<\/strong>: Set up a connection for Google Cloud<\/strong> using a service account (we’ll go into this below).\u00a0<\/li>\r\n<\/ol>\r\n
          [ Also Read Part 1: How to Optimize Amazon Redshift for Faster and Seamless Data Migration<\/a>]<\/strong><\/p>\r\n\r\n\r\n\r\n
          Step 2: Setting Up Google Cloud Resources<\/strong>\u00a0<\/h2>\r\n\r\n\r\n\r\n
          Before we can transfer data from S3 to Google Cloud Storage (GCS) and eventually to BigQuery, we need to configure the necessary resources in Google Cloud<\/strong>.\u00a0<\/p>\r\n\r\n\r\n\r\n
          Create a Google Cloud Project<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n
            \r\n
          1. Go to the Google Cloud Console<\/strong>.\u00a0<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
              \r\n
            1. Create a new project (e.g., migration-project).\u00a0

              Enable the Necessary APIs<\/strong>\u00a0<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
              To interact with Google Cloud services, you need to enable the following APIs:\u00a0<\/p>\r\n\r\n\r\n\r\n