{"id":215,"date":"2019-01-21T06:49:00","date_gmt":"2019-01-21T06:49:00","guid":{"rendered":""},"modified":"2026-01-08T12:23:54","modified_gmt":"2026-01-08T06:53:54","slug":"log-parsing-of-windows-servers-on-instance-termination","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2019\/01\/21\/log-parsing-of-windows-servers-on-instance-termination\/","title":{"rendered":"Log Parsing of Windows Servers on Instance Termination"},"content":{"rendered":"

Introduction<\/h2>\n

Logs play a critical role in any application or system. They provide deep visibility into what the application is doing, how requests are processed, and what caused an error. Depending on how logging is configured, logs may contain transaction history, timestamps, request details, and even financial information such as debits or credits.<\/p>\n

In enterprise environments, applications usually run across multiple hosts. Managing logs across hundreds of servers can quickly become complex. Debugging issues by manually searching log files on multiple instances is time consuming and inefficient. This is why centralizing logs is considered a best practice.<\/p>\n

Recently, I encountered a common challenge in AWS environments where application logs need to be retained from instances running behind an Auto Scaling Group. This blog explains a practical solution to ensure logs are preserved even when instances are terminated.<\/p>\n

Problem Scenario<\/h2>\n

Assume your application writes logs to the following directory on a Windows instance.<\/p>\n

\n
C<\/span>:<\/span>\\Source<\/span>\\Application<\/span>\\web<\/span>\\logs<\/span>
\n<\/code><\/div>\n<\/div>\n

Traffic to the application is variable. At low traffic, two EC2 instances may be sufficient. During peak traffic, the Auto Scaling Group may scale out to twenty or more instances.<\/p>\n

When traffic increases, new EC2 instances are launched and logs are generated normally. However, when traffic drops, Auto Scaling triggers scale-down events and terminates instances. When an instance is terminated, all logs stored locally on that instance are lost.<\/p>\n

This makes post-incident debugging and auditing difficult.<\/p>\n

Solution Overview<\/h2>\n

The goal is to synchronize logs from terminating EC2 instances before they are fully removed.<\/p>\n

This solution uses AWS services to trigger a PowerShell script through AWS Systems Manager at instance termination time. The script archives logs and uploads them to an S3 bucket with identifying information such as IP address and date.<\/p>\n

To achieve this, two prerequisites are required.<\/p>\n

    \n
  1. \n

    Systems Manager must be able to communicate with EC2 instances<\/p>\n<\/li>\n

  2. \n

    EC2 instances must have permission to write logs to Amazon S3<\/p>\n<\/li>\n<\/ol>\n

    Environment Used<\/h2>\n

    For this setup, the following AMI was used.<\/p>\n

    \n
    \n
    \n
    \u00a0<\/div>\n<\/div>\n<\/div>\n
    Microsoft<\/span> Windows<\/span> Server<\/span> 2012 <\/span>R2<\/span> Base<\/span>
    \nAMI ID:<\/span> ami-0f7af6e605e2d2db5<\/span>
    \n<\/code><\/div>\n
    <\/div>\n<\/div>\n

    Step 1 Configuring Systems Manager Access on EC2<\/h2>\n

    SSM Agent is installed by default on Windows Server 2016 and on Windows Server 2003 to 2012 R2 AMIs published after November 2016.<\/p>\n

    For older Windows AMIs, EC2Config must be upgraded and SSM Agent installed alongside it.<\/p>\n

    The following PowerShell script upgrades EC2Config, installs SSM Agent, and installs AWS CLI.
    Use this script only for instructional and controlled environments.<\/p>\n

    PowerShell Script to Install Required Components<\/h3>\n
    \n
    \n
    \n
    \u00a0<\/div>\n<\/div>\n<\/div>\n
    # Create temporary directory if not present<\/span>
    \nif<\/span> (!(Test-Path<\/span> -Path<\/span> C:\\Tmp)) {
    \n New-Item<\/span> -ItemType<\/span> Directory -Path<\/span> C:\\Tmp
    \n}<\/p>\n

    Set-Location<\/span> C:\\Tmp<\/p>\n

    # Download installers<\/span>
    \nInvoke-WebRequest<\/span> \"https:\/\/s3.ap-south-1.amazonaws.com\/asg-termination-logs\/Ec2Install.exe\"<\/span> -OutFile<\/span> Ec2Config.exe
    \nInvoke-WebRequest<\/span> \"https:\/\/s3.amazonaws.com\/ec2-downloads-windows\/SSMAgent\/latest\/windows_amd64\/AmazonSSMAgentSetup.exe\"<\/span> -OutFile<\/span> ssmagent.exe
    \nInvoke-WebRequest<\/span> \"https:\/\/s3.amazonaws.com\/aws-cli\/AWSCLISetup.exe\"<\/span> -OutFile<\/span> awscli.exe<\/p>\n

    # Install EC2Config<\/span>
    \nStart-Process<\/span> C:\\Tmp\\Ec2Config.exe -ArgumentList<\/span> \"\/Ec \/S \/v\/qn\"<\/span> -Wait<\/span>
    \nStart-Sleep<\/span> -Seconds<\/span> 20<\/span><\/p>\n

    # Install AWS CLI<\/span>
    \nStart-Process<\/span> C:\\Tmp\\awscli.exe -ArgumentList<\/span> \"\/Ec \/S \/v\/qn\"<\/span> -Wait<\/span>
    \nStart-Sleep<\/span> -Seconds<\/span> 20<\/span><\/p>\n

    # Install SSM Agent<\/span>
    \nStart-Process<\/span> C:\\Tmp\\ssmagent.exe -ArgumentList<\/span> \"\/Ec \/S \/v\/qn\"<\/span> -Wait<\/span>
    \nStart-Sleep<\/span> -Seconds<\/span> 10<\/span><\/p>\n

    Restart-Service<\/span> AmazonSSMAgent<\/p>\n

    Remove-Item<\/span> C:\\Tmp -Recurse<\/span> -Force<\/span>
    \n<\/code><\/div>\n<\/div>\n

    IAM Role for Systems Manager<\/h2>\n

    The EC2 instance must have an IAM role that allows it to communicate with Systems Manager.<\/p>\n

    Attach the following managed policy to the instance role.<\/p>\n

    \n
    \n
    \n
    \u00a0<\/div>\n<\/div>\n<\/div>\n
    AmazonEC2RoleforSSM<\/span>
    \n<\/code><\/div>\n<\/div>\n

    Once attached, the role should appear under the instance IAM configuration.<\/p>\n

    \"index\"<\/p>\n

    Step 2 Allowing EC2 to Write Logs to S3<\/h2>\n

    The EC2 instance also needs permission to upload logs to S3.<\/p>\n

    Attach the following policy to the same IAM role.<\/p>\n

    \n
    \n
    \n
    \u00a0<\/div>\n<\/div>\n<\/div>\n
    AmazonS3FullAccess<\/span>
    \n<\/code><\/div>\n<\/div>\n

    In production environments, it is recommended to scope this permission to a specific bucket.<\/p>\n

    \"index\"<\/p>\n

    PowerShell Script for Log Archival and Upload<\/h2>\n

    Save the following script as shown below.<\/p>\n

    \n
    \n
    \n
    \u00a0<\/div>\n<\/div>\n<\/div>\n
    C:\\Scripts\\termination.ps1<\/span>
    \n<\/code><\/div>\n<\/div>\n

    This script performs the following actions.<\/p>\n