On an enterprise level application, your system goes to multiple hosts, managing the logs across multiple hosts can be complicated. Debugging the error in the application across hundreds of log files on hundreds of servers can be very time consuming and complicated and not the right approach so it is always better to move the logs to a centralized location.
Faced a similar situation ? No worries, now in order to retain logs I figured out an absolute solution.
Here, in this blog, the motive is to sync the logs from dying instances at the time of their termination. This will be done using AWS Services, the goal is to trigger a Powershell Script in the instance using SSM which sync logs to S3 Bucket with sufficient information about the dying instances. For this we will require 2 things:
For the tutorial we will be using Microsoft Windows Server 2012 R2 Base with the AMI ID: ami-0f7af6e605e2d2db5
A Blueprint of the scenario to be understood below:
#ScriptBlock
}
cd C:/Tmp
wget https://s3.ap-south-1.amazonaws.com/asg-termination-logs/Ec2Install.exe -OutFile Ec2Config.exe
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/windows_amd64/AmazonSSMAgentSetup.exe -OutFile ssmagent.exe
wget https://s3.amazonaws.com/aws-cli/AWSCLI64PY3.msi -OutFile awscli.msi
wget https://s3.amazonaws.com/aws-cli/AWSCLISetup.exe -OutFile awscli.exe
Invoke-Command -ScriptBlock {C:\Tmp\Ec2Config.exe /Ec /S /v/qn }
sleep 20
Invoke-Command -ScriptBlock {C:\Tmp\awscli.exe /Ec /S /v/qn }
sleep 20
Invoke-Command -ScriptBlock {C:\Tmp\ssmagent.exe /Ec /S /v/qn }
sleep 10
Restart-Service AmazonSSMAgent
Remove-Item C:\Tmp
-
Open the IAM console at https://console.aws.amazon.com/iam/.
-
In the navigation pane, choose Roles, and then choose the existing role you want to associate with an instance profile for Systems Manager operations.
-
On the Permissions tab, choose Attach policy.
-
On the Attach policy page, select the check box next to AmazonEC2RoleforSSM, and then choose Attach policy.
-
Open the IAM console at https://console.aws.amazon.com/iam/.
-
In the navigation pane, choose Roles, and then choose the existing role you want to associate with an instance profile for Systems Manager operations.
-
On the Permissions tab, choose Attach policy.
-
On the Attach policy page, select the check box next to AmazonS3FullAccess, and then choose Attach policy.
This Powershell script saved in C:/Scripts/termination.ps1 will pick up log files from:
$SourcePathWeb:
and will output logs into:
Make sure that the s3 bucket name and –region and source of log files is changed according to the preferences.
#ScriptBlock
$Date=Get-Date -Format yyyy-MM-dd
$InstanceName=”TerminationEc2″
$LocalIP=curl http://169.254.169.254/latest/meta-data/local-ipv4 -UseBasicParsing
if((Test-Path -Path C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date )){
Remove-Item “C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date” -Force -Recurse
}
New-Item -path “C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date” -type directory
$SourcePathWeb=”C:\Source\Application\web\logs”
$DestFileWeb=”C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date/logs.zip”
Add-Type -assembly “system.io.compression.filesystem”
[io.compression.zipfile]::CreateFromDirectory($SourcePathWeb, $DestFileWeb)
C:\’Program Files’\Amazon\AWSCLI\bin\aws.cmd s3 cp C:\Users\Administrator\workdir s3://terminationec2 –recursive –exclude “*.ok” –include “*” –region us-east-1
Check your S3, Bucket for seeing if it has synced logs to there. Now, because the focus of this blog trigger a Powershell Script in the instance using SSM which syncs the logs to S3 Bucket so we will try running the script through SSM > Run Command.
Select and run of the instances having the above script and configuration. The output should be pleasing.
The AMI used by the ASG should have the above configuration (Can be archived via created a ami from ec2 having above config and then adding it into Launch Configuration of the ASG). The ASG we have here for the tutorial is named after my last name : “group_kaien”.
Now, the last and the Most important step is configuration theCloudwatch > Event > Rules.
This would return the following JSON config:
“
-
Document: AwsRunPowerShellScript
-
Target key: “Instanceids or tag:
-
Target Values:
-
Commands: .\termination.ps1
-
WorkingDirectory: C:\Scripts.ps1
-
ExecutionTimeout: 3600 is default
For more on setting up Cloudwatch Events refer :
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_GettingStarted.html
Conclusion
Hence, fulfilling the scope of the blog.
Stay tuned for more
