{"id":9798,"date":"2022-02-08T16:00:34","date_gmt":"2022-02-08T10:30:34","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=9798"},"modified":"2022-02-08T16:00:34","modified_gmt":"2022-02-08T10:30:34","slug":"learn-the-hacks-for-running-custom-scripts-at-spot-termination","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2022\/02\/08\/learn-the-hacks-for-running-custom-scripts-at-spot-termination\/","title":{"rendered":"Learn the Hacks for Running Custom Scripts at Spot Termination"},"content":{"rendered":"\n

Nowadays, it is very common to run applications on Spot instances. In this scenario, where a spot instance could be terminated at any point of time because of AWS pulling back their resource or ASG Scale-In incident, we need to have something in place to handle the termination smoothly so that we can complete our final tasks before the system shutdown. It could be executing some scripts, unmounting some storage device, shipping final log files to S3, or uploading cache data in a centralized server like Redis.
Today, I will attempt to cater to this problem.

First of all, let’s think of trying to run a custom script prior to shutdown in our local system. If everything works fine, the same would be applicable for ec2 spot instances too.<\/p>\n\n\n\n\n\n\n\n

I will be using a Ubuntu 20 machine that uses systemd <\/em>as the service manager for this entire blog (works for Ubuntu 16\/18\/20\/22 as well).

As all of you might know, we can add our custom service file in the \/etc\/systemd\/system\/ location and install it in any of the target levels to run your program\/script as a service.

Below is an example of a service file that can run your custom script prior to shutdown, and will hold the shutdown process until your script is completed.<\/p>\n\n\n\n

[root@ss ~]# cat \/etc\/systemd\/system\/run-pre-shutdown.service\n[Unit]\nDescription=Run custom task prior shutdown\nDefaultDependencies=no\nBefore=shutdown.target\n\n[Service]\nType=oneshot\nExecStart=<your script here>\nTimeoutStartSec=0\n\n[Install]\nWantedBy=shutdown.target\n<\/code><\/pre>\n\n\n\n
So, this script basically tells the system to run the ExecStart<\/code> to run before shutdown. The  TimeoutStartSec<\/code> directive tells the time the system will wait for the process to start (or completed in this case). One can read details about the directives here<\/a>.

Next, make sure you have executable permissions on the run-pre-shutdown.service<\/strong> and the custom script mentioned in ExecStart<\/code>.
Then run systemctl daemon-reload<\/code> followed by systemctl enable run-pre-shutdown.service<\/code>.

So far so good, but this script won’t be able to access the internet or do a curl and hence will not be able to ship logs to the S3 bucket or put data into Redis. This is because the network will be already shut down when systemd<\/em> reaches to execute this script. To handle this we can use a systemd<\/em> unit file similar to this  below. This unit file ships data from the \/var\/log\/<\/code> directory in your system to your S3 bucket. <\/p>\n\n\n\n
ss@ubuntumini:\/etc\/systemd\/system$ cat run-before-shutdown.service\n[Unit]\nDescription=Send Logs\nRequires=network-online.target\nAfter=network.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStop=\/usr\/bin\/aws s3 --region ap-south-1 sync \/var\/log\/ s3:\/\/bucket-name\/path-to-folder\/\n\n\n[Install]\nWantedBy=network.target\n<\/code><\/pre>\n\n\n\n
Let’s break down the parts of this unit file.

The shutdown ordering of units in systemd<\/em> is the reverse of the startup order. Hence, any unit that is ordered After=network.target<\/code> can be sure that it is stopped before the network is shut down if the system is powered off\/terminated. And our purpose is solved here.

Note<\/strong>: It is not mandatory here to install it under network.target<\/strong>, we can also specify as WantedBy=multi-user.target<\/code>

For better understanding, I will just rename the file name, since systemd<\/em> orders units in the alphabet order as well, I will rename it to aa-run-before-shutdown.service<\/code>
<\/p>\n\n\n\n
root@ubuntumini:\/etc\/systemd\/system# mv run-before-shutdown.service aa-run-before-shutdown.service<\/strong>\n\nroot@ubuntumini:\/etc\/systemd\/system# systemctl daemon-reload\nroot@ubuntumini:\/etc\/systemd\/system# systemctl enable aa-run-before-shutdown.service\nCreated symlink \/etc\/systemd\/system\/network.target.wants\/aa-run-before-shutdown.service \u2192 \/etc\/systemd\/system\/aa-run-before-shutdown.service.\n\nroot@ubuntumini:\/etc\/systemd\/system# systemctl start aa-run-before-shutdown.service\nroot@ubuntumini:\/etc\/systemd\/system# systemctl status aa-run-before-shutdown.service\n\u25cf aa-run-before-shutdown.service - Send Logs\n     Loaded: loaded (\/etc\/systemd\/system\/aa-run-before-shutdown.service; enabled; vendor preset: enabled)\n     Active<\/strong>: active (exited) since Fri 2022-02-04 06:35:19 UTC; 2s ago\n\nFeb 04 06:35:19 ubuntumini systemd[1]: Finished Send Logs.\nroot@ubuntumini:\/etc\/systemd\/system#<\/code><\/pre>\n\n\n\n
Note:  It is mandatory that the service is Started and the status is Active as above.<\/p>\n\n\n\n
[optional]:<\/strong> Now if you run the command – systemctl list-dependencies --after shutdown.target<\/code>  we will be able to see the service with green <\/strong>(meaning active state) at the top.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n

In our case, make sure you have awscli<\/em><\/strong> installed and have Access\/Secret keys configured with proper permissions to Get\/Put on the S3 bucket.

Next, let’s shut down the system to see this working.

<\/p>\n\n\n\n
You will see a job is running that has halted the shutdown process.  Finally when the logs are sent the process will be stopped, the shutdown will be complete. Check your S3 bucket and the files should be there.


The same procedure can be applied on a Spot instance and it works absolutely perfectly. And the entire steps can be automated via Ansible<\/strong> as well.


Challenges<\/strong>:-<\/p>\n\n\n\n