Why Opt for Integrating Alerts with OpenSearch?
Monitoring the OpenSearch visualization dashboard 24*7 becomes challenging when application or system logs fail to appear. Therefore, it’s essential to establish alerts that promptly notify us when such issues arise. The flexibility and scalability of OpenSearch make it a compelling choice for implementing alerting solutions in various domains, from IT operations and security to business intelligence and analytics.
Overview of OpenSearch cluster
OpenSearch, a powerful and versatile search and analytics engine, plays a crucial role in helping you gain valuable insights from your data. However, staying on top of potential issues and anomalies in your OpenSearch cluster requires a proactive approach.
OpenSearch provides a robust alerting and monitoring framework that enables you to set up customized alerts based on your specific use cases and requirements. By defining alert conditions that trigger notifications, you can be instantly notified about critical events such as:
- Abnormal CPU utilization
- Out-of-memory incidents
- Query performance degradation
- Data indexing failures
What is SNS (Simple Notification Service)
AWS SNS is a versatile notification service that empowers you to build event-driven, cloud-native applications. In this blog post, we’ll explore the process of integrating OpenSearch alerts with AWS SNS, and we’ll uncover how this integration can greatly enhance your monitoring capabilities.
So now we will deep-dive into the integration of OpenSearch alerts with AWS SNS.
Step 1
Now we will have to create an SNS topic.
Log in to the AWS Management Console, then navigate to the SNS service. Click on ‘Create topic’ and provide a name along with a display name for your topic. Afterward, click ‘Create topic.’
Remember to note down the Amazon Resource Name (ARN) of the newly created SNS topic.
Step 2
As we have created an SNS Topic, we will now need to create a subscription endpoint where we can receive an alert. So in our case, I want to get a notification on my email ID hence email is selected as an endpoint for the subscription.
Step 3
Next, create an OpenSearch role that grants access to the SNS service, enabling it to initiate alerts.
Step 4
Now attach the SNS Full access policy for this role.
Update the trust relationship to openSearch for the same.
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Principal": {
"Service": [
"opensearchservice.amazonaws.com"
]
}
}
]
}
Step 5
Once we have created a role in IAM. Now navigate to your OpenSearch cluster and update the access policy of openSearch cluster.
Go to the security panel to update the access policy.
Having established a role with complete SNS access and having designated Amazon OpenSearch as a trusted entity, we need to incorporate this role into the access policy of your OpenSearch cluster.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:ap-south-1:<account-id>:domain/<opensearch-domain-name>/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:role/SNS-ACCESS"
},
"Action": "es:ESHttpPost",
"Resource": "arn:aws:es:ap-south-1:<account-id>:domain/<opensearch-domain-name>/_doc/_bulk"
}
]
}
Step 6
Now Login into OpenSearch then go to the notification channel and create a channel for SNS.
Step 7
To complete the setup, proceed by selecting “Create Channel” and opt for SES as the chosen channel type. Input the SNS Topic ARN and the Role ARN of the previously established role. This configuration ensures the activation of alerts effectively.
Now let’s check an example for setting up alerts on openSearch.
Step 1
Once you log into OpenSearch, navigate to the Alerting section.
Step 2
Once you’re inside the alerting system click on Monitor and click on Create monitor to create an alert.
Step 3
Now we will create an alert to monitor the index. Give a name and time interval that it will monitor the index.
Step 4
Provide the index name for which monitoring needs to be enabled.
Step 5
We will write a query for monitoring the index, aiming to identify instances where data hasn’t been received. In this illustration, will use the env: dev tag that accompanies each log entering the OpenSearch of the fmpp-goal index.
Step 6
Once you’ve established the monitoring query for the index, proceed to configure the trigger. This enables the generation of alerts whenever logs are not generated.
Step 7
Now, define the alert condition based on the specific use case. In my scenario, I’ve configured it to trigger when the count reaches zero. This condition is applicable when no logs are present, meaning the env: dev tag is absent in the fmpp-goal index. Therefore, I’ve precisely set the trigger condition to zero.
Step 8
Now set the alert message and notification channel which we have configured earlier.
Step 9
Click on Create and that will complete the alerting setup on OpenSearch
Step 10
Once the alert is set up using openSearch. You will receive the notification below.
In the same way, you can set up alerts for API Failure as well.
Conclusion
Real-time monitoring and alerts are not just a convenience but a necessity. The integration of OpenSearch alerts with AWS SNS empowers you to take control of your cluster’s health by providing instant notifications that enable swift action. With this integration, you can ensure that potential issues are addressed before they impact your users or disrupt your operations.
Blog Pundits: Deepak Gupta and Sandeep Rawat
OpsTree is an End-to-End DevOps Solution Provider.
Connect with Us