In the current era, organizations demand high-quality working data, and management systems that can scale, deploy quickly, robustly, are highly available, and highly secure for any unfortunate incidents. Traditionally, applications used relational databases as the primary data stores but in today’s need for data-driven applications, developers lean towards alternative databases like NoSQL(Not Only Structured Query Language).
NoSQL databases enable speed, flexibility, and scalability in this era of growing development in the cloud. Moreover, NoSQL databases also support JSON-like documents which are commonly used formats to share data in modern web applications.
MongoDB is an open-source NoSQL database that stores data in flexible, JSON-like documents gathered together in collections unlike data stored in a relational database in tables, rows, and columns, with the relationship of entities. MongoDB is a document-oriented database which means it works on principles of dealing with “documents”; it allows you to express data in its natural form, the way it’s meant to be. Moreover, it provides more flexibility, as fields can vary from document to document and data structure can change over time as well.
Need for Monitoring of MongoDB
The database is an extremely critical component for the high performance of applications, and most of the time its performance is directly proportional to application performance. Yet in quite a few cases, organizations ignore this fact leading to performance degradation and incidentally failing to identify issues proactively. In addition, it also helps you calculate your database resource requirements in various situations and helps you provision your database with the right amount of resources in the future.
In this blog, we are going to cover the key metrics which are needed to monitor the MongoDB database. We are using the Prometheus stack to capture the metrics using MongoDB exporter.
Key Metrics of Mongodb we are collecting (Mongodb Dashboard)
MongoDB Service UP: This metric depicts whether your MongoDB database service is running on the server or not. So, prompt alerting on this one is self-explanatory.
MongoDB Uptime: How long was MongoDB service/database running? This is a time-based metric and it will reset once the MongoDB service gets restarted. It is a frequently looked-up metric for troubleshooting, especially, in containerized environments.
MongoDB Connections
Once the database starts, it listens to a particular IP address and port for requests. This is how connections are established with clients. By running db.serverStatus().connections we can check the current, total, and active connections of the MongoDB database. If we plot these connections in a graph with history, we can also measure the peak, normal, and low traffic time. This helps us schedule the patching and upgrading activities of databases and applications.
Instant increases in traffic can lead to performance issues. This can be visualized by using the current connections metric. It can notify about the drastic change in traffic patterns via alerting as well.
- Current Connections: The number of incoming connections from the client to the database server, also includes any shell connections or connections from other servers as well.
- Total Created Connections: Include all incoming connections created to the server, and includes connections that have since closed.
- Available Connections: The number of unused incoming connections available
- Active Connections: Number of client connections that currently have operations in progress.
OPCounters
A document that reports the average number of operations in a database by type since the MongoDB instance last started is known as OPCounters. We are going to monitor opcounters.insert, opcounters.query, opcounters.update, opcounters.delete, opcounters.getmore, and opcounters.command metrics from MongoDB which tell us the number of inserts, update, delete, getMore operations respectively. Moreover, also going to get the metrics about the number of queries and commands received on MongoDB.
Total Collections Per DB
It is always beneficial to track the total number of collections in MongoDB. A collection is a group of documents just like the tables in Relational databases. These metrics can help to keep a record of the number of collections per database.
Total Indexes Per DB
The indexes are special data structures that support the efficient execution of queries in MongoDB. It stores some information like the field or sub-field of the document to find the right data in less time. This metric lets you know how many indexes are created per DB.
Allocated Storage to Collections and Indexes
The metrics illustrate the total amount of storage in bytes allocated to each collection and index in the MongoDB database. So, for example, it is necessary to monitor these metrics in scenarios where a chunk of data got deleted. It will reflect in these metrics and should trigger an alert about this intended or unintended activity. It will also be helpful where storage drastically increases as it can lead to a crash of database service.
Replication Metrics
MongoDB can be deployed in Primary and Secondary replication mode over multiple servers which ensures that your database is highly available, and it can store your data in multiple physical partitions called shards. Normally, in a production environment replica set is recommended. So, we are going to monitor the following key metrics.
- OPLog Size: The OPLog(Operation Logs) is a special capped collection that keeps a record of all operations that modify the data stored in your database. So this OPLog size should be consistent, if this is less than 25% or more than 50% this metric should trigger an alert.
- Replication Lag: In replication mode, the primary MongoDB server replicates its data to the secondary MongoDB server. For better performance this replication should be up to date and as fast as possible. Replication lags tell us how far a secondary MongoDB server is behind from Primary one. We want replication lag as small as possible. There can be multiple reasons like the issue with networking Latency or Throughput etc. So, this alert should trigger when replication lag crosses a particular threshold.
MongoDB Server Metrics
The monitoring and alerting of MongoDB server metrics are as important as capturing the MongoDB Database metrics. For example, CPU, and memory utilization of the server. This can tell us the overall load on the server, the server can also crash because of a high load, or your MongoDB service consumes the whole memory of the server. So, decide on a suitable threshold like 70% or 80%, and trigger an alert whenever utilization of these metrics crosses this threshold. Apart from that, we also need to consider Network I/O and Disk I/O metrics as well. If these metrics cross the threshold on regular basis with the normal working of the application and normal traffic, it indicates that we have under-provisioned our MongoDB cluster.
Tools We are going to use to Monitor MongoDB
- Prometheus – An open-source time series database for event monitoring and alert manager by Cloud Native Computing Foundation(CNCF).
- Grafana – An open-source project used for analytics and visualizations of the metrics from any database.
- MongoDB Exporter – A Prometheus exporter which acts like an agent to publish the metrics (http://your_server_ip:9216/metrics)
- Node Exporter – Another Prometheus exporter is used to collect server metrics, such as CPU and memory utilization, disk space-related metrics, etc.
Setup MongoDB Exporter
We are going to use MongoDB Exporter developed by Percona. In the following steps, we will run MongoDB Exporter as a service along with Node Exporter on the MongoDB Server.
To isolate the ownership of the Prometheus Exporters services, we will create separate users without a home directory and shell so that nobody can log in to these users.
sudo useradd --no-create-home --shell /bin/false exporterAfterward, Download the latest MongoDB Exporter from here. Extract the archive file and copy the binary file to /use/local/bin.
wget https://github.com/percona/mongodb_exporter/releases/download/v0.34.0/mongodb_exporter-0.34.0.linux-amd64.tar.gz
tar xvzf mongodb_exporter-0.34.0.linux-amd64.tar.gz
sudo mv mongodb_exporter /usr/local/bin/Now change the ownership of the binary file.
sudo chown exporter:exporter /usr/local/bin/mongodb_exporterAccessing MongoDB should be authenticated and authorized. MongoDB Exporter going to get metrics from MongoDB Cluster. So, we need a user who can authenticate the MongoDB Database and should have the right set of permissions to read all the data. We need to create an Administrator account with clusterMonitor role.
use admin
db.createUser(
{
user: "mongodb_exporter",
pwd: "password",
roles: [
{ role: "clusterMonitor", db: "admin" },
{ role: "read", db: "local" }
]
}
)The output should be like this:
Successfully added user: {
"user" : "mongodb_exporter",
"roles" : [
{
"role" : "clusterMonitor",
"db" : "admin"
},
{
"role" : "read",
"db" : "local"
}
]
} Now We have to run the MongoDB Exporter service with the above mongodb_exporter user authentication with the following command in which we are going to provide the MONGODB_URI the parameter in the command line.
/usr/local/bin/mongodb_exporter --mongodb.uri=mongodb://mongodb_exporter:password@localhost:27017 --collect-allNow we have MongoDB Exporter binary, user, and mongodb_exporter user to authenticate the Database. The next step would be to create a service to run MongoDB Exporter Binary and collect metrics. Create the following file in /etc/systemd/system/ directory to create a mongodb_exporter service.
[Unit]
Description=Mongodb Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=exporter
Group=exporter
Type=simple
ExecStart=/usr/local/bin/mongodb_exporter --mongodb.uri=mongodb://mongodb_exporter:password@localhost:27017 --collect-all
[Install]
WantedBy=multi-user.targetTo use the newly created service, reload systemd.
sudo systemctl daemon-reloadWe can restart the service after reloading the system daemon.
sudo systemctl start mongodb_exporter.serviceTo check the status of the service, run the following command.
sudo systemctl status mongodb_exporter.serviceIf we want to make sure that MongoDB Exporter service runs even after rebooting the server as well, we need to enable the service.
sudo systemctl enable mongodb_exporter.serviceSetup Node Exporter
Download the latest Node Exporter binary, click here
To set up the Node Exporter through Ansible, click here
To Install Node Exporter on Linux with Systemd service, click here.
Setup Prometheus
Download the latest Prometheus binary from here.
To set up The Prometheus through Ansible, please click here.
If you want to install Prometheus as a service, you can follow the steps from here.
Setup Grafana
To Download and Install Grafana please check here.
To Set up the Grafana through Ansible, please click here.
Configure MongoDB Exporter and Node Exporter with Prometheus
To Configure MongoDB Exporter and Node Exporter with Prometheus, we need to add the following configuration in prometheus.yml.
- job_name: 'mongodb'
scrape_interval: 5s
static_configs:
- targets: ['mongodb_exporter_ip:9216', 'mongodb_exporter_ip:9100']In targets, we need to add the server IP where exporters are running, and 9216 and 9100 is the port of MongoDB and Node Exporter.
Set Prometheus As Data Source in Grafana
Firstly we need to add Prometheus as the data source in Grafana to fetch the metrics data. Add the Datasource like below.
Import MongoDB Dashboard in Grafana
After setting up Prometheus as a data source, we are going to import the MongoDB dashboard with the ID. This Id is a unique ID given by Grafana when we upload a custom dashboard.
We need to click the Plus sign on the left menu bar from the Garfana homepage. Then click on the import, we will be on the import page, where we can provide a unique Garfana Dashboard ID and then load the dashboard. In this blog, we are going to import https://grafana.com/grafana/dashboards/16490-opstree-mongodb-dashboard/ this dashboard and the ID of this dashboard is 16490.
Or if we have a Dashboard in JSON format, or we downloaded the dashboard, we can import it by uploading the JSON file from the local machine.
Screenshots of MongoDB Dashboard created by us
Conclusion
The purpose of this blog was to dig a little deeper into MongoDB Concepts. The use of the MongoDB database and identifying the key metrics which can be monitored. We also set up a fully operational end-to-end monitoring pipeline for MongoDB with respect to the metrics we talked about in this blog. We published the custom dashboard in Grafana, check here.
Now, in our next blog, we will talk about MongoDB Monitoring from an SRE perspective.
Want to check other dashboards published by us, please check here.
Blog Pundits: Naveen Verma and Sandeep Rawat
Opstree is an End to End DevOps solution provider.
Connect with Us