{"id":14033,"date":"2023-05-30T12:03:48","date_gmt":"2023-05-30T06:33:48","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=14033"},"modified":"2023-05-30T12:03:48","modified_gmt":"2023-05-30T06:33:48","slug":"basic-logging-setup-of-loki-grafana","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2023\/05\/30\/basic-logging-setup-of-loki-grafana\/","title":{"rendered":"Basic Logging Setup of Loki Grafana"},"content":{"rendered":"\n

Let’s say you are the DevOps lead for a large e-commerce platform that runs on a microservices architecture with hundreds of services. You need to monitor the logs of all these services to quickly identify issues, troubleshoot problems, and optimize the system’s performance. You also want to be able to search and analyze logs across all services in real time and be alerted when any critical issues arise.<\/p>\n\n\n\n

To address this scenario, you could use Grafana Loki as your centralized logging system. Loki is a lightweight and cost-effective solution that can handle high volumes of logs and store them in a distributed manner. You can configure each service to send logs to Loki, which will automatically index them and make them available for search and analysis.<\/p>\n\n\n\n

Introduction<\/h2>\n\n\n\n

Loki and Grafana are two open-source projects that are commonly used together for log aggregation, analysis, and visualization.<\/p>\n\n\n\n

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be a cost-effective solution for storing and querying logs and uses a unique indexing approach to provide fast and efficient searching of log data. Loki is also highly extensible, allowing users to easily add custom logging drivers and integrate with other systems.<\/p>\n\n\n\n

Grafana, on the other hand, is a popular open-source platform for visualizing and analyzing time-series data, including logs. It provides a powerful and flexible dashboarding system that allows users to create customized visualizations and alerts based on their log data. Grafana also integrates with many different data sources, including Loki, which makes it a great choice for log analysis and visualization.<\/p>\n\n\n\n\n\n\n\n

Why prefer Loki over other logging tools?<\/h3>\n\n\n\n

There are other logging tools available but Loki’s combination of scalability, cost-effectiveness, Prometheus integration, centralized log aggregation, powerful querying capabilities, log stream processing and community support make it a compelling choice for many organizations seeking an efficient and effective logging solution.<\/p>\n\n\n\n

    \n
  1. Efficient scaling: Loki is highly scalable and able to handle massive volumes of log data efficiently thanks to its distributed architecture and compressed, indexed log storage.<\/li>\n\n\n\n
  2. Cost-effectiveness: When compared to conventional log management solutions, Loki’s use of object storage\u2014such as Amazon S3 or Google Cloud Storage\u2014helps lower storage costs.<\/li>\n\n\n\n
  3. Seamless integration with Prometheus: Loki integrates smoothly with Prometheus, allowing for correlation between logs and metrics. This correlation enhances troubleshooting and analysis capabilities.<\/li>\n\n\n\n
  4. Centralized log aggregation and querying: Loki provides a centralized log aggregation solution, enabling logs from multiple sources to be collected and queried in a unified manner. Its powerful filtering and querying capabilities, inspired by Prometheus, make log analysis and troubleshooting more efficient.<\/li>\n\n\n\n
  5. Log stream processing: Loki supports log stream processing, allowing for the extraction of labels and application of filters to logs. This enables actions such as routing logs to different destinations or creating alerting rules based on log content.<\/li>\n\n\n\n
  6. Open-source and community-driven: As an open-source project with an active community, Loki benefits from a wide range of community-contributed integrations, enhancements, and support. This makes it a flexible and well-supported logging solution.<\/li>\n<\/ol>\n\n\n\n

    Loki Architecture<\/h3>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    Installation <\/h2>\n\n\n\n

    There are diff methods available to install Loki e.g. via helm, docker-compose, binary and etc<\/p>\n\n\n\n

    Choose the specific version from site Loki-Releases<\/a> and run via below instructions.<\/p>\n\n\n\n

    While running binary we need to pass config-file. Please find an example of config file below<\/p>\n\n\n\n

    auth_enabled: false\n\nserver:\n  http_listen_port: 3100\n\ningester:\n  lifecycler:\n    address: 127.0.0.1\n    ring:\n      kvstore:\n        store: inmemory\n      replication_factor: 1\n    final_sleep: 0s\n  chunk_idle_period: 5m\n  chunk_retain_period: 30s\n  max_transfer_retries: 0\n\nschema_config:\n  configs:\n    - from: 2018-04-15\n      store: boltdb\n      object_store: filesystem\n      schema: v11\n      index:\n        prefix: index_\n        period: 168h\n\nstorage_config:\n  boltdb:\n    directory: \/data\/loki\/index\n\n  filesystem:\n    directory: \/data\/loki\/chunks\n\nlimits_config:\n  enforce_metric_name: false\n  reject_old_samples: true\n  reject_old_samples_max_age: 168h\n\nchunk_store_config:\n  max_look_back_period: 0s\n\ntable_manager:\n  retention_deletes_enabled: false\n  retention_period: 0s\n<\/code><\/pre>\n\n\n\n
    To learn more about config-file follow link<\/a>. <\/p>\n\n\n\n
    Install Grafana  for visualization<\/h3>\n\n\n\n
    We can install Grafana through linux package manager.<\/p>\n\n\n\n
    Use the given command to install some requisite packages \u2013\nsudo apt install -y gnupg2 curl\n\n# Use the following command to add the Grafana GPG key to your system \u2013\ncurl https:\/\/packages.grafana.com\/gpg.key | sudo apt-key add -\n\n# Run the given command to add Grafana package repository to your system \u2013\nsudo add-apt-repository \"deb https:\/\/packages.grafana.com\/oss\/deb stable main\"\n\nsudo apt update\nsudo apt -y install grafana\n\n# Once the package is installed, use the following command to start and enable Grafana services \u2013\n\nsudo systemctl start grafana-server\nsudo systemctl enable grafana-server\nsudo systemctl status grafana-server<\/code><\/pre>\n\n\n\n
    Install Loki Agent <\/h3>\n\n\n\n
    Grafana Loki supports the following official clients for sending logs:<\/p>\n\n\n\n