{"id":29484,"date":"2025-08-19T13:43:06","date_gmt":"2025-08-19T08:13:06","guid":{"rendered":"https:\/\/opstree.com\/blog\/?p=29484"},"modified":"2025-11-21T12:24:56","modified_gmt":"2025-11-21T06:54:56","slug":"logs-to-unclog-the-complete-guide-to-logging","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2025\/08\/19\/logs-to-unclog-the-complete-guide-to-logging\/","title":{"rendered":"Logs to Unclog: The Complete Guide to Logging"},"content":{"rendered":"<h2 id=\"introduction-to-logging\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Introduction to Logging<\/h2>\n<h3 id=\"b990\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">What Are Logs?<\/h3>\n<p>Logs are chronological records of events that occur within software applications, operating systems, and network devices. They serve as the digital equivalent of a ship\u2019s logbook, documenting what happened, when it happened, and often providing context about why it happened.<\/p>\n<h3 id=\"0620\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Why Logging Matters<\/h3>\n<p>In today\u2019s distributed systems and microservices architectures, logging is not just helpful \u2014 it\u2019s essential. Here\u2019s why:<\/p>\n<ul class=\"\">\n<li id=\"a848\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Debugging<\/strong>: Logs provide crucial information for identifying and fixing bugs<\/li>\n<li id=\"ef67\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Monitoring<\/strong>: They enable real-time monitoring of system health and performance<\/li>\n<li id=\"f0e8\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Security<\/strong>: Logs help <a href=\"https:\/\/opstree.com\/blog\/2025\/02\/10\/cloud-security-posture-management-how-to-stay-compliant\/\">detect security<\/a> incidents and unauthorized access<\/li>\n<li id=\"f64e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Compliance<\/strong>: Many regulations require comprehensive logging for audit trails<\/li>\n<li id=\"e66b\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Performance Analysis<\/strong>: They help identify bottlenecks and optimization opportunities<\/li>\n<li id=\"922e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Business Intelligence<\/strong>: Application logs can provide insights into user behavior and business metrics<\/li>\n<\/ul>\n<p><!--more--><\/p>\n<div class=\"y\">\n<section>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<article>\n<div class=\"y\">\n<div class=\"y\">\n<section>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<div class=\"advanced-toc-box\">\n<div class=\"toc-content-box\">\n<div style=\"background: #ffffff; border: 1px solid #e2e8f0; border-radius: 8px; padding: 20px; margin: 25px 0; box-shadow: 0 1px 3px rgba(0,0,0,0.1);\">\n<h2 style=\"color: #1e40af; margin-top: 0; border-bottom: 2px solid #dbeafe; padding-bottom: 10px;\">Table of Contents<\/h2>\n<ul>\n<li><a href=\"#introduction-to-logging\">Introduction to Logging<\/a><\/li>\n<li><a href=\"#fundamentals-of-logging\">Fundamentals of Logging<\/a><\/li>\n<li><a href=\"#log-levels-and-best-practices\">Log Levels and Best Practices<\/a><\/li>\n<li><a href=\"#structured-vs-unstructured-logging\">Structured vs Unstructured Logging<\/a><\/li>\n<li><a href=\"#popular-logging-frameworks\">Popular Logging Frameworks<\/a><\/li>\n<li><a href=\"#log-management-and-aggregation\">Log Management and Aggregation<\/a><\/li>\n<li><a href=\"#observability-and-the-three-pillars\">Observability and the Three Pillars<\/a><\/li>\n<li><a href=\"#real-world-case-studies\">Real-World Case Studies<\/a><\/li>\n<li><a href=\"#security-and-compliance\">Security and Compliance<\/a><\/li>\n<li><a href=\"#performance-optimization\">Performance Optimization<\/a><\/li>\n<li><a href=\"#industry-news-and-trends\">Industry News and Trends<\/a><\/li>\n<li><a href=\"#tools-and-technologies\">Tools and Technologies<\/a><\/li>\n<li><a href=\"#troubleshooting-common-issues\">Troubleshooting Common Issues<\/a><\/li>\n<li><a href=\"#final-thoughts\">Final Thoughts<\/a><\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"fundamentals-of-logging\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Fundamentals of Logging<\/h2>\n<h3 id=\"8932\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Core Components of a Log Entry<\/h3>\n<p>Every effective log entry should contain these essential elements:<\/p>\n<ol class=\"\">\n<li id=\"f5cf\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe xu xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Timestamp<\/strong>: When the event occurred (preferably in UTC)<\/li>\n<li id=\"4649\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe xu xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Log Level<\/strong>: The severity or importance of the event<\/li>\n<li id=\"611e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe xu xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Source<\/strong>: Which component, service, or module generated the log<\/li>\n<li id=\"96f4\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe xu xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Message<\/strong>: A human-readable description of the event<\/li>\n<li id=\"39bc\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe xu xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Context<\/strong>: Additional metadata that helps understand the event<\/li>\n<\/ol>\n<h3 id=\"f7bd\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Example of a Well-Formed Log Entry<\/h3>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"c255\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-number\">2024<\/span>-08-07T14:<span class=\"hljs-number\">30<\/span>:<span class=\"hljs-number\">15.123<\/span>Z [INFO] UserService: User login successful - userID=<span class=\"hljs-number\">12345<\/span>, sessionID=abc-<span class=\"hljs-keyword\">def<\/span>-ghi, clientIP=<span class=\"hljs-number\">192.168<\/span><span class=\"hljs-number\">.1<\/span><span class=\"hljs-number\">.100<\/span>, userAgent=<span class=\"hljs-string\">\"Mozilla\/5.0...\"<\/span><\/span><\/pre>\n<h3 id=\"6452\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Log Anatomy Breakdown<\/h3>\n<ul class=\"\">\n<li id=\"cf58\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Timestamp<\/strong>: <code class=\"cu zc zd ze yu b\">2024-08-07T14:30:15.123Z<\/code> (ISO 8601 format)<\/li>\n<li id=\"bfcf\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Level<\/strong>: <code class=\"cu zc zd ze yu b\">INFO<\/code><\/li>\n<li id=\"e455\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Source<\/strong>: <code class=\"cu zc zd ze yu b\">UserService<\/code><\/li>\n<li id=\"f83a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Event<\/strong>: <code class=\"cu zc zd ze yu b\">User login successful<\/code><\/li>\n<li id=\"f925\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Context<\/strong>: User ID, session ID, client IP, user agent<\/li>\n<\/ul>\n<p><strong>[ Also\u00a0Read: <a href=\"https:\/\/opstree.com\/blog\/2023\/05\/30\/basic-logging-setup-of-loki-grafana\/\">Basic Logging Setup of Loki Grafana<\/a>]<\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"log-levels-and-best-practices\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Log Levels and Best Practices<\/h2>\n<h3 id=\"f272\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Standard Log Levels (RFC 5424)<\/h3>\n<p>Understanding when to use each log level is crucial for effective logging:<\/p>\n<h4 id=\"452a\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. TRACE<\/h4>\n<ul class=\"\">\n<li id=\"bc64\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Use Case<\/strong>: Extremely detailed information, typically only of interest when diagnosing problems<\/li>\n<li id=\"22a1\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Example<\/strong>: <code class=\"cu zc zd ze yu b\">TRACE: Entering method calculateDiscount() with parameters: price=100, discountRate=0.15<\/code><\/li>\n<\/ul>\n<h4 id=\"543f\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. DEBUG<\/h4>\n<ul class=\"\">\n<li id=\"1cb6\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Use Case<\/strong>: Information useful for debugging applications<\/li>\n<li id=\"2d18\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Example<\/strong>: <code class=\"cu zc zd ze yu b\">DEBUG: Database query executed in 45ms: SELECT * FROM users WHERE active=true<\/code><\/li>\n<\/ul>\n<h4 id=\"c1f5\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">3. INFO<\/h4>\n<ul class=\"\">\n<li id=\"395f\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Use Case<\/strong>: General informational messages that highlight system progress<\/li>\n<li id=\"aaab\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Example<\/strong>: <code class=\"cu zc zd ze yu b\">INFO: Application started successfully on port 8080<\/code><\/li>\n<\/ul>\n<h4 id=\"ff8f\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">4. WARN<\/h4>\n<ul class=\"\">\n<li id=\"7d14\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Use Case<\/strong>: Potentially harmful situations that don\u2019t stop the application<\/li>\n<li id=\"aef6\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Example<\/strong>: <code class=\"cu zc zd ze yu b\">WARN: Database connection pool is 85% full, consider increasing pool size<\/code><\/li>\n<\/ul>\n<h4 id=\"7d7e\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">5. ERROR<\/h4>\n<ul class=\"\">\n<li id=\"7765\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Use Case<\/strong>: Error events that might still allow the application to continue<\/li>\n<li id=\"2bd2\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Example<\/strong>: <code class=\"cu zc zd ze yu b\">ERROR: Failed to process payment for order 12345: Payment gateway timeout<\/code><\/li>\n<\/ul>\n<h4 id=\"bc52\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">6. FATAL<\/h4>\n<ul class=\"\">\n<li id=\"f5d3\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Use Case<\/strong>: Very severe error events that will presumably lead to application termination<\/li>\n<li id=\"868a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Example<\/strong>: <code class=\"cu zc zd ze yu b\">FATAL: Cannot connect to primary database, application shutting down<\/code><\/li>\n<\/ul>\n<h3 id=\"ce3b\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Best Practices for Log Levels<\/h3>\n<h4 id=\"78e4\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">DO:<\/h4>\n<ul class=\"\">\n<li id=\"edf7\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Use INFO for business-significant events<\/li>\n<li id=\"6974\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Use ERROR for exceptions that are handled<\/li>\n<li id=\"6a4f\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Use WARN for recoverable error conditions<\/li>\n<li id=\"1711\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Use DEBUG for diagnostic information useful during development<\/li>\n<li id=\"50b7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Be consistent across your application<\/li>\n<\/ul>\n<h4 id=\"3cb9\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">DON\u2019T:<\/h4>\n<ul class=\"\">\n<li id=\"eeb5\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Log sensitive information (passwords, credit cards, SSNs)<\/li>\n<li id=\"3db3\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Use DEBUG logs in production without log level filtering<\/li>\n<li id=\"340a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Create \u201clog spam\u201d with excessive INFO messages<\/li>\n<li id=\"3c7e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Mix log levels inconsistently<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"structured-vs-unstructured-logging\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Structured vs Unstructured Logging<\/h2>\n<h3 id=\"9abc\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Unstructured Logging<\/h3>\n<p>Traditional logging often produces human-readable but machine-unparseable text:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"c826\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">User<\/span> komal.jaiswal<span class=\"hljs-variable\">@example<\/span>.com logged <span class=\"hljs-keyword\">in<\/span> successfully <span class=\"hljs-keyword\">at<\/span> <span class=\"hljs-number\">2024<\/span><span class=\"hljs-number\">-08<\/span><span class=\"hljs-number\">-07<\/span> <span class=\"hljs-number\">14<\/span>:<span class=\"hljs-number\">30<\/span>:<span class=\"hljs-number\">15<\/span> <span class=\"hljs-keyword\">from<\/span> IP <span class=\"hljs-number\">192.168<\/span><span class=\"hljs-number\">.1<\/span><span class=\"hljs-number\">.100<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">Pros<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"85ef\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Human-readable<\/li>\n<li id=\"2846\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Simple to implement<\/li>\n<li id=\"ab41\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Familiar to developers<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Cons<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"55b2\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Difficult to parse programmatically<\/li>\n<li id=\"cbfe\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Limited querying capabilities<\/li>\n<li id=\"d4d7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Hard to aggregate and analyze<\/li>\n<\/ul>\n<h3 id=\"5c8f\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Structured Logging<\/h3>\n<p>Structured logging produces machine-parseable output, typically in JSON format:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"6850\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\">{\r\n  <span class=\"hljs-string\">\"timestamp\"<\/span>: <span class=\"hljs-string\">\"2024-08-07T14:30:15.123Z\"<\/span>,\r\n  <span class=\"hljs-string\">\"level\"<\/span>: <span class=\"hljs-string\">\"INFO\"<\/span>,\r\n  <span class=\"hljs-string\">\"service\"<\/span>: <span class=\"hljs-string\">\"auth-service\"<\/span>,\r\n  <span class=\"hljs-string\">\"event\"<\/span>: <span class=\"hljs-string\">\"user_login_success\"<\/span>,\r\n  <span class=\"hljs-string\">\"user_email\"<\/span>: <span class=\"hljs-string\">\"john.doe@example.com\"<\/span>,\r\n  <span class=\"hljs-string\">\"client_ip\"<\/span>: <span class=\"hljs-string\">\"192.168.1.100\"<\/span>,\r\n  <span class=\"hljs-string\">\"session_id\"<\/span>: <span class=\"hljs-string\">\"abc-def-ghi-123\"<\/span>,\r\n  <span class=\"hljs-string\">\"response_time_ms\"<\/span>: <span class=\"hljs-number\">45<\/span>\r\n}<\/span><\/pre>\n<p><strong class=\"wj gl\">Pros<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"d670\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Machine-parseable<\/li>\n<li id=\"f6f3\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Easy to query and filter<\/li>\n<li id=\"93fb\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Excellent for aggregation and analytics<\/li>\n<li id=\"3056\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Enables powerful log analysis tools<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Cons<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"bd00\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Less human-readable in raw form<\/li>\n<li id=\"3932\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Slightly more complex to implement<\/li>\n<li id=\"eaf0\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Can be more verbose<\/li>\n<\/ul>\n<h3 id=\"3400\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">When to Use Which?<\/h3>\n<ul class=\"\">\n<li id=\"34d5\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Structured Logging<\/strong>: Production systems, <strong><a href=\"https:\/\/opstree.com\/blog\/2021\/12\/27\/introduction-to-microservices\/\">microservices<\/a><\/strong>, any system requiring analysis<\/li>\n<li id=\"356d\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Unstructured Logging<\/strong>: Simple applications, development environments, legacy systems<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"popular-logging-frameworks\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Popular Logging Frameworks<\/h2>\n<h3 id=\"4844\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Java Ecosystem<\/h3>\n<h4 id=\"7aef\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Logback<\/h4>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"cdf6\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> org.slf4j.Logger;\r\n<span class=\"hljs-keyword\">import<\/span> org.slf4j.LoggerFactory;\r\n<span class=\"hljs-keyword\">public<\/span> <span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">UserService<\/span> {\r\n    <span class=\"hljs-keyword\">private<\/span> <span class=\"hljs-keyword\">static<\/span> <span class=\"hljs-keyword\">final<\/span> <span class=\"hljs-type\">Logger<\/span> <span class=\"hljs-variable\">logger<\/span> <span class=\"hljs-operator\">=<\/span> LoggerFactory.getLogger(UserService.class);\r\n    \r\n    <span class=\"hljs-keyword\">public<\/span> <span class=\"hljs-keyword\">void<\/span> <span class=\"hljs-title.function\">loginUser<\/span><span class=\"hljs-params\">(String userId)<\/span> {\r\n        logger.info(<span class=\"hljs-string\">\"User login attempt for userId: {}\"<\/span>, userId);\r\n        <span class=\"hljs-keyword\">try<\/span> {\r\n            <span class=\"hljs-comment\">\/\/ Login logic here<\/span>\r\n            logger.info(<span class=\"hljs-string\">\"User {} logged in successfully\"<\/span>, userId);\r\n        } <span class=\"hljs-keyword\">catch<\/span> (Exception e) {\r\n            logger.error(<span class=\"hljs-string\">\"Login failed for userId: {}\"<\/span>, userId, e);\r\n        }\r\n    }\r\n}<\/span><\/pre>\n<h4 id=\"c9cb\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Log4j2<\/h4>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"fe85\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\">&lt;!-- log4j2.xml --&gt;<\/span>\r\n<span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">Configuration<\/span> <span class=\"hljs-attr\">status<\/span>=<span class=\"hljs-string\">\"WARN\"<\/span>&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">Appenders<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">Console<\/span> <span class=\"hljs-attr\">name<\/span>=<span class=\"hljs-string\">\"Console\"<\/span> <span class=\"hljs-attr\">target<\/span>=<span class=\"hljs-string\">\"SYSTEM_OUT\"<\/span>&gt;<\/span>\r\n            <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">PatternLayout<\/span> <span class=\"hljs-attr\">pattern<\/span>=<span class=\"hljs-string\">\"%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n\"<\/span>\/&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">Console<\/span>&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">Appenders<\/span>&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">Loggers<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">Root<\/span> <span class=\"hljs-attr\">level<\/span>=<span class=\"hljs-string\">\"info\"<\/span>&gt;<\/span>\r\n            <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">AppenderRef<\/span> <span class=\"hljs-attr\">ref<\/span>=<span class=\"hljs-string\">\"Console\"<\/span>\/&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">Root<\/span>&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">Loggers<\/span>&gt;<\/span>\r\n<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">Configuration<\/span>&gt;<\/span><\/span><\/pre>\n<h3 id=\"8b52\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Python Ecosystem<\/h3>\n<h4 id=\"dcba\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Python Logging Module<\/h4>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"e57d\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> logging\r\n<span class=\"hljs-keyword\">import<\/span> json\r\n\r\n<span class=\"hljs-comment\"># Configure structured logging<\/span>\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">JsonFormatter<\/span>(logging.Formatter):\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">format<\/span>(<span class=\"hljs-params\">self, record<\/span>):\r\n        log_entry = {\r\n            <span class=\"hljs-string\">'timestamp'<\/span>: self.formatTime(record),\r\n            <span class=\"hljs-string\">'level'<\/span>: record.levelname,\r\n            <span class=\"hljs-string\">'message'<\/span>: record.getMessage(),\r\n            <span class=\"hljs-string\">'module'<\/span>: record.module,\r\n            <span class=\"hljs-string\">'function'<\/span>: record.funcName\r\n        }\r\n        <span class=\"hljs-keyword\">return<\/span> json.dumps(log_entry)\r\n<span class=\"hljs-comment\"># Setup logger<\/span>\r\nlogger = logging.getLogger(__name__)\r\nhandler = logging.StreamHandler()\r\nhandler.setFormatter(JsonFormatter())\r\nlogger.addHandler(handler)\r\nlogger.setLevel(logging.INFO)\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">process_order<\/span>(<span class=\"hljs-params\">order_id<\/span>):\r\n    logger.info(<span class=\"hljs-string\">f\"Processing order <span class=\"hljs-subst\">{order_id}<\/span>\"<\/span>)\r\n    <span class=\"hljs-keyword\">try<\/span>:\r\n        <span class=\"hljs-comment\"># Order processing logic<\/span>\r\n        logger.info(<span class=\"hljs-string\">f\"Order <span class=\"hljs-subst\">{order_id}<\/span> processed successfully\"<\/span>)\r\n    <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n        logger.error(<span class=\"hljs-string\">f\"Failed to process order <span class=\"hljs-subst\">{order_id}<\/span>: <span class=\"hljs-subst\">{<span class=\"hljs-built_in\">str<\/span>(e)}<\/span>\"<\/span>)<\/span><\/pre>\n<h4 id=\"7ddc\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Structlog<\/h4>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"f09a\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> structlog\r\n<span class=\"hljs-comment\"># Configure structlog<\/span>\r\nstructlog.configure(\r\n    processors=[\r\n        structlog.stdlib.filter_by_level,\r\n        structlog.stdlib.add_logger_name,\r\n        structlog.stdlib.add_log_level,\r\n        structlog.stdlib.PositionalArgumentsFormatter(),\r\n        structlog.processors.TimeStamper(fmt=<span class=\"hljs-string\">\"iso\"<\/span>),\r\n        structlog.processors.JSONRenderer()\r\n    ],\r\n    context_class=<span class=\"hljs-built_in\">dict<\/span>,\r\n    logger_factory=structlog.stdlib.LoggerFactory(),\r\n    wrapper_class=structlog.stdlib.BoundLogger,\r\n    cache_logger_on_first_use=<span class=\"hljs-literal\">True<\/span>,\r\n)\r\nlogger = structlog.get_logger()\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">transfer_money<\/span>(<span class=\"hljs-params\">from_account, to_account, amount<\/span>):\r\n    logger = logger.bind(\r\n        from_account=from_account,\r\n        to_account=to_account,\r\n        amount=amount,\r\n        transaction_id=generate_transaction_id()\r\n    )\r\n    \r\n    logger.info(<span class=\"hljs-string\">\"Starting money transfer\"<\/span>)\r\n    <span class=\"hljs-keyword\">try<\/span>:\r\n        <span class=\"hljs-comment\"># Transfer logic<\/span>\r\n        logger.info(<span class=\"hljs-string\">\"Money transfer completed successfully\"<\/span>)\r\n    <span class=\"hljs-keyword\">except<\/span> InsufficientFundsError <span class=\"hljs-keyword\">as<\/span> e:\r\n        logger.error(<span class=\"hljs-string\">\"Transfer failed due to insufficient funds\"<\/span>)<\/span><\/pre>\n<h3 id=\"4fb5\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">JavaScript\/Node.js Ecosystem<\/h3>\n<h4 id=\"5f1a\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Winston<\/h4>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"7b25\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">const<\/span> <span class=\"hljs-variable.constant\">winston<\/span> = <span class=\"hljs-keyword\">require<\/span>(<span class=\"hljs-string\">'winston'<\/span>);\r\n<span class=\"hljs-keyword\">const<\/span> <span class=\"hljs-variable.constant\">logger<\/span> = winston.<span class=\"hljs-title.function.invoke\">createLogger<\/span>({\r\n  <span class=\"hljs-attr\">level<\/span>: <span class=\"hljs-string\">'info'<\/span>,\r\n  <span class=\"hljs-attr\">format<\/span>: winston.format.<span class=\"hljs-title.function.invoke\">combine<\/span>(\r\n    winston.format.<span class=\"hljs-title.function.invoke\">timestamp<\/span>(),\r\n    winston.format.<span class=\"hljs-title.function.invoke\">errors<\/span>({ <span class=\"hljs-attr\">stack<\/span>: <span class=\"hljs-literal\">true<\/span> }),\r\n    winston.format.<span class=\"hljs-title.function.invoke\">json<\/span>()\r\n  ),\r\n  <span class=\"hljs-attr\">defaultMeta<\/span>: { <span class=\"hljs-attr\">service<\/span>: <span class=\"hljs-string\">'user-service'<\/span> },\r\n  <span class=\"hljs-attr\">transports<\/span>: [\r\n    <span class=\"hljs-keyword\">new<\/span> winston.transports.<span class=\"hljs-title.function.invoke\">File<\/span>({ <span class=\"hljs-attr\">filename<\/span>: <span class=\"hljs-string\">'error.log'<\/span>, <span class=\"hljs-attr\">level<\/span>: <span class=\"hljs-string\">'error'<\/span> }),\r\n    <span class=\"hljs-keyword\">new<\/span> winston.transports.<span class=\"hljs-title.function.invoke\">File<\/span>({ <span class=\"hljs-attr\">filename<\/span>: <span class=\"hljs-string\">'combined.log'<\/span> }),\r\n    <span class=\"hljs-keyword\">new<\/span> winston.transports.<span class=\"hljs-title.function.invoke\">Console<\/span>({\r\n      <span class=\"hljs-attr\">format<\/span>: winston.format.<span class=\"hljs-title.function.invoke\">simple<\/span>()\r\n    })\r\n  ]\r\n});\r\n<span class=\"hljs-function\"><span class=\"hljs-keyword\">function<\/span> <span class=\"hljs-title\">authenticateUser<\/span>(<span class=\"hljs-params\">username, password<\/span>) <\/span>{\r\n  logger.<span class=\"hljs-title.function.invoke\">info<\/span>(<span class=\"hljs-string\">'User authentication attempt'<\/span>, { username });\r\n  \r\n  <span class=\"hljs-keyword\">try<\/span> {\r\n    <span class=\"hljs-comment\">\/\/ Authentication logic<\/span>\r\n    logger.<span class=\"hljs-title.function.invoke\">info<\/span>(<span class=\"hljs-string\">'User authenticated successfully'<\/span>, { \r\n      username, \r\n      <span class=\"hljs-attr\">sessionId<\/span>: <span class=\"hljs-title.function.invoke\">generateSessionId<\/span>() \r\n    });\r\n  } <span class=\"hljs-keyword\">catch<\/span> (error) {\r\n    logger.<span class=\"hljs-title.function.invoke\">error<\/span>(<span class=\"hljs-string\">'Authentication failed'<\/span>, { \r\n      username, \r\n      <span class=\"hljs-attr\">error<\/span>: error.message,\r\n      <span class=\"hljs-attr\">stack<\/span>: error.stack \r\n    });\r\n  }\r\n}<\/span><\/pre>\n<h4 id=\"fcf7\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Pino<\/h4>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"fbe5\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">const<\/span> <span class=\"hljs-variable.constant\">pino<\/span> = <span class=\"hljs-keyword\">require<\/span>(<span class=\"hljs-string\">'pino'<\/span>);\r\n<span class=\"hljs-keyword\">const<\/span> <span class=\"hljs-variable.constant\">logger<\/span> = <span class=\"hljs-title.function.invoke\">pino<\/span>({\r\n  <span class=\"hljs-attr\">level<\/span>: <span class=\"hljs-string\">'info'<\/span>,\r\n  <span class=\"hljs-attr\">formatters<\/span>: {\r\n    <span class=\"hljs-title.function.invoke\">level<\/span>(label) {\r\n      <span class=\"hljs-keyword\">return<\/span> { <span class=\"hljs-attr\">level<\/span>: label };\r\n    }\r\n  },\r\n  <span class=\"hljs-attr\">timestamp<\/span>: pino.stdTimeFunctions.isoTime\r\n});\r\n<span class=\"hljs-function\"><span class=\"hljs-keyword\">function<\/span> <span class=\"hljs-title\">processPayment<\/span>(<span class=\"hljs-params\">paymentId, amount, currency<\/span>) <\/span>{\r\n  <span class=\"hljs-keyword\">const<\/span> <span class=\"hljs-variable.constant\">childLogger<\/span> = logger.<span class=\"hljs-title.function.invoke\">child<\/span>({ \r\n    paymentId, \r\n    amount, \r\n    currency,\r\n    <span class=\"hljs-attr\">correlationId<\/span>: <span class=\"hljs-title.function.invoke\">generateCorrelationId<\/span>() \r\n  });\r\n  \r\n  childLogger.<span class=\"hljs-title.function.invoke\">info<\/span>(<span class=\"hljs-string\">'Payment processing started'<\/span>);\r\n  \r\n  <span class=\"hljs-keyword\">try<\/span> {\r\n    <span class=\"hljs-comment\">\/\/ Payment processing logic<\/span>\r\n    childLogger.<span class=\"hljs-title.function.invoke\">info<\/span>(<span class=\"hljs-string\">'Payment processed successfully'<\/span>);\r\n  } <span class=\"hljs-keyword\">catch<\/span> (error) {\r\n    childLogger.<span class=\"hljs-title.function.invoke\">error<\/span>({ <span class=\"hljs-attr\">err<\/span>: error }, <span class=\"hljs-string\">'Payment processing failed'<\/span>);\r\n  }\r\n}<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"log-management-and-aggregation\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Log Management and Aggregation<\/h2>\n<h3 id=\"7a6a\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">The Challenge of Distributed Logging<\/h3>\n<p>In <a href=\"https:\/\/opstree.com\/blog\/2021\/06\/02\/microservices-unlock-great-power\/\">microservices architectures<\/a>, a single user request might traverse multiple services, each generating logs. Managing and correlating these logs becomes a significant challenge.<\/p>\n<h3 id=\"8de0\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Log Aggregation Patterns<\/h3>\n<h4 id=\"e4c0\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Push-Based Logging<\/h4>\n<p>Services actively send logs to a central aggregation point.<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"2103\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Example: Fluentd configuration<\/span>\r\n&lt;<span class=\"hljs-built_in\">source<\/span>&gt;\r\n  @<span class=\"hljs-built_in\">type<\/span> forward\r\n  port 24224\r\n  <span class=\"hljs-built_in\">bind<\/span> 0.0.0.0\r\n&lt;\/source&gt;\r\n\r\n&lt;match app.**&gt;\r\n  @<span class=\"hljs-built_in\">type<\/span> elasticsearch\r\n  host elasticsearch.logging.svc.cluster.local\r\n  port 9200\r\n  index_name application_logs\r\n  type_name <span class=\"hljs-built_in\">log<\/span>\r\n&lt;\/match&gt;<\/span><\/pre>\n<h4 id=\"2830\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Pull-Based Logging<\/h4>\n<p>A central system pulls logs from various sources.<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"18ba\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Example: Promtail configuration for Loki<\/span>\r\n<span class=\"hljs-attr\">server:<\/span>\r\n  <span class=\"hljs-attr\">http_listen_port:<\/span> <span class=\"hljs-number\">9080<\/span>\r\n  <span class=\"hljs-attr\">grpc_listen_port:<\/span> <span class=\"hljs-number\">0<\/span>\r\n<span class=\"hljs-attr\">positions:<\/span>\r\n  <span class=\"hljs-attr\">filename:<\/span> <span class=\"hljs-string\">\/tmp\/positions.yaml<\/span>\r\n<span class=\"hljs-attr\">clients:<\/span>\r\n  <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">url:<\/span> <span class=\"hljs-string\">http:\/\/loki:3100\/loki\/api\/v1\/push<\/span>\r\n<span class=\"hljs-attr\">scrape_configs:<\/span>\r\n  <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">job_name:<\/span> <span class=\"hljs-string\">containers<\/span>\r\n    <span class=\"hljs-attr\">static_configs:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">targets:<\/span>\r\n          <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">localhost<\/span>\r\n        <span class=\"hljs-attr\">labels:<\/span>\r\n          <span class=\"hljs-attr\">job:<\/span> <span class=\"hljs-string\">containerlogs<\/span>\r\n          <span class=\"hljs-attr\">__path__:<\/span> <span class=\"hljs-string\">\/var\/log\/containers\/*log<\/span><\/span><\/pre>\n<h3 id=\"e991\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Popular Log Aggregation Solutions<\/h3>\n<h4 id=\"47cc\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. ELK Stack (Elasticsearch, Logstash, Kibana)<\/h4>\n<p><strong class=\"wj gl\">Architecture Overview<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"25fe\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Elasticsearch<\/strong>: Search and analytics engine<\/li>\n<li id=\"46f5\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Logstash<\/strong>: <a href=\"https:\/\/opstree.com\/blog\/2025\/08\/06\/llm-powered-etl-genai-data-transformation\/\">Data processing pipeline<\/a><\/li>\n<li id=\"b8e2\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Kibana<\/strong>: Visualization and dashboards<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Sample Logstash Configuration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"ea12\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\">input {\r\n  beats {\r\n    port =&gt; <span class=\"hljs-number\">5044<\/span>\r\n  }\r\n}\r\n\r\nfilter {\r\n  <span class=\"hljs-keyword\">if<\/span> [fields][log_type] == <span class=\"hljs-string\">\"application\"<\/span> {\r\n    json {\r\n      source =&gt; <span class=\"hljs-string\">\"message\"<\/span>\r\n    }\r\n    \r\n    date {\r\n      match =&gt; [ <span class=\"hljs-string\">\"timestamp\"<\/span>, <span class=\"hljs-string\">\"ISO8601\"<\/span> ]\r\n    }\r\n    \r\n    mutate {\r\n      remove_field =&gt; [ <span class=\"hljs-string\">\"message\"<\/span> ]\r\n    }\r\n  }\r\n}\r\noutput {\r\n  elasticsearch {\r\n    hosts =&gt; [<span class=\"hljs-string\">\"elasticsearch:9200\"<\/span>]\r\n    index =&gt; <span class=\"hljs-string\">\"application-logs-%{+YYYY.MM.dd}\"<\/span>\r\n  }\r\n}<\/span><\/pre>\n<h4 id=\"d326\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Grafana Loki<\/h4>\n<p>Loki is designed to be cost-effective and easy to operate, focusing on logs you\u2019re already collecting with Prometheus.<strong class=\"wj gl\">Sample Configuration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"43fe\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-attr\">auth_enabled:<\/span> <span class=\"hljs-literal\">false<\/span>\r\n<span class=\"hljs-attr\">server:<\/span>\r\n  <span class=\"hljs-attr\">http_listen_port:<\/span> <span class=\"hljs-number\">3100<\/span>\r\n  <span class=\"hljs-attr\">grpc_listen_port:<\/span> <span class=\"hljs-number\">9096<\/span>\r\n<span class=\"hljs-attr\">ingester:<\/span>\r\n  <span class=\"hljs-attr\">wal:<\/span>\r\n    <span class=\"hljs-attr\">enabled:<\/span> <span class=\"hljs-literal\">true<\/span>\r\n    <span class=\"hljs-attr\">dir:<\/span> <span class=\"hljs-string\">\/loki\/wal<\/span>\r\n  <span class=\"hljs-attr\">lifecycler:<\/span>\r\n    <span class=\"hljs-attr\">address:<\/span> <span class=\"hljs-number\">127.0<\/span><span class=\"hljs-number\">.0<\/span><span class=\"hljs-number\">.1<\/span>\r\n    <span class=\"hljs-attr\">ring:<\/span>\r\n      <span class=\"hljs-attr\">kvstore:<\/span>\r\n        <span class=\"hljs-attr\">store:<\/span> <span class=\"hljs-string\">inmemory<\/span>\r\n      <span class=\"hljs-attr\">replication_factor:<\/span> <span class=\"hljs-number\">1<\/span>\r\n    <span class=\"hljs-attr\">final_sleep:<\/span> <span class=\"hljs-string\">0s<\/span>\r\n  <span class=\"hljs-attr\">chunk_idle_period:<\/span> <span class=\"hljs-string\">1h<\/span>\r\n  <span class=\"hljs-attr\">max_chunk_age:<\/span> <span class=\"hljs-string\">1h<\/span>\r\n  <span class=\"hljs-attr\">chunk_target_size:<\/span> <span class=\"hljs-number\">1048576<\/span>\r\n  <span class=\"hljs-attr\">chunk_retain_period:<\/span> <span class=\"hljs-string\">30s<\/span>\r\n<span class=\"hljs-attr\">schema_config:<\/span>\r\n  <span class=\"hljs-attr\">configs:<\/span>\r\n    <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">from:<\/span> <span class=\"hljs-number\">2020-10-24<\/span>\r\n      <span class=\"hljs-attr\">store:<\/span> <span class=\"hljs-string\">boltdb-shipper<\/span>\r\n      <span class=\"hljs-attr\">object_store:<\/span> <span class=\"hljs-string\">filesystem<\/span>\r\n      <span class=\"hljs-attr\">schema:<\/span> <span class=\"hljs-string\">v11<\/span>\r\n      <span class=\"hljs-attr\">index:<\/span>\r\n        <span class=\"hljs-attr\">prefix:<\/span> <span class=\"hljs-string\">index_<\/span>\r\n        <span class=\"hljs-attr\">period:<\/span> <span class=\"hljs-string\">24h<\/span>\r\n<span class=\"hljs-attr\">storage_config:<\/span>\r\n  <span class=\"hljs-attr\">boltdb_shipper:<\/span>\r\n    <span class=\"hljs-attr\">active_index_directory:<\/span> <span class=\"hljs-string\">\/loki\/boltdb-shipper-active<\/span>\r\n    <span class=\"hljs-attr\">cache_location:<\/span> <span class=\"hljs-string\">\/loki\/boltdb-shipper-cache<\/span>\r\n    <span class=\"hljs-attr\">resync_interval:<\/span> <span class=\"hljs-string\">24h<\/span>\r\n    <span class=\"hljs-attr\">shared_store:<\/span> <span class=\"hljs-string\">filesystem<\/span>\r\n  <span class=\"hljs-attr\">filesystem:<\/span>\r\n    <span class=\"hljs-attr\">directory:<\/span> <span class=\"hljs-string\">\/loki\/chunks<\/span>\r\n<span class=\"hljs-attr\">compactor:<\/span>\r\n  <span class=\"hljs-attr\">working_directory:<\/span> <span class=\"hljs-string\">\/loki\/boltdb-shipper-compactor<\/span>\r\n  <span class=\"hljs-attr\">shared_store:<\/span> <span class=\"hljs-string\">filesystem<\/span>\r\n<span class=\"hljs-attr\">limits_config:<\/span>\r\n  <span class=\"hljs-attr\">reject_old_samples:<\/span> <span class=\"hljs-literal\">true<\/span>\r\n  <span class=\"hljs-attr\">reject_old_samples_max_age:<\/span> <span class=\"hljs-string\">168h<\/span>\r\n<span class=\"hljs-attr\">chunk_store_config:<\/span>\r\n  <span class=\"hljs-attr\">max_look_back_period:<\/span> <span class=\"hljs-string\">0s<\/span>\r\n<span class=\"hljs-attr\">table_manager:<\/span>\r\n  <span class=\"hljs-attr\">retention_deletes_enabled:<\/span> <span class=\"hljs-literal\">false<\/span>\r\n  <span class=\"hljs-attr\">retention_period:<\/span> <span class=\"hljs-string\">0s<\/span>\r\n<span class=\"hljs-attr\">ruler:<\/span>\r\n  <span class=\"hljs-attr\">storage:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">local<\/span>\r\n    <span class=\"hljs-attr\">local:<\/span>\r\n      <span class=\"hljs-attr\">directory:<\/span> <span class=\"hljs-string\">\/loki\/rules<\/span>\r\n  <span class=\"hljs-attr\">rule_path:<\/span> <span class=\"hljs-string\">\/loki\/rules-temp<\/span>\r\n  <span class=\"hljs-attr\">alertmanager_url:<\/span> <span class=\"hljs-string\">http:\/\/localhost:9093<\/span>\r\n  <span class=\"hljs-attr\">ring:<\/span>\r\n    <span class=\"hljs-attr\">kvstore:<\/span>\r\n      <span class=\"hljs-attr\">store:<\/span> <span class=\"hljs-string\">inmemory<\/span>\r\n  <span class=\"hljs-attr\">enable_api:<\/span> <span class=\"hljs-literal\">true<\/span><\/span><\/pre>\n<h4 id=\"260e\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">3. Fluentd\/Fluent Bit<\/h4>\n<p>Unified logging layers that help unify data collection and consumption.<strong class=\"wj gl\">Fluent Bit Configuration Example<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"9248\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\">[INPUT]\r\n    Name <span class=\"hljs-built_in\">tail<\/span>\r\n    Path \/var\/log\/containers\/*.<span class=\"hljs-built_in\">log<\/span>\r\n    Parser docker\r\n    Tag kube.*\r\n    Refresh_Interval 5\r\n    Mem_Buf_Limit 50MB<\/span><\/pre>\n<pre class=\"zw yt yu zx zy bh zz bx\"><span id=\"853f\" class=\"zf xh sd yu b fn aba abb y abc zb\" data-selectable-paragraph=\"\">[FILTER]\r\n    Name kubernetes\r\n    Match kube.*\r\n    Kube_URL https:\/\/kubernetes.default.svc:443\r\n    Kube_CA_File \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt\r\n    Kube_Token_File \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token\r\n    Kube_Tag_Prefix kube.var.log.containers.\r\n    Merge_Log On<\/span><span id=\"ce89\" class=\"zf xh sd yu b fn abd abb y abc zb\" data-selectable-paragraph=\"\">[OUTPUT]\r\n    Name elasticsearch\r\n    Match *\r\n    Host elasticsearch.logging.svc.cluster.local\r\n    Port 9200\r\n    Index application_logs\r\n    Type _doc<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"observability-and-the-three-pillars\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Observability and the Three Pillars<\/h2>\n<h3 id=\"1f0c\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Understanding Observability<\/h3>\n<p><a href=\"https:\/\/opstree.com\/services\/observability-sre-production-engineering\/\"><strong>Observability<\/strong><\/a> is the ability to understand the internal state of a system by examining its external outputs. In software systems, this traditionally involves three key pillars:<\/p>\n<h4 id=\"793d\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Logs<\/h4>\n<p><strong class=\"wj gl\">What<\/strong>: Discrete events that happened at a specific time <strong class=\"wj gl\">When to Use<\/strong>: Debugging specific issues, understanding application flow <strong class=\"wj gl\">Example<\/strong>: \u201cUser 12345 failed to authenticate at 2024\u201308\u201307T14:30:15Z\u201d<\/p>\n<h4 id=\"e8c9\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Metrics<\/h4>\n<p><strong class=\"wj gl\">What<\/strong>: Numerical measurements aggregated over time <strong class=\"wj gl\">When to Use<\/strong>: Monitoring system health, alerting, capacity planning <strong class=\"wj gl\">Example<\/strong>: \u201cAverage response time: 150ms, Error rate: 2.3%\u201d<\/p>\n<h4 id=\"b231\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">3. Traces<\/h4>\n<p><strong class=\"wj gl\">What<\/strong>: Records of requests as they flow through distributed systems <strong class=\"wj gl\">When to Use<\/strong>: Understanding request flow, identifying bottlenecks <strong class=\"wj gl\">Example<\/strong>: Request journey through API Gateway \u2192 Auth Service \u2192 User Service \u2192 Database<\/p>\n<h3 id=\"ed82\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Correlation Between Pillars<\/h3>\n<p>The real power comes from correlating these three data types:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"6471\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-punctuation\">{<\/span>\r\n  <span class=\"hljs-attr\">\"timestamp\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"2024-08-07T14:30:15.123Z\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"level\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"ERROR\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"message\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"Database connection failed\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"service\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"user-service\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"trace_id\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"abc123def456\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"span_id\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"789ghi012\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"user_id\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"12345\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"endpoint\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"\/api\/users\/profile\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"response_time_ms\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-number\">5000<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"error_code\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"DB_CONNECTION_TIMEOUT\"<\/span>\r\n<span class=\"hljs-punctuation\">}<\/span><\/span><\/pre>\n<p>This log entry contains:<\/p>\n<ul class=\"\">\n<li id=\"8f82\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Log data<\/strong>: The error message and context<\/li>\n<li id=\"8dc6\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Metric data<\/strong>: Response time<\/li>\n<li id=\"034e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Trace data<\/strong>: Trace and span IDs for correlation<\/li>\n<\/ul>\n<h3 id=\"what-is-devops\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">OpenTelemetry: The Future of Observability<\/h3>\n<p><a href=\"https:\/\/opstree.com\/blog\/2025\/06\/17\/supervisor-process-monitoring-with-open-telemetry\/\"><strong>OpenTelemetry<\/strong><\/a> provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from applications.<strong class=\"wj gl\">Example: OpenTelemetry Implementation in Python<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"e239\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">from<\/span> opentelemetry <span class=\"hljs-keyword\">import<\/span> trace\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.exporter.jaeger.thrift <span class=\"hljs-keyword\">import<\/span> JaegerExporter\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.sdk.trace <span class=\"hljs-keyword\">import<\/span> TracerProvider\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.sdk.trace.export <span class=\"hljs-keyword\">import<\/span> BatchSpanProcessor\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.instrumentation.flask <span class=\"hljs-keyword\">import<\/span> FlaskInstrumentor\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.instrumentation.requests <span class=\"hljs-keyword\">import<\/span> RequestsInstrumentor\r\n<span class=\"hljs-keyword\">import<\/span> logging\r\n<span class=\"hljs-keyword\">import<\/span> structlog\r\n\r\n<span class=\"hljs-comment\"># Configure tracing<\/span>\r\ntrace.set_tracer_provider(TracerProvider())\r\ntracer = trace.get_tracer(__name__)\r\njaeger_exporter = JaegerExporter(\r\n    agent_host_name=<span class=\"hljs-string\">\"jaeger\"<\/span>,\r\n    agent_port=<span class=\"hljs-number\">6831<\/span>,\r\n)\r\nspan_processor = BatchSpanProcessor(jaeger_exporter)\r\ntrace.get_tracer_provider().add_span_processor(span_processor)\r\n<span class=\"hljs-comment\"># Configure structured logging with trace correlation<\/span>\r\nstructlog.configure(\r\n    processors=[\r\n        structlog.stdlib.filter_by_level,\r\n        structlog.stdlib.add_logger_name,\r\n        structlog.stdlib.add_log_level,\r\n        structlog.processors.TimeStamper(fmt=<span class=\"hljs-string\">\"iso\"<\/span>),\r\n        <span class=\"hljs-keyword\">lambda<\/span> _, __, event_dict: {\r\n            **event_dict,\r\n            <span class=\"hljs-string\">'trace_id'<\/span>: trace.format_trace_id(trace.get_current_span().get_span_context().trace_id),\r\n            <span class=\"hljs-string\">'span_id'<\/span>: trace.format_span_id(trace.get_current_span().get_span_context().span_id)\r\n        },\r\n        structlog.processors.JSONRenderer()\r\n    ],\r\n    context_class=<span class=\"hljs-built_in\">dict<\/span>,\r\n    logger_factory=structlog.stdlib.LoggerFactory(),\r\n    wrapper_class=structlog.stdlib.BoundLogger,\r\n    cache_logger_on_first_use=<span class=\"hljs-literal\">True<\/span>,\r\n)\r\nlogger = structlog.get_logger()\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">process_user_request<\/span>(<span class=\"hljs-params\">user_id<\/span>):\r\n    <span class=\"hljs-keyword\">with<\/span> tracer.start_as_current_span(<span class=\"hljs-string\">\"process_user_request\"<\/span>) <span class=\"hljs-keyword\">as<\/span> span:\r\n        span.set_attribute(<span class=\"hljs-string\">\"user.id\"<\/span>, user_id)\r\n        logger.info(<span class=\"hljs-string\">\"Processing user request\"<\/span>, user_id=user_id)\r\n        \r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            <span class=\"hljs-comment\"># Business logic here<\/span>\r\n            user_data = fetch_user_data(user_id)\r\n            span.set_attribute(<span class=\"hljs-string\">\"user.email\"<\/span>, user_data.get(<span class=\"hljs-string\">\"email\"<\/span>))\r\n            \r\n            logger.info(<span class=\"hljs-string\">\"User request processed successfully\"<\/span>, \r\n                       user_id=user_id, \r\n                       email=user_data.get(<span class=\"hljs-string\">\"email\"<\/span>))\r\n            \r\n            <span class=\"hljs-keyword\">return<\/span> user_data\r\n        <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n            span.record_exception(e)\r\n            span.set_status(trace.Status(trace.StatusCode.ERROR, <span class=\"hljs-built_in\">str<\/span>(e)))\r\n            logger.error(<span class=\"hljs-string\">\"Failed to process user request\"<\/span>, \r\n                        user_id=user_id, \r\n                        error=<span class=\"hljs-built_in\">str<\/span>(e))\r\n            <span class=\"hljs-keyword\">raise<\/span>\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">fetch_user_data<\/span>(<span class=\"hljs-params\">user_id<\/span>):\r\n    <span class=\"hljs-keyword\">with<\/span> tracer.start_as_current_span(<span class=\"hljs-string\">\"fetch_user_data\"<\/span>) <span class=\"hljs-keyword\">as<\/span> span:\r\n        span.set_attribute(<span class=\"hljs-string\">\"db.operation\"<\/span>, <span class=\"hljs-string\">\"SELECT\"<\/span>)\r\n        span.set_attribute(<span class=\"hljs-string\">\"db.table\"<\/span>, <span class=\"hljs-string\">\"users\"<\/span>)\r\n        \r\n        logger.debug(<span class=\"hljs-string\">\"Fetching user data from database\"<\/span>, user_id=user_id)\r\n        \r\n        <span class=\"hljs-comment\"># Database query simulation<\/span>\r\n        <span class=\"hljs-keyword\">import<\/span> time\r\n        time.sleep(<span class=\"hljs-number\">0.1<\/span>)  <span class=\"hljs-comment\"># Simulate DB query time<\/span>\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> {<span class=\"hljs-string\">\"id\"<\/span>: user_id, <span class=\"hljs-string\">\"email\"<\/span>: <span class=\"hljs-string\">f\"@example.com\"<\/span>&gt;user{user_id}@example.com<span class=\"hljs-string\">\"}<\/span><\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"real-world-case-studies\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Real-World Case Studies<\/h2>\n<h3 id=\"9e39\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Case Study 1: Netflix \u2014 Microservices Logging at Scale<\/h3>\n<p><strong class=\"wj gl\">Challenge<\/strong>: Netflix operates thousands of microservices serving millions of users. Traditional logging approaches couldn\u2019t handle the scale and complexity.<strong class=\"wj gl\">Solution<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"783a\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Distributed Tracing<\/strong>: Implemented Zipkin for request tracing across services<\/li>\n<li id=\"321d\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Centralized Logging<\/strong>: Built a custom log aggregation system handling 1+ billion log events per day<\/li>\n<li id=\"8f70\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Real-time Processing<\/strong>: Uses\u00a0<a href=\"https:\/\/opstree.com\/blog\/2024\/09\/27\/apache-flink-for-real-time-stream-processing\/\">Apache Kafka for real-time<\/a> log streaming<\/li>\n<li id=\"4228\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Machine Learning<\/strong>: Automated anomaly detection in log patterns<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Key Innovations<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"c3ab\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-punctuation\">{<\/span>\r\n  <span class=\"hljs-attr\">\"timestamp\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"2024-08-07T14:30:15.123Z\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"level\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"INFO\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"service\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"recommendation-service\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"trace_id\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"netflix_trace_123\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"user_id\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"user_456\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"title_id\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"movie_789\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"event\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"recommendation_served\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"algorithm\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"collaborative_filtering_v2\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"response_time_ms\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-number\">45<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"region\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"us-east-1\"<\/span><span class=\"hljs-punctuation\">,<\/span>\r\n  <span class=\"hljs-attr\">\"device_type\"<\/span><span class=\"hljs-punctuation\">:<\/span> <span class=\"hljs-string\">\"smart_tv\"<\/span>\r\n<span class=\"hljs-punctuation\">}<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">Results<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"66db\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Reduced mean time to resolution (MTTR) by 60%<\/li>\n<li id=\"fb42\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Improved system reliability to 99.99% uptime<\/li>\n<li id=\"c8d8\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Enabled data-driven product decisions<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Lessons Learned<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"9bfb\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Correlation IDs are essential for distributed systems<\/li>\n<li id=\"0013\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Automated log analysis is necessary at scale<\/li>\n<li id=\"73a4\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Context-rich logging enables better debugging<\/li>\n<\/ul>\n<h3 id=\"e150\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Case Study 2: Airbnb \u2014 Logging Infrastructure Evolution<\/h3>\n<p><strong class=\"wj gl\">Challenge<\/strong>: Airbnb\u2019s rapid growth from startup to global platform required evolving their logging strategy multiple times.<strong class=\"wj gl\">Evolution Timeline<\/strong>:<strong class=\"wj gl\">Phase 1: Simple File Logging (2008\u20132010)<\/strong><\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"7d4d\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Simple Rails logging<\/span>\r\nRails.logger.info <span class=\"hljs-string\">\"Booking created: <span class=\"hljs-subst\">#{booking.id}<\/span>\"<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">Phase 2: Centralized Logging (2010\u20132014)<\/strong><\/p>\n<ul class=\"\">\n<li id=\"5e0b\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><a href=\"https:\/\/opstree.com\/blog\/2025\/06\/04\/cloud-performance-monitoring-a-complete-setup-using-elk-stack\/\">Implemented ELK stack<\/a><\/li>\n<li id=\"47c8\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Added structured logging<\/li>\n<li id=\"a5b1\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Introduced log rotation and retention policies<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Phase 3: Real-time Analytics (2014\u20132018)<\/strong><\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"424c\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Enhanced structured logging with business context<\/span>\r\nlogger.info(<span class=\"hljs-string\">\"booking_created\"<\/span>, {\r\n    <span class=\"hljs-string\">\"booking_id\"<\/span>: booking.<span class=\"hljs-built_in\">id<\/span>,\r\n    <span class=\"hljs-string\">\"host_id\"<\/span>: booking.host_id,\r\n    <span class=\"hljs-string\">\"guest_id\"<\/span>: booking.guest_id,\r\n    <span class=\"hljs-string\">\"property_type\"<\/span>: booking.<span class=\"hljs-built_in\">property<\/span>.<span class=\"hljs-built_in\">type<\/span>,\r\n    <span class=\"hljs-string\">\"booking_value\"<\/span>: booking.total_price,\r\n    <span class=\"hljs-string\">\"market\"<\/span>: booking.<span class=\"hljs-built_in\">property<\/span>.market,\r\n    <span class=\"hljs-string\">\"check_in_date\"<\/span>: booking.check_in,\r\n    <span class=\"hljs-string\">\"nights\"<\/span>: booking.nights,\r\n    <span class=\"hljs-string\">\"guest_count\"<\/span>: booking.guest_count\r\n})<\/span><\/pre>\n<p><strong class=\"wj gl\">Phase 4: ML-Powered Insights (2018-Present)<\/strong><\/p>\n<ul class=\"\">\n<li id=\"717a\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Implemented anomaly detection<\/li>\n<li id=\"d4f6\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Built predictive models from log data<\/li>\n<li id=\"2917\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Real-time fraud detection using log patterns<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Current Architecture<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"7d82\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Apache Kafka for log streaming<\/li>\n<li id=\"1531\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Apache Spark for real-time processing<\/li>\n<li id=\"497b\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Custom ML models for pattern recognition<\/li>\n<li id=\"6b5e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Grafana dashboards for visualization<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Business Impact<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"4010\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Fraud detection improved by 80%<\/li>\n<li id=\"f34e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Customer support response time reduced by 40%<\/li>\n<li id=\"1c4c\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Revenue optimization through better recommendation algorithms<\/li>\n<\/ul>\n<h3 id=\"df89\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Case Study 3: Shopify \u2014 Black Friday Logging Strategy<\/h3>\n<p><strong class=\"wj gl\">Challenge<\/strong>: Handle logging for the biggest shopping day of the year with 10x normal traffic.<strong class=\"wj gl\">Preparation Strategy<\/strong>:<strong class=\"wj gl\">1. Load Testing with Realistic Logging<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"bc40\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Load test simulation with full logging<\/span>\r\n<span class=\"hljs-string\">class<\/span> <span class=\"hljs-string\">LoadTestController<\/span> <span class=\"hljs-string\">&lt;<\/span> <span class=\"hljs-string\">ApplicationController<\/span>\r\n  <span class=\"hljs-string\">before_action<\/span> <span class=\"hljs-string\">:setup_logging_context<\/span>\r\n  \r\n  <span class=\"hljs-string\">def<\/span> <span class=\"hljs-string\">checkout<\/span>\r\n    <span class=\"hljs-string\">Rails.logger.info<\/span> <span class=\"hljs-string\">\"checkout_started\"<\/span><span class=\"hljs-string\">,<\/span> {\r\n      <span class=\"hljs-attr\">session_id:<\/span> <span class=\"hljs-string\">session.id<\/span>,\r\n      <span class=\"hljs-attr\">cart_value:<\/span> <span class=\"hljs-string\">params<\/span>[<span class=\"hljs-string\">:cart_value<\/span>],\r\n      <span class=\"hljs-attr\">item_count:<\/span> <span class=\"hljs-string\">params<\/span>[<span class=\"hljs-string\">:item_count<\/span>],\r\n      <span class=\"hljs-attr\">shop_id:<\/span> <span class=\"hljs-string\">current_shop.id<\/span>,\r\n      <span class=\"hljs-attr\">customer_type:<\/span> <span class=\"hljs-string\">customer_classification<\/span>\r\n    }\r\n    \r\n    <span class=\"hljs-string\">begin<\/span>\r\n      <span class=\"hljs-string\">result<\/span> <span class=\"hljs-string\">=<\/span> <span class=\"hljs-string\">CheckoutService.new(checkout_params).process<\/span>\r\n      \r\n      <span class=\"hljs-string\">Rails.logger.info<\/span> <span class=\"hljs-string\">\"checkout_completed\"<\/span><span class=\"hljs-string\">,<\/span> {\r\n        <span class=\"hljs-attr\">session_id:<\/span> <span class=\"hljs-string\">session.id<\/span>,\r\n        <span class=\"hljs-attr\">order_id:<\/span> <span class=\"hljs-string\">result.order_id<\/span>,\r\n        <span class=\"hljs-attr\">processing_time_ms:<\/span> <span class=\"hljs-string\">result.processing_time<\/span>,\r\n        <span class=\"hljs-attr\">payment_method:<\/span> <span class=\"hljs-string\">result.payment_method<\/span>\r\n      }\r\n      \r\n      <span class=\"hljs-attr\">render json:<\/span> <span class=\"hljs-string\">result<\/span>\r\n    <span class=\"hljs-string\">rescue<\/span> <span class=\"hljs-string\">CheckoutError<\/span> <span class=\"hljs-string\">=&gt;<\/span> <span class=\"hljs-string\">e<\/span>\r\n      <span class=\"hljs-string\">Rails.logger.error<\/span> <span class=\"hljs-string\">\"checkout_failed\"<\/span><span class=\"hljs-string\">,<\/span> {\r\n        <span class=\"hljs-attr\">session_id:<\/span> <span class=\"hljs-string\">session.id<\/span>,\r\n        <span class=\"hljs-attr\">error_type:<\/span> <span class=\"hljs-string\">e.class.name<\/span>,\r\n        <span class=\"hljs-attr\">error_message:<\/span> <span class=\"hljs-string\">e.message<\/span>,\r\n        <span class=\"hljs-attr\">cart_state:<\/span> <span class=\"hljs-string\">cart.to_log_hash<\/span>\r\n      }\r\n      \r\n      <span class=\"hljs-attr\">render json:<\/span> { <span class=\"hljs-attr\">error:<\/span> <span class=\"hljs-string\">e.message<\/span> }<span class=\"hljs-string\">,<\/span> <span class=\"hljs-attr\">status:<\/span> <span class=\"hljs-number\">422<\/span>\r\n    <span class=\"hljs-string\">end<\/span>\r\n  <span class=\"hljs-string\">end<\/span>\r\n<span class=\"hljs-string\">end<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">2. Dynamic Log Level Management<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"e4d2\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Runtime log level adjustment<\/span>\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">LogLevelController<\/span> &lt; <span class=\"hljs-title.class.inherited\">ApplicationController<\/span>\r\n  <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">update<\/span>\r\n    <span class=\"hljs-keyword\">if<\/span> params[<span class=\"hljs-symbol\">:emergency_mode<\/span>]\r\n      <span class=\"hljs-comment\"># Reduce log verbosity during high traffic<\/span>\r\n      Rails.logger.level = Logger::<span class=\"hljs-variable.constant\">WARN<\/span>\r\n      disable_debug_logging\r\n    <span class=\"hljs-keyword\">else<\/span>\r\n      Rails.logger.level = Logger::<span class=\"hljs-variable.constant\">INFO<\/span>\r\n      enable_normal_logging\r\n    <span class=\"hljs-keyword\">end<\/span>\r\n    \r\n    Rails.logger.warn <span class=\"hljs-string\">\"log_level_changed\"<\/span>, {\r\n      <span class=\"hljs-symbol\">new_level:<\/span> Rails.logger.level,\r\n      <span class=\"hljs-symbol\">changed_by:<\/span> current_user.id,\r\n      <span class=\"hljs-symbol\">reason:<\/span> params[<span class=\"hljs-symbol\">:reason<\/span>]\r\n    }\r\n  <span class=\"hljs-keyword\">end<\/span>\r\n<span class=\"hljs-keyword\">end<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">3. Machine Learning Integration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"ca2e\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># ML model integration for advanced fraud detection<\/span>\r\n<span class=\"hljs-keyword\">import<\/span> joblib\r\n<span class=\"hljs-keyword\">import<\/span> pandas <span class=\"hljs-keyword\">as<\/span> pd\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">MLFraudDetector<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.model = joblib.load(<span class=\"hljs-string\">'fraud_detection_model.pkl'<\/span>)\r\n        self.logger = structlog.get_logger(<span class=\"hljs-string\">\"ml_fraud_detector\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">analyze_ride_patterns<\/span>(<span class=\"hljs-params\">self, user_id, recent_events<\/span>):\r\n        <span class=\"hljs-comment\"># Extract features from recent ride events<\/span>\r\n        features = self.extract_features(recent_events)\r\n        \r\n        <span class=\"hljs-comment\"># Generate fraud probability<\/span>\r\n        fraud_probability = self.model.predict_proba([features])[<span class=\"hljs-number\">0<\/span>][<span class=\"hljs-number\">1<\/span>]\r\n        \r\n        self.logger.info(<span class=\"hljs-string\">\"fraud_analysis_completed\"<\/span>,\r\n                        user_id=user_id,\r\n                        fraud_probability=fraud_probability,\r\n                        feature_count=<span class=\"hljs-built_in\">len<\/span>(features),\r\n                        model_version=<span class=\"hljs-string\">\"v2.1\"<\/span>)\r\n        \r\n        <span class=\"hljs-keyword\">if<\/span> fraud_probability &gt; <span class=\"hljs-number\">0.8<\/span>:\r\n            self.logger.error(<span class=\"hljs-string\">\"high_fraud_probability_detected\"<\/span>,\r\n                            user_id=user_id,\r\n                            fraud_probability=fraud_probability,\r\n                            risk_factors=self.identify_risk_factors(features))\r\n            \r\n            <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-string\">\"HIGH_RISK\"<\/span>\r\n        <span class=\"hljs-keyword\">elif<\/span> fraud_probability &gt; <span class=\"hljs-number\">0.5<\/span>:\r\n            self.logger.warning(<span class=\"hljs-string\">\"moderate_fraud_probability_detected\"<\/span>,\r\n                              user_id=user_id,\r\n                              fraud_probability=fraud_probability)\r\n            <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-string\">\"MODERATE_RISK\"<\/span>\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-string\">\"LOW_RISK\"<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">extract_features<\/span>(<span class=\"hljs-params\">self, events<\/span>):\r\n        <span class=\"hljs-comment\"># Convert log events to ML features<\/span>\r\n        df = pd.DataFrame(events)\r\n        \r\n        features = {\r\n            <span class=\"hljs-string\">'avg_trip_distance'<\/span>: df[<span class=\"hljs-string\">'trip_distance'<\/span>].mean(),\r\n            <span class=\"hljs-string\">'avg_trip_duration'<\/span>: df[<span class=\"hljs-string\">'trip_duration'<\/span>].mean(),\r\n            <span class=\"hljs-string\">'unique_pickup_locations'<\/span>: df[<span class=\"hljs-string\">'pickup_location'<\/span>].nunique(),\r\n            <span class=\"hljs-string\">'night_rides_ratio'<\/span>: (df[<span class=\"hljs-string\">'hour'<\/span>] &lt; <span class=\"hljs-number\">6<\/span>).<span class=\"hljs-built_in\">sum<\/span>() \/ <span class=\"hljs-built_in\">len<\/span>(df),\r\n            <span class=\"hljs-string\">'payment_method_changes'<\/span>: df[<span class=\"hljs-string\">'payment_method'<\/span>].nunique(),\r\n            <span class=\"hljs-string\">'surge_rides_ratio'<\/span>: (df[<span class=\"hljs-string\">'surge_multiplier'<\/span>] &gt; <span class=\"hljs-number\">1.0<\/span>).<span class=\"hljs-built_in\">sum<\/span>() \/ <span class=\"hljs-built_in\">len<\/span>(df)\r\n        }\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-built_in\">list<\/span>(features.values())<\/span><\/pre>\n<p><strong class=\"wj gl\">Results<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"c3ea\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Reduced fraudulent transactions by 75%<\/li>\n<li id=\"1080\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Decreased false positive rate by 60%<\/li>\n<li id=\"ee26\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Saved over $100M annually in fraud losses<\/li>\n<li id=\"b39e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Real-time detection with &lt;100ms latency<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Key Innovations<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"b03a\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Real-time feature engineering from log streams<\/li>\n<li id=\"859d\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Multi-model ensemble for different fraud types<\/li>\n<li id=\"4be8\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Continuous model retraining based on new fraud patterns<\/li>\n<li id=\"416f\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Integration of business rules with ML predictions<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h3 id=\"23ab\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Case Study 4: Uber \u2014 Real-time Fraud Detection Through Logs<\/h3>\n<p><strong class=\"wj gl\">Challenge<\/strong>: Detect fraudulent activities in real-time across ride requests, payments, and driver behavior.<strong class=\"wj gl\">Solution Architecture<\/strong>:<strong class=\"wj gl\">1. Structured Event Logging<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"1d86\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Ride request logging with fraud detection context<\/span>\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">RideRequestHandler<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span><\/span>):\r\n        <span class=\"hljs-variable.language\">self<\/span>.logger = structlog.get_logger(<span class=\"hljs-string\">\"fraud_detection\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">handle_ride_request<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, request<\/span>):\r\n        fraud_context = {\r\n            <span class=\"hljs-string\">\"user_id\"<\/span>: request.user_id,\r\n            <span class=\"hljs-string\">\"pickup_lat\"<\/span>: request.pickup_location.lat,\r\n            <span class=\"hljs-string\">\"pickup_lng\"<\/span>: request.pickup_location.lng,\r\n            <span class=\"hljs-string\">\"dropoff_lat\"<\/span>: request.dropoff_location.lat,\r\n            <span class=\"hljs-string\">\"dropoff_lng\"<\/span>: request.dropoff_location.lng,\r\n            <span class=\"hljs-string\">\"request_timestamp\"<\/span>: request.timestamp,\r\n            <span class=\"hljs-string\">\"device_id\"<\/span>: request.device_id,\r\n            <span class=\"hljs-string\">\"app_version\"<\/span>: request.app_version,\r\n            <span class=\"hljs-string\">\"payment_method\"<\/span>: request.payment_method,\r\n            <span class=\"hljs-string\">\"estimated_fare\"<\/span>: request.estimated_fare,\r\n            <span class=\"hljs-string\">\"user_account_age_days\"<\/span>: request.user.account_age_days,\r\n            <span class=\"hljs-string\">\"user_ride_count\"<\/span>: request.user.total_rides,\r\n            <span class=\"hljs-string\">\"recent_location_changes\"<\/span>: request.user.recent_location_changes\r\n        }\r\n        \r\n        <span class=\"hljs-variable.language\">self<\/span>.logger.info(<span class=\"hljs-string\">\"ride_request_received\"<\/span>, **fraud_context)\r\n        \r\n        <span class=\"hljs-comment\"># Real-time fraud scoring<\/span>\r\n        fraud_score = <span class=\"hljs-variable.language\">self<\/span>.calculate_fraud_score(fraud_context)\r\n        \r\n        <span class=\"hljs-keyword\">if<\/span> fraud_score &gt; <span class=\"hljs-variable.constant\">FRAUD_THRESHOLD<\/span>:\r\n            <span class=\"hljs-variable.language\">self<\/span>.logger.warning(<span class=\"hljs-string\">\"potential_fraud_detected\"<\/span>, \r\n                              fraud_score=fraud_score,\r\n                              **fraud_context)\r\n            <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-variable.language\">self<\/span>.handle_suspicious_request(request, fraud_score)\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-variable.language\">self<\/span>.process_normal_request(request)<\/span><\/pre>\n<p><strong class=\"wj gl\">2. Real-time Stream Processing<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"5067\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Kafka Streams processor for real-time fraud detection<\/span>\r\n<span class=\"hljs-keyword\">from<\/span> kafka <span class=\"hljs-keyword\">import<\/span> KafkaConsumer\r\n<span class=\"hljs-keyword\">import<\/span> json\r\n<span class=\"hljs-keyword\">import<\/span> redis\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">FraudDetectionProcessor<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.consumer = KafkaConsumer(\r\n            <span class=\"hljs-string\">'ride_requests'<\/span>,\r\n            bootstrap_servers=[<span class=\"hljs-string\">'kafka1:9092'<\/span>, <span class=\"hljs-string\">'kafka2:9092'<\/span>],\r\n            value_deserializer=<span class=\"hljs-keyword\">lambda<\/span> m: json.loads(m.decode(<span class=\"hljs-string\">'utf-8'<\/span>))\r\n        )\r\n        self.redis_client = redis.Redis(host=<span class=\"hljs-string\">'redis-cluster'<\/span>)\r\n        self.logger = structlog.get_logger(<span class=\"hljs-string\">\"fraud_processor\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">process_events<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-keyword\">for<\/span> message <span class=\"hljs-keyword\">in<\/span> self.consumer:\r\n            event_data = message.value\r\n            \r\n            <span class=\"hljs-comment\"># Pattern detection: Multiple requests from same device<\/span>\r\n            device_key = <span class=\"hljs-string\">f\"device_requests:<span class=\"hljs-subst\">{event_data['device_id']}<\/span>\"<\/span>\r\n            request_count = self.redis_client.incr(device_key)\r\n            self.redis_client.expire(device_key, <span class=\"hljs-number\">300<\/span>)  <span class=\"hljs-comment\"># 5-minute window<\/span>\r\n            \r\n            <span class=\"hljs-keyword\">if<\/span> request_count &gt; <span class=\"hljs-number\">10<\/span>:  <span class=\"hljs-comment\"># Suspicious: &gt;10 requests in 5 minutes<\/span>\r\n                self.logger.warning(<span class=\"hljs-string\">\"device_spam_detected\"<\/span>,\r\n                                  device_id=event_data[<span class=\"hljs-string\">'device_id'<\/span>],\r\n                                  request_count=request_count,\r\n                                  user_id=event_data[<span class=\"hljs-string\">'user_id'<\/span>])\r\n                \r\n                self.trigger_fraud_alert(event_data, <span class=\"hljs-string\">\"device_spam\"<\/span>)\r\n            \r\n            <span class=\"hljs-comment\"># Geographic impossibility detection<\/span>\r\n            <span class=\"hljs-keyword\">if<\/span> self.detect_geographic_impossibility(event_data):\r\n                self.logger.warning(<span class=\"hljs-string\">\"geographic_impossibility_detected\"<\/span>,\r\n                                  user_id=event_data[<span class=\"hljs-string\">'user_id'<\/span>],\r\n                                  **event_data)\r\n                \r\n                self.trigger_fraud_alert(event_data, <span class=\"hljs-string\">\"geographic_impossibility\"<\/span>)<\/span><\/pre>\n<p><strong class=\"wj gl\">3. Machine Learning and Automated Root Cause Analysis<\/strong>Large-scale systems increasingly rely on machine learning to detect complex fraud patterns and automatically identify root causes from logs. This includes anomaly detection, behavioral clustering, and predictive alerting. Log data enriched with business context allows <strong><a href=\"https:\/\/opstree.com\/services\/generative-ai-solutions\/\">ML models<\/a><\/strong> to improve accuracy and reduce false positives.Example architecture:<\/p>\n<ul class=\"\">\n<li id=\"dc3f\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Structured event logging with rich context (user, device, location, transaction metadata)<\/li>\n<li id=\"96b9\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><a href=\"https:\/\/opstree.com\/services\/middleware-database-and-data-engineering\/\"><strong>Real-time stream processing<\/strong><\/a> with Kafka or Pulsar<\/li>\n<li id=\"d302\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">ML models trained on historical log data to identify anomalies<\/li>\n<li id=\"5a56\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><a href=\"https:\/\/www.buildpiper.io\/blogs\/exploring-devops-observability\/\" target=\"_blank\" rel=\"noopener\"><strong>Automated alerts and dashboards<\/strong><\/a> for fraud investigators<\/li>\n<\/ul>\n<h3 id=\"4b46\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">4. Lessons Learned from Uber \u2019s Logging Strategy<\/h3>\n<ul class=\"\">\n<li id=\"d1a5\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Structured logs enable fine-grained event tracking essential for ML models.<\/li>\n<li id=\"7a76\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Real-time processing is critical for rapid fraud detection and mitigation.<\/li>\n<li id=\"3204\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Correlation across logs, metrics, and traces improves investigation efficiency.<\/li>\n<li id=\"a7a0\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Continuous retraining of models on latest data prevents degradation in detection capabilities.<\/li>\n<\/ul>\n<h2 id=\"security-and-compliance\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Security and Compliance<\/h2>\n<h3 id=\"bb24\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Log Security Best Practices<\/h3>\n<h4 id=\"2d21\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Sensitive Data Protection<\/h4>\n<p><strong class=\"wj gl\">Never Log These Items<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"b8d2\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Passwords or authentication tokens<\/li>\n<li id=\"85ed\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Credit card numbers or SSNs<\/li>\n<li id=\"8406\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Personal identification information<\/li>\n<li id=\"e42f\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><a href=\"https:\/\/opstree.com\/blog\/2022\/04\/26\/google-python-api-the-easy-way\/\">API keys<\/a> or secrets<\/li>\n<li id=\"88d7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Medical records or other regulated data<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Safe Logging Example<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"be76\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> hashlib\r\n<span class=\"hljs-keyword\">import<\/span> re\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">SecureLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.logger = structlog.get_logger()\r\n        self.pii_patterns = [\r\n            <span class=\"hljs-string\">r'\\b\\d{4}[-\\s]?\\d{4}[-\\s]?\\d{4}[-\\s]?\\d{4}\\b'<\/span>,  <span class=\"hljs-comment\"># Credit cards<\/span>\r\n            <span class=\"hljs-string\">r'\\b\\d{3}-\\d{2}-\\d{4}\\b'<\/span>,  <span class=\"hljs-comment\"># SSNs<\/span>\r\n            <span class=\"hljs-string\">r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b'<\/span>  <span class=\"hljs-comment\"># Emails<\/span>\r\n        ]\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log_user_action<\/span>(<span class=\"hljs-params\">self, user_id, action, details<\/span>):\r\n        <span class=\"hljs-comment\"># Hash sensitive identifiers<\/span>\r\n        user_hash = hashlib.sha256(<span class=\"hljs-built_in\">str<\/span>(user_id).encode()).hexdigest()[:<span class=\"hljs-number\">8<\/span>]\r\n        \r\n        <span class=\"hljs-comment\"># Sanitize details<\/span>\r\n        safe_details = self.sanitize_data(details)\r\n        \r\n        self.logger.info(<span class=\"hljs-string\">\"user_action\"<\/span>,\r\n                        user_hash=user_hash,\r\n                        action=action,\r\n                        details=safe_details)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">sanitize_data<\/span>(<span class=\"hljs-params\">self, data<\/span>):\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">isinstance<\/span>(data, <span class=\"hljs-built_in\">str<\/span>):\r\n            <span class=\"hljs-keyword\">for<\/span> pattern <span class=\"hljs-keyword\">in<\/span> self.pii_patterns:\r\n                data = re.sub(pattern, <span class=\"hljs-string\">'[REDACTED]'<\/span>, data)\r\n        <span class=\"hljs-keyword\">elif<\/span> <span class=\"hljs-built_in\">isinstance<\/span>(data, <span class=\"hljs-built_in\">dict<\/span>):\r\n            <span class=\"hljs-keyword\">return<\/span> {k: self.sanitize_data(v) <span class=\"hljs-keyword\">for<\/span> k, v <span class=\"hljs-keyword\">in<\/span> data.items()}\r\n        <span class=\"hljs-keyword\">elif<\/span> <span class=\"hljs-built_in\">isinstance<\/span>(data, <span class=\"hljs-built_in\">list<\/span>):\r\n            <span class=\"hljs-keyword\">return<\/span> [self.sanitize_data(item) <span class=\"hljs-keyword\">for<\/span> item <span class=\"hljs-keyword\">in<\/span> data]\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> data\r\n<span class=\"hljs-comment\"># Usage example<\/span>\r\nsecure_logger = SecureLogger()\r\nsecure_logger.log_user_action(\r\n    user_id=<span class=\"hljs-number\">12345<\/span>,\r\n    action=<span class=\"hljs-string\">\"payment_processed\"<\/span>,\r\n    details={\r\n        <span class=\"hljs-string\">\"amount\"<\/span>: <span class=\"hljs-number\">99.99<\/span>,\r\n        <span class=\"hljs-string\">\"payment_method\"<\/span>: <span class=\"hljs-string\">\"credit_card_ending_1234\"<\/span>,  <span class=\"hljs-comment\"># Safe<\/span>\r\n        <span class=\"hljs-string\">\"email\"<\/span>: <span class=\"hljs-string\">\"user@example.com\"<\/span>  <span class=\"hljs-comment\"># Will be redacted<\/span>\r\n    }\r\n)<\/span><\/pre>\n<h4 id=\"7485\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Log Encryption and Transport Security<\/h4>\n<p><strong class=\"wj gl\">Encryption in Transit<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"7ad7\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Filebeat configuration with TLS<\/span>\r\nfilebeat.inputs:\r\n- <span class=\"hljs-built_in\">type<\/span>: <span class=\"hljs-built_in\">log<\/span>\r\n  paths:\r\n    - \/var\/log\/app\/*.<span class=\"hljs-built_in\">log<\/span>\r\n  \r\noutput.elasticsearch:\r\n  hosts: [<span class=\"hljs-string\">\"elasticsearch.example.com:9200\"<\/span>]\r\n  protocol: <span class=\"hljs-string\">\"https\"<\/span>\r\n  ssl.certificate_authorities: [<span class=\"hljs-string\">\"\/path\/to\/ca.crt\"<\/span>]\r\n  ssl.certificate: <span class=\"hljs-string\">\"\/path\/to\/client.crt\"<\/span>\r\n  ssl.key: <span class=\"hljs-string\">\"\/path\/to\/client.key\"<\/span>\r\n  ssl.verification_mode: <span class=\"hljs-string\">\"strict\"<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">Encryption at Rest<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"7850\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># Elasticsearch configuration for encryption at rest<\/span>\r\nxpack.security.enabled: <span class=\"hljs-literal\">true<\/span>\r\nxpack.security.encryption_key.enabled: <span class=\"hljs-literal\">true<\/span>\r\nxpack.security.encryption_key.path: \/path\/to\/encryption.key\r\n\r\n<span class=\"hljs-comment\"># Enable field-level encryption for sensitive data<\/span>\r\nPUT \/secure_logs\r\n{\r\n  <span class=\"hljs-string\">\"mappings\"<\/span>: {\r\n    <span class=\"hljs-string\">\"properties\"<\/span>: {\r\n      <span class=\"hljs-string\">\"sensitive_field\"<\/span>: {\r\n        <span class=\"hljs-string\">\"type\"<\/span>: <span class=\"hljs-string\">\"text\"<\/span>,\r\n        <span class=\"hljs-string\">\"fields\"<\/span>: {\r\n          <span class=\"hljs-string\">\"encrypted\"<\/span>: {\r\n            <span class=\"hljs-string\">\"type\"<\/span>: <span class=\"hljs-string\">\"text\"<\/span>,\r\n            <span class=\"hljs-string\">\"store\"<\/span>: <span class=\"hljs-literal\">true<\/span>,\r\n            <span class=\"hljs-string\">\"index\"<\/span>: <span class=\"hljs-literal\">false<\/span>\r\n          }\r\n        }\r\n      }\r\n    }\r\n  }\r\n}<\/span><\/pre>\n<h3 id=\"08ea\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Compliance Requirements<\/h3>\n<h4 id=\"cc4f\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">GDPR (General Data Protection Regulation)<\/h4>\n<p><strong class=\"wj gl\">Key Requirements for Logging<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"4b9b\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Right to be forgotten: Ability to delete user data from logs<\/li>\n<li id=\"f088\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Data minimization: Log only necessary information<\/li>\n<li id=\"a7e5\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Consent: Clear purpose for data collection<\/li>\n<li id=\"02cc\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Data retention: Automated deletion after specified periods<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">GDPR-Compliant Logging Implementation<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"04a3\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">GDPRCompliantLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.logger = structlog.get_logger()\r\n        self.retention_days = <span class=\"hljs-number\">365<\/span>  <span class=\"hljs-comment\"># Configurable retention period<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log_with_retention<\/span>(<span class=\"hljs-params\">self, event_type, data, retention_override=<span class=\"hljs-literal\">None<\/span><\/span>):\r\n        retention_date = datetime.utcnow() + timedelta(\r\n            days=retention_override <span class=\"hljs-keyword\">or<\/span> self.retention_days\r\n        )\r\n        \r\n        log_entry = {\r\n            <span class=\"hljs-string\">\"timestamp\"<\/span>: datetime.utcnow().isoformat(),\r\n            <span class=\"hljs-string\">\"event_type\"<\/span>: event_type,\r\n            <span class=\"hljs-string\">\"data\"<\/span>: self.pseudonymize_data(data),\r\n            <span class=\"hljs-string\">\"retention_date\"<\/span>: retention_date.isoformat(),\r\n            <span class=\"hljs-string\">\"data_subject_id\"<\/span>: self.generate_subject_id(data)\r\n        }\r\n        \r\n        self.logger.info(<span class=\"hljs-string\">\"gdpr_compliant_event\"<\/span>, **log_entry)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">pseudonymize_data<\/span>(<span class=\"hljs-params\">self, data<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Replace direct identifiers with pseudonyms\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-string\">'user_id'<\/span> <span class=\"hljs-keyword\">in<\/span> data:\r\n            data[<span class=\"hljs-string\">'user_pseudonym'<\/span>] = self.generate_pseudonym(data[<span class=\"hljs-string\">'user_id'<\/span>])\r\n            <span class=\"hljs-keyword\">del<\/span> data[<span class=\"hljs-string\">'user_id'<\/span>]\r\n        \r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-string\">'email'<\/span> <span class=\"hljs-keyword\">in<\/span> data:\r\n            data[<span class=\"hljs-string\">'email_domain'<\/span>] = data[<span class=\"hljs-string\">'email'<\/span>].split(<span class=\"hljs-string\">'@'<\/span>)[<span class=\"hljs-number\">1<\/span>]\r\n            <span class=\"hljs-keyword\">del<\/span> data[<span class=\"hljs-string\">'email'<\/span>]\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> data\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">generate_pseudonym<\/span>(<span class=\"hljs-params\">self, identifier<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Generate consistent pseudonym for an identifier\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">return<\/span> hashlib.sha256(<span class=\"hljs-string\">f\"<span class=\"hljs-subst\">{identifier}<\/span>:<span class=\"hljs-subst\">{os.getenv('PSEUDONYM_SALT')}<\/span>\"<\/span>.encode()).hexdigest()[:<span class=\"hljs-number\">12<\/span>]<\/span><\/pre>\n<h3 id=\"f18d\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">SOX (Sarbanes-Oxley) Compliance<\/h3>\n<p><strong class=\"wj gl\">Requirements<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"650d\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Immutable audit logs<\/li>\n<li id=\"d7d2\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Access logging for financial systems<\/li>\n<li id=\"dc71\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Change tracking and approval workflows<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">SOX-Compliant Audit Logging<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"db29\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">SOXAuditLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.logger = structlog.get_logger(<span class=\"hljs-string\">\"sox_audit\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log_financial_transaction<\/span>(<span class=\"hljs-params\">self, transaction_id, user_id, action, amount, details<\/span>):\r\n        <span class=\"hljs-comment\"># Create immutable audit entry<\/span>\r\n        audit_entry = {\r\n            <span class=\"hljs-string\">\"audit_id\"<\/span>: self.generate_audit_id(),\r\n            <span class=\"hljs-string\">\"transaction_id\"<\/span>: transaction_id,\r\n            <span class=\"hljs-string\">\"user_id\"<\/span>: user_id,\r\n            <span class=\"hljs-string\">\"action\"<\/span>: action,\r\n            <span class=\"hljs-string\">\"amount\"<\/span>: <span class=\"hljs-built_in\">float<\/span>(amount),\r\n            <span class=\"hljs-string\">\"details\"<\/span>: details,\r\n            <span class=\"hljs-string\">\"timestamp\"<\/span>: datetime.utcnow().isoformat(),\r\n            <span class=\"hljs-string\">\"system_state_hash\"<\/span>: self.calculate_system_state_hash(),\r\n            <span class=\"hljs-string\">\"compliance_framework\"<\/span>: <span class=\"hljs-string\">\"SOX\"<\/span>\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Digital signature for tamper detection<\/span>\r\n        audit_entry[<span class=\"hljs-string\">\"signature\"<\/span>] = self.sign_entry(audit_entry)\r\n        \r\n        self.logger.info(<span class=\"hljs-string\">\"sox_audit_event\"<\/span>, **audit_entry)\r\n        \r\n        <span class=\"hljs-comment\"># Also send to immutable storage<\/span>\r\n        self.store_in_blockchain_ledger(audit_entry)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">sign_entry<\/span>(<span class=\"hljs-params\">self, entry<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Create digital signature for audit entry\"\"\"<\/span>\r\n        entry_string = json.dumps(entry, sort_keys=<span class=\"hljs-literal\">True<\/span>)\r\n        <span class=\"hljs-keyword\">return<\/span> hmac.new(\r\n            os.getenv(<span class=\"hljs-string\">'AUDIT_SIGNING_KEY'<\/span>).encode(),\r\n            entry_string.encode(),\r\n            hashlib.sha256\r\n        ).hexdigest()<\/span><\/pre>\n<h3 id=\"6499\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">HIPAA (Health Insurance Portability and Accountability Act)<\/h3>\n<p><strong class=\"wj gl\">Requirements<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"1087\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">PHI (Protected Health Information) handling<\/li>\n<li id=\"0ab9\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Access controls and audit trails<\/li>\n<li id=\"8e02\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Data breach notification capabilities<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">HIPAA-Compliant Medical System Logging<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"8525\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">HIPAALogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.logger = structlog.get_logger(<span class=\"hljs-string\">\"hipaa_audit\"<\/span>)\r\n        self.phi_fields = [<span class=\"hljs-string\">'ssn'<\/span>, <span class=\"hljs-string\">'medical_record_number'<\/span>, <span class=\"hljs-string\">'patient_name'<\/span>, <span class=\"hljs-string\">'date_of_birth'<\/span>]\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log_phi_access<\/span>(<span class=\"hljs-params\">self, user_id, patient_id, action, phi_accessed, justification<\/span>):\r\n        <span class=\"hljs-comment\"># All PHI access must be logged<\/span>\r\n        audit_entry = {\r\n            <span class=\"hljs-string\">\"access_id\"<\/span>: <span class=\"hljs-built_in\">str<\/span>(uuid.uuid4()),\r\n            <span class=\"hljs-string\">\"user_id\"<\/span>: user_id,\r\n            <span class=\"hljs-string\">\"patient_id\"<\/span>: self.hash_patient_id(patient_id),\r\n            <span class=\"hljs-string\">\"action\"<\/span>: action,\r\n            <span class=\"hljs-string\">\"phi_fields_accessed\"<\/span>: phi_accessed,\r\n            <span class=\"hljs-string\">\"access_justification\"<\/span>: justification,\r\n            <span class=\"hljs-string\">\"timestamp\"<\/span>: datetime.utcnow().isoformat(),\r\n            <span class=\"hljs-string\">\"user_role\"<\/span>: self.get_user_role(user_id),\r\n            <span class=\"hljs-string\">\"access_method\"<\/span>: <span class=\"hljs-string\">\"system\"<\/span>,\r\n            <span class=\"hljs-string\">\"patient_consent_verified\"<\/span>: self.verify_patient_consent(patient_id)\r\n        }\r\n        \r\n        self.logger.info(<span class=\"hljs-string\">\"hipaa_phi_access\"<\/span>, **audit_entry)\r\n        \r\n        <span class=\"hljs-comment\"># Alert for suspicious access patterns<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> self.detect_suspicious_access(user_id, patient_id, action):\r\n            self.logger.warning(<span class=\"hljs-string\">\"suspicious_phi_access_detected\"<\/span>,\r\n                              user_id=user_id,\r\n                              patient_id=self.hash_patient_id(patient_id),\r\n                              reason=<span class=\"hljs-string\">\"unusual_access_pattern\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">hash_patient_id<\/span>(<span class=\"hljs-params\">self, patient_id<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Create consistent but non-reversible patient identifier\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">return<\/span> hashlib.sha256(<span class=\"hljs-string\">f\"<span class=\"hljs-subst\">{patient_id}<\/span>:<span class=\"hljs-subst\">{os.getenv('PATIENT_ID_SALT')}<\/span>\"<\/span>.encode()).hexdigest()[:<span class=\"hljs-number\">16<\/span>]<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"performance-optimization\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Performance Optimization<\/h2>\n<h3 id=\"deba\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Log Performance Impact<\/h3>\n<p>Logging can significantly impact application performance if not implemented carefully. Here\u2019s how to optimize:<\/p>\n<h4 id=\"2924\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Asynchronous Logging<\/h4>\n<p><strong class=\"wj gl\">Problem<\/strong>: Synchronous logging blocks application threads <strong class=\"wj gl\">Solution<\/strong>: Use asynchronous logging frameworks<strong class=\"wj gl\">Java Example with Logback<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"2f9d\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\">&lt;!-- logback-spring.xml --&gt;<\/span>\r\n<span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">configuration<\/span>&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">appender<\/span> <span class=\"hljs-attr\">name<\/span>=<span class=\"hljs-string\">\"ASYNC\"<\/span> <span class=\"hljs-attr\">class<\/span>=<span class=\"hljs-string\">\"ch.qos.logback.classic.AsyncAppender\"<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">queueSize<\/span>&gt;<\/span>1000<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">queueSize<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">discardingThreshold<\/span>&gt;<\/span>0<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">discardingThreshold<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">includeCallerData<\/span>&gt;<\/span>false<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">includeCallerData<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">appender-ref<\/span> <span class=\"hljs-attr\">ref<\/span>=<span class=\"hljs-string\">\"FILE\"<\/span>\/&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">appender<\/span>&gt;<\/span>\r\n    \r\n    <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">appender<\/span> <span class=\"hljs-attr\">name<\/span>=<span class=\"hljs-string\">\"FILE\"<\/span> <span class=\"hljs-attr\">class<\/span>=<span class=\"hljs-string\">\"ch.qos.logback.core.rolling.RollingFileAppender\"<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">file<\/span>&gt;<\/span>app.log<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">file<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">rollingPolicy<\/span> <span class=\"hljs-attr\">class<\/span>=<span class=\"hljs-string\">\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\"<\/span>&gt;<\/span>\r\n            <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">fileNamePattern<\/span>&gt;<\/span>app.%d{yyyy-MM-dd}.%i.gz<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">fileNamePattern<\/span>&gt;<\/span>\r\n            <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">maxFileSize<\/span>&gt;<\/span>100MB<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">maxFileSize<\/span>&gt;<\/span>\r\n            <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">maxHistory<\/span>&gt;<\/span>30<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">maxHistory<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">rollingPolicy<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">encoder<\/span>&gt;<\/span>\r\n            <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">pattern<\/span>&gt;<\/span>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">pattern<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">encoder<\/span>&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">appender<\/span>&gt;<\/span>\r\n    \r\n    <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">root<\/span> <span class=\"hljs-attr\">level<\/span>=<span class=\"hljs-string\">\"INFO\"<\/span>&gt;<\/span>\r\n        <span class=\"hljs-tag\">&lt;<span class=\"hljs-name\">appender-ref<\/span> <span class=\"hljs-attr\">ref<\/span>=<span class=\"hljs-string\">\"ASYNC\"<\/span>\/&gt;<\/span>\r\n    <span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">root<\/span>&gt;<\/span>\r\n<span class=\"hljs-tag\">&lt;\/<span class=\"hljs-name\">configuration<\/span>&gt;<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">Python Asyncio Example<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"be7e\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> asyncio\r\n<span class=\"hljs-keyword\">import<\/span> logging\r\n<span class=\"hljs-keyword\">from<\/span> concurrent.futures <span class=\"hljs-keyword\">import<\/span> ThreadPoolExecutor\r\n<span class=\"hljs-keyword\">import<\/span> queue\r\n<span class=\"hljs-keyword\">import<\/span> threading\r\n<span class=\"hljs-keyword\">import<\/span> json\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">AsyncLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, max_queue_size=<span class=\"hljs-number\">10000<\/span><\/span>):\r\n        self.log_queue = queue.Queue(maxsize=max_queue_size)\r\n        self.executor = ThreadPoolExecutor(max_workers=<span class=\"hljs-number\">2<\/span>, thread_name_prefix=<span class=\"hljs-string\">\"log-worker\"<\/span>)\r\n        self.running = <span class=\"hljs-literal\">True<\/span>\r\n        \r\n        <span class=\"hljs-comment\"># Start background logging thread<\/span>\r\n        self.log_thread = threading.Thread(target=self._process_logs, daemon=<span class=\"hljs-literal\">True<\/span>)\r\n        self.log_thread.start()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_process_logs<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Background thread to process log queue\"\"\"<\/span>\r\n        logger = logging.getLogger(<span class=\"hljs-string\">\"async_app\"<\/span>)\r\n        handler = logging.FileHandler(<span class=\"hljs-string\">\"app.log\"<\/span>)\r\n        formatter = logging.Formatter(<span class=\"hljs-string\">'%(asctime)s - %(levelname)s - %(message)s'<\/span>)\r\n        handler.setFormatter(formatter)\r\n        logger.addHandler(handler)\r\n        logger.setLevel(logging.INFO)\r\n        \r\n        <span class=\"hljs-keyword\">while<\/span> self.running:\r\n            <span class=\"hljs-keyword\">try<\/span>:\r\n                log_entry = self.log_queue.get(timeout=<span class=\"hljs-number\">1<\/span>)\r\n                <span class=\"hljs-keyword\">if<\/span> log_entry <span class=\"hljs-keyword\">is<\/span> <span class=\"hljs-literal\">None<\/span>:  <span class=\"hljs-comment\"># Shutdown signal<\/span>\r\n                    <span class=\"hljs-keyword\">break<\/span>\r\n                \r\n                level, message, extra = log_entry\r\n                logger.log(level, message, extra=extra)\r\n                self.log_queue.task_done()\r\n                \r\n            <span class=\"hljs-keyword\">except<\/span> queue.Empty:\r\n                <span class=\"hljs-keyword\">continue<\/span>\r\n            <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n                <span class=\"hljs-built_in\">print<\/span>(<span class=\"hljs-string\">f\"Logging error: <span class=\"hljs-subst\">{e}<\/span>\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log<\/span>(<span class=\"hljs-params\">self, level, message, **kwargs<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Non-blocking log method\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            self.log_queue.put_nowait((level, message, kwargs))\r\n        <span class=\"hljs-keyword\">except<\/span> queue.Full:\r\n            <span class=\"hljs-comment\"># Drop logs if queue is full to prevent blocking<\/span>\r\n            <span class=\"hljs-keyword\">pass<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">info<\/span>(<span class=\"hljs-params\">self, message, **kwargs<\/span>):\r\n        self.log(logging.INFO, message, **kwargs)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">error<\/span>(<span class=\"hljs-params\">self, message, **kwargs<\/span>):\r\n        self.log(logging.ERROR, message, **kwargs)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">shutdown<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.running = <span class=\"hljs-literal\">False<\/span>\r\n        self.log_queue.put(<span class=\"hljs-literal\">None<\/span>)  <span class=\"hljs-comment\"># Shutdown signal<\/span>\r\n        self.log_thread.join()\r\n        self.executor.shutdown(wait=<span class=\"hljs-literal\">True<\/span>)\r\n<span class=\"hljs-comment\"># Usage<\/span>\r\nasync_logger = AsyncLogger()\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">process_request<\/span>(<span class=\"hljs-params\">request_id<\/span>):\r\n    async_logger.info(<span class=\"hljs-string\">\"Processing request\"<\/span>, request_id=request_id)\r\n    \r\n    <span class=\"hljs-comment\"># Simulate work<\/span>\r\n    <span class=\"hljs-keyword\">import<\/span> time\r\n    time.sleep(<span class=\"hljs-number\">0.1<\/span>)\r\n    \r\n    async_logger.info(<span class=\"hljs-string\">\"Request completed\"<\/span>, request_id=request_id, duration_ms=<span class=\"hljs-number\">100<\/span>)<\/span><\/pre>\n<h4 id=\"11a1\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Log Level Filtering<\/h4>\n<p><strong class=\"wj gl\">Performance Impact<\/strong>: Creating log messages that are discarded wastes CPU cycles<strong class=\"wj gl\">Efficient Log Level Checking<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"7f1d\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> logging\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">OptimizedLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, name<\/span>):\r\n        self.logger = logging.getLogger(name)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">debug<\/span>(<span class=\"hljs-params\">self, message, **kwargs<\/span>):\r\n        <span class=\"hljs-comment\"># Check level before expensive operations<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> self.logger.isEnabledFor(logging.DEBUG):\r\n            <span class=\"hljs-comment\"># Only format message if debug logging is enabled<\/span>\r\n            formatted_message = self._format_message(message, **kwargs)\r\n            self.logger.debug(formatted_message)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_format_message<\/span>(<span class=\"hljs-params\">self, message, **kwargs<\/span>):\r\n        <span class=\"hljs-comment\"># Expensive formatting operation<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> kwargs:\r\n            <span class=\"hljs-keyword\">import<\/span> json\r\n            context = json.dumps(kwargs)\r\n            <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-string\">f\"<span class=\"hljs-subst\">{message}<\/span> - Context: <span class=\"hljs-subst\">{context}<\/span>\"<\/span>\r\n        <span class=\"hljs-keyword\">return<\/span> message\r\n<span class=\"hljs-comment\"># Lazy evaluation with lambdas<\/span>\r\nlogger = logging.getLogger(__name__)\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">expensive_computation<\/span>():\r\n    <span class=\"hljs-comment\"># Simulate expensive operation<\/span>\r\n    <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-built_in\">sum<\/span>(<span class=\"hljs-built_in\">range<\/span>(<span class=\"hljs-number\">100000<\/span>))\r\n<span class=\"hljs-comment\"># Bad: Always computes even if debug is disabled<\/span>\r\nlogger.debug(<span class=\"hljs-string\">f\"Result: <span class=\"hljs-subst\">{expensive_computation()}<\/span>\"<\/span>)\r\n<span class=\"hljs-comment\"># Good: Only computes if debug is enabled<\/span>\r\n<span class=\"hljs-keyword\">if<\/span> logger.isEnabledFor(logging.DEBUG):\r\n    logger.debug(<span class=\"hljs-string\">f\"Result: <span class=\"hljs-subst\">{expensive_computation()}<\/span>\"<\/span>)\r\n<span class=\"hljs-comment\"># Better: Use lazy evaluation<\/span>\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log_debug_lazy<\/span>(<span class=\"hljs-params\">logger, message_func<\/span>):\r\n    <span class=\"hljs-keyword\">if<\/span> logger.isEnabledFor(logging.DEBUG):\r\n        logger.debug(message_func())\r\nlog_debug_lazy(logger, <span class=\"hljs-keyword\">lambda<\/span>: <span class=\"hljs-string\">f\"Result: <span class=\"hljs-subst\">{expensive_computation()}<\/span>\"<\/span>)<\/span><\/pre>\n<h4 id=\"3faf\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">3. Structured Logging Performance<\/h4>\n<p><strong class=\"wj gl\">JSON Serialization Optimization<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"544a\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> orjson  <span class=\"hljs-comment\"># Fast JSON serialization<\/span>\r\n<span class=\"hljs-keyword\">import<\/span> ujson   <span class=\"hljs-comment\"># Alternative fast JSON library<\/span>\r\n<span class=\"hljs-keyword\">import<\/span> logging\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">HighPerformanceStructuredLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.logger = logging.getLogger(<span class=\"hljs-string\">\"high_perf\"<\/span>)\r\n        \r\n        <span class=\"hljs-comment\"># Use fastest JSON serializer available<\/span>\r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            <span class=\"hljs-keyword\">import<\/span> orjson\r\n            self.json_dumps = <span class=\"hljs-keyword\">lambda<\/span> obj: orjson.dumps(obj).decode()\r\n        <span class=\"hljs-keyword\">except<\/span> ImportError:\r\n            <span class=\"hljs-keyword\">try<\/span>:\r\n                <span class=\"hljs-keyword\">import<\/span> ujson\r\n                self.json_dumps = ujson.dumps\r\n            <span class=\"hljs-keyword\">except<\/span> ImportError:\r\n                <span class=\"hljs-keyword\">import<\/span> json\r\n                self.json_dumps = json.dumps\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">structured_log<\/span>(<span class=\"hljs-params\">self, level, event, **context<\/span>):\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-keyword\">not<\/span> self.logger.isEnabledFor(level):\r\n            <span class=\"hljs-keyword\">return<\/span>\r\n        \r\n        log_entry = {\r\n            <span class=\"hljs-string\">\"timestamp\"<\/span>: self._get_timestamp(),\r\n            <span class=\"hljs-string\">\"level\"<\/span>: logging.getLevelName(level),\r\n            <span class=\"hljs-string\">\"event\"<\/span>: event,\r\n            **context\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Fast JSON serialization<\/span>\r\n        message = self.json_dumps(log_entry)\r\n        self.logger.log(level, message)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_get_timestamp<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-comment\"># Optimized timestamp generation<\/span>\r\n        <span class=\"hljs-keyword\">import<\/span> time\r\n        <span class=\"hljs-keyword\">return<\/span> time.time()\r\n<span class=\"hljs-comment\"># Benchmark different approaches<\/span>\r\n<span class=\"hljs-keyword\">import<\/span> time\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">benchmark_logging<\/span>():\r\n    logger = HighPerformanceStructuredLogger()\r\n    \r\n    <span class=\"hljs-comment\"># Measure performance<\/span>\r\n    start_time = time.time()\r\n    \r\n    <span class=\"hljs-keyword\">for<\/span> i <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-built_in\">range<\/span>(<span class=\"hljs-number\">10000<\/span>):\r\n        logger.structured_log(\r\n            logging.INFO,\r\n            <span class=\"hljs-string\">\"test_event\"<\/span>,\r\n            iteration=i,\r\n            user_id=<span class=\"hljs-string\">f\"user_<span class=\"hljs-subst\">{i}<\/span>\"<\/span>,\r\n            processing_time=<span class=\"hljs-number\">0.1<\/span> * i\r\n        )\r\n    \r\n    end_time = time.time()\r\n    <span class=\"hljs-built_in\">print<\/span>(<span class=\"hljs-string\">f\"Logged 10,000 entries in <span class=\"hljs-subst\">{end_time - start_time:<span class=\"hljs-number\">.3<\/span>f}<\/span> seconds\"<\/span>)<\/span><\/pre>\n<h4 id=\"b7f9\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">4. Log Sampling and Rate Limiting<\/h4>\n<p><strong class=\"wj gl\">High-Volume Log Management<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"4284\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> random\r\n<span class=\"hljs-keyword\">import<\/span> time\r\n<span class=\"hljs-keyword\">from<\/span> collections <span class=\"hljs-keyword\">import<\/span> defaultdict, deque\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">SamplingLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, sample_rates=<span class=\"hljs-literal\">None<\/span>, rate_limits=<span class=\"hljs-literal\">None<\/span><\/span>):\r\n        self.logger = logging.getLogger(<span class=\"hljs-string\">\"sampling\"<\/span>)\r\n        \r\n        <span class=\"hljs-comment\"># Default sampling rates by log level<\/span>\r\n        self.sample_rates = sample_rates <span class=\"hljs-keyword\">or<\/span> {\r\n            logging.DEBUG: <span class=\"hljs-number\">0.01<\/span>,    <span class=\"hljs-comment\"># 1% of debug logs<\/span>\r\n            logging.INFO: <span class=\"hljs-number\">0.1<\/span>,      <span class=\"hljs-comment\"># 10% of info logs<\/span>\r\n            logging.WARNING: <span class=\"hljs-number\">0.5<\/span>,   <span class=\"hljs-comment\"># 50% of warning logs<\/span>\r\n            logging.ERROR: <span class=\"hljs-number\">1.0<\/span>,     <span class=\"hljs-comment\"># 100% of error logs<\/span>\r\n            logging.CRITICAL: <span class=\"hljs-number\">1.0<\/span>   <span class=\"hljs-comment\"># 100% of critical logs<\/span>\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Rate limits (logs per second)<\/span>\r\n        self.rate_limits = rate_limits <span class=\"hljs-keyword\">or<\/span> {\r\n            logging.DEBUG: <span class=\"hljs-number\">10<\/span>,\r\n            logging.INFO: <span class=\"hljs-number\">100<\/span>,\r\n            logging.WARNING: <span class=\"hljs-number\">1000<\/span>,\r\n            logging.ERROR: <span class=\"hljs-built_in\">float<\/span>(<span class=\"hljs-string\">'inf'<\/span>),\r\n            logging.CRITICAL: <span class=\"hljs-built_in\">float<\/span>(<span class=\"hljs-string\">'inf'<\/span>)\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Rate limiting state<\/span>\r\n        self.rate_limit_windows = defaultdict(<span class=\"hljs-keyword\">lambda<\/span>: deque())\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">should_log<\/span>(<span class=\"hljs-params\">self, level<\/span>):\r\n        <span class=\"hljs-comment\"># Check sampling rate<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> random.random() &gt; self.sample_rates.get(level, <span class=\"hljs-number\">1.0<\/span>):\r\n            <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-literal\">False<\/span>\r\n        \r\n        <span class=\"hljs-comment\"># Check rate limit<\/span>\r\n        current_time = time.time()\r\n        window = self.rate_limit_windows[level]\r\n        \r\n        <span class=\"hljs-comment\"># Remove old entries (older than 1 second)<\/span>\r\n        <span class=\"hljs-keyword\">while<\/span> window <span class=\"hljs-keyword\">and<\/span> window[<span class=\"hljs-number\">0<\/span>] &lt; current_time - <span class=\"hljs-number\">1<\/span>:\r\n            window.popleft()\r\n        \r\n        <span class=\"hljs-comment\"># Check if we're under the rate limit<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">len<\/span>(window) &gt;= self.rate_limits.get(level, <span class=\"hljs-built_in\">float<\/span>(<span class=\"hljs-string\">'inf'<\/span>)):\r\n            <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-literal\">False<\/span>\r\n        \r\n        <span class=\"hljs-comment\"># Add current time to window<\/span>\r\n        window.append(current_time)\r\n        <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-literal\">True<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log<\/span>(<span class=\"hljs-params\">self, level, message, **kwargs<\/span>):\r\n        <span class=\"hljs-keyword\">if<\/span> self.should_log(level):\r\n            <span class=\"hljs-keyword\">if<\/span> kwargs:\r\n                <span class=\"hljs-comment\"># Add sampling metadata<\/span>\r\n                kwargs[<span class=\"hljs-string\">'sampled'<\/span>] = <span class=\"hljs-literal\">True<\/span>\r\n                kwargs[<span class=\"hljs-string\">'sample_rate'<\/span>] = self.sample_rates.get(level, <span class=\"hljs-number\">1.0<\/span>)\r\n            \r\n            self.logger.log(level, message, extra=kwargs)\r\n<span class=\"hljs-comment\"># Usage example<\/span>\r\nsampling_logger = SamplingLogger()\r\n<span class=\"hljs-comment\"># This will only log 10% of these info messages<\/span>\r\n<span class=\"hljs-keyword\">for<\/span> i <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-built_in\">range<\/span>(<span class=\"hljs-number\">1000<\/span>):\r\n    sampling_logger.log(logging.INFO, <span class=\"hljs-string\">f\"Processing item <span class=\"hljs-subst\">{i}<\/span>\"<\/span>, item_id=i)<\/span><\/pre>\n<h4 id=\"6e46\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">5. Log Batching and Buffering<\/h4>\n<p><strong class=\"wj gl\">Batch Processing for Better Performance<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"ec38\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> threading\r\n<span class=\"hljs-keyword\">import<\/span> time\r\n<span class=\"hljs-keyword\">from<\/span> queue <span class=\"hljs-keyword\">import<\/span> Queue, Empty\r\n<span class=\"hljs-keyword\">import<\/span> logging\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">BatchingLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, batch_size=<span class=\"hljs-number\">100<\/span>, flush_interval=<span class=\"hljs-number\">5.0<\/span><\/span>):\r\n        self.batch_size = batch_size\r\n        self.flush_interval = flush_interval\r\n        self.log_queue = Queue()\r\n        self.buffer = []\r\n        self.last_flush = time.time()\r\n        \r\n        <span class=\"hljs-comment\"># Setup actual logger<\/span>\r\n        self.logger = logging.getLogger(<span class=\"hljs-string\">\"batching\"<\/span>)\r\n        handler = logging.StreamHandler()\r\n        formatter = logging.Formatter(<span class=\"hljs-string\">'%(asctime)s - %(levelname)s - %(message)s'<\/span>)\r\n        handler.setFormatter(formatter)\r\n        self.logger.addHandler(handler)\r\n        self.logger.setLevel(logging.INFO)\r\n        \r\n        <span class=\"hljs-comment\"># Start background processing<\/span>\r\n        self.running = <span class=\"hljs-literal\">True<\/span>\r\n        self.worker_thread = threading.Thread(target=self._process_logs, daemon=<span class=\"hljs-literal\">True<\/span>)\r\n        self.worker_thread.start()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_process_logs<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-keyword\">while<\/span> self.running:\r\n            <span class=\"hljs-keyword\">try<\/span>:\r\n                <span class=\"hljs-comment\"># Try to get a log entry<\/span>\r\n                log_entry = self.log_queue.get(timeout=<span class=\"hljs-number\">1<\/span>)\r\n                self.buffer.append(log_entry)\r\n                \r\n                <span class=\"hljs-comment\"># Flush if buffer is full or enough time has passed<\/span>\r\n                current_time = time.time()\r\n                should_flush = (\r\n                    <span class=\"hljs-built_in\">len<\/span>(self.buffer) &gt;= self.batch_size <span class=\"hljs-keyword\">or<\/span>\r\n                    current_time - self.last_flush &gt;= self.flush_interval\r\n                )\r\n                \r\n                <span class=\"hljs-keyword\">if<\/span> should_flush:\r\n                    self._flush_buffer()\r\n                    \r\n            <span class=\"hljs-keyword\">except<\/span> Empty:\r\n                <span class=\"hljs-comment\"># Check if we need to flush due to time<\/span>\r\n                current_time = time.time()\r\n                <span class=\"hljs-keyword\">if<\/span> (self.buffer <span class=\"hljs-keyword\">and<\/span> \r\n                    current_time - self.last_flush &gt;= self.flush_interval):\r\n                    self._flush_buffer()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_flush_buffer<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-keyword\">not<\/span> self.buffer:\r\n            <span class=\"hljs-keyword\">return<\/span>\r\n        \r\n        <span class=\"hljs-comment\"># Process all buffered logs<\/span>\r\n        <span class=\"hljs-keyword\">for<\/span> log_entry <span class=\"hljs-keyword\">in<\/span> self.buffer:\r\n            level, message, extra = log_entry\r\n            self.logger.log(level, message, extra=extra)\r\n        \r\n        <span class=\"hljs-comment\"># Clear buffer and update flush time<\/span>\r\n        self.buffer.clear()\r\n        self.last_flush = time.time()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log<\/span>(<span class=\"hljs-params\">self, level, message, **kwargs<\/span>):\r\n        self.log_queue.put((level, message, kwargs))\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">info<\/span>(<span class=\"hljs-params\">self, message, **kwargs<\/span>):\r\n        self.log(logging.INFO, message, **kwargs)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">error<\/span>(<span class=\"hljs-params\">self, message, **kwargs<\/span>):\r\n        self.log(logging.ERROR, message, **kwargs)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">shutdown<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.running = <span class=\"hljs-literal\">False<\/span>\r\n        self._flush_buffer()  <span class=\"hljs-comment\"># Flush remaining logs<\/span>\r\n        self.worker_thread.join()\r\n<span class=\"hljs-comment\"># Performance comparison<\/span>\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">benchmark_batching<\/span>():\r\n    <span class=\"hljs-keyword\">import<\/span> time\r\n    \r\n    <span class=\"hljs-comment\"># Regular logger<\/span>\r\n    regular_logger = logging.getLogger(<span class=\"hljs-string\">\"regular\"<\/span>)\r\n    regular_logger.addHandler(logging.StreamHandler())\r\n    regular_logger.setLevel(logging.INFO)\r\n    \r\n    <span class=\"hljs-comment\"># Batching logger<\/span>\r\n    batching_logger = BatchingLogger(batch_size=<span class=\"hljs-number\">50<\/span>, flush_interval=<span class=\"hljs-number\">1.0<\/span>)\r\n    \r\n    <span class=\"hljs-comment\"># Benchmark regular logging<\/span>\r\n    start_time = time.time()\r\n    <span class=\"hljs-keyword\">for<\/span> i <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-built_in\">range<\/span>(<span class=\"hljs-number\">1000<\/span>):\r\n        regular_logger.info(<span class=\"hljs-string\">f\"Regular log message <span class=\"hljs-subst\">{i}<\/span>\"<\/span>)\r\n    regular_time = time.time() - start_time\r\n    \r\n    <span class=\"hljs-comment\"># Benchmark batching logging<\/span>\r\n    start_time = time.time()\r\n    <span class=\"hljs-keyword\">for<\/span> i <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-built_in\">range<\/span>(<span class=\"hljs-number\">1000<\/span>):\r\n        batching_logger.info(<span class=\"hljs-string\">f\"Batching log message <span class=\"hljs-subst\">{i}<\/span>\"<\/span>)\r\n    \r\n    <span class=\"hljs-comment\"># Wait for batch processing to complete<\/span>\r\n    time.sleep(<span class=\"hljs-number\">2<\/span>)\r\n    batching_time = time.time() - start_time\r\n    \r\n    <span class=\"hljs-built_in\">print<\/span>(<span class=\"hljs-string\">f\"Regular logging: <span class=\"hljs-subst\">{regular_time:<span class=\"hljs-number\">.3<\/span>f}<\/span> seconds\"<\/span>)\r\n    <span class=\"hljs-built_in\">print<\/span>(<span class=\"hljs-string\">f\"Batching logging: <span class=\"hljs-subst\">{batching_time:<span class=\"hljs-number\">.3<\/span>f}<\/span> seconds\"<\/span>)\r\n    \r\n    batching_logger.shutdown()<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"industry-news-and-trends\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Industry News and Trends<\/h2>\n<h3 id=\"3bcd\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">2024\u20132025 Logging Trends<\/h3>\n<h4 id=\"089e\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. OpenTelemetry Adoption Surge<\/h4>\n<p><strong class=\"wj gl\">Market Trend<\/strong>: <a href=\"https:\/\/opstree.com\/blog\/2025\/06\/10\/monitor-open-telemetry-collector-performance\/\">OpenTelemetry<\/a> has become the de facto standard for observability data collection.<strong class=\"wj gl\">Key Developments<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"b91d\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Native support in major cloud platforms (AWS X-Ray, Google Cloud Trace, Azure Monitor)<\/li>\n<li id=\"b6bd\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Integration with popular frameworks (Spring Boot, Django, Express.js)<\/li>\n<li id=\"0b08\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Standardized semantic conventions for logs, metrics, and traces<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Implementation Example<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"89c0\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">from<\/span> opentelemetry <span class=\"hljs-keyword\">import<\/span> trace, logs\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.exporter.otlp.proto.grpc.trace_exporter <span class=\"hljs-keyword\">import<\/span> OTLPSpanExporter\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.exporter.otlp.proto.grpc.logs_exporter <span class=\"hljs-keyword\">import<\/span> OTLPLogsExporter\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.sdk.trace <span class=\"hljs-keyword\">import<\/span> TracerProvider\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.sdk.logs <span class=\"hljs-keyword\">import<\/span> LoggerProvider\r\n<span class=\"hljs-keyword\">from<\/span> opentelemetry.instrumentation.auto_instrumentation <span class=\"hljs-keyword\">import<\/span> sitecustomize\r\n\r\n<span class=\"hljs-comment\"># Configure OpenTelemetry<\/span>\r\ntrace.set_tracer_provider(TracerProvider())\r\nlogs.set_logger_provider(LoggerProvider())\r\n<span class=\"hljs-comment\"># Export to multiple backends<\/span>\r\ntrace.get_tracer_provider().add_span_processor(\r\n    BatchSpanProcessor(OTLPSpanExporter(endpoint=<span class=\"hljs-string\">\"http:\/\/jaeger:14250\"<\/span>))\r\n)\r\nlogs.get_logger_provider().add_log_record_processor(\r\n    BatchLogRecordProcessor(OTLPLogsExporter(endpoint=<span class=\"hljs-string\">\"http:\/\/loki:9095\"<\/span>))\r\n)<\/span><\/pre>\n<h4 id=\"eac3\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. AI-Powered Log Analysis<\/h4>\n<p><strong class=\"wj gl\">Market Trend<\/strong>: Machine learning is transforming log analysis from reactive to predictive.<strong class=\"wj gl\">Key Applications<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"0194\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Anomaly Detection<\/strong>: Automatically identify unusual patterns<\/li>\n<li id=\"a11a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Root Cause Analysis<\/strong>: AI suggests probable causes of incidents<\/li>\n<li id=\"d71b\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Predictive Alerting<\/strong>: Warn of potential issues before they occur<\/li>\n<li id=\"1d64\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Log Summarization<\/strong>: Generate human-readable summaries of log events<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Example: AI Log Analyzer<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"e98e\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> pandas <span class=\"hljs-keyword\">as<\/span> pd\r\n<span class=\"hljs-keyword\">from<\/span> sklearn.ensemble <span class=\"hljs-keyword\">import<\/span> IsolationForest\r\n<span class=\"hljs-keyword\">from<\/span> sklearn.feature_extraction.text <span class=\"hljs-keyword\">import<\/span> TfidfVectorizer\r\n<span class=\"hljs-keyword\">import<\/span> numpy <span class=\"hljs-keyword\">as<\/span> np\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">AILogAnalyzer<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        self.anomaly_detector = IsolationForest(contamination=<span class=\"hljs-number\">0.1<\/span>)\r\n        self.text_vectorizer = TfidfVectorizer(max_features=<span class=\"hljs-number\">1000<\/span>)\r\n        self.trained = <span class=\"hljs-literal\">False<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">train_on_logs<\/span>(<span class=\"hljs-params\">self, log_messages<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Train anomaly detection on historical logs\"\"\"<\/span>\r\n        <span class=\"hljs-comment\"># Extract features from log messages<\/span>\r\n        text_features = self.text_vectorizer.fit_transform(log_messages)\r\n        \r\n        <span class=\"hljs-comment\"># Train anomaly detector<\/span>\r\n        self.anomaly_detector.fit(text_features.toarray())\r\n        self.trained = <span class=\"hljs-literal\">True<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">analyze_realtime_logs<\/span>(<span class=\"hljs-params\">self, new_log_messages<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Analyze incoming logs for anomalies\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-keyword\">not<\/span> self.trained:\r\n            <span class=\"hljs-keyword\">raise<\/span> ValueError(<span class=\"hljs-string\">\"Model not trained yet\"<\/span>)\r\n        \r\n        <span class=\"hljs-comment\"># Vectorize new messages<\/span>\r\n        new_features = self.text_vectorizer.transform(new_log_messages)\r\n        \r\n        <span class=\"hljs-comment\"># Predict anomalies<\/span>\r\n        anomaly_scores = self.anomaly_detector.decision_function(new_features.toarray())\r\n        is_anomaly = self.anomaly_detector.predict(new_features.toarray()) == -<span class=\"hljs-number\">1<\/span>\r\n        \r\n        results = []\r\n        <span class=\"hljs-keyword\">for<\/span> i, (message, score, is_anom) <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-built_in\">enumerate<\/span>(<span class=\"hljs-built_in\">zip<\/span>(new_log_messages, anomaly_scores, is_anomaly)):\r\n            results.append({\r\n                <span class=\"hljs-string\">'message'<\/span>: message,\r\n                <span class=\"hljs-string\">'anomaly_score'<\/span>: score,\r\n                <span class=\"hljs-string\">'is_anomaly'<\/span>: is_anom,\r\n                <span class=\"hljs-string\">'confidence'<\/span>: <span class=\"hljs-built_in\">abs<\/span>(score)\r\n            })\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> results\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">suggest_root_cause<\/span>(<span class=\"hljs-params\">self, error_logs, context_logs<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Use AI to suggest root cause of errors\"\"\"<\/span>\r\n        <span class=\"hljs-comment\"># This is a simplified example - real implementations use more sophisticated NLP<\/span>\r\n        <span class=\"hljs-keyword\">from<\/span> collections <span class=\"hljs-keyword\">import<\/span> Counter\r\n        \r\n        <span class=\"hljs-comment\"># Extract common patterns from error logs<\/span>\r\n        error_keywords = []\r\n        <span class=\"hljs-keyword\">for<\/span> log <span class=\"hljs-keyword\">in<\/span> error_logs:\r\n            words = log.lower().split()\r\n            error_keywords.extend([word <span class=\"hljs-keyword\">for<\/span> word <span class=\"hljs-keyword\">in<\/span> words <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">len<\/span>(word) &gt; <span class=\"hljs-number\">3<\/span>])\r\n        \r\n        <span class=\"hljs-comment\"># Find most common error patterns<\/span>\r\n        common_patterns = Counter(error_keywords).most_common(<span class=\"hljs-number\">5<\/span>)\r\n        \r\n        <span class=\"hljs-comment\"># Analyze context logs for correlation<\/span>\r\n        context_keywords = []\r\n        <span class=\"hljs-keyword\">for<\/span> log <span class=\"hljs-keyword\">in<\/span> context_logs:\r\n            words = log.lower().split()\r\n            context_keywords.extend([word <span class=\"hljs-keyword\">for<\/span> word <span class=\"hljs-keyword\">in<\/span> words <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">len<\/span>(word) &gt; <span class=\"hljs-number\">3<\/span>])\r\n        \r\n        context_patterns = Counter(context_keywords).most_common(<span class=\"hljs-number\">10<\/span>)\r\n        \r\n        <span class=\"hljs-comment\"># Simple correlation analysis<\/span>\r\n        correlations = []\r\n        <span class=\"hljs-keyword\">for<\/span> error_word, error_count <span class=\"hljs-keyword\">in<\/span> common_patterns:\r\n            <span class=\"hljs-keyword\">for<\/span> context_word, context_count <span class=\"hljs-keyword\">in<\/span> context_patterns:\r\n                <span class=\"hljs-keyword\">if<\/span> context_word <span class=\"hljs-keyword\">in<\/span> error_word <span class=\"hljs-keyword\">or<\/span> error_word <span class=\"hljs-keyword\">in<\/span> context_word:\r\n                    correlations.append({\r\n                        <span class=\"hljs-string\">'error_pattern'<\/span>: error_word,\r\n                        <span class=\"hljs-string\">'context_pattern'<\/span>: context_word,\r\n                        <span class=\"hljs-string\">'correlation_strength'<\/span>: <span class=\"hljs-built_in\">min<\/span>(error_count, context_count)\r\n                    })\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-built_in\">sorted<\/span>(correlations, key=<span class=\"hljs-keyword\">lambda<\/span> x: x[<span class=\"hljs-string\">'correlation_strength'<\/span>], reverse=<span class=\"hljs-literal\">True<\/span>)\r\n<span class=\"hljs-comment\"># Usage example<\/span>\r\nanalyzer = AILogAnalyzer()\r\n<span class=\"hljs-comment\"># Train on historical logs<\/span>\r\nhistorical_logs = [\r\n    <span class=\"hljs-string\">\"INFO: User login successful\"<\/span>,\r\n    <span class=\"hljs-string\">\"INFO: Database connection established\"<\/span>,\r\n    <span class=\"hljs-string\">\"WARN: High memory usage detected\"<\/span>,\r\n    <span class=\"hljs-string\">\"ERROR: Connection timeout\"<\/span>,\r\n    <span class=\"hljs-comment\"># ... more logs<\/span>\r\n]\r\nanalyzer.train_on_logs(historical_logs)\r\n<span class=\"hljs-comment\"># Analyze new logs<\/span>\r\nnew_logs = [\r\n    <span class=\"hljs-string\">\"ERROR: Unexpected null pointer exception\"<\/span>,\r\n    <span class=\"hljs-string\">\"INFO: User login successful\"<\/span>\r\n]\r\nresults = analyzer.analyze_realtime_logs(new_logs)\r\n<span class=\"hljs-keyword\">for<\/span> result <span class=\"hljs-keyword\">in<\/span> results:\r\n    <span class=\"hljs-keyword\">if<\/span> result[<span class=\"hljs-string\">'is_anomaly'<\/span>]:\r\n        <span class=\"hljs-built_in\">print<\/span>(<span class=\"hljs-string\">f\"Anomaly detected: <span class=\"hljs-subst\">{result['message']}<\/span> (confidence: <span class=\"hljs-subst\">{result['confidence']:<span class=\"hljs-number\">.2<\/span>f}<\/span>)\"<\/span>)<\/span><\/pre>\n<h4 id=\"214c\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">3. Cloud-Native Logging Evolution<\/h4>\n<p><strong class=\"wj gl\">Market Trend<\/strong>: Serverless and container-native logging solutions are becoming mainstream.<strong class=\"wj gl\">Key Developments<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"e738\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">AWS CloudWatch Insights<\/strong>: SQL-like queries for log analysis<\/li>\n<li id=\"4c1e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Google Cloud Logging<\/strong>: <a href=\"https:\/\/opstree.com\/services\/middleware-database-and-data-engineering\/\">BigQuery integration<\/a> for log analytics<\/li>\n<li id=\"adef\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Azure Monitor Logs<\/strong>: KQL (Kusto Query Language) for advanced analysis<\/li>\n<li id=\"9b69\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Kubernetes-native<\/strong>: Fluent Bit, Vector, and other CNCF projects<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Example: Serverless Logging with AWS Lambda<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"9b5f\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> json\r\n<span class=\"hljs-keyword\">import<\/span> logging\r\n<span class=\"hljs-keyword\">import<\/span> boto3\r\n<span class=\"hljs-keyword\">from<\/span> datetime <span class=\"hljs-keyword\">import<\/span> datetime\r\n<span class=\"hljs-keyword\">import<\/span> os\r\n\r\n<span class=\"hljs-comment\"># Configure structured logging for Lambda<\/span>\r\nlogger = logging.getLogger()\r\nlogger.setLevel(logging.INFO)\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">LambdaStructuredFormatter<\/span>(logging.Formatter):\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">format<\/span>(<span class=\"hljs-params\">self, record<\/span>):\r\n        log_entry = {\r\n            <span class=\"hljs-string\">\"timestamp\"<\/span>: datetime.utcnow().isoformat(),\r\n            <span class=\"hljs-string\">\"level\"<\/span>: record.levelname,\r\n            <span class=\"hljs-string\">\"message\"<\/span>: record.getMessage(),\r\n            <span class=\"hljs-string\">\"function_name\"<\/span>: os.environ.get(<span class=\"hljs-string\">'AWS_LAMBDA_FUNCTION_NAME'<\/span>),\r\n            <span class=\"hljs-string\">\"function_version\"<\/span>: os.environ.get(<span class=\"hljs-string\">'AWS_LAMBDA_FUNCTION_VERSION'<\/span>),\r\n            <span class=\"hljs-string\">\"request_id\"<\/span>: <span class=\"hljs-built_in\">getattr<\/span>(record, <span class=\"hljs-string\">'aws_request_id'<\/span>, <span class=\"hljs-literal\">None<\/span>),\r\n            <span class=\"hljs-string\">\"cold_start\"<\/span>: <span class=\"hljs-built_in\">getattr<\/span>(record, <span class=\"hljs-string\">'cold_start'<\/span>, <span class=\"hljs-literal\">False<\/span>)\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Add exception info if present<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> record.exc_info:\r\n            log_entry[<span class=\"hljs-string\">\"exception\"<\/span>] = self.formatException(record.exc_info)\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> json.dumps(log_entry)\r\n<span class=\"hljs-comment\"># Set up formatter<\/span>\r\n<span class=\"hljs-keyword\">for<\/span> handler <span class=\"hljs-keyword\">in<\/span> logger.handlers:\r\n    handler.setFormatter(LambdaStructuredFormatter())\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">lambda_handler<\/span>(<span class=\"hljs-params\">event, context<\/span>):\r\n    <span class=\"hljs-comment\"># Add request ID to all logs<\/span>\r\n    logger = logging.getLogger()\r\n    <span class=\"hljs-keyword\">for<\/span> handler <span class=\"hljs-keyword\">in<\/span> logger.handlers:\r\n        handler.addFilter(<span class=\"hljs-keyword\">lambda<\/span> record: <span class=\"hljs-built_in\">setattr<\/span>(record, <span class=\"hljs-string\">'aws_request_id'<\/span>, context.aws_request_id) <span class=\"hljs-keyword\">or<\/span> <span class=\"hljs-literal\">True<\/span>)\r\n    \r\n    logger.info(<span class=\"hljs-string\">\"Lambda function started\"<\/span>, extra={\r\n        <span class=\"hljs-string\">\"event_type\"<\/span>: event.get(<span class=\"hljs-string\">\"Records\"<\/span>, [{}])[<span class=\"hljs-number\">0<\/span>].get(<span class=\"hljs-string\">\"eventName\"<\/span>) <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-string\">\"Records\"<\/span> <span class=\"hljs-keyword\">in<\/span> event <span class=\"hljs-keyword\">else<\/span> <span class=\"hljs-string\">\"unknown\"<\/span>,\r\n        <span class=\"hljs-string\">\"event_source\"<\/span>: event.get(<span class=\"hljs-string\">\"Records\"<\/span>, [{}])[<span class=\"hljs-number\">0<\/span>].get(<span class=\"hljs-string\">\"eventSource\"<\/span>) <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-string\">\"Records\"<\/span> <span class=\"hljs-keyword\">in<\/span> event <span class=\"hljs-keyword\">else<\/span> <span class=\"hljs-string\">\"unknown\"<\/span>\r\n    })\r\n    \r\n    <span class=\"hljs-keyword\">try<\/span>:\r\n        <span class=\"hljs-comment\"># Process the event<\/span>\r\n        result = process_event(event)\r\n        \r\n        logger.info(<span class=\"hljs-string\">\"Lambda function completed successfully\"<\/span>, extra={\r\n            <span class=\"hljs-string\">\"processing_time_ms\"<\/span>: context.get_remaining_time_in_millis(),\r\n            <span class=\"hljs-string\">\"result_size\"<\/span>: <span class=\"hljs-built_in\">len<\/span>(json.dumps(result))\r\n        })\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> {\r\n            <span class=\"hljs-string\">'statusCode'<\/span>: <span class=\"hljs-number\">200<\/span>,\r\n            <span class=\"hljs-string\">'body'<\/span>: json.dumps(result)\r\n        }\r\n        \r\n    <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n        logger.error(<span class=\"hljs-string\">\"Lambda function failed\"<\/span>, extra={\r\n            <span class=\"hljs-string\">\"error_type\"<\/span>: <span class=\"hljs-built_in\">type<\/span>(e).__name__,\r\n            <span class=\"hljs-string\">\"error_message\"<\/span>: <span class=\"hljs-built_in\">str<\/span>(e),\r\n            <span class=\"hljs-string\">\"remaining_time_ms\"<\/span>: context.get_remaining_time_in_millis()\r\n        }, exc_info=<span class=\"hljs-literal\">True<\/span>)\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> {\r\n            <span class=\"hljs-string\">'statusCode'<\/span>: <span class=\"hljs-number\">500<\/span>,\r\n            <span class=\"hljs-string\">'body'<\/span>: json.dumps({<span class=\"hljs-string\">'error'<\/span>: <span class=\"hljs-string\">'Internal server error'<\/span>})\r\n        }\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">process_event<\/span>(<span class=\"hljs-params\">event<\/span>):\r\n    <span class=\"hljs-comment\"># Your business logic here<\/span>\r\n    <span class=\"hljs-keyword\">return<\/span> {<span class=\"hljs-string\">\"processed\"<\/span>: <span class=\"hljs-literal\">True<\/span>, <span class=\"hljs-string\">\"event_count\"<\/span>: <span class=\"hljs-built_in\">len<\/span>(event.get(<span class=\"hljs-string\">\"Records\"<\/span>, []))}<\/span><\/pre>\n<h4 id=\"04e1\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">4. Real-Time Log Streaming and Processing<\/h4>\n<p><strong class=\"wj gl\">Market Trend<\/strong>: Organizations need immediate insights from log data, driving adoption of stream processing.<strong class=\"wj gl\">Key Technologies<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"25ee\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Apache Kafka<\/strong>: High-throughput log streaming<\/li>\n<li id=\"e3c5\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Apache Pulsar<\/strong>: <a href=\"https:\/\/opstree.com\/services\/cloud-engineering-modernisation-migrations\/\">Cloud-native<\/a> alternative to Kafka<\/li>\n<li id=\"803e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Amazon Kinesis<\/strong>: Managed streaming service<\/li>\n<li id=\"4768\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Vector<\/strong>: High-performance log router<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Example: Real-Time Log Processing with Kafka Streams<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"89de\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">from<\/span> kafka <span class=\"hljs-keyword\">import<\/span> KafkaConsumer, KafkaProducer\r\n<span class=\"hljs-keyword\">import<\/span> json\r\n<span class=\"hljs-keyword\">import<\/span> logging\r\n<span class=\"hljs-keyword\">from<\/span> collections <span class=\"hljs-keyword\">import<\/span> defaultdict, deque\r\n<span class=\"hljs-keyword\">import<\/span> threading\r\n<span class=\"hljs-keyword\">import<\/span> time\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">RealTimeLogProcessor<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, kafka_servers=[<span class=\"hljs-string\">'localhost:9092'<\/span>]<\/span>):\r\n        self.consumer = KafkaConsumer(\r\n            <span class=\"hljs-string\">'application-logs'<\/span>,\r\n            bootstrap_servers=kafka_servers,\r\n            value_deserializer=<span class=\"hljs-keyword\">lambda<\/span> m: json.loads(m.decode(<span class=\"hljs-string\">'utf-8'<\/span>)),\r\n            group_id=<span class=\"hljs-string\">'log-processor'<\/span>\r\n        )\r\n        \r\n        self.producer = KafkaProducer(\r\n            bootstrap_servers=kafka_servers,\r\n            value_serializer=<span class=\"hljs-keyword\">lambda<\/span> v: json.dumps(v).encode(<span class=\"hljs-string\">'utf-8'<\/span>)\r\n        )\r\n        \r\n        self.logger = logging.getLogger(__name__)\r\n        \r\n        <span class=\"hljs-comment\"># Real-time metrics<\/span>\r\n        self.error_rates = defaultdict(<span class=\"hljs-keyword\">lambda<\/span>: deque(maxlen=<span class=\"hljs-number\">60<\/span>))  <span class=\"hljs-comment\"># 1 minute window<\/span>\r\n        self.response_times = defaultdict(<span class=\"hljs-keyword\">lambda<\/span>: deque(maxlen=<span class=\"hljs-number\">100<\/span>))\r\n        self.alert_thresholds = {\r\n            <span class=\"hljs-string\">'error_rate'<\/span>: <span class=\"hljs-number\">0.05<\/span>,  <span class=\"hljs-comment\"># 5% error rate<\/span>\r\n            <span class=\"hljs-string\">'avg_response_time'<\/span>: <span class=\"hljs-number\">1000<\/span>  <span class=\"hljs-comment\"># 1 second<\/span>\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Start processing<\/span>\r\n        self.running = <span class=\"hljs-literal\">True<\/span>\r\n        self.processor_thread = threading.Thread(target=self._process_logs, daemon=<span class=\"hljs-literal\">True<\/span>)\r\n        self.processor_thread.start()\r\n        \r\n        <span class=\"hljs-comment\"># Start metrics calculator<\/span>\r\n        self.metrics_thread = threading.Thread(target=self._calculate_metrics, daemon=<span class=\"hljs-literal\">True<\/span>)\r\n        self.metrics_thread.start()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_process_logs<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Process incoming log messages in real-time\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">for<\/span> message <span class=\"hljs-keyword\">in<\/span> self.consumer:\r\n            <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-keyword\">not<\/span> self.running:\r\n                <span class=\"hljs-keyword\">break<\/span>\r\n            \r\n            log_data = message.value\r\n            \r\n            <span class=\"hljs-keyword\">try<\/span>:\r\n                <span class=\"hljs-comment\"># Extract metrics from log<\/span>\r\n                service = log_data.get(<span class=\"hljs-string\">'service'<\/span>, <span class=\"hljs-string\">'unknown'<\/span>)\r\n                level = log_data.get(<span class=\"hljs-string\">'level'<\/span>, <span class=\"hljs-string\">'INFO'<\/span>)\r\n                response_time = log_data.get(<span class=\"hljs-string\">'response_time_ms'<\/span>, <span class=\"hljs-number\">0<\/span>)\r\n                \r\n                <span class=\"hljs-comment\"># Track error rates<\/span>\r\n                is_error = level <span class=\"hljs-keyword\">in<\/span> [<span class=\"hljs-string\">'ERROR'<\/span>, <span class=\"hljs-string\">'CRITICAL'<\/span>]\r\n                current_time = time.time()\r\n                self.error_rates[service].append((current_time, is_error))\r\n                \r\n                <span class=\"hljs-comment\"># Track response times<\/span>\r\n                <span class=\"hljs-keyword\">if<\/span> response_time &gt; <span class=\"hljs-number\">0<\/span>:\r\n                    self.response_times[service].append((current_time, response_time))\r\n                \r\n                <span class=\"hljs-comment\"># Detect specific patterns<\/span>\r\n                self._detect_patterns(log_data)\r\n                \r\n                <span class=\"hljs-comment\"># Check for immediate alerts<\/span>\r\n                self._check_immediate_alerts(log_data)\r\n                \r\n            <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n                self.logger.error(<span class=\"hljs-string\">f\"Error processing log: <span class=\"hljs-subst\">{e}<\/span>\"<\/span>, exc_info=<span class=\"hljs-literal\">True<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_detect_patterns<\/span>(<span class=\"hljs-params\">self, log_data<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Detect specific patterns in logs\"\"\"<\/span>\r\n        message = log_data.get(<span class=\"hljs-string\">'message'<\/span>, <span class=\"hljs-string\">''<\/span>).lower()\r\n        \r\n        <span class=\"hljs-comment\"># Database connection issues<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">any<\/span>(keyword <span class=\"hljs-keyword\">in<\/span> message <span class=\"hljs-keyword\">for<\/span> keyword <span class=\"hljs-keyword\">in<\/span> [<span class=\"hljs-string\">'connection refused'<\/span>, <span class=\"hljs-string\">'timeout'<\/span>, <span class=\"hljs-string\">'connection reset'<\/span>]):\r\n            alert = {\r\n                <span class=\"hljs-string\">'alert_type'<\/span>: <span class=\"hljs-string\">'database_connection_issue'<\/span>,\r\n                <span class=\"hljs-string\">'severity'<\/span>: <span class=\"hljs-string\">'high'<\/span>,\r\n                <span class=\"hljs-string\">'service'<\/span>: log_data.get(<span class=\"hljs-string\">'service'<\/span>),\r\n                <span class=\"hljs-string\">'message'<\/span>: log_data.get(<span class=\"hljs-string\">'message'<\/span>),\r\n                <span class=\"hljs-string\">'timestamp'<\/span>: log_data.get(<span class=\"hljs-string\">'timestamp'<\/span>),\r\n                <span class=\"hljs-string\">'pattern_matched'<\/span>: <span class=\"hljs-string\">'database_connectivity'<\/span>\r\n            }\r\n            self._send_alert(alert)\r\n        \r\n        <span class=\"hljs-comment\"># Memory issues<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">any<\/span>(keyword <span class=\"hljs-keyword\">in<\/span> message <span class=\"hljs-keyword\">for<\/span> keyword <span class=\"hljs-keyword\">in<\/span> [<span class=\"hljs-string\">'out of memory'<\/span>, <span class=\"hljs-string\">'memory leak'<\/span>, <span class=\"hljs-string\">'gc overhead'<\/span>]):\r\n            alert = {\r\n                <span class=\"hljs-string\">'alert_type'<\/span>: <span class=\"hljs-string\">'memory_issue'<\/span>,\r\n                <span class=\"hljs-string\">'severity'<\/span>: <span class=\"hljs-string\">'critical'<\/span>,\r\n                <span class=\"hljs-string\">'service'<\/span>: log_data.get(<span class=\"hljs-string\">'service'<\/span>),\r\n                <span class=\"hljs-string\">'message'<\/span>: log_data.get(<span class=\"hljs-string\">'message'<\/span>),\r\n                <span class=\"hljs-string\">'timestamp'<\/span>: log_data.get(<span class=\"hljs-string\">'timestamp'<\/span>),\r\n                <span class=\"hljs-string\">'pattern_matched'<\/span>: <span class=\"hljs-string\">'memory_exhaustion'<\/span>\r\n            }\r\n            self._send_alert(alert)\r\n        \r\n        <span class=\"hljs-comment\"># Security events<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-built_in\">any<\/span>(keyword <span class=\"hljs-keyword\">in<\/span> message <span class=\"hljs-keyword\">for<\/span> keyword <span class=\"hljs-keyword\">in<\/span> [<span class=\"hljs-string\">'unauthorized'<\/span>, <span class=\"hljs-string\">'forbidden'<\/span>, <span class=\"hljs-string\">'authentication failed'<\/span>]):\r\n            alert = {\r\n                <span class=\"hljs-string\">'alert_type'<\/span>: <span class=\"hljs-string\">'security_event'<\/span>,\r\n                <span class=\"hljs-string\">'severity'<\/span>: <span class=\"hljs-string\">'medium'<\/span>,\r\n                <span class=\"hljs-string\">'service'<\/span>: log_data.get(<span class=\"hljs-string\">'service'<\/span>),\r\n                <span class=\"hljs-string\">'user_id'<\/span>: log_data.get(<span class=\"hljs-string\">'user_id'<\/span>),\r\n                <span class=\"hljs-string\">'client_ip'<\/span>: log_data.get(<span class=\"hljs-string\">'client_ip'<\/span>),\r\n                <span class=\"hljs-string\">'message'<\/span>: log_data.get(<span class=\"hljs-string\">'message'<\/span>),\r\n                <span class=\"hljs-string\">'timestamp'<\/span>: log_data.get(<span class=\"hljs-string\">'timestamp'<\/span>),\r\n                <span class=\"hljs-string\">'pattern_matched'<\/span>: <span class=\"hljs-string\">'security_violation'<\/span>\r\n            }\r\n            self._send_alert(alert)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_calculate_metrics<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Calculate real-time metrics\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">while<\/span> self.running:\r\n            current_time = time.time()\r\n            \r\n            <span class=\"hljs-keyword\">for<\/span> service <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-built_in\">list<\/span>(self.error_rates.keys()):\r\n                <span class=\"hljs-comment\"># Calculate error rate (last 1 minute)<\/span>\r\n                recent_errors = [\r\n                    (timestamp, is_error) \r\n                    <span class=\"hljs-keyword\">for<\/span> timestamp, is_error <span class=\"hljs-keyword\">in<\/span> self.error_rates[service]\r\n                    <span class=\"hljs-keyword\">if<\/span> current_time - timestamp &lt;= <span class=\"hljs-number\">60<\/span>\r\n                ]\r\n                \r\n                <span class=\"hljs-keyword\">if<\/span> recent_errors:\r\n                    error_count = <span class=\"hljs-built_in\">sum<\/span>(is_error <span class=\"hljs-keyword\">for<\/span> _, is_error <span class=\"hljs-keyword\">in<\/span> recent_errors)\r\n                    total_count = <span class=\"hljs-built_in\">len<\/span>(recent_errors)\r\n                    error_rate = error_count \/ total_count\r\n                    \r\n                    <span class=\"hljs-keyword\">if<\/span> error_rate &gt; self.alert_thresholds[<span class=\"hljs-string\">'error_rate'<\/span>]:\r\n                        alert = {\r\n                            <span class=\"hljs-string\">'alert_type'<\/span>: <span class=\"hljs-string\">'high_error_rate'<\/span>,\r\n                            <span class=\"hljs-string\">'severity'<\/span>: <span class=\"hljs-string\">'high'<\/span>,\r\n                            <span class=\"hljs-string\">'service'<\/span>: service,\r\n                            <span class=\"hljs-string\">'error_rate'<\/span>: error_rate,\r\n                            <span class=\"hljs-string\">'threshold'<\/span>: self.alert_thresholds[<span class=\"hljs-string\">'error_rate'<\/span>],\r\n                            <span class=\"hljs-string\">'time_window'<\/span>: <span class=\"hljs-string\">'1_minute'<\/span>,\r\n                            <span class=\"hljs-string\">'timestamp'<\/span>: time.time()\r\n                        }\r\n                        self._send_alert(alert)\r\n                \r\n                <span class=\"hljs-comment\"># Calculate average response time<\/span>\r\n                recent_response_times = [\r\n                    (timestamp, response_time)\r\n                    <span class=\"hljs-keyword\">for<\/span> timestamp, response_time <span class=\"hljs-keyword\">in<\/span> self.response_times.get(service, [])\r\n                    <span class=\"hljs-keyword\">if<\/span> current_time - timestamp &lt;= <span class=\"hljs-number\">60<\/span>\r\n                ]\r\n                \r\n                <span class=\"hljs-keyword\">if<\/span> recent_response_times:\r\n                    avg_response_time = <span class=\"hljs-built_in\">sum<\/span>(rt <span class=\"hljs-keyword\">for<\/span> _, rt <span class=\"hljs-keyword\">in<\/span> recent_response_times) \/ <span class=\"hljs-built_in\">len<\/span>(recent_response_times)\r\n                    \r\n                    <span class=\"hljs-keyword\">if<\/span> avg_response_time &gt; self.alert_thresholds[<span class=\"hljs-string\">'avg_response_time'<\/span>]:\r\n                        alert = {\r\n                            <span class=\"hljs-string\">'alert_type'<\/span>: <span class=\"hljs-string\">'high_response_time'<\/span>,\r\n                            <span class=\"hljs-string\">'severity'<\/span>: <span class=\"hljs-string\">'medium'<\/span>,\r\n                            <span class=\"hljs-string\">'service'<\/span>: service,\r\n                            <span class=\"hljs-string\">'avg_response_time_ms'<\/span>: avg_response_time,\r\n                            <span class=\"hljs-string\">'threshold_ms'<\/span>: self.alert_thresholds[<span class=\"hljs-string\">'avg_response_time'<\/span>],\r\n                            <span class=\"hljs-string\">'time_window'<\/span>: <span class=\"hljs-string\">'1_minute'<\/span>,\r\n                            <span class=\"hljs-string\">'timestamp'<\/span>: time.time()\r\n                        }\r\n                        self._send_alert(alert)\r\n            \r\n            time.sleep(<span class=\"hljs-number\">10<\/span>)  <span class=\"hljs-comment\"># Check every 10 seconds<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_check_immediate_alerts<\/span>(<span class=\"hljs-params\">self, log_data<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Check for alerts that need immediate attention\"\"\"<\/span>\r\n        level = log_data.get(<span class=\"hljs-string\">'level'<\/span>)\r\n        \r\n        <span class=\"hljs-comment\"># Critical errors always generate alerts<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> level == <span class=\"hljs-string\">'CRITICAL'<\/span>:\r\n            alert = {\r\n                <span class=\"hljs-string\">'alert_type'<\/span>: <span class=\"hljs-string\">'critical_error'<\/span>,\r\n                <span class=\"hljs-string\">'severity'<\/span>: <span class=\"hljs-string\">'critical'<\/span>,\r\n                <span class=\"hljs-string\">'service'<\/span>: log_data.get(<span class=\"hljs-string\">'service'<\/span>),\r\n                <span class=\"hljs-string\">'message'<\/span>: log_data.get(<span class=\"hljs-string\">'message'<\/span>),\r\n                <span class=\"hljs-string\">'timestamp'<\/span>: log_data.get(<span class=\"hljs-string\">'timestamp'<\/span>),\r\n                <span class=\"hljs-string\">'requires_immediate_attention'<\/span>: <span class=\"hljs-literal\">True<\/span>\r\n            }\r\n            self._send_alert(alert)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_send_alert<\/span>(<span class=\"hljs-params\">self, alert<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Send alert to alert management system\"\"\"<\/span>\r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            <span class=\"hljs-comment\"># Send to alerts topic<\/span>\r\n            self.producer.send(<span class=\"hljs-string\">'alerts'<\/span>, value=alert)\r\n            \r\n            <span class=\"hljs-comment\"># Also log the alert<\/span>\r\n            self.logger.warning(<span class=\"hljs-string\">f\"Alert generated: <span class=\"hljs-subst\">{alert['alert_type']}<\/span>\"<\/span>, extra=alert)\r\n            \r\n        <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n            self.logger.error(<span class=\"hljs-string\">f\"Failed to send alert: <span class=\"hljs-subst\">{e}<\/span>\"<\/span>, exc_info=<span class=\"hljs-literal\">True<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">shutdown<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-string\">\"\"\"Gracefully shutdown the processor\"\"\"<\/span>\r\n        self.running = <span class=\"hljs-literal\">False<\/span>\r\n        self.consumer.close()\r\n        self.producer.close()\r\n<span class=\"hljs-comment\"># Usage<\/span>\r\nprocessor = RealTimeLogProcessor()\r\n<span class=\"hljs-comment\"># Let it run for processing<\/span>\r\n<span class=\"hljs-comment\"># processor.shutdown() when done<\/span><\/span><\/pre>\n<h4 id=\"1a18\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">5. Privacy-First Logging<\/h4>\n<p><strong class=\"wj gl\">Market Trend<\/strong>: Increasing focus on data privacy and compliance drives new logging approaches.<strong class=\"wj gl\">Key Concepts<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"03a8\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Data Minimization<\/strong>: Log only what\u2019s necessary<\/li>\n<li id=\"baf7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Pseudonymization<\/strong>: Replace identifiers with pseudonyms<\/li>\n<li id=\"0b58\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Differential Privacy<\/strong>: Add noise to prevent identification<\/li>\n<li id=\"0737\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Homomorphic Encryption<\/strong>: Perform computations on encrypted logs<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Example: Privacy-Preserving Logger<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"2c5e\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\">import hashlib\r\nimport hmac\r\nimport random\r\nimport os\r\nfrom typing import Dict, Any, Optional\r\n\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">PrivacyPreservingLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, privacy_level=<span class=\"hljs-string\">'standard'<\/span><\/span>):\r\n        <span class=\"hljs-variable.language\">self<\/span>.logger = logging.getLogger(<span class=\"hljs-string\">\"privacy_logger\"<\/span>)\r\n        <span class=\"hljs-variable.language\">self<\/span>.privacy_level = privacy_level\r\n        <span class=\"hljs-variable.language\">self<\/span>.salt = os.environ.get(<span class=\"hljs-string\">'PRIVACY_SALT'<\/span>, <span class=\"hljs-string\">'default_salt'<\/span>)\r\n        <span class=\"hljs-variable.language\">self<\/span>.sensitive_fields = {\r\n            <span class=\"hljs-string\">'email'<\/span>, <span class=\"hljs-string\">'phone'<\/span>, <span class=\"hljs-string\">'ssn'<\/span>, <span class=\"hljs-string\">'credit_card'<\/span>, <span class=\"hljs-string\">'user_id'<\/span>,\r\n            <span class=\"hljs-string\">'ip_address'<\/span>, <span class=\"hljs-string\">'session_id'<\/span>, <span class=\"hljs-string\">'device_id'<\/span>\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Differential privacy parameters<\/span>\r\n        <span class=\"hljs-variable.language\">self<\/span>.epsilon = <span class=\"hljs-number\">1.0<\/span>  <span class=\"hljs-comment\"># Privacy budget<\/span>\r\n        <span class=\"hljs-variable.language\">self<\/span>.sensitivity = <span class=\"hljs-number\">1.0<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log_with_privacy<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, <span class=\"hljs-symbol\">level:<\/span> str, <span class=\"hljs-symbol\">event:<\/span> str, <span class=\"hljs-symbol\">data:<\/span> Dict[str, Any]<\/span>):\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Log data with privacy protection\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        protected_data = <span class=\"hljs-variable.language\">self<\/span>._apply_privacy_protection(data)\r\n        \r\n        log_entry = {\r\n            <span class=\"hljs-string\">'timestamp'<\/span>: time.time(),\r\n            <span class=\"hljs-string\">'level'<\/span>: level,\r\n            <span class=\"hljs-string\">'event'<\/span>: event,\r\n            <span class=\"hljs-string\">'data'<\/span>: protected_data,\r\n            <span class=\"hljs-string\">'privacy_level'<\/span>: <span class=\"hljs-variable.language\">self<\/span>.privacy_level\r\n        }\r\n        \r\n        <span class=\"hljs-variable.language\">self<\/span>.logger.info(json.dumps(log_entry))\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_apply_privacy_protection<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, <span class=\"hljs-symbol\">data:<\/span> Dict[str, Any]<\/span>) -&gt; Dict[str, Any]:\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Apply privacy protection based on privacy level\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        protected = {}\r\n        \r\n        <span class=\"hljs-keyword\">for<\/span> key, value <span class=\"hljs-keyword\">in<\/span> data.items():\r\n            <span class=\"hljs-keyword\">if<\/span> key <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-variable.language\">self<\/span>.<span class=\"hljs-symbol\">sensitive_fields:<\/span>\r\n                <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-variable.language\">self<\/span>.privacy_level == <span class=\"hljs-string\">'minimal'<\/span>:\r\n                    protected[f<span class=\"hljs-string\">'{key}_hash'<\/span>] = <span class=\"hljs-variable.language\">self<\/span>._hash_value(value)\r\n                elif <span class=\"hljs-variable.language\">self<\/span>.privacy_level == <span class=\"hljs-string\">'standard'<\/span>:\r\n                    protected[f<span class=\"hljs-string\">'{key}_pseudonym'<\/span>] = <span class=\"hljs-variable.language\">self<\/span>._pseudonymize(value)\r\n                elif <span class=\"hljs-variable.language\">self<\/span>.privacy_level == <span class=\"hljs-string\">'maximum'<\/span>:\r\n                    <span class=\"hljs-comment\"># Don't log sensitive fields at all<\/span>\r\n                    continue\r\n            <span class=\"hljs-symbol\">else:<\/span>\r\n                <span class=\"hljs-comment\"># Apply differential privacy to numeric values<\/span>\r\n                <span class=\"hljs-keyword\">if<\/span> isinstance(value, (int, float)) <span class=\"hljs-keyword\">and<\/span> <span class=\"hljs-variable.language\">self<\/span>.privacy_level <span class=\"hljs-keyword\">in<\/span> [<span class=\"hljs-string\">'standard'<\/span>, <span class=\"hljs-string\">'maximum'<\/span>]:\r\n                    protected[key] = <span class=\"hljs-variable.language\">self<\/span>._add_differential_privacy_noise(value)\r\n                <span class=\"hljs-symbol\">else:<\/span>\r\n                    protected[key] = value\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> protected\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_hash_value<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, <span class=\"hljs-symbol\">value:<\/span> Any<\/span>) -&gt; <span class=\"hljs-symbol\">str:<\/span>\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Create irreversible hash of value\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        value_str = str(value)\r\n        <span class=\"hljs-keyword\">return<\/span> hashlib.sha256(f<span class=\"hljs-string\">\"{value_str}:{self.salt}\"<\/span>.encode()).hexdigest()[<span class=\"hljs-symbol\">:<\/span><span class=\"hljs-number\">12<\/span>]\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_pseudonymize<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, <span class=\"hljs-symbol\">value:<\/span> Any<\/span>) -&gt; <span class=\"hljs-symbol\">str:<\/span>\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Create consistent pseudonym that can be correlated\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        value_str = str(value)\r\n        <span class=\"hljs-keyword\">return<\/span> hmac.new(\r\n            <span class=\"hljs-variable.language\">self<\/span>.salt.encode(),\r\n            value_str.encode(),\r\n            hashlib.sha256\r\n        ).hexdigest()[<span class=\"hljs-symbol\">:<\/span><span class=\"hljs-number\">16<\/span>]\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_add_differential_privacy_noise<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, <span class=\"hljs-symbol\">value:<\/span> float<\/span>) -&gt; <span class=\"hljs-symbol\">float:<\/span>\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Add Laplace noise for differential privacy\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        <span class=\"hljs-keyword\">if<\/span> <span class=\"hljs-variable.language\">self<\/span>.privacy_level == <span class=\"hljs-string\">'maximum'<\/span>:\r\n            <span class=\"hljs-comment\"># More noise for maximum privacy<\/span>\r\n            scale = <span class=\"hljs-variable.language\">self<\/span>.sensitivity \/ (<span class=\"hljs-variable.language\">self<\/span>.epsilon \/ <span class=\"hljs-number\">2<\/span>)\r\n        <span class=\"hljs-symbol\">else:<\/span>\r\n            scale = <span class=\"hljs-variable.language\">self<\/span>.sensitivity \/ <span class=\"hljs-variable.language\">self<\/span>.epsilon\r\n        \r\n        <span class=\"hljs-comment\"># Generate Laplace noise<\/span>\r\n        u = random.random() - <span class=\"hljs-number\">0.5<\/span>\r\n        noise = -scale * (<span class=\"hljs-number\">1<\/span> <span class=\"hljs-keyword\">if<\/span> u &gt;= <span class=\"hljs-number\">0<\/span> <span class=\"hljs-keyword\">else<\/span> -<span class=\"hljs-number\">1<\/span>) * math.log(<span class=\"hljs-number\">1<\/span> - <span class=\"hljs-number\">2<\/span> * abs(u))\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> value + noise\r\n<span class=\"hljs-comment\"># Example usage<\/span>\r\nprivacy_logger = PrivacyPreservingLogger(privacy_level=<span class=\"hljs-string\">'standard'<\/span>)\r\n<span class=\"hljs-comment\"># Original sensitive data<\/span>\r\nuser_data = {\r\n    <span class=\"hljs-string\">'user_id'<\/span>: <span class=\"hljs-number\">12345<\/span>,\r\n    <span class=\"hljs-string\">'email'<\/span>: <span class=\"hljs-string\">'john.doe@example.com'<\/span>,\r\n    <span class=\"hljs-string\">'age'<\/span>: <span class=\"hljs-number\">30<\/span>,\r\n    <span class=\"hljs-string\">'purchase_amount'<\/span>: <span class=\"hljs-number\">99.99<\/span>,\r\n    <span class=\"hljs-string\">'product_category'<\/span>: <span class=\"hljs-string\">'electronics'<\/span>\r\n}\r\n<span class=\"hljs-comment\"># Log with privacy protection<\/span>\r\nprivacy_logger.log_with_privacy(<span class=\"hljs-string\">'INFO'<\/span>, <span class=\"hljs-string\">'user_purchase'<\/span>, user_data)\r\n<span class=\"hljs-comment\"># Logged output will be:<\/span>\r\n<span class=\"hljs-comment\"># {<\/span>\r\n<span class=\"hljs-comment\">#   \"timestamp\": 1691234567.89,<\/span>\r\n<span class=\"hljs-comment\">#   \"level\": \"INFO\", <\/span>\r\n<span class=\"hljs-comment\">#   \"event\": \"user_purchase\",<\/span>\r\n<span class=\"hljs-comment\">#   \"data\": {<\/span>\r\n<span class=\"hljs-comment\">#     \"user_id_pseudonym\": \"a1b2c3d4e5f6g7h8\",<\/span>\r\n<span class=\"hljs-comment\">#     \"email_pseudonym\": \"h8g7f6e5d4c3b2a1\", <\/span>\r\n<span class=\"hljs-comment\">#     \"age\": 30.234,  # With differential privacy noise<\/span>\r\n<span class=\"hljs-comment\">#     \"purchase_amount\": 99.87,  # With differential privacy noise<\/span>\r\n<span class=\"hljs-comment\">#     \"product_category\": \"electronics\"<\/span>\r\n<span class=\"hljs-comment\">#   },<\/span>\r\n<span class=\"hljs-comment\">#   \"privacy_level\": \"standard\"<\/span>\r\n<span class=\"hljs-comment\"># }<\/span><\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"o q mt fg in yd\" role=\"separator\"><\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"tools-and-technologies\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Tools and Technologies<\/h2>\n<h3 id=\"ed3a\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Popular Log Management Platforms<\/h3>\n<h4 id=\"817e\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">1. Elastic Stack (ELK\/ELK+)<\/h4>\n<p><strong class=\"wj gl\">Components<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"9bec\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Elasticsearch<\/strong>: Distributed search and analytics engine<\/li>\n<li id=\"30e1\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Logstash<\/strong>: <a href=\"https:\/\/opstree.com\/services\/middleware-database-and-data-engineering\/\">Data processing pipeline<\/a><\/li>\n<li id=\"d7b7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Kibana<\/strong>: Visualization and dashboards<\/li>\n<li id=\"6614\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Beats<\/strong>: Lightweight data shippers<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Pros<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"d552\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Mature ecosystem with extensive plugins<\/li>\n<li id=\"e155\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Powerful search capabilities<\/li>\n<li id=\"07fb\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Rich visualization options<\/li>\n<li id=\"0ae3\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Large community support<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Cons<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"66a4\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Resource intensive<\/li>\n<li id=\"0f95\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Complex operational overhead<\/li>\n<li id=\"0ddc\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Licensing changes (Elastic License vs Open Source)<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Best For<\/strong>: Organizations needing powerful search and analytics capabilities<strong class=\"wj gl\">Sample Configuration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"431a\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># docker-compose.yml for ELK Stack<\/span>\r\n<span class=\"hljs-attr\">version:<\/span> <span class=\"hljs-string\">'3.8'<\/span>\r\n<span class=\"hljs-attr\">services:<\/span>\r\n  <span class=\"hljs-attr\">elasticsearch:<\/span>\r\n    <span class=\"hljs-attr\">image:<\/span> <span class=\"hljs-string\">docker.elastic.co\/elasticsearch\/elasticsearch:8.8.0<\/span>\r\n    <span class=\"hljs-attr\">environment:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">discovery.type=single-node<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\"ES_JAVA_OPTS=-Xms2g -Xmx2g\"<\/span>\r\n    <span class=\"hljs-attr\">ports:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\"9200:9200\"<\/span>\r\n    <span class=\"hljs-attr\">volumes:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">elasticsearch-data:\/usr\/share\/elasticsearch\/data<\/span>\r\n<span class=\"hljs-attr\">logstash:<\/span>\r\n    <span class=\"hljs-attr\">image:<\/span> <span class=\"hljs-string\">docker.elastic.co\/logstash\/logstash:8.8.0<\/span>\r\n    <span class=\"hljs-attr\">volumes:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">.\/logstash\/pipeline:\/usr\/share\/logstash\/pipeline<\/span>\r\n    <span class=\"hljs-attr\">ports:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\"5044:5044\"<\/span>\r\n    <span class=\"hljs-attr\">depends_on:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">elasticsearch<\/span>\r\n  <span class=\"hljs-attr\">kibana:<\/span>\r\n    <span class=\"hljs-attr\">image:<\/span> <span class=\"hljs-string\">docker.elastic.co\/kibana\/kibana:8.8.0<\/span>\r\n    <span class=\"hljs-attr\">ports:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\"5601:5601\"<\/span>\r\n    <span class=\"hljs-attr\">environment:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">ELASTICSEARCH_HOSTS=http:\/\/elasticsearch:9200<\/span>\r\n    <span class=\"hljs-attr\">depends_on:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">elasticsearch<\/span>\r\n  <span class=\"hljs-attr\">filebeat:<\/span>\r\n    <span class=\"hljs-attr\">image:<\/span> <span class=\"hljs-string\">docker.elastic.co\/beats\/filebeat:8.8.0<\/span>\r\n    <span class=\"hljs-attr\">volumes:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">.\/filebeat.yml:\/usr\/share\/filebeat\/filebeat.yml<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\/var\/log:\/var\/log:ro<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\/var\/lib\/docker\/containers:\/var\/lib\/docker\/containers:ro<\/span>\r\n    <span class=\"hljs-attr\">depends_on:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">logstash<\/span>\r\n<span class=\"hljs-attr\">volumes:<\/span>\r\n  <span class=\"hljs-attr\">elasticsearch-data:<\/span><\/span><\/pre>\n<h4 id=\"b660\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">2. Grafana Loki + Promtail<\/h4>\n<p><strong class=\"wj gl\">Architecture<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"b58b\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Loki<\/strong>: Log aggregation system inspired by <a href=\"https:\/\/opstree.com\/blog\/2024\/10\/08\/prometheus-the-prom-king-part-2\/\">Prometheus<\/a><\/li>\n<li id=\"864f\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Promtail<\/strong>: Log shipper agent<\/li>\n<li id=\"eb9e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><strong class=\"wj gl\">Grafana<\/strong>: Visualization and querying<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Pros<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"3c6c\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Cost-effective (indexes only metadata)<\/li>\n<li id=\"a927\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Excellent integration with Prometheus metrics<\/li>\n<li id=\"08f7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">LogQL provides powerful querying<\/li>\n<li id=\"4640\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Horizontal scaling<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Cons<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"309c\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Less mature than Elasticsearch<\/li>\n<li id=\"3f88\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Limited full-text search capabilities<\/li>\n<li id=\"e25d\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Smaller ecosystem<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Best For<\/strong>: Organizations already using Prometheus\/Grafana stack<strong class=\"wj gl\">Sample Configuration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"c5e0\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># loki-config.yaml<\/span>\r\n<span class=\"hljs-attr\">auth_enabled:<\/span> <span class=\"hljs-literal\">false<\/span>\r\n\r\n<span class=\"hljs-attr\">server:<\/span>\r\n  <span class=\"hljs-attr\">http_listen_port:<\/span> <span class=\"hljs-number\">3100<\/span>\r\n  <span class=\"hljs-attr\">grpc_listen_port:<\/span> <span class=\"hljs-number\">9096<\/span>\r\n\r\n<span class=\"hljs-attr\">common:<\/span>\r\n  <span class=\"hljs-attr\">path_prefix:<\/span> <span class=\"hljs-string\">\/loki<\/span>\r\n  <span class=\"hljs-attr\">storage:<\/span>\r\n    <span class=\"hljs-attr\">filesystem:<\/span>\r\n      <span class=\"hljs-attr\">chunks_directory:<\/span> <span class=\"hljs-string\">\/loki\/chunks<\/span>\r\n      <span class=\"hljs-attr\">rules_directory:<\/span> <span class=\"hljs-string\">\/loki\/rules<\/span>\r\n  <span class=\"hljs-attr\">replication_factor:<\/span> <span class=\"hljs-number\">1<\/span>\r\n  <span class=\"hljs-attr\">ring:<\/span>\r\n    <span class=\"hljs-attr\">instance_addr:<\/span> <span class=\"hljs-number\">127.0<\/span><span class=\"hljs-number\">.0<\/span><span class=\"hljs-number\">.1<\/span>\r\n    <span class=\"hljs-attr\">kvstore:<\/span>\r\n      <span class=\"hljs-attr\">store:<\/span> <span class=\"hljs-string\">inmemory<\/span>\r\n<span class=\"hljs-attr\">schema_config:<\/span>\r\n  <span class=\"hljs-attr\">configs:<\/span>\r\n    <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">from:<\/span> <span class=\"hljs-number\">2020-10-24<\/span>\r\n      <span class=\"hljs-attr\">store:<\/span> <span class=\"hljs-string\">boltdb-shipper<\/span>\r\n      <span class=\"hljs-attr\">object_store:<\/span> <span class=\"hljs-string\">filesystem<\/span>\r\n      <span class=\"hljs-attr\">schema:<\/span> <span class=\"hljs-string\">v11<\/span>\r\n      <span class=\"hljs-attr\">index:<\/span>\r\n        <span class=\"hljs-attr\">prefix:<\/span> <span class=\"hljs-string\">index_<\/span>\r\n        <span class=\"hljs-attr\">period:<\/span> <span class=\"hljs-string\">24h<\/span>\r\n<span class=\"hljs-attr\">storage_config:<\/span>\r\n  <span class=\"hljs-attr\">boltdb_shipper:<\/span>\r\n    <span class=\"hljs-attr\">active_index_directory:<\/span> <span class=\"hljs-string\">\/loki\/boltdb-shipper-active<\/span>\r\n    <span class=\"hljs-attr\">cache_location:<\/span> <span class=\"hljs-string\">\/loki\/boltdb-shipper-cache<\/span>\r\n    <span class=\"hljs-attr\">shared_store:<\/span> <span class=\"hljs-string\">filesystem<\/span>\r\n  <span class=\"hljs-attr\">filesystem:<\/span>\r\n    <span class=\"hljs-attr\">directory:<\/span> <span class=\"hljs-string\">\/loki\/chunks<\/span>\r\n<span class=\"hljs-attr\">limits_config:<\/span>\r\n  <span class=\"hljs-attr\">reject_old_samples:<\/span> <span class=\"hljs-literal\">true<\/span>\r\n  <span class=\"hljs-attr\">reject_old_samples_max_age:<\/span> <span class=\"hljs-string\">168h<\/span>\r\n  <span class=\"hljs-attr\">ingestion_rate_mb:<\/span> <span class=\"hljs-number\">16<\/span>\r\n  <span class=\"hljs-attr\">ingestion_burst_size_mb:<\/span> <span class=\"hljs-number\">32<\/span>\r\n<span class=\"hljs-meta\">---<\/span>\r\n<span class=\"hljs-comment\"># promtail-config.yaml<\/span>\r\n<span class=\"hljs-attr\">server:<\/span>\r\n  <span class=\"hljs-attr\">http_listen_port:<\/span> <span class=\"hljs-number\">9080<\/span>\r\n  <span class=\"hljs-attr\">grpc_listen_port:<\/span> <span class=\"hljs-number\">0<\/span>\r\n<span class=\"hljs-attr\">positions:<\/span>\r\n  <span class=\"hljs-attr\">filename:<\/span> <span class=\"hljs-string\">\/tmp\/positions.yaml<\/span>\r\n<span class=\"hljs-attr\">clients:<\/span>\r\n  <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">url:<\/span> <span class=\"hljs-string\">http:\/\/loki:3100\/loki\/api\/v1\/push<\/span>\r\n<span class=\"hljs-attr\">scrape_configs:<\/span>\r\n  <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">job_name:<\/span> <span class=\"hljs-string\">containers<\/span>\r\n    <span class=\"hljs-attr\">static_configs:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">targets:<\/span>\r\n          <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">localhost<\/span>\r\n        <span class=\"hljs-attr\">labels:<\/span>\r\n          <span class=\"hljs-attr\">job:<\/span> <span class=\"hljs-string\">containerlogs<\/span>\r\n          <span class=\"hljs-attr\">__path__:<\/span> <span class=\"hljs-string\">\/var\/log\/containers\/*log<\/span>\r\n    \r\n    <span class=\"hljs-attr\">pipeline_stages:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">json:<\/span>\r\n          <span class=\"hljs-attr\">expressions:<\/span>\r\n            <span class=\"hljs-attr\">output:<\/span> <span class=\"hljs-string\">log<\/span>\r\n            <span class=\"hljs-attr\">stream:<\/span> <span class=\"hljs-string\">stream<\/span>\r\n            <span class=\"hljs-attr\">attrs:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">json:<\/span>\r\n          <span class=\"hljs-attr\">expressions:<\/span>\r\n            <span class=\"hljs-attr\">tag:<\/span>\r\n          <span class=\"hljs-attr\">source:<\/span> <span class=\"hljs-string\">attrs<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">regex:<\/span>\r\n          <span class=\"hljs-attr\">expression:<\/span> <span class=\"hljs-string\">(?P&lt;container_name&gt;(?:[^_]+_){2})(?P&lt;pod_name&gt;[^_]+)_(?P&lt;namespace&gt;[^_]+)<\/span>\r\n          <span class=\"hljs-attr\">source:<\/span> <span class=\"hljs-string\">tag<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">timestamp:<\/span>\r\n          <span class=\"hljs-attr\">format:<\/span> <span class=\"hljs-string\">RFC3339Nano<\/span>\r\n          <span class=\"hljs-attr\">source:<\/span> <span class=\"hljs-string\">time<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">labels:<\/span>\r\n          <span class=\"hljs-attr\">stream:<\/span>\r\n          <span class=\"hljs-attr\">container_name:<\/span>\r\n          <span class=\"hljs-attr\">pod_name:<\/span>\r\n          <span class=\"hljs-attr\">namespace:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-attr\">output:<\/span>\r\n          <span class=\"hljs-attr\">source:<\/span> <span class=\"hljs-string\">output<\/span><\/span><\/pre>\n<h4 id=\"47ac\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">3. Fluentd\/Fluent Bit<\/h4>\n<p><strong class=\"wj gl\">Purpose<\/strong>: Unified logging layer for data collection and routing<strong class=\"wj gl\">Fluentd<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"f367\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Ruby-based, feature-rich<\/li>\n<li id=\"047a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Large plugin ecosystem<\/li>\n<li id=\"2ad1\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Higher resource usage<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Fluent Bit<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"67a5\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">C-based, lightweight<\/li>\n<li id=\"31df\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Lower resource footprint<\/li>\n<li id=\"e16d\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Better for edge\/container environments<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Best For<\/strong>: Organizations needing flexible log routing and transformation<strong class=\"wj gl\">Sample Fluent Bit Configuration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"9de3\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-punctuation\">[<\/span>SERVICE<span class=\"hljs-punctuation\">]<\/span>\r\n    Flush         <span class=\"hljs-number\">1<\/span>\r\n    Log_Level     info\r\n    Daemon        off\r\n    Parsers_File  parsers.conf\r\n    HTTP_Server   <span class=\"hljs-keyword\">On<\/span>\r\n    HTTP_Listen   <span class=\"hljs-number\">0.0<\/span>.<span class=\"hljs-number\">0.0<\/span>\r\n    HTTP_Port     <span class=\"hljs-number\">2020<\/span>\r\n\r\n<span class=\"hljs-punctuation\">[<\/span><span class=\"hljs-keyword\">INPUT<\/span><span class=\"hljs-punctuation\">]<\/span>\r\n    Name              tail\r\n    Path              \/var\/log\/containers\/*.log\r\n    Parser            docker\r\n    Tag               kube.*\r\n    Refresh_Interval  <span class=\"hljs-number\">5<\/span>\r\n    Mem_Buf_Limit     <span class=\"hljs-number\">50<\/span>MB\r\n    Skip_Long_Lines   <span class=\"hljs-keyword\">On<\/span>\r\n\r\n<span class=\"hljs-punctuation\">[<\/span><span class=\"hljs-keyword\">INPUT<\/span><span class=\"hljs-punctuation\">]<\/span>\r\n    Name              systemd\r\n    Tag               host.*\r\n    Systemd_Filter    _SYSTEMD_UNIT<span class=\"hljs-punctuation\">=<\/span>docker.service\r\n    Read_From_Tail    <span class=\"hljs-keyword\">On<\/span>\r\n\r\n\r\n<span class=\"hljs-punctuation\">[<\/span>FILTER<span class=\"hljs-punctuation\">]<\/span>\r\n    Name                kubernetes\r\n    Match               kube.*\r\n    Kube_URL            <span class=\"hljs-symbol\">https<\/span><span class=\"hljs-punctuation\">:<\/span>\/\/kubernetes.default.<span class=\"hljs-symbol\">svc<\/span><span class=\"hljs-punctuation\">:<\/span><span class=\"hljs-number\">443<\/span>\r\n    Kube_CA_File        \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt\r\n    Kube_Token_File     \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token\r\n    Kube_Tag_Prefix     kube.var.log.containers.\r\n    Merge_Log           <span class=\"hljs-keyword\">On<\/span>\r\n    Keep_Log            Off\r\n    K8S-Logging.Parser  <span class=\"hljs-keyword\">On<\/span>\r\n    K8S-Logging.Exclude <span class=\"hljs-keyword\">On<\/span>\r\n\r\n\r\n<span class=\"hljs-punctuation\">[<\/span>FILTER<span class=\"hljs-punctuation\">]<\/span>\r\n    Name                modify\r\n    Match               *\r\n    Add                 cluster_name production\r\n    Add                 environment prod\r\n\r\n\r\n<span class=\"hljs-punctuation\">[<\/span>OUTPUT<span class=\"hljs-punctuation\">]<\/span>\r\n    Name                es\r\n    Match               kube.*\r\n    Host                elasticsearch.logging.svc.cluster.local\r\n    Port                <span class=\"hljs-number\">9200<\/span>\r\n    Index               kubernetes_cluster\r\n    <span class=\"hljs-keyword\">Type<\/span>                _doc\r\n    Retry_Limit         <span class=\"hljs-literal\">False<\/span>\r\n\r\n\r\n<span class=\"hljs-punctuation\">[<\/span>OUTPUT<span class=\"hljs-punctuation\">]<\/span>\r\n    Name                loki\r\n    Match               host.*\r\n    Host                loki.logging.svc.cluster.local\r\n    Port                <span class=\"hljs-number\">3100<\/span>\r\n    Labels              job<span class=\"hljs-punctuation\">=<\/span>systemd<\/span><\/pre>\n<h4 id=\"d10d\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">4. Vector by Timber.io<\/h4>\n<p><strong class=\"wj gl\">Purpose<\/strong>: High-performance observability data pipeline<strong class=\"wj gl\">Key Features<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"c441\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Written in Rust for performance<\/li>\n<li id=\"27db\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\"><a href=\"https:\/\/opstree.com\/blog\/2025\/01\/28\/end-to-end-data-pipeline-for-real-time-stock-market-data-%f0%9f%93%88%f0%9f%92%bc\/\">End-to-end data routing<\/a><\/li>\n<li id=\"f250\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Built-in transformations<\/li>\n<li id=\"04c4\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Vendor-neutral<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Best For<\/strong>: High-throughput environments needing performance<strong class=\"wj gl\">Sample Configuration<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"c72f\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-comment\"># vector.yaml<\/span>\r\n<span class=\"hljs-attr\">sources:<\/span>\r\n  <span class=\"hljs-attr\">file_logs:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"file\"<\/span>\r\n    <span class=\"hljs-attr\">include:<\/span>\r\n      <span class=\"hljs-bullet\">-<\/span> <span class=\"hljs-string\">\"\/var\/log\/application\/*.log\"<\/span>\r\n    <span class=\"hljs-attr\">ignore_older_secs:<\/span> <span class=\"hljs-number\">86400<\/span>\r\n\r\n<span class=\"hljs-attr\">kubernetes_logs:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"kubernetes_logs\"<\/span>\r\n    \r\n  <span class=\"hljs-attr\">syslog:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"syslog\"<\/span>\r\n    <span class=\"hljs-attr\">address:<\/span> <span class=\"hljs-string\">\"0.0.0.0:514\"<\/span>\r\n    <span class=\"hljs-attr\">mode:<\/span> <span class=\"hljs-string\">\"tcp\"<\/span>\r\n\r\n<span class=\"hljs-attr\">transforms:<\/span>\r\n  <span class=\"hljs-attr\">parse_json:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"json_parser\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"file_logs\"<\/span>]\r\n    <span class=\"hljs-attr\">drop_invalid:<\/span> <span class=\"hljs-literal\">true<\/span>\r\n\r\n\r\n<span class=\"hljs-attr\">add_metadata:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"add_fields\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"parse_json\"<\/span>]\r\n    <span class=\"hljs-attr\">fields:<\/span>\r\n      <span class=\"hljs-attr\">environment:<\/span> <span class=\"hljs-string\">\"production\"<\/span>\r\n      <span class=\"hljs-attr\">datacenter:<\/span> <span class=\"hljs-string\">\"us-east-1\"<\/span>\r\n\r\n<span class=\"hljs-attr\">sample_data:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"sample\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"add_metadata\"<\/span>]\r\n    <span class=\"hljs-attr\">rate:<\/span> <span class=\"hljs-number\">10<\/span>\r\n    <span class=\"hljs-attr\">key_field:<\/span> <span class=\"hljs-string\">\"level\"<\/span>\r\n    <span class=\"hljs-attr\">exclude:<\/span>\r\n      <span class=\"hljs-attr\">level:<\/span> <span class=\"hljs-string\">\"DEBUG\"<\/span>\r\n\r\n<span class=\"hljs-attr\">sinks:<\/span>\r\n  <span class=\"hljs-attr\">elasticsearch:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"elasticsearch\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"sample_data\"<\/span>]\r\n    <span class=\"hljs-attr\">hosts:<\/span> [<span class=\"hljs-string\">\"http:\/\/elasticsearch:9200\"<\/span>]\r\n    <span class=\"hljs-attr\">index:<\/span> <span class=\"hljs-string\">\"application-logs-%Y.%m.%d\"<\/span>\r\n    <span class=\"hljs-attr\">doc_type:<\/span> <span class=\"hljs-string\">\"_doc\"<\/span>\r\n    <span class=\"hljs-attr\">compression:<\/span> <span class=\"hljs-string\">\"gzip\"<\/span>\r\n\r\n<span class=\"hljs-attr\">s3_archive:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"aws_s3\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"add_metadata\"<\/span>]\r\n    <span class=\"hljs-attr\">bucket:<\/span> <span class=\"hljs-string\">\"log-archive-bucket\"<\/span>\r\n    <span class=\"hljs-attr\">key_prefix:<\/span> <span class=\"hljs-string\">\"date=%Y-%m-%d\/\"<\/span>\r\n    <span class=\"hljs-attr\">compression:<\/span> <span class=\"hljs-string\">\"gzip\"<\/span>\r\n    <span class=\"hljs-attr\">encoding:<\/span>\r\n      <span class=\"hljs-attr\">codec:<\/span> <span class=\"hljs-string\">\"ndjson\"<\/span>\r\n\r\n\r\n<span class=\"hljs-attr\">datadog:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"datadog_logs\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"sample_data\"<\/span>]\r\n    <span class=\"hljs-attr\">endpoint:<\/span> <span class=\"hljs-string\">\"https:\/\/http-intake.logs.datadoghq.com\"<\/span>\r\n    <span class=\"hljs-attr\">default_api_key:<\/span> <span class=\"hljs-string\">\"${DD_API_KEY}\"<\/span>\r\n\r\n\r\n<span class=\"hljs-attr\">prometheus_metrics:<\/span>\r\n    <span class=\"hljs-attr\">type:<\/span> <span class=\"hljs-string\">\"prometheus_exporter\"<\/span>\r\n    <span class=\"hljs-attr\">inputs:<\/span> [<span class=\"hljs-string\">\"add_metadata\"<\/span>]\r\n    <span class=\"hljs-attr\">address:<\/span> <span class=\"hljs-string\">\"0.0.0.0:9598\"<\/span>\r\n    <span class=\"hljs-attr\">namespace:<\/span> <span class=\"hljs-string\">\"vector\"<\/span><\/span><\/pre>\n<h4 id=\"e575\" class=\"zf xh sd as xi zg zh zi kj zj zk zl ko ws zm zn zo ww zp zq zr xa zs zt zu zv bx\">5. Cloud-Native Solutions<\/h4>\n<p><strong class=\"wj gl\"><a href=\"https:\/\/opstree.com\/blog\/2025\/07\/01\/logs-to-alerts-with-cloudwatch-filters\/\">AWS CloudWatch<\/a> Logs<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"4ee2\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">import<\/span> boto3\r\n<span class=\"hljs-keyword\">import<\/span> json\r\n<span class=\"hljs-keyword\">from<\/span> datetime <span class=\"hljs-keyword\">import<\/span> datetime\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">CloudWatchLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, log_group, log_stream<\/span>):\r\n        self.client = boto3.client(<span class=\"hljs-string\">'logs'<\/span>)\r\n        self.log_group = log_group\r\n        self.log_stream = log_stream\r\n        self.sequence_token = <span class=\"hljs-literal\">None<\/span>\r\n        \r\n        <span class=\"hljs-comment\"># Create log group and stream if they don't exist<\/span>\r\n        self._ensure_log_infrastructure()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_ensure_log_infrastructure<\/span>(<span class=\"hljs-params\">self<\/span>):\r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            self.client.create_log_group(logGroupName=self.log_group)\r\n        <span class=\"hljs-keyword\">except<\/span> self.client.exceptions.ResourceAlreadyExistsException:\r\n            <span class=\"hljs-keyword\">pass<\/span>\r\n        \r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            self.client.create_log_stream(\r\n                logGroupName=self.log_group,\r\n                logStreamName=self.log_stream\r\n            )\r\n        <span class=\"hljs-keyword\">except<\/span> self.client.exceptions.ResourceAlreadyExistsException:\r\n            <span class=\"hljs-keyword\">pass<\/span>\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log<\/span>(<span class=\"hljs-params\">self, level, message, **context<\/span>):\r\n        log_entry = {\r\n            <span class=\"hljs-string\">'timestamp'<\/span>: datetime.utcnow().isoformat(),\r\n            <span class=\"hljs-string\">'level'<\/span>: level,\r\n            <span class=\"hljs-string\">'message'<\/span>: message,\r\n            **context\r\n        }\r\n        \r\n        event = {\r\n            <span class=\"hljs-string\">'timestamp'<\/span>: <span class=\"hljs-built_in\">int<\/span>(datetime.utcnow().timestamp() * <span class=\"hljs-number\">1000<\/span>),\r\n            <span class=\"hljs-string\">'message'<\/span>: json.dumps(log_entry)\r\n        }\r\n        \r\n        kwargs = {\r\n            <span class=\"hljs-string\">'logGroupName'<\/span>: self.log_group,\r\n            <span class=\"hljs-string\">'logStreamName'<\/span>: self.log_stream,\r\n            <span class=\"hljs-string\">'logEvents'<\/span>: [event]\r\n        }\r\n        \r\n        <span class=\"hljs-keyword\">if<\/span> self.sequence_token:\r\n            kwargs[<span class=\"hljs-string\">'sequenceToken'<\/span>] = self.sequence_token\r\n        \r\n        <span class=\"hljs-keyword\">try<\/span>:\r\n            response = self.client.put_log_events(**kwargs)\r\n            self.sequence_token = response.get(<span class=\"hljs-string\">'nextSequenceToken'<\/span>)\r\n        <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n            <span class=\"hljs-built_in\">print<\/span>(<span class=\"hljs-string\">f\"Failed to send log to CloudWatch: <span class=\"hljs-subst\">{e}<\/span>\"<\/span>)<\/span><\/pre>\n<pre class=\"zw yt yu zx zy bh zz bx\"><span id=\"d040\" class=\"zf xh sd yu b fn aba abb y abc zb\" data-selectable-paragraph=\"\"># Usage\r\nlogger = CloudWatchLogger('my-application', 'production-instance-1')\r\nlogger.log('INFO', 'User logged in', user_id=12345, session_id='abc123')<\/span><\/pre>\n<p><strong class=\"wj gl\">Google Cloud Logging<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"aef5\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\"><span class=\"hljs-keyword\">from<\/span> google.cloud <span class=\"hljs-keyword\">import<\/span> logging <span class=\"hljs-keyword\">as<\/span> gcp_logging\r\n<span class=\"hljs-keyword\">import<\/span> json\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">GCPStructuredLogger<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\">self, project_id, log_name<\/span>):\r\n        self.client = gcp_logging.Client(project=project_id)\r\n        self.logger = self.client.logger(log_name)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">log<\/span>(<span class=\"hljs-params\">self, severity, message, **structured_data<\/span>):\r\n        <span class=\"hljs-comment\"># GCP expects severity in uppercase<\/span>\r\n        severity = severity.upper()\r\n        \r\n        <span class=\"hljs-comment\"># Structure the log entry<\/span>\r\n        entry = {\r\n            <span class=\"hljs-string\">'message'<\/span>: message,\r\n            <span class=\"hljs-string\">'structured_data'<\/span>: structured_data,\r\n            <span class=\"hljs-string\">'service'<\/span>: structured_data.get(<span class=\"hljs-string\">'service'<\/span>, <span class=\"hljs-string\">'unknown'<\/span>),\r\n            <span class=\"hljs-string\">'version'<\/span>: structured_data.get(<span class=\"hljs-string\">'version'<\/span>, <span class=\"hljs-string\">'1.0.0'<\/span>)\r\n        }\r\n        \r\n        <span class=\"hljs-comment\"># Send to Cloud Logging<\/span>\r\n        self.logger.log_struct(\r\n            entry,\r\n            severity=severity,\r\n            labels=structured_data.get(<span class=\"hljs-string\">'labels'<\/span>, {})\r\n        )\r\n\r\n<span class=\"hljs-comment\"># Usage with Cloud Functions<\/span>\r\n<span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">cloud_function_handler<\/span>(<span class=\"hljs-params\">request<\/span>):\r\n    logger = GCPStructuredLogger(<span class=\"hljs-string\">'my-project-id'<\/span>, <span class=\"hljs-string\">'cloud-function-logs'<\/span>)\r\n    \r\n    logger.log(<span class=\"hljs-string\">'INFO'<\/span>, <span class=\"hljs-string\">'Function started'<\/span>, \r\n               function_name=<span class=\"hljs-string\">'process-data'<\/span>,\r\n               request_id=request.headers.get(<span class=\"hljs-string\">'X-Request-ID'<\/span>),\r\n               user_agent=request.headers.get(<span class=\"hljs-string\">'User-Agent'<\/span>))\r\n    \r\n    <span class=\"hljs-keyword\">try<\/span>:\r\n        <span class=\"hljs-comment\"># Function logic here<\/span>\r\n        result = process_data(request.get_json())\r\n        \r\n        logger.log(<span class=\"hljs-string\">'INFO'<\/span>, <span class=\"hljs-string\">'Function completed successfully'<\/span>,\r\n                   function_name=<span class=\"hljs-string\">'process-data'<\/span>,\r\n                   processing_time_ms=<span class=\"hljs-number\">100<\/span>,\r\n                   records_processed=<span class=\"hljs-built_in\">len<\/span>(result))\r\n        \r\n        <span class=\"hljs-keyword\">return<\/span> {<span class=\"hljs-string\">'status'<\/span>: <span class=\"hljs-string\">'success'<\/span>, <span class=\"hljs-string\">'data'<\/span>: result}\r\n        \r\n    <span class=\"hljs-keyword\">except<\/span> Exception <span class=\"hljs-keyword\">as<\/span> e:\r\n        logger.log(<span class=\"hljs-string\">'ERROR'<\/span>, <span class=\"hljs-string\">'Function failed'<\/span>,\r\n                   function_name=<span class=\"hljs-string\">'process-data'<\/span>,\r\n                   error_type=<span class=\"hljs-built_in\">type<\/span>(e).__name__,\r\n                   error_message=<span class=\"hljs-built_in\">str<\/span>(e))\r\n        <span class=\"hljs-keyword\">return<\/span> {<span class=\"hljs-string\">'status'<\/span>: <span class=\"hljs-string\">'error'<\/span>, <span class=\"hljs-string\">'message'<\/span>: <span class=\"hljs-string\">'Processing failed'<\/span>}, <span class=\"hljs-number\">500<\/span><\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"iw qe rz sa sb\">\n<div class=\"o q\">\n<div class=\"de n df dg dh di\">\n<h2 id=\"troubleshooting-common-issues\" class=\"xg xh sd as xi kf yi kg kj kk yj kl ko kp yk kq kt ku yl kv ky kz ym la ld xo bx\">Troubleshooting Common Issues<\/h2>\n<h3 id=\"ccec\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Problem 1: Log Volume Overwhelming Systems<\/h3>\n<p><strong class=\"wj gl\">Symptoms<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"d43a\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Slow log ingestion<\/li>\n<li id=\"9d53\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Disk space running out<\/li>\n<li id=\"447a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Search queries timing out<\/li>\n<li id=\"5793\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Application performance degradation<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Root Causes<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"19bc\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Excessive DEBUG logging in production<\/li>\n<li id=\"6e40\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Chatty applications logging every minor event<\/li>\n<li id=\"5506\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">No log rotation or retention policies<\/li>\n<li id=\"2800\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Inadequate infrastructure sizing<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Solutions<\/strong>:<strong class=\"wj gl\">1. Implement Dynamic Log Level Control<\/strong>:<\/p>\n<pre class=\"yo yp yq yr ys yt yu yv ng yw bq bx\"><span id=\"dcef\" class=\"yx xh sd yu b bu yy yz y za zb\" data-selectable-paragraph=\"\">import logging\r\nimport os\r\nimport signal\r\nimport threading\r\n    \r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">DynamicLogLevelController<\/span>:\r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">__init__<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span><\/span>):\r\n        <span class=\"hljs-variable.language\">self<\/span>.logger = logging.getLogger()\r\n        <span class=\"hljs-variable.language\">self<\/span>.current_level = logging.<span class=\"hljs-variable.constant\">INFO<\/span>\r\n        <span class=\"hljs-variable.language\">self<\/span>.emergency_mode = False\r\n        \r\n        <span class=\"hljs-comment\"># Setup signal handlers for dynamic control<\/span>\r\n        signal.signal(signal.<span class=\"hljs-variable.constant\">SIGUSR1<\/span>, <span class=\"hljs-variable.language\">self<\/span>._increase_log_level)\r\n        signal.signal(signal.<span class=\"hljs-variable.constant\">SIGUSR2<\/span>, <span class=\"hljs-variable.language\">self<\/span>._decrease_log_level)\r\n        \r\n        <span class=\"hljs-comment\"># Monitor system resources<\/span>\r\n        <span class=\"hljs-variable.language\">self<\/span>.monitor_thread = threading.Thread(target=<span class=\"hljs-variable.language\">self<\/span>._monitor_resources, daemon=True)\r\n        <span class=\"hljs-variable.language\">self<\/span>.monitor_thread.start()\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_increase_log_level<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, signum, frame<\/span>):\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Reduce log verbosity (increase log level)\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        levels = [logging.<span class=\"hljs-variable.constant\">DEBUG<\/span>, logging.<span class=\"hljs-variable.constant\">INFO<\/span>, logging.<span class=\"hljs-variable.constant\">WARNING<\/span>, logging.<span class=\"hljs-variable.constant\">ERROR<\/span>, logging.<span class=\"hljs-variable.constant\">CRITICAL<\/span>]\r\n        current_index = levels.index(<span class=\"hljs-variable.language\">self<\/span>.current_level)\r\n        \r\n        <span class=\"hljs-keyword\">if<\/span> current_index &lt; len(levels) - <span class=\"hljs-number\">1<\/span>:\r\n            <span class=\"hljs-variable.language\">self<\/span>.current_level = levels[current_index + <span class=\"hljs-number\">1<\/span>]\r\n            <span class=\"hljs-variable.language\">self<\/span>.logger.setLevel(<span class=\"hljs-variable.language\">self<\/span>.current_level)\r\n            <span class=\"hljs-variable.language\">self<\/span>.logger.warning(f<span class=\"hljs-string\">\"Log level increased to {logging.getLevelName(self.current_level)}\"<\/span>)\r\n    \r\n    <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">_decrease_log_level<\/span>(<span class=\"hljs-params\"><span class=\"hljs-variable.language\">self<\/span>, signum, frame<\/span>):\r\n        <span class=\"hljs-string\">\"\"<\/span><span class=\"hljs-string\">\"Increase log verbosity (decrease log level)\"<\/span><span class=\"hljs-string\">\"\"<\/span>\r\n        levels = [logging.<span class=\"hljs-variable.constant\">DEBUG<\/span>, logging.<span class=\"hljs-variable.constant\">INFO<\/span>, logging.<span class=\"hljs-variable.constant\">WARNING<\/span>, logging.<span class=\"hljs-variable.constant\">ERROR<\/span>, logging.<span class=\"hljs-variable.constant\">CRITICAL<\/span>] Intelligent Log Sampling**:\r\n\r\n<span class=\"hljs-comment\"># Sample logs during high traffic to prevent overwhelming systems<\/span>\r\n<span class=\"hljs-keyword\">class<\/span> <span class=\"hljs-title.class\">SamplingLogger<\/span>\r\n  <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">initialize<\/span>(<span class=\"hljs-params\"><span class=\"hljs-symbol\">sample_rate:<\/span> <span class=\"hljs-number\">0.1<\/span><\/span>)\r\n    <span class=\"hljs-variable\">@sample_rate<\/span> = sample_rate\r\n    <span class=\"hljs-variable\">@base_logger<\/span> = Rails.logger\r\n  <span class=\"hljs-keyword\">end<\/span>\r\n  \r\n  <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">info<\/span>(<span class=\"hljs-params\">message, context = {}<\/span>)\r\n    <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-keyword\">unless<\/span> should_log?\r\n    \r\n    <span class=\"hljs-variable\">@base_logger<\/span>.info(message, context.merge({\r\n      <span class=\"hljs-symbol\">sampled:<\/span> <span class=\"hljs-literal\">true<\/span>,\r\n      <span class=\"hljs-symbol\">sample_rate:<\/span> <span class=\"hljs-variable\">@sample_rate<\/span>\r\n    }))\r\n  <span class=\"hljs-keyword\">end<\/span>\r\n  \r\n  private\r\n  \r\n  <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">should_log?<\/span>\r\n    rand &lt; <span class=\"hljs-variable\">@sample_rate<\/span> || high_priority_event?\r\n  <span class=\"hljs-keyword\">end<\/span>\r\n  \r\n  <span class=\"hljs-keyword\">def<\/span> <span class=\"hljs-title.function\">high_priority_event?<\/span>\r\n    <span class=\"hljs-comment\"># Always log errors, security events, etc.<\/span>\r\n    Thread.current[<span class=\"hljs-symbol\">:log_priority<\/span>] == <span class=\"hljs-symbol\">:high<\/span>\r\n  <span class=\"hljs-keyword\">end<\/span>\r\n<span class=\"hljs-keyword\">end<\/span><\/span><\/pre>\n<p><strong class=\"wj gl\">Results<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"ca0d\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Successfully handled 10x traffic increase<\/li>\n<li id=\"5f80\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Maintained sub-second log ingestion times<\/li>\n<li id=\"cb92\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Zero logging-related performance degradation<\/li>\n<li id=\"c23e\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Complete audit trail for post-event analysis<\/li>\n<\/ul>\n<p><strong class=\"wj gl\">Key Takeaways<\/strong>:<\/p>\n<ul class=\"\">\n<li id=\"334a\" class=\"wh wi sd wj b wk wl wm wn wo wp wq wr ws wt wu wv ww wx wy wz xa xb xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Plan for traffic spikes in logging infrastructure<\/li>\n<li id=\"9b13\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Implement dynamic log sampling<\/li>\n<li id=\"75cf\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Always maintain high-priority event logging<\/li>\n<li id=\"5850\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Test logging <a href=\"https:\/\/opstree.com\/blog\/2023\/12\/05\/ecs-capacity-provider-strategy\/\">infrastructure<\/a> under load<\/li>\n<\/ul>\n<h3 id=\"1f28\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">How to Start Your Own Logging Journey<\/h3>\n<h4 id=\"e5bd\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Step 1: Define Your Logging Goals<\/h4>\n<ul class=\"\">\n<li id=\"a58c\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">What problems do you want to solve? Debugging, security, compliance, performance?<\/li>\n<li id=\"b2dc\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">What systems and applications need logging?<\/li>\n<li id=\"50a2\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">What stakeholders will use the logs and for what purposes.<\/li>\n<\/ul>\n<h4 id=\"5c85\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Step 2: Choose the Right Tools and Frameworks<\/h4>\n<ul class=\"\">\n<li id=\"b797\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">For basic logging: Use language-native frameworks like Logback, Python logging, or Winston.<\/li>\n<li id=\"fab5\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">For log aggregation: Consider ELK Stack or Grafana Loki + Promtail.<\/li>\n<li id=\"2233\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">For observability: Adopt OpenTelemetry for unified telemetry data.<\/li>\n<\/ul>\n<h4 id=\"1f41\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Step 3: Establish Logging Best Practices<\/h4>\n<ul class=\"\">\n<li id=\"c0ea\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Include timestamps, log levels, and contextual metadata.<\/li>\n<li id=\"27e7\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Prefer structured logging for better automation.<\/li>\n<li id=\"b364\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Avoid logging sensitive data or use pseudonymization.<\/li>\n<li id=\"1248\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Implement log rotation and retention policies.<\/li>\n<\/ul>\n<h4 id=\"e648\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Step 4: Build Observability and Alerting<\/h4>\n<ul class=\"\">\n<li id=\"4f76\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Correlate logs with metrics and traces for full context.<\/li>\n<li id=\"709a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Set up dashboards and real-time alerts for anomalies.<\/li>\n<li id=\"963b\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Use AI-powered analytics for early detection of issues.<\/li>\n<\/ul>\n<h4 id=\"7df3\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Step 5: Continuously Improve<\/h4>\n<ul class=\"\">\n<li id=\"4efd\" class=\"wh wi sd wj b wk xp wm wn wo xq wq wr ws xr wu wv ww xs wy wz xa xt xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Periodically review log verbosity and quality.<\/li>\n<li id=\"694a\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Automate log analysis and pattern detection.<\/li>\n<li id=\"814c\" class=\"wh wi sd wj b wk xy wm wn wo xz wq wr ws ya wu wv ww yb wy wz xa yc xc xd xe yn xv xw bx\" data-selectable-paragraph=\"\">Stay updated on industry best practices and tools.<\/li>\n<\/ul>\n<h2 id=\"final-thoughts\" class=\"xg xh sd as xi kf xj kg kj kk xk kl ko kp xl kq kt ku xm kv ky kz xn la ld xo bx\">Final Thoughts<\/h2>\n<p>Logging is the backbone of modern software operations and <a href=\"https:\/\/opstree.com\/blog\/2022\/02\/11\/observability-for-monitoring-microservices-top-5-ways\/\">observability<\/a>. With increasing system complexity, high-volume distributed logging, and evolving compliance requirements, mastering logging techniques and tools is crucial for any organization.\u201cLogs Unclog\u201d aims to empower you with the knowledge and practical guidance to build efficient, secure, and insightful logging systems that provide real value.Happy logging and observability journey!If you want, I can help you draft detailed articles or technical tutorials for these new sections or any part of the blog. Let me know how you\u2019d like to proceed!<\/p>\n<p><strong>Related Searches &#8211; <a href=\"https:\/\/opstree.com\/services\/observability-sre-production-engineering\/\">Comprehensive Observability Setup<\/a><\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/section>\n<\/div>\n<\/div>\n<\/article>\n<\/div>\n<\/div>\n<\/div>\n<\/section>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Introduction to Logging What Are Logs? Logs are chronological records of events that occur within software applications, operating systems, and network devices. They serve as the digital equivalent of a ship\u2019s logbook, documenting what happened, when it happened, and often providing context about why it happened. Why Logging Matters In today\u2019s distributed systems and microservices &hellip; <a href=\"https:\/\/opstree.com\/blog\/2025\/08\/19\/logs-to-unclog-the-complete-guide-to-logging\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Logs to Unclog: The Complete Guide to Logging&#8221;<\/span><\/a><\/p>\n","protected":false},"author":244582682,"featured_media":29520,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474],"tags":[69816,768739276,62087,17060625,16279507,343865],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/08\/Logs-to-UnclogThe-Complete-Guide-to-Logging-.jpg","jetpack_likes_enabled":false,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-7Fy","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/29484"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/244582682"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=29484"}],"version-history":[{"count":9,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/29484\/revisions"}],"predecessor-version":[{"id":29972,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/29484\/revisions\/29972"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29520"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=29484"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=29484"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=29484"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}