{"id":19830,"date":"2024-12-31T16:51:08","date_gmt":"2024-12-31T11:21:08","guid":{"rendered":"https:\/\/opstree.com\/blog\/?p=19830"},"modified":"2025-07-15T21:16:57","modified_gmt":"2025-07-15T15:46:57","slug":"stream-and-analyze-postgresql-data-from-s3-using-kafka-and-ksqldb-part-2","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2024\/12\/31\/stream-and-analyze-postgresql-data-from-s3-using-kafka-and-ksqldb-part-2\/","title":{"rendered":"Stream and Analyze PostgreSQL Data from S3 Using Kafka and ksqlDB: Part 2"},"content":{"rendered":"<h2 class=\"ds-markdown-paragraph\"><strong>Introduction<\/strong><\/h2>\n<p class=\"ds-markdown-paragraph\">In\u00a0<strong><a href=\"https:\/\/opstree.com\/blog\/2024\/12\/17\/stream-postgresql-data-to-s3-via-kafka-using-jdbc-and-s3-sink-connectors-part-1\/\" target=\"_blank\" rel=\"noopener noreferrer\">Part 1<\/a><\/strong>, we set up a\u00a0<strong>real-time data pipeline<\/strong>\u00a0that streams PostgreSQL changes to\u00a0<strong>Amazon S3 using Kafka Connect<\/strong>. Here\u2019s what we accomplished:<\/p>\n<ul>\n<li class=\"ds-markdown-paragraph\"><strong>Configured PostgreSQL for CDC<\/strong> (using logical decoding\/WAL)<\/li>\n<li class=\"ds-markdown-paragraph\"><strong>Deployed Kafka Connect with JDBC Source Connector<\/strong> (to capture PostgreSQL changes)<\/li>\n<li class=\"ds-markdown-paragraph\"><strong>Set up an S3 Sink Connector<\/strong>\u00a0(to persist data in S3 in Avro\/Parquet format)<\/li>\n<\/ul>\n<p>In Part 2 of our journey, we dive deeper into the process of streaming data from <strong>PostgreSQL to S3 via Kafka.<\/strong> This time, we explore how to set up connectors, create a sample PostgreSQL table with large datasets, and leverage ksqlDB for real-time data analysis. Additionally, we\u2019ll cover the steps to configure AWS IAM policies for secure S3 access. Whether you&#8217;re building a data pipeline or experimenting with Kafka integrations, this guide will help you navigate the essentials with ease.<\/p>\n<p><!--more--><\/p>\n<h2><strong>Visual Data Flow Diagram<\/strong><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-29396\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57.png\" alt=\"\" width=\"3464\" height=\"910\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57.png 3464w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57-300x79.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57-1024x269.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57-768x202.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57-1536x404.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57-2048x538.png 2048w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/deepseek_mermaid_20250715_04cf57-1200x315.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<div class=\"md-code-block md-code-block-dark\">\n<div class=\"md-code-block-banner-wrap\">\n<div class=\"md-code-block-banner md-code-block-banner-lite\">\n<div class=\"_121d384\">\n<div class=\"d2a24f03\">\n<div class=\"ds-segmented _81e8954\" role=\"tablist\">\n<h4><strong>Role of Each Component<\/strong><\/h4>\n<div class=\"markdown-table-wrapper\">\n<table>\n<thead>\n<tr>\n<th>Component<\/th>\n<th>Responsibility<\/th>\n<th>Key Tools Used<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>PostgreSQL<\/strong><\/td>\n<td>Source database emitting Change Data Capture (CDC) events via WAL<\/td>\n<td><code>wal2json<\/code>,\u00a0<code>DECODING<\/code><\/td>\n<\/tr>\n<tr>\n<td><strong>Kafka (Brokers)<\/strong><\/td>\n<td>Ingests and buffers real-time data changes from PostgreSQL<\/td>\n<td><code>JDBC Source Connector<\/code>,\u00a0<code>Topics<\/code><\/td>\n<\/tr>\n<tr>\n<td><strong>Kafka Connect<\/strong><\/td>\n<td>Bridges PostgreSQL \u2192 Kafka \u2192 S3 with fault-tolerant pipelines<\/td>\n<td><code>S3 Sink Connector<\/code>,\u00a0<code>Avro\/Parquet<\/code><\/td>\n<\/tr>\n<tr>\n<td><strong>Amazon S3<\/strong><\/td>\n<td>Stores raw CDC logs for historical analysis (data lake)<\/td>\n<td><code>Parquet format<\/code>,\u00a0<code>Partitioning<\/code><\/td>\n<\/tr>\n<tr>\n<td><strong>ksqlDB<\/strong><\/td>\n<td>Processes streams in real-time (filtering, joins, aggregations)<\/td>\n<td><code>CREATE STREAM<\/code>,\u00a0<code>PUSH QUERIES<\/code><\/td>\n<\/tr>\n<tr>\n<td><strong>Downstream Apps<\/strong><\/td>\n<td>Consume processed data (dashboards, alerts, APIs)<\/td>\n<td><code>Kafka Consumers<\/code>,\u00a0<code>REST Endpoints<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<h2><strong>Step-by-Step Implementation<\/strong><\/h2>\n<p>Let\u2019s pick up where we left off in Part 1 and continue building our <a href=\"https:\/\/opstree.com\/blog\/2025\/01\/28\/end-to-end-data-pipeline-for-real-time-stock-market-data-%f0%9f%93%88%f0%9f%92%bc\/\"><strong>data pipeline<\/strong><\/a> by setting up connectors, creating sample data, and configuring S3 for seamless integration.<\/p>\n<h3>Step 1<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19832\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-153534-300x228.png\" alt=\"\" width=\"597\" height=\"454\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-153534-300x228.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-153534.png 664w\" sizes=\"(max-width: 597px) 85vw, 597px\" \/><\/p>\n<p>Enter<\/p>\n<p>Output<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19833\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-153657-300x124.png\" alt=\"\" width=\"990\" height=\"409\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-153657-300x124.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-153657.png 755w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<h3>Step 2<\/h3>\n<p>docker exec &#8211;tty &#8211;interactive postgres bash -c &#8216;psql -U $POSTGRES_USER $POSTGRES_DB&#8217;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19646\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-165711-300x114.png\" alt=\"\" width=\"850\" height=\"323\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-165711-300x114.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-165711.png 679w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19835\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154245-300x174.png\" alt=\"\" width=\"607\" height=\"352\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154245-300x174.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154245-768x444.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154245.png 814w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h3>Step 3<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19648\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-172709-300x109.png\" alt=\"\" width=\"669\" height=\"243\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-172709-300x109.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-172709-1024x372.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-172709-768x279.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-172709-1200x435.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-172709.png 1502w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19836\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154444-256x300.png\" alt=\"\" width=\"478\" height=\"560\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154444-256x300.png 256w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154444.png 578w\" sizes=\"(max-width: 478px) 85vw, 478px\" \/><\/p>\n<p>Enter<\/p>\n<p>Output<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19837\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-154936.png\" alt=\"\" width=\"272\" height=\"122\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19838\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-155043-300x275.png\" alt=\"\" width=\"624\" height=\"572\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-155043-300x275.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-31-155043.png 604w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h3>Step 4<\/h3>\n<p>docker exec -it ksqldb ksql http:\/\/ksqldb:8088<br \/>\nshow connectors;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19649\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219-300x110.png\" alt=\"\" width=\"736\" height=\"270\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219-300x110.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219-1024x377.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219-768x283.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219-1536x565.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219-1200x442.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173219.png 1864w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h3>Step 5<\/h3>\n<p>kafkacat -b localhost:9092 -t postgres-employees -C -o beginning -f &#8216;%o %s\\n&#8217;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19650\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424-300x109.png\" alt=\"\" width=\"611\" height=\"222\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424-300x109.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424-1024x370.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424-768x278.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424-1536x556.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424-1200x434.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173424.png 1869w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h3>Step-6<\/h3>\n<p>on aws<br \/>\ncreate a user through iam console<br \/>\nand add a policy into the same<br \/>\nby making a new policy<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19651\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758-300x108.png\" alt=\"\" width=\"703\" height=\"253\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758-300x108.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758-1024x369.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758-768x277.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758-1536x554.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758-1200x433.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173758.png 1871w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19652\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910-300x137.png\" alt=\"\" width=\"622\" height=\"284\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910-300x137.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910-1024x468.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910-768x351.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910-1536x702.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910-1200x549.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-173910.png 1704w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p>here you can manage the policy by specifying the s3 service required into your policy and bucket to which you want to apply this policy<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19653\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001-300x138.png\" alt=\"\" width=\"633\" height=\"291\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001-300x138.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001-1024x472.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001-768x354.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001-1536x708.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001-1200x553.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174001.png 1575w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h3>Final Step<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19654\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056-300x136.png\" alt=\"\" width=\"721\" height=\"327\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056-300x136.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056-1024x463.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056-768x348.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056-1536x695.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056-1200x543.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174056.png 1750w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19655\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140-300x136.png\" alt=\"\" width=\"662\" height=\"300\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140-300x136.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140-1024x465.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140-768x349.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140-1536x697.png 1536w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140-1200x545.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174140.png 1747w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19657\" src=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174313-300x137.png\" alt=\"\" width=\"709\" height=\"324\" srcset=\"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174313-300x137.png 300w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174313-1024x467.png 1024w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174313-768x350.png 768w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174313-1200x547.png 1200w, https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-17-174313.png 1385w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<p><strong>Blog Pundit<\/strong>:\u00a0<a href=\"https:\/\/opstree.com\/blog\/author\/sandeep7c51ad81ba\/\" target=\"_blank\" rel=\"noopener\">Sandeep Rawat<\/a>\u00a0Opstree is an End to End DevOps solution provider<\/p>\n<div class=\"wp-block-buttons\">\n<div class=\"wp-block-button is-style-fill\"><a class=\"wp-block-button__link\" href=\"https:\/\/www.opstree.com\/contact-us\" target=\"_blank\" rel=\"noreferrer noopener\">CONTACT US<\/a><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Introduction In\u00a0Part 1, we set up a\u00a0real-time data pipeline\u00a0that streams PostgreSQL changes to\u00a0Amazon S3 using Kafka Connect. Here\u2019s what we accomplished: Configured PostgreSQL for CDC (using logical decoding\/WAL) Deployed Kafka Connect with JDBC Source Connector (to capture PostgreSQL changes) Set up an S3 Sink Connector\u00a0(to persist data in S3 in Avro\/Parquet format) In Part 2 &hellip; <a href=\"https:\/\/opstree.com\/blog\/2024\/12\/31\/stream-and-analyze-postgresql-data-from-s3-using-kafka-and-ksqldb-part-2\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Stream and Analyze PostgreSQL Data from S3 Using Kafka and ksqlDB: Part 2&#8221;<\/span><\/a><\/p>\n","protected":false},"author":242684335,"featured_media":19842,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474],"tags":[768739426,207392,15989,133339,4996032],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2024\/12\/stream_and_analyze_postgresql_data_from_s3_using_kafka_and_ksqldb_part_2_720.png","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-59Q","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/19830"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/242684335"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=19830"}],"version-history":[{"count":7,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/19830\/revisions"}],"predecessor-version":[{"id":29397,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/19830\/revisions\/29397"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/19842"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=19830"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=19830"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=19830"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}