Apache Hudi (Hadoop Upserts Deletes and Incrementals) is a powerful framework designed for managing large datasets on cloud storage systems, enabling efficient data ingestion, storage, and retrieval. One of the key features of Hudi is its support for two distinct storage types: Copy-On-Write (COW) and Merge-On-Read (MOR). Each of these storage strategies has unique characteristics and serves different use cases. In this blog, we will explore COW and MOR. Continue reading “Understanding COW and MOR in Apache Hudi: Choosing the Right Storage Strategy “
Tag: Apache
Securing Nifi Cluster with TLS Toolkit
Apache NiFi is an easy-to-use, powerful, and reliable system to process and distribute data. As data flows between different systems it’s a good practice to ensure the integrity of the data being processed. Securing a NiFi cluster is essential for various reasons, primarily to protect sensitive data, ensure system integrity, and prevent unauthorized access. Apache NiFi provides a TLS toolkit using which we can self-sign Certificate Authority (CA) and easily issue and sign certificates in the format expected by NiFi. Continue reading “Securing Nifi Cluster with TLS Toolkit”
Where there is a shell, There is a way.
Well, as a DevOps; I like to play around with shell scripts and shell commands especially on a remote system as it just adds some level of fun in it. But what’s more thrilling than running shell scripts and command on the remote server, making them return the dynamic web pages or JSON from that remote system.
Yes for most of us it comes as a surprise that just like PHP, JSP, ASP shell scripts can also return us dynamic web pages but, as long time ago a wise man said: “where there is a shell there is a way”.
Isn’t PHP or JSP a better option for web development?
Why do we need this exactly?
- We can use it as a time series based data exporter.
- We might want an API that returns us the system info in the form of JSON, and we don’t have access to PHP.
- We might want to see the system information as a web page when we hit a URL.
- It’s not only limited to system info you can do whatever you want from it.
- With bare minimum on your machine, you can get the max out of it
Let’s get started
$ cd /etc/apache2/mods-enabled $ sudo ln -s ../mods-available/cgi.load
$ cd /usr/lib/cgi-bin
$ vim hello.sh
#!/bin/bash
echo "Content-type: text/html"
echo ""
echo "hello world! from shell script"
$ sudo systemctl restart apache2.service
Let’s take it to the next level
So let’s get started
<!doctype html> <html lang="en"> <head> <!-- Required meta tags --> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <!-- Bootstrap CSS --> <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> <title>Hello, world!</title> </head> <body> <h1>All the user using /usr/sbin/nologin shell</h1> <table class="table"> <thead> <tr> <th scope="col">Name</th> <th scope="col">User Id</th> <th scope="col">Group Id</th> </tr> </thead> <tbody>
</tbody> </table> <!-- Optional JavaScript --> <!-- jQuery first, then Popper.js, then Bootstrap JS --> https://code.jquery.com/jquery-3.3.1.slim.min.js https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js </body> </html>
hello.sh
#!/bin/bash
echo "Content-type: text/html"
echo ""
cat header
cat /etc/passwd | awk -F ':' '{if($7 == "/usr/sbin/nologin"){print ""$1""$3""$4""}}'
cat footer