{"id":9939,"date":"2022-02-22T16:09:37","date_gmt":"2022-02-22T10:39:37","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=9939"},"modified":"2022-02-22T16:58:06","modified_gmt":"2022-02-22T11:28:06","slug":"handling-private-affair-a-guide-to-secrets-management-system","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2022\/02\/22\/handling-private-affair-a-guide-to-secrets-management-system\/","title":{"rendered":"Handling Private Affair: A Guide to Secrets Management System"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Needless to say an automated system, no matter how big or small, must be designed with scale in mind. We\u2019ll talk about laying foundation for a robust and malleable setup which is a useful read for everyone.<\/p><\/blockquote>\n\n\n\n

In my experience as DevOps and SRE, I\u2019ve enjoyed quite a lot of things this profession offered. From the satisfaction of fulfilled curiosities to the anxiety of unforeseen mishaps, it delivered one day after the other. The nervousness in the face of new challenges, happiness on receiving appreciation, thrill during troubleshooting, the pride after a successful implementation, and a lot more. But the one I found myself seeking was boredom. Yes, plain old silence<\/em> where no surprises are met, everything runs exactly as it should and you enjoy listening to dropping pins. This is especially true when the system in question is your own design. There\u2019s no greater sentiment. It is like watching a<\/em> bird that you\u2019ve freed soar.<\/em><\/p>\n\n\n\n\n\n\n\n

For the same reason, in this article, we\u2019ll talk about the considerations while designing a secret management system for large-scale infrastructure. Having a grip over our secret flow is paramount w.r.t functioning of our distributed system as well the safety of our intellectual property. Therefore, keeping it stable is a huge part of preserving that silence<\/em> on a big scale. Needless to say an automated system, no matter how big or small, must be designed with scale in mind. We\u2019ll talk about laying the foundation for a robust and malleable setup which is a useful read for everyone. This article assumes that the reader is aware of secrets and their importance in a microservices infrastructure. However, if not so, a quick google search should yield ample information.<\/p>\n\n\n\n

Encrypted Store<\/strong><\/p>\n\n\n\n

Secrets Management System<\/strong> consists of a lot more than just a secret store. It consists of processes to map secrets to appropriate CI pipelines, maintain different versions, and ways to incorporate secrets during deployments to various services regardless of changes in infrastructure. We\u2019ll talk more about it later but all these facts do not downplay the importance of a hardened, highly-available storage application at all. HashiCorp Vault, for example, is quite popular in this area. It is a distributed system, which has below desirable features:<\/p>\n\n\n\n

  1. RESTful API<\/em><\/li>
  2. Scalable<\/em><\/li>
  3. Highly-available<\/em><\/li>
  4. Secure storage<\/em><\/li><\/ol>\n\n\n\n

    A secret storage system should be, preferably, central. One place where all clients can request the secrets they need. This type of setup has its undeniable advantages:<\/p>\n\n\n\n

    1. It is simple to manage secrets<\/em><\/li>
    2. Chances of spill or leak are minimized<\/em><\/li>
    3. Facilitates coordination among teams<\/em><\/li>
    4. Keeps code-base secure<\/em><\/li><\/ol>\n\n\n\n

      HashiCorp Vault<\/strong><\/em> is one good example but AWS Secret Manager<\/a> is also highly recommended. It may not have as many features as the vault, however, if your infra is mostly on AWS, it is a compelling choice with seamless services integration and IAM authentication.<\/em><\/p>\n\n\n\n

      Certificates<\/strong><\/p>\n\n\n\n

      All official documents need to be attested. That is how we validate the authenticity of their source. Why should it be any different for our microservices? I am talking, of course, about certificates. It is recommended to use them during any or all communication even in a private network. Our focus, however, is secrets management, hence we\u2019ll discuss certificates and their importance while adding and fetching secrets from the secret vault. <\/p>\n\n\n\n

      Ideally, we\u2019ll read\/write secrets using RESTful APIs. From a security point of view, these requests must go over SSL\/TLS with certificates issued from a trusted CA. In this regard, there are a few things we must understand:<\/p>\n\n\n\n

      1. Depending on the scale, a large number of clients will be requesting secrets from our secret vault<\/em><\/li>
      2. We cannot keep issuing certificates for these clients and forget about them<\/em><\/li>
      3. Certificates must come from a trusted CA or we\u2019ll have major security loopholes in our system.<\/em><\/li><\/ol>\n\n\n\n

        Due to the above reasons, there are best practices in regard to PKI\u2019s that we must follow. These practices ensure that we not only have visibility over all issued certificates but can also backtrack and pinpoint culprits during unprecedented security incidents. Here are some recommendations:<\/p>\n\n\n\n

        1. Integrate secrets management with policy-compliant certificate issuers (public or private CA)<\/strong><\/em><\/li>
        2. Track every certificate being issued for audit purposes<\/strong><\/em><\/li>
        3. Certificates must have a managed life-cycle. No certificate must stay valid longer than required.<\/strong><\/em><\/li><\/ol>\n\n\n\n

          If you have a private trusted CA setup, you\u2019re golden. If not, it is definitely worth looking into.<\/p>\n\n\n\n

          Automation<\/strong><\/p>\n\n\n\n

          Finally! here we are. Time to write some code. Like I mentioned before, secrets management is not just about storing and fetching. We also need to govern transmission, CRUD, availability, visibility, scalability of deployment infra, etc.  We can address these requirements by writing utilities and automation to streamline them as much as possible. Let\u2019s start by reflecting on what we want from scalable automation around our secrets:<\/p>\n\n\n\n

          1. Should support distributing secrets to multiple environments and sub-environments in run-time<\/strong><\/em>. It is the best way as it omits the possibility of storing or committing secrets anywhere else except the secret vault. Secrets are pulled via APIs directly from the vault as the deployment is happening. Here\u2019s a related interesting tool<\/a> for Kubernetes.<\/li>
          2. Provide an easy way to do CRUD operations on a secret vault as per the specific design of our system<\/em><\/strong>. The application we\u2019re using as vault may provide easy ways to do this but, often, in my experience, writing a wrapper on top of it has been crucial. Particularly when the goal is to design a self-service system where the dependency on DevOps is minimal.<\/li>
          3. Authenticate users via Central Identity Management<\/em><\/strong>. It may be Active Directory or IAM but automation must have checks in place to regulate access. This way we can have tight control over the system.<\/li>
          4. Expose metrics for monitoring the automation. <\/em><\/strong>A tailored system would require custom metrics for insightful monitoring. The more detailed metrics, the better dashboard and alerting. Also, goes without saying, integrate it with your monitoring system.<\/li>
          5. Write test cases for automation<\/em><\/strong>. Whether it is Go, Python, Shell, or HCL, test cases can be written for all of them. As the system scales up, it might overflow with features and fixes. Having a test-suite handy will make the whole thing reliable by reducing the chances of something breaking exponentially.<\/li><\/ol>\n\n\n\n

            What’s next?<\/strong><\/p>\n\n\n\n

            That\u2019s all for now. I know what would help you more in grasping these concepts. An already implemented, large-scale, intricate secrets management system that we can take apart and study, like Rancho with the refrigerator i<\/strong>n 3 idiots<\/a>.<\/em> Well, I am planning something like that for my next article, which will be in continuation to this one. Thanks for reading. Stay tuned!<\/p>\n\n\n\n

            Image Source<\/a><\/p>\n\n\n\n


            Blog Pundit:<\/strong> Sanjeev Pandey<\/strong><\/p>\n\n\n\n

            Opstree<\/a><\/strong> is an End to End DevOps solution provider<\/p>\n\n\n\n

            \n
            CONTACT US<\/a><\/div>\n<\/div>\n\n\n\n

            Connect Us <\/strong><\/p>\n\n\n\n