AWS SECRET MANAGER

Introduction

Most of the IT companies are working or are migrating their infrastructure to the cloud environment for cost reduction, high availability, data security, and hassle-free setup. Companies create or use applications/databases on the cloud, where they need to authenticate via secrets (or credentials). These days protection of passwords is one of the challenges, and hard-coding the secrets into the system would be a major security issue. Hence, we can make use of the AWS service named Secret Manager that will be responsible for the management of secrets.

What is a Secret?

A secret is not just a pair of usernames and passwords. It might contain a set of credentials holding key-value pairs or connection details to access the related services. In AWS Secret manager, a secret has metadata:

An Amazon Resource Name, name of the secret, a description, a resource policy, and tags, ARN for an encryption key (an AWS KMS key that Secrets Manager uses to encrypt and decrypt the secret value), Information of rotation of secret.

AWS Secret Manager (ASM)

AWS secret manager stores secrets and programmatically retrieves the same by an API call that will be used to access your applications, databases, etc. It automatically rotates the secret as per the specified schedule, and you don’t need to update the secret in the application manually. Hence, the secret can’t be compromised. It easily rotates and manages the lifecycle of the secrets. You can also provide granular permission and controlled access to the secrets and audit secret rotation for your resources. Overall, it is a very strong tool for secret management.

Secrets Manager offers pay-as-you-go pricing. You pay for the number of secrets and the number of Secrets Manager API calls that are made. For pricing please refer to this link https://aws.amazon.com/secrets-manager/pricing/

Authentication And Authorization

Secrets Manager uses IAM policies that define only approved identities can securely access the secrets and can modify them. ASM uses the below 2 permissions to control access:
SecretManagerReadWrite – provides access to performing actions in secret manager service.
IAMFullAccess – requires to enable the secret rotation. Without this permission, a cloud formation stack will not be created which is used for rotation.

Types of permission policies that are used by ASM:

Identity-based policy (attaching permission to an identity)

Grants permission to an identity or IAM group to access the secret.
Controls who can create new secrets.

Resource-based policy (attaching permission to a secret)
Allows the secrets to be accessed by multiple identities of same account or other AWS accounts.
(We must allow access both in a resource policy and in an identity policy in case of cross-account access, otherwise AWS denies the request.)
ASM looks over the resource-based policy attached to the secret and all the identity-based policies attached to the IAM identities sending the request.

Versions & Labels in ASM

ASM stores encrypted secret data that include connection details containing server name, IP address, port number, username and password for authentication.
Versions of secret hold copies of the encrypted secret value. When you change the secret value, or the secret is rotated, Secrets Manager creates a new version. ASM stores many versions of single secret depending on our requirements.
Two versions of a secret can’t have the same staging label. Different regions have different quota for ASM.
During rotation, Secrets Manager uses staging labels to indicate the different versions of a secret:

AWSCURRENT – version is active on clients. A secret always has an AWSCURRENT version.
AWSPENDING – version will become AWSCURRENT after rotation is completed.
AWSPREVIOUS – previous AWSCURRENT version (or the last known good version).

You can retrieve older versions of secret, if needed, by mentioning version id when you access a secret version. If you don’t specify any version, either by version ID or staging label, Secrets Manager gets the version with the staging label AWSCURRENT attached.

Command to list all active versions of the secret and staging labels attached to each version.
# aws secretmanager describe-secret –secret-id <name of secret>

Command to view the encrypted text in your secret.
# aws secretmanager get-secret-value –secret-id <name of secret>

For getting only secret value
# aws secretsmanager get-secret-value –secret-id <name of secret> –query SecretString –output text

**If details 0f any specific label of version is required, you can make use of –version-stage option following by label with the same command.

If there are more than 100 versions, ASM will remove the outdated versions but it doesn’t remove the versions that are created within 24 hours.

Secret And Data Encryption

Secrets Manager uses envelope encryption with AWS KMS keys to encrypt secrets and its value. Both AWS and Customer managed keys can be used. Every secret is associated with a key. Envelope encryption is a practice of encrypting plaintext data with a data key, and then encrypting that data key under another key.
When a secret is stored, Secrets Manager calls AWS KMS with the ID of KMS key to generate and encrypt the 256-bit symmetric plaintext data key and an encrypted data key. The plaintext data key is used to encrypt the secret value, and then removes it from memory. The encrypted data key is stored in the metadata of the secret so that it will be used to decrypt the secret values.
When a secret is retrieved, Secrets Manager decrypts the encrypted data key (using the AWS KMS keys) and returns the plaintext data key that is used to decrypt the secret value. The data key is stored encrypted and is never written to disk in plaintext. Also, Secrets Manager does not write or cache the plaintext secret to persistent storage. It removes the data key from the memory as soon as possible.
Whenever Secrets Manager encrypt a new version of the protected secret data, it requests AWS KMS to generate a new data key. You can use the same KMS key or different KMS keys for each secret in your account. Secrets Manager supports only symmetric KMS keys.

Secret Rotation in ASM

What is rotation?

Rotation is a process of updating the secrets. By automatic rotation, secret manager updates the secrets in itself as well as in the database or service at the scheduled time without any user intervention. Secret having automatic rotation ‘on’ can only be rotated. And each type of secret requires one separate lambda function for automatic rotation. It is always recommended to verify whether the secret is able to connect with the database or service before configuring automatic rotation.
Applications that retrieve the secret from Secrets Manager will automatically get the new credentials after rotation. Whenever you enable automatic rotation, you will see an extra cloud formation stack that shows the serverless application repository stack.

Permission required for rotation.

ASM uses lambda function for rotation. Lambda function must have access to secret and database as it has an IAM execution role attached to it.
For an Amazon RDS, Amazon DocumentDB, or Amazon Redshift secret, Secrets Manager creates the role for you and attaches permissions to it.
For any other type of secret, if you create the Lambda function through the Secrets Manager console, the role is created for you. But if you create the Lambda function another way, you need to also create an execution role and make sure it has the correct permissions.

Rotation supported services in AWS

For Amazon Relational Database Service (Amazon RDS) databases, you get AWS written and tested lambda rotation functions templates which means you don’t need to do any programming. These databases are listed below:

Amazon Aurora on Amazon RDS
MySQL on Amazon RDS
PostgreSQL on Amazon RDS
Oracle on Amazon RDS
MariaDB on Amazon RDS
Microsoft SQL Server on Amazon RDS

There are two more services that support lambda rotation functional templates:
Amazon DocumentDB
Amazon Redshift

Also, you can store any secrets of a different kind, but for automatic rotation, you need to create and test a custom lambda rotation function.

How rotation works?

Secrets Manager uses staging labels to label secret versions during rotation. You can also choose the number of days after which the rotation will happen. It must be between 1 and 365 days. For rotation, the rotation function must be able to communicate both with protected resources and with the Secret manager. The preferred way to access the ASM is through VPC interface endpoints. You must look up the AZs that offers VPC endpoints. Not all the AZs with a region offer interface endpoints to the service.

Step 1: Create a new version of the secret with identical privileges. Depending on your rotation strategy, the new version can contain a new password, a new username, and password, or more secret information. Secrets Manager labels the new version with the staging label “AWSPENDING”.

Step 2: Rotation changes the credentials in the database or service to match with the new credentials in the AWSPENDING version of the secret. Depending on your rotation strategy, this step can create a new user with the same permissions as the existing user.

Step 3: Rotation tests the AWSPENDING version of the secret by using it to access the database or service. Rotation functions based on Rotation function templates test the new secret by using read access. Depending on the type of access your applications need, you can update the function to include other access such as write access.

Step 4: Rotation moves the label AWSCURRENT from the previous secret version to the latest version. Secrets Manager adds the AWSPREVIOUS staging label to the previous version so that you retain the last known good version of the secret.

ASM doesn’t allow you to immediately delete a secret. You can schedule for a minimum of 7 days in the future. Also, you can cancel the deletion of a secret anytime prior to its deletion.

You can reuse a rotation function you used for another secret if the rotation strategy is the same. After rotation is successful, applications that from Secrets Manager automatically get the updated credentials. In case of any problem, you can make use of these staging labels to help with the recovery.

Conclusion

Secret Manager is a very powerful tool for secret management, and it also supports the whole lifecycle of secrets. It also creates logs events that indicate the state of rotation. You can audit AWS CloudTrail logs to see when rotation happened & also capture all the API calls made from console or code. You can also configure CloudWatch events whenever the secret gets rotated & deleted.

Blog Pundit: Kapendra Singh and Sanjeev Pandey

Opstree is an End to End DevOps solution provider

Connect Us