Cert-Manager Issuer for Cross-Account Route 53 [ EKS ]

Cert-Manager is a very powerful tool when we talk about managing TLS certificates & issuers and no other tool comes near the Cert-Manager for kubernetes in terms of open source, visibility, documentation, installation option, integration, and many more. Even with the same account or cross-account option, there is a direct integration option provided by cert-manager CRDs. This will lead to ease of setting of certificates and managing those created certificates.

ASSUMPTION

For this session/blog, we are going to use ACME certificates [or Let’s encrypt certificates] using DNS01 challenger.

Before setting up we need to have the clarity of account and their functionality.

ACCOUNT-X — EKS SETUP

ACCOUNT Y — ROUTE 53

NOTE: For the same account, you can use serviceaccount to make a call through OIDC To AWS IAM Role.

SETUP

IAM User credentials

Under Account Y, Create IAM user and provide privileged [Administrator] access or you can use following IAM policy to provide limited access to Route 53. This permission required to change or read information of the Hosted zone.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "route53:GetChange",
"Resource": "arn:aws:route53:::change/*"
},
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets",
"route53:ListResourceRecordSets"
],
"Resource": "arn:aws:route53:::hostedzone/*"
},
{
"Effect": "Allow",
"Action": "route53:ListHostedZonesByName",
"Resource": "*"
}
]
}
view raw route-53-acccess-policy-for-cert-manager.json hosted with ❤ by GitHub

Once created. Copy ACCESS KEY & SECRET KEY of that IAM user. This is required further when we will configure Issuer.

ACCESS-KEY - AKIAXXXXXXXXXXXXXXX
SECRET-ACCESS-KEY - 5CE3sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Get Hosted Zone ID

This hosted zone we need to copy from Route 53 of Account Y, dsf

Hosted Zone ID - ZO61XXXXXXXXXXXXX

Cert-Manager Setup

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.9.1 --set installCRDs=true

Check This Link for detailed installation of Cert-Manager.

BASE64 encoded secret access key

First, copy the secret access key that we created for IAM User for Account Y.

echo "5CE3sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" | base64

It will give base 64 encoded output which we will use to create a kubernetes secret for secretAccessKeySecretRef reference.

NUNFMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=

AWS Secret key Secret creation

Now, we need to create that secret before applying the changes.

apiVersion: v1
kind: Secret
metadata:
name: awssecretkey
namespace: cert-manager
type: Opaque
data:
secret-access-key: NUNFMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
view raw accounty-secret-key.yaml hosted with ❤ by GitHub

and apply the following

kubectl apply -f accounty-secret-key.yaml

Check if the secret is created or not.

kubectl get secret -n cert-manager

ClusterIssuer Breakdown

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cert-manager-staging-test

The above information is related to CRD of cert-manager. This includes apiVersion, kind & metadata

spec:
  acme:
    email: [email protected]xx
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cert-manager-staging-test-secret

ACME BREAKDOWN:

  • email [Require for general information for certification creation]
  • server [This specifies what kind of certificates user want’s to create Like staging or prod certificates]
  • privateKeySecretRef [Name of the secret]
solvers:
    - selector:
        dnsZones:
          - "*.example.example"
          - "example.com"

Solvers BREAKDOWN:

  • Selectors [Selector specifies the configuration, For let’s encrypt, we need to specify dnsZones]
  • dnsZones [dnsZones specifies the domain & wildcard information]
dns01:
        route53:
          region: XX-XXXX-X
          hostedZoneID: ZO61XXXXXXXXXXXXX
          accessKeyID: AKIAXXXXXXXXXXXXXX
          secretAccessKeySecretRef:
            name: awssecretkey
            key: accounty-secret-key

dns01 [This specifies the actual information related to the route53 like region, hostedZoneId, access keys, role id]

ClusterIssuer file

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: cert-manager-staging-test
spec:
acme:
email: [email protected]
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cert-manager-staging-test-secret
solvers:
– selector:
dnsZones:
– "*.example.com"
– "example.com"
dns01:
route53:
region: us-east-1
hostedZoneID: ZO61XXXXXXXXXXXXX
accessKeyID: AKIAXXXXXXXXXXXXXX
secretAccessKeySecretRef:
name: awssecretkey
key: secret-access-key
view raw issuer.yaml hosted with ❤ by GitHub

Use below command to apply the clusterissuer configuration.

kubectl apply -f issuer.yaml

Once you apply the changes, you can check the cert-manager controller pod logs to check the behaviour or validate the logs.

LOGS:

I0817 19:16:29.779442       1 setup.go:111] cert-manager/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"
I0817 19:16:29.906133       1 setup.go:219] cert-manager/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"
I0817 19:16:30.770143       1 setup.go:309] cert-manager/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"
I0817 19:16:30.770174       1 conditions.go:95] Setting lastTransitionTime for Issuer "cert-manager-staging-test" condition "Ready" to 2022-08-17 19:16:30.770167831 +0000 UTC m=+2333.520794129
I0817 19:16:30.783535       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"
I0817 19:16:34.907590       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-staging-test-secret" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="cert-manager-staging-test" "resource_namespace"="" "resource_version"="v1"

You can also validate the changes by using the below command to check READY status. If it is showing True, it means it is ready to serve.

NOTE: If it is not showing True, inspect the cert-manager controller logs to rectify the issue.

kubectl get clusterissuer

Certificate creation

Once, you create the clusterissuer with True ready status. You need to create certificates which responsible for creating secrets containing certificates.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cert-manager-staging-test-cert
namespace: default
spec:
secretName: cert-manager-staging-test-staging-secret
issuerRef:
name: cert-manager-staging-test
kind: ClusterIssuer
commonName: '*.example.com'
dnsNames:
– '*.example.com'
– 'example.com'
view raw cert-test.yaml hosted with ❤ by GitHub

Once you created manifest file containing certificates changes. Apply the above changes

kubectl apply -f cert-test.yaml

Once, you create, you can check Route 53 [On account Y] to validate the TXT record created by cert-manager.

_acme-challenge.example.com.xyz TXT Multivalue answer-  "QCGSHgXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Once, you check or validate, you can use the below command to validate whether the certificates are ready to use or not.

kubectl get certificates -n test

You can also describe or check the content secret created by certificate CR.

kubectl describe secret cert-manager-staging-test-staging-secret -n test

OUTPUT:

Data
====
tls.key:  1675 bytes
tls.crt:  5753 bytes

Cert manager Certificate logs

I0817 19:21:19.993429       1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Ready" to 2022-08-17 19:21:19.993418389 +0000 UTC m=+2622.744044690
I0817 19:21:19.994199       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="test/cert-manager-staging-test-cert" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"
I0817 19:21:19.994220       1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Issuing" to 2022-08-17 19:21:19.994216003 +0000 UTC m=+2622.744842295
I0817 19:21:20.073511       1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"cert-manager-staging-test-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert"
I0817 19:21:20.073583       1 conditions.go:201] Setting lastTransitionTime for Certificate "cert-manager-staging-test-cert" condition "Ready" to 2022-08-17 19:21:20.073575848 +0000 UTC m=+2622.824202143
I0817 19:21:20.358802       1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"cert-manager-staging-test-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert"
I0817 19:21:20.388505       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Approved" to 2022-08-17 19:21:20.388478055 +0000 UTC m=+2623.139104353
I0817 19:21:20.439792       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Ready" to 2022-08-17 19:21:20.439781619 +0000 UTC m=+2623.190407917
I0817 19:21:20.455394       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cert-manager-staging-test-cert-qd447" condition "Ready" to 2022-08-17 19:21:20.455383292 +0000 UTC m=+2623.206009583
I0817 19:21:20.470152       1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cert-manager-staging-test-cert-qd447\": the object has been modified; please apply your changes to the latest version and try again" "key"="test/cert-manager-staging-test-cert-qd447"

Now, you can use the above certificates in any kind of resource like an ingress controller, service mesh, etc.

NOTE: To know more about certificates & cluster issuers, visit the official documentation provided by the cert-manager. 

How-To

Now, these certificates can use anywhere where you specify these certificates. But for this section, we will only focus on ingress object.

Check the below ingress manifest containing rules and it also contains the secret that was created through a cert-manager certificate.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
# cert-manager.io/issuer: "cert-manager-staging-test"
spec:
rules:
– host: example.com
http:
paths:
– path: /
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
tls:
– hosts:
– example.com
secretName: cert-manager-staging-test-cert
view raw ingress-to-test-cert-manager.yaml hosted with ❤ by GitHub

Once applied, check browser and check TLS

Check certificate details, you will see staging certificate issued by Let’s encrypt.

The above certificate is just to validate for testing purposes or only for staging purposes.

To implement for a production environment, use the following server URL in clusterIssuer YAML.

https://acme-v02.api.letsencrypt.org/directory

YAML:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: cert-manager-staging-test
spec:
acme:
email: [email protected]
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cert-manager-staging-test-secret
solvers:
– selector:
dnsZones:
– "*.example.com"
– "example.com"
dns01:
route53:
region: us-east-1
hostedZoneID: ZO61XXXXXXXXXXXXX
accessKeyID: AKIAXXXXXXXXXXXXXX
secretAccessKeySecretRef:
name: awssecretkey
key: secret-access-key
view raw clusterissuer-prod.yaml hosted with ❤ by GitHub

Once, you added and repeated the above steps, you will get valid certificates.

REFERENCES

CONNECT WITH ME

Blog Pundit: Sanjeev Pandey and Sandeep Rawat

Opstree is an End to End DevOps solution provider.

CONTACT US

Connect with Us

One thought on “Cert-Manager Issuer for Cross-Account Route 53 [ EKS ]”

  1. Hi bupendhar I have a question is it possible to connect specific EC2 instance to a specific S3 bucket without attaching IAM role and policy to the EC2 instance, instead of attaching IAM ROLE and policy to the EC2 instance can we attach EC2 ARN to specific S3 bucket policy will this work? how can i do this i tried but giving some error.

    Reply

Leave a Reply