Certificates with Azure Key Vault and Nginx Ingress Controller

Let’s face it. If you deploy web applications and APIs of any sort, you need certificates. If you have been long enough in IT, there’s just no escape! In this article, we will take a look at getting a certificate from Azure Key Vault to Azure Kubernetes service. Next, we will use that certificate with Nginx Ingress Controller and check what happens when the certificate gets renewed.

If you are more into videos, check out the video below from my channel:

Video from https://youtube.com/geertbaeke

Prerequisites

What do you need to following along?

  • Azure subscription: see https://azure.microsoft.com/en-us/free/
  • Azure Key Vault: see the quickstart to create it with the Azure Portal
  • Azure Kubernetes Services (AKS): see the quickstart to deploy it via the portal
  • Azure CLI: see the installation options
  • Kubectl: the Kubernetes administration tool; check the installation instructions here; use a package manager such as brew of choco to easily install it
  • Helm: required to install Helm charts; use a package manager such as brew of choco to install it; use v3 and higher

When AKS is up and running and you have authenticated with the Azure CLI using az login, get the credentials to AKS with:

az aks get-credentials -n <clustername> -g <resourcegroup>

We can now proceed to install nginx ingress controller.

Installing nginx ingress controller

Use the Helm chart to install nginx. First add the repo:

helm repo add https://kubernetes.github.io/ingress-nginx
helm repo update

Now install the chart:

helm install my-release ingress-nginx/ingress-nginx

More information can be found here: https://kubernetes.github.io/ingress-nginx/deploy/. The Helm chart will result in an nginx pod on your cluster. It will use a Kubernetes service exposed via an Azure Public Load Balancer. Later, we will publish an application on our cluster via this endpoint. We will do that by creating a resource of kind Ingress.

The procedure below works equally well with an ingress controller on an internal IP address and potentially, internal DNS names and certificates. We just happen to use an external IP address and a self-signed certificate here.

Installing the akv2k8s controller

To sync a Key Vault certificate to Kubernetes, we need some extra software. You will often come across the secrets store CSI driver, which has a provider for Azure Key Vault. Although this works well and is probably the way forward in the future, I often use another solution that is just a bit easier to use: the Azure Key Vault to Kubernetes controller. Check out the documentation over at https://akv2k8s.io.

The controller can be configured to sync a certificate in Azure Key Vault to a secret of type kubernetes.io/tls. Normally, you would create such a secret with the following command:

kubectl create secret tls my-tls-secret --cert=path/to/cert/file --key=path/to/key/file

Indeed, you would need the certificate and private key files to create such a secret. The akv2k8s controller does that work for you, grabbing the certificate and private key from Key Vault. Do note that what we are doing here is creating a regular Kubernetes secret. Such a secret contains the certificate and key in base64 encoded format. Anyone with the proper access rights on your cluster can easily decode the secret and use it as they please. Check out the following document about the risks of regular secrets in Kubernetes.

To install the controller, see https://akv2k8s.io/installation/installing-with-helm.

Creating the certificate in Key Vault

There are many ways to generate certificates and store them in Key Vault. In general, you should automate as much as possible especially when it comes to renewing the certificate. However, this post focuses on getting a certificate to Kubernetes. That is the reason why we will generate a self-signed certificate in Key Vault.

In your Key Vault, navigate to Certificates and click Generate/Import:

Certificates in Key Vault

In Create a certificate, fill in the blanks. If you want to use a real domain, make sure you specify it in the DNS Names. I used test.baeke.info with a validity of 12 months. The content type can either be PKCS #12 or PEM. The akv2k8s controller can handle both formats.

New self-signed certificate

After clicking Create and refreshing the list a few times, you should see the certificate listed:

mycert lis in the list

Note: in what follows, I will use the nginx certificate in the list; it was created in the same way although it is valid for 24 months

Access Policy

The akv2k8s controller needs access to your Key Vault to retrieve the certificate. It used the service principal or managed identity of the cluster to do so. My cluster was setup with managed identity. You can retrieve the identity with the Azure CLI:

az aks show -n <clustername> -g <resourcegroup> | jq .identityProfile.kubeletidentity.objectId -r

jq is a tool to parse JSON content. We use it here to retrieve the objectId of the managed identity. Once you have the objectId, you can grant it the required access rights:

az keyvault set-policy --name <KeyVault> --object-id  <objectId> --certificate-permissions get

The above Azure CLI command gives the objectId of our managed identity access to retrieve certificates from the specified Key Vault. You can use the short name of the Key Vault in –name.

Syncing the certificate

With the controller installed and granted sufficient access rights, we can now instruct it to sync the certificate. We do so with the following YAML:

apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
  name: cert-sync
  namespace: certsync
spec:
  vault:
    name: gebakv
    object:
      name: nginx
      type: certificate
  output:
    secret:
      name: nginx-cert
      type: kubernetes.io/tls

Note that all the resources I deploy from now are in the certsync namespace. The above YAML is pretty clear: it syncs the nginx certificate in Key Vault to a Kubernetes secret called nginx-cert. The type of the secret is kubernetes.io/tls. After synchronization, it will appear in the namespace:

NAME                  TYPE                                  DATA   AGE
nginx-cert            kubernetes.io/tls                     2      19s

On my system, I have installed the krew view-cert plugin. The command kubectl view-cert in the namespace certsync results in the following output (it enumerates all certs as a JSON array but there is only one):

[
    {
        "SecretName": "nginx-cert",
        "Namespace": "certsync",
        "Version": 3,
        "SerialNumber": "15fd15ed11384d31a0a21f96f5e457c6",
        "Issuer": "CN=test.baeke.info",
        "Validity": {
            "NotBefore": "2020-12-05T14:09:53Z",
            "NotAfter": "2022-12-05T14:19:53Z"
        },
        "Subject": "CN=test.baeke.info",
        "IsCA": false
    }
]

When I check the serial number in Key Vault, it matches with the serial number above. The certificate is valid for two years.

Using the secret with nginx-ingress

In the certsync namespace, I installed a simple app that uses a service called realtime. We will expose that service on the Internet via the nginx ingress controller (version v0.41.2; image k8s.gcr.io/ingress-nginx/controller). We use the following Ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: testingress
  namespace: certsync
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - test.baeke.info
    secretName: nginx-cert
  rules:
  - host: test.baeke.info
    http:
      paths:
      - path: /
        backend:
          serviceName: realtime
          servicePort: 80

Important: my Kubernetes version is 1.18.8 so the above definition is still valid; for 1.19, check the docs

The above creates an ingress for test.baeke.info and requires tls with the certificate in the nginx-cert secret. After a while, you will see the address and ports the ingress uses. Use kubectl get ingress to check:

NAME          CLASS    HOSTS             ADDRESS       PORTS     AGE
testingress   <none>   test.baeke.info   20.73.37.74   80, 443   41s

At https://test.baeke.info, the following certificate is offered:

Self-signed certificate offered by nginx ingress for test.baeke.info

Note: you need to ensure the FQDN (test.baeke.info here) resolves to the IP of the ingress; on my cluster this is done automatically by external dns. Note that the certificate is valid for two years.

Renewing the certificate

While the renewal process can be configured to be automatic, we will configure a new certificate from Azure Key Vault. Just navigate to your certificate and click New Version:

Creating a new version of the certificate

In the screen that follows, you can adjust the settings of the new certificate. I changed the lifetime back to 12 months. When you save your changes, the akv2k8s controller will pick up the change and modify the certificate in the Kubernetes secret. It will not delete and create a new secret. With kubectl view-cert, I now get the following output:

[
    {
        "SecretName": "nginx-cert",
        "Namespace": "certsync",
        "Version": 3,
        "SerialNumber": "27f95965e2644e0a58a878bc8a86f7d",
        "Issuer": "CN=test.baeke.info",
        "Validity": {
            "NotBefore": "2020-12-07T09:05:27Z",
            "NotAfter": "2021-12-07T09:15:27Z"
        },
        "Subject": "CN=test.baeke.info",
        "IsCA": false
    }
]

The serial number has changed. You can also see that the validity period has changed to 12 months.

What about our ingress?

Nginx ingress controller is smart enough to detect the changed certificate and offer it to clients. I used SHIFT-F5 to refresh the page and ingore cached content. Here is the offered certificate:

New certificate with 12 month lifetime

Conclusion

When you work with certificates in Kubernetes, always automate as much as possible. You can do that with a solution such as cert-manager that can request certificates dynamically (e.g. from Let’s Encrypt). In many other cases though, there are other certificate management practices in place that might prevent you from using a tool like cert-manager. In that case, try to get the certificates into a system like Key Vault and create your automation from there.

10 thoughts on “Certificates with Azure Key Vault and Nginx Ingress Controller”

  1. Does the cert has to be a self-signed cert?
    I tried with a certificate issued by an integrated CA in KeyVault and it does not work.

    1. We have used self-signed certs and certs issued by a commercial CA. As long as the cert is valid, it should work. However, I do not have CA integration to verify so I cannot say much about that scenario.

      1. Thanks much for the reply. I tried with a self-signed cert and it works as expected as in this article shows. But when I try with a cert issued by integrate CA(Microsoft Azure TLS in my case), it did create a tls secret. But no cert showS up when I run Kubectl view-cert. When I try use this tls secret in Ingress controller, after deploy the cert shows issued by Kubernetes Ingress Controller Fake Certificate and does not have the same serial number as the cert in Keyvault.

      2. Can you verify the certificate without the view-cert plugin and just check that there is a secret that contains both the certificate and the private key? So just kubectl get secret SECRETNAME -o yaml… You should see tls.crt and tls.key in the data field. If you cannot find that secret, or it does not contain those fields, something went wrong with syncing the certificate. When you manually download the cert from Key Vault in pfx/pem format, that also works? Is it valid when you retrieve it directly from KV?

      3. That is the problem. The TLS Secret do contains both tls.crt and tls.key. But when I use it in Ingress, the cert is wrong, the seral number does not match and the issued by shows as Kubernetes Ingress Controller Fake Certificate
        If I manually download pfx and use openssl to convert it into cert and key pem and create a tls secret based on them, it works.

        Below is my cert-sync deployment yaml

        apiVersion: spv.no/v1
        kind: AzureKeyVaultSecret
        metadata:
        name: cert-sync
        namespace: default
        spec:
        vault:
        name: rduimiddlewaretestkv
        object:
        name: middleware-ppe-ssl-cert
        type: certificate
        output:
        secret:
        name: middleware-ppe-ssl-cert
        type: kubernetes.io/tls

        When do kubectl get secret middleware-ppe-ssl-cert -o yaml it shows below (replaced the really cert and key value below)

        apiVersion: v1
        data:
        tls.crt: mycert
        tls.key: mykey
        kind: Secret
        metadata:
        annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
        {“apiVersion”:”spv.no/v1″,”kind”:”AzureKeyVaultSecret”,”metadata”:{“annotations”:{},”name”:”cert-sync”,”namespace”:”default”},”spec”:{“output”:{“secret”:{“name”:”middleware-ppe-ssl-cert”,”type”:”kubernetes.io/tls”}},”vault”:{“name”:”rduimiddlewaretestkv”,”object”:{“name”:”middleware-ppe-ssl-cert”,”type”:”certificate”}}}}
        creationTimestamp: “2021-01-20T19:25:48Z”
        name: middleware-ppe-ssl-cert
        namespace: default
        ownerReferences:
        – apiVersion: spv.no/v1
        blockOwnerDeletion: true
        controller: true
        kind: AzureKeyVaultSecret
        name: cert-sync
        uid: 7ab49f1f-eac7-416f-9bdb-309acaa48cdf
        resourceVersion: “46620218”
        selfLink: /api/v1/namespaces/default/secrets/middleware-ppe-ssl-cert
        uid: 75ebec46-456d-4efb-a89d-e4a680aee628
        type: kubernetes.io/tls

        And below is my ingress yaml
        apiVersion: extensions/v1beta1
        kind: Ingress
        metadata:
        name: rduimiddleware
        namespace: default
        annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/enable-cors: “true”
        nginx.ingress.kubernetes.io/cors-allow-origin: “*”
        nginx.ingress.kubernetes.io/ssl-ciphers: “ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-CBC-SHA256:ECDHE-RSA-AES256-CBC-SHA384”
        nginx.ingress.kubernetes.io/proxy-buffering: “on”
        nginx.ingress.kubernetes.io/proxy-buffers-number: “512”
        nginx.ingress.kubernetes.io/proxy-buffer-size: “512k”
        nginx.ingress.kubernetes.io/client-header-buffer-size: “512k”
        nginx.ingress.kubernetes.io/proxy-max-temp-file-size: “1024m”
        nginx.ingress.kubernetes.io/large-client-header-buffers: “4 512k”
        spec:
        tls:
        – hosts:
        – rduimiddlewareppenew.trafficmanager.net
        secretName: middleware-ppe-ssl-cert
        rules:
        – host: rduimiddlewareppenew.trafficmanager.net
        http:
        paths:
        – backend:
        serviceName: rduimiddleware-service
        servicePort: 8080
        path: /

  2. Does the cert has to be a self-signed cert? I tried with a cert issued by an integrated CA and it does not work.

    1. When I get the fake cert, it’s usually due to a mismatch between the hostname I am using in the browser and the configured hostname in the ingress definition or CN in the cert. I see you are using rduimiddlewareppenew.trafficmanager.net directly. I never do that and use a custom hostname (e.g. something.baeke.info) and the cert is for that hostname as well. Of course, if you are using traffic manager, the custom hostname you are using should be CNAME for rduimiddlewareppenew.trafficmanager.net. Can you check that?

      1. Yes, I am using traffic manager so the host is rduimiddlewareppenew.trafficmanager.net. the cert CN is the same.
        As I said, If I manually download the pfx from Azure key vault and convert it into TLS secret, it works. I only got the fake cert when using the tls cert synced by akv2k8s. Do you know if there is any support group from akv2k8s that can help brainstorm?

Leave a Reply to Anna Cui Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s