Month: October 2020

I often work with customers that build web applications on cloud platforms like Azure, AWS or Digital Ocean. The web application is usually built by a third party that specializes in e-Commerce, logistics or industrial applications in a wide range of industries. More often than not, these applications use CloudFlare for DNS, caching, and security.

In this post, we will take a look at such a case with the application running in containers on Azure Kubernetes Service (AKS). I have substituted the application with one of my own, the go-realtime app.

There’s also a video:

The big picture

The application runs in containers on an AKS cluster. Although we could expose the application using an Azure load balancer, a layer 7 load balancer such as Azure Application Gateway, referred to as AG below, is more appropriate here because it allows routing based on URLs and paths and much more.

Because Kubernetes is a dynamic environment, a component is required that configures AG automatically. Application Gateway Ingress Controller (AGIC) plays that part. AGIC configures the AG based on the ingresses we create in the cluster. In essence, that will result in a listener on the public IP that is associated with AG.

In Cloudflare, we will need to configure DNS records that use proxying. The records will point to the IP address of the AG. Below is an example of a DNS record with proxying turned on (orange cloud):

Let’s look at these components in a bit more detail.

Application Gateway

Microsoft has a lot of documentation on AG, including the AGIC component. There are many options and approaches when it comes to using AG together with AKS. Some are listed below:

• Install AKS, AG and AGIC in one step: see the docs for more information; in general, I would not follow this approach and use the next option
• Install AKS and AG separately: you can find an example here; this allows you to deploy AKS and AG (plus its public IP) using your automation tools of choice such as ARM, Terraform or Pulumi

In most cases, we deploy AKS with Azure CNI networking. This requires a virtual network (VNet) with a subnet specifically for your AKS cluster. Only one cluster should be in the subnet.

AG also requires a subnet. You can create that subnet in the same VNet and size it according to the documentation. In virtually all cases, you should go for AG v2.

In the video above, I install AG with Azure CLI. Once AKS and AG are deployed, you will need to deploy the AGIC component.

Application Gateway Ingress Controller

You basically have two options to install AGIC:

• Install via an AKS addon: discussed further
• Install with a Helm chart: see Helm greenfield and Helm brownfield deployment for more information

Although the installation via an AKS addon is preferred, at the time of writing (October 2020), this method is in preview. After configuring your subscription to enable this feature and after installing the aks-preview addon for Azure CLI, you can use the following command to install AGIC:

appgwId=$(az network application-gateway show -n AGname -g AGresourcegroup -o tsv --query "id")
az aks enable-addons -n AKSclustername -g AKSResourcegroup -a ingress-appgw --appgw-id $appgwId

Indeed, you first need to find the id of the AG you deployed. This id can be found in the portal or with the first command above, which saves the result in a variable (Linux shell). The az aks enable-addons command is the command to install any addon in AKS, including the AGIC addon. The AGIC addon is called ingress-appgw.

Installation via the addon is preferred because that makes the AGIC installation part of AKS and part of the managed service for maintenance and upgrades. If you install AGIC via Helm, you are responsible for maintaining and upgrading it. In addition, the Helm deployment requires AAD pod identity, which complicates matters further. From the moment the addon is GA (generally available), I would recommend to use it exclusively, as long as your scenario supports it.

That last sentence is important because there are quite some differences between AGIC installed with Helm and AGIC installed with the addon. Those differences should disappear over time though.

Required access rights for AGIC

AGIC configures AG via ARM (Azure Resource Manager). As such, AGIC requires read and write access to AG. To check if AGIC has the correct access, use the AGIC pod logs to do so.

Indeed, the AGIC installation results in a pod in the kube-system namespace. On my system, it looks like this (from kubectl get pods -n kube-system)

ingress-appgw-deployment-7dd969fddb-jfps5 1/1 Running 0 6h50m

When you check the logs of that pod, you should see output like below:

The logs show that AGIC can connect to AG properly. If however, you get 403 errors, AGIC does not have the correct access rights. That can easily be fixed by granting the Contributor role on your AG to the user managed identity used by AGIC (if the AKS addon was used). In my case, that is the following account:

Configuring AG via Ingresses

Now that AG and AGIC are installed and AGIC has read and write access to AG, we can created Kubernetes Ingress objects like we usually do. Below is an example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: realtimeapp-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: "origin"
    

spec:
  rules:
  - host: rt.baeke.info
    http:
      paths:
      - path: /
        backend:
          serviceName: realtimeapp
          servicePort: 80

This is a regular Ingress definition. The ingress.class annotation is super important because it tells AGIC to do its job. The second annotation is part of our use case because we want AG to create an HTTPS listener and we want to use a certificate that is already installed on AG. That certificate contains a Cloudflare origin certificate valid for *.baeke.info and expiring somewhere in 2035. I must make sure I update that certificate at that time! 😉

Note that this is just one way of configuring the certificate. You can also save the certificate as a Kubernetes secret and refer to it in your Ingress definition. AGIC will then push that certificate to AG. AGIC also supports Let’s Encrypt, with some help from certmgr. I will let you have some fun with that though! Tell me how it went!

From the moment we create the Ingress, AGIC will pick it up and configure AG. Here’s the listener for instance:

By the way, to create the certificate in AG, use the command below with a cert.pfx file containing the certificate and private key in the same folder:

az network application-gateway ssl-cert create -g resourcegroupname --gateway-name AGname -n origin --cert-file cert.pfx --cert-password SomePassword123

Of course, you can choose any name you like for the -n parameter.

CloudFlare Configuration

As mentioned before, you need to create proxied A or CNAME records. The user connection will go to Cloudflare, Cloudflare will do its thing and then connect to the public IP of AG, returning the results to the user.

To enforce end-to-end encryption, set the mode to Full (strict):

As your Edge certificate (used at the Cloudflare edge locations), you have several options. One of those options is to use a CloudFlare Universal SSL Certificate which is free. Another option is to use the Advanced Certificate Manager which comes at an extra cost. On higher plans, you can upload your own certificates. In my case, I have Universal SSL applied but we mostly use the other two options in production scenarios:

Via the edge certificate, users can connect securely to a Cloudflare edge location. Cloudflare itself, needs to connect securely to AG. We now need to generate an origin certificate we can install on AG:

The questions that follow are straightforward and not discussed here. Here they are:

After clicking next, you will get your certificate and private key in PEM format (default), which you can use to create the .pfx file. You can use the openssl tool as discussed here. Just copy and paste the certificate and private key to separate text files, for example cert.pem and cert.key, and use them as input to the openssl command. Once you have the .pfx file, use the command shown earlier to upload it to AG.

In the Edge Certificates section, it is recommended to also enable Always use HTTPS which redirects HTTP to HTTPS traffic.

Restricting AG to Cloudflare traffic

Application Gateway v2 is automatically deployed with a public IP. You can restrict access to that IP address with an NSG.

It is important to understand how the NSG works before you start creating it. The documentation provides all the information you need, but be aware the steps are different for AG v2 compared to AG v1.

Here is a screenshot of my NSG inbound rules, outbound rules were left at the default:

Note that the second rule only allows access on port 443 from Cloudflare addresses as found here.

Let’s check if only Cloudflare has access with curl. First I run the following command without the NSG applied:

curl --header "Host: rt.baeke.info" https://52.224.72.167 --insecure

The above command responds with:

When I apply the NSG and give it some time, the curl command times out.

Conclusion

In this post, we looked at using Application Gateway Ingress Controller, which configures Application Gateway based on Kubernetes Ingress definitions. We have also looked at combining Application Gateway with Cloudflare, by using Cloudflare proxying in combination with an Azure Network Security Group that only allows access to Application Gateway from well-known IP addresses. I hoped you liked this and if you have any remarks or spotted errors, let me know!

Recently (October, 2020) I posted an introduction to HashiCorp Waypoint on my YouTube channel. It shows how to build, push, deploy and release applications to Kubernetes with a single waypoint up command. If you want to check out that video first, see below ⬇⬇⬇

After watching that video, it should be clear that you drive the process from a file called waypoint.hcl. The waypoint.hcl to deploy the Azure Function app in the video, is shown below:

project = "wptest-hello"

app "wptest-hello" {
  labels = {
    "service" = "wptest-hello",
    "env" = "dev"
  }

  build {
    use "docker" {}
    registry {
        use "docker" {
          image = "gbaeke/wptest-hello"
          tag = "latest"
          local = false
        }
    }
  }

  deploy {
    use "kubernetes" {
        service_port = 80
        probe_path = "/"
    }
  }

  release {
    use "kubernetes" {
       load_balancer =  true
    }
  }
}

In the build stanza, use “docker” tells Waypoint to build the container image from a local Dockerfile. With registry, we push that image to, in this case, Docker Hub. Instead of Docker Hub, other registries can be used as well. Before the image is pushed to the registry, it is first tagged with the tag you specify. Here, that is the latest tag. Although that is easy, you should not use that tag in your workflow because you will not get different images per application version. And you certainly want that when you do multiple deploys based on different code.

To make the tag unique, you can replace “latest” with the gitrefpretty() function, as shown below:

build {
    use "docker" {}
    registry {
        use "docker" {
          image = "gbaeke/wptest-hello"
          tag = gitrefpretty()
          local = false
        }
    }
  }

Assuming you work with git and commit your code changes 😉, gitrefpretty() will return the git commit sha at the time of build.

You can check the commit sha of each commit with git log:

When you use gitrefpretty() and you issue the waypoint build command, the images will be tagged with the sha-1 checksum. In Docker Hub, that is clearly shown:

That’s it for this quick post. If you have further questions, just hit me up on Twitter or leave a comment!