Azure Kubernetes Service and Azure Private Link Integration

If you have done any work with Azure, you have probably come across terms such as Azure Private Link Service (PLS) and Private Endpoints (PEs). To quickly illustrate what Azure PLS is, let’s look at a diagram from the Microsoft documentation for Azure SQL database:

PLS with Azure SQL

Above, Azure SQL Database uses Azure Private Link Service (PLS) to provide connectivity to the database from inside a virtual network that you control. Without a private link, you would need to connect to Azure SQL via a public IP address over the Internet. In order to connect privately, a private endpoint connection (PE) is created inside a subnet in your virtual network. Above, that interface gets IP address 10.0.0.5. The PE can be seen as a network interface that is connected to Azure SQL Database via Azure PLS. The green arrow from the PE to Azure SQL Database can be seen as the private connection.

Azure SQL Database is not the only service offering this functionality. For example, when you deploy Azure Kubernetes Service (AKS) with a private Kubernetes API service, a private endpoint connection is created to access the Kubernetes control plane via Azure PLS.

When you go to Private Link Center in the Azure Portal, you can see all your private endpoints and their connection state. Below, a private endpoint for a private AKS cluster is shown. It shows as connected via private link.

Private endpoint to access the Microsoft managed AKS control plane

Creating your own Private link services

In the two examples above, Azure SQL Database and AKS use Azure PLS to enable a private connection. But what if you build your own service and you want to offer private connectivity to consumers such as your customers or other Azure services? That is where the creation of your own private link services comes into play. These services can be created from Private Link Center by enabling private connections to a standard load balancer:

Creating your own private link service

More information about this process can be found in the documentation.

In summary, when you have a standard load balancer that load balances traffic to an application, you can offer a private connection to that load balancer via Azure Private Link Service.

The load balancer can be in front of traditional virtual machines or Kubernetes pods. In the next section, we’ll look at the second scenario: creating a private link service from an internal load balancer (ILB) that AKS creates for a Kubernetes service.

Creating a Private Link Service from an AKS internal load balancer

Although it was technically possible to create a Private Link Service from an internal load balancer controlled by AKS in the past, it was a cumbersome process. In addition, AKS was not aware of the Private Link Service configuration. A new capability in the Azure Cloud Provider changes this.

When you create a Kubernetes service of type LoadBalancer, you can now provide annotations that instruct the AKS Azure Cloud Provider to create a private link service from the internal load balancer it creates. Here’s an example:

apiVersion: v1
kind: Service
metadata:
  name: super-api
  annotations:
    # create ILB instead of ELB; this functionality predates the PLS functionality
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-pls-create: "true"
    service.beta.kubernetes.io/azure-pls-name: myPLS
    service.beta.kubernetes.io/azure-pls-ip-configuration-subnet: YOUR SUBNET
    service.beta.kubernetes.io/azure-pls-ip-configuration-ip-address-count: "1"
    service.beta.kubernetes.io/azure-pls-ip-configuration-ip-address: 10.224.10.10
    service.beta.kubernetes.io/azure-pls-proxy-protocol: "false"
    service.beta.kubernetes.io/azure-pls-visibility: "*"
    # does not apply here because we will use Front Door later
    service.beta.kubernetes.io/azure-pls-auto-approval: "YOUR SUBSCRIPTION ID"
spec:
  selector:
    app: super-api
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080

This works with both Kubenet and the Azure CNI. You can use the subnet that your AKS nodes are in. Above, replace YOUR SUBNET with the name of your subnet, not its resource id.

When the above YAML is submitted to Kubernetes, the private link service myPLS gets created. Record the alias for later use:

Creation of the PLS

Note that the annotation service.beta.kubernetes.io/azure-load-balancer-internal: "true" creates the load balancer in the AKS node resource group.

Note that a private link service also creates a network interface in the subnet for NATting purposes. NAT ensures that the networking configuration of the consumer does not lead to IP address conflicts. The NAT IP above is 10.224.10.10. You can configure multiple NAT IP addresses to avoid port exhaustion.

The PLS will be visible in the Private Link Center without connections. Later, when you add services that use this private link service, the number of connections will be shown as below:

myPLS with one connection (from Azure Front Door, see below 😉)

But what can we connect to this? We already know the answer: a private endpoint. You could create a private endpoint in any network, in any subscription, and link it up to myPLS. In fact, other customers from different Azure AD tenants can use myPLS as well, provided that the usage is approved by you. We will not do that in this example, and instead, wire up Azure Front Door to our AKS service.

Azure Front Door Premium

Azure Front Door Premium supports private endpoints that connect to your own private link services. Those private endpoints are not owned by you but by the Front Door service. You will not be able to see those private endpoints in your subscription(s) because they do not live there. It’s as if someone from another organization and tenant connects to your private link service. In this case, that other organization is Microsoft! 😉

With the configuration of Front Door, we get the full picture below:

AKS service via ILB with PLS consumed by Front Door Premium Private Endpoint 🧠

The configuration of the private endpoint and wiring it up to your private link service is done in the origin group configuration, as shown above. When you add an origin to the origin group, one of the options is to connect to a private link service. Below, you see an already configured origin group:

Origin group with a private link service origin

Above, the origin host name is the alias of the private link service created earlier (myPLS).

Here’s a screenshot of the Add an origin UI:

Adding an origin using private link service

The Origin type should be custom, and the Host name should be the private link service alias. Then, you can check Enable private link service and select the private link that was created by AKS based on the service annotations.

Remember that you will still have to approve the usage of the private link service by Azure Front Door! Check Pending Connections in Private Link Center.

Does it work?

In Front Door manager, you should have an endpoint and a route that uses the origin group. In my case, that is aksdemo-agfcfwgkgyctgyhs.z01.azurefd.net. The AKS service publishes a deployment of ghcr.io/gbaeke/super:1.0.7 which just prints Hello from Super API:

Tadaaa, it works!

Conclusion

This new feature makes it super easy to create Azure Private Link Services from internal load balancers created by AKS. Combined with Azure Front Door Premium, you can publish these services to the Internet without having to provide public connectivity at the AKS level. In addition, you can enable other Front Door features such as WAF (web application firewall). Maybe in the future, we’ll see some extra integration with Azure Front Door so it can act as an AKS Ingress Controller, all controlled from Kubernetes manifests? 😉

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: