Azure Private Link and DNS

When you are just starting out with Azure Private Link, it can be hard figuring out how name resolution works and how DNS has to be configured. In this post, we will take a look at some of the internals and try to clear up some of the confusion. If you end up even more confused then I’m sorry in advance. Drop me your questions in the comments if that happens. 😉 I will illustrate the inner workings with a Cosmos DB account. It is similar for other services.

Wait! What is Private Link?

Azure Private Link provides private IP addresses for services such as Cosmos DB, Azure SQL Database and many more. You choose where the private IP address comes from by specifying a VNET and subnet. Without private link, these services are normally accessed via a public IP address or via Network Service Endpoints (also the public IP but over the Azure network and restricted to selected subnets). There are several issues or shortcomings with those options:

• for most customers, accessing databases and other services over the public Internet is just not acceptable
• although network service endpoints provide a solution, this only works for systems that run inside an Azure Virtual Network (VNET)

When you want to access a service like Cosmos DB from on-premises networks and keep the traffic limited to your on-premises networks and Azure virtual networks, Azure Private Link is the way to go. In addition, you can filter the traffic with Azure Firewall or a virtual appliance, typically installed in a hub site. Now let’s take a look at how this works with Cosmos DB.

Azure Private Link for Cosmos DB

I deployed a Cosmos DB account in East US and called it geba-cosmos. To access this account and work with collections, I can use the following name: https://geba-cosmos.documents.azure.com:443/. As explained before, geba-cosmos.document.azure.com resolves to a public IP address. Note that you can still control who can connect to this public IP address. Below, only my home IP address is allowed to connect:

In order to connect to Cosmos DB using a private IP address in your Azure Virtual Network, just click Private Endpoint Connections below Firewall and virtual networks:

To create a new private endpoint, click + Private Endpoint and follow the steps. The private endpoint is a resource on its own which needs a name and region. It should be in the same region as the virtual network you want to grab an IP address from. In the second screen, you can select the resource you want the private IP to point to (can be in a different region):

In the next step, you select the virtual network and subnet you want to grab an IP address from:

In this third step (Configuration), you will be asked if you want Private DNS integration. The default is Yes but I will select No for now.

Note: it is not required to use a Private DNS zone with Private Link

When you finish the wizard and look at the created private endpoint, it will look similar to the screenshot below:

In the background, a network interface was created and attached to the selected virtual network. Above, the network interface is pe-geba-cosmos.nic.a755f7ad-9d54-4074-996c-8a14e9434898. The network interface screen will look like the screenshot below:

The interesting part is the Custom DNS Settings. How can you resolve the name geba-cosmos.documents.azure.com to 10.1.0.5 when a client (either in Azure or on-premises) requests it? Let’s look at DNS resolution next…

DNS Resolution

Let’s use dig to check what a request for a Cosmos DB account return without private link. I have another account, geba-test, that I can use for that:

The above DNS request was made on my local machine, using public DNS servers. The response from Microsoft DNS servers for geba-test.documents.azure.com is a CNAME to a cloudapp.net name which results in IP address 40.78.226.8.

The response from the DNS server will be different when private link is configured. When I resolve geba-cosmos.documents.azure.com, I get the following:

As you can see, the Microsoft DNS servers respond with a CNAME of accountname.privatelink.documents.azure.com. but by default that CNAME goes to a cloudapp.net name that resolves to a public IP.

This means that, if you don’t take specific action to resolve accountname.privatelink.documents.azure.com to the private IP, you will just end up with the public IP address. In most cases, you will not be able to connect because you will restrict public access to Cosmos DB. It’s important to note that you do not have to restrict public access and that you can enable both private and public access. Most customers I work with though, restrict public access.

Resolving to the private IP address

Before continuing, it’s important to state that developers should connect to https://accountname.documents.azure.com (if they use the gateway mode). In fact, Cosmos DB expects you to use that name. Don’t try to connect with the IP address or some other name because it will not work. This is similar for services other than Cosmos DB. In the background though, we will make sure that accountname.documents.azure.com goes to the internal IP. So how do we make that happen? In what follows, I will list a couple of solutions. I will not discuss using a hosts file on your local pc, although it is possible to make that work.

Create privatelink DNS zones on your DNS servers
Note: this approach is not recommended; it can present problems later when you need to connect to private link enabled services that are not under your control

This means that in this case, we create a zone for privatelink.documents.azure.com on our own DNS servers and add the following records:

• geba-cosmos.privatelink.documents.azure.com. IN A 10.1.0.5
• geba-cosmos-eastus.privatelink.documents.azure.com. IN A 10.1.0.6

Note: use a low TTL like 10s (similar to Azure Private DNS; see below)

When the DNS server has to resolve geba-cosmos.documents.azure.com, it will get the CNAME response of geba-cosmos.privatelink.documents.azure.com and will be able to answer authoritatively that that is 10.1.0.5.

If you use this solution, you need to make sure that you register the custom DNS settings listed by the private endpoint resource manually. If you want to try this yourself, you can easily do this with a Windows virtual machine with the DNS role or a Linux VM with bind.

Use Azure Private DNS zones
If you do not want to register the custom DNS settings of the private endpoint manually in your own DNS servers, you can use Azure Private DNS. You can create the private DNS zone during the creation of the private endpoint. An internal zone for privatelink.documents.azure.com will be created and Azure will automatically add the required DNS configuration the private endpoint requires:

This is great for systems running in Azure virtual networks that are associated with the private DNS zone and that use the DNS servers provided by Azure but you still need to integrate your on-premises DNS servers with these private DNS zones. The way to do that is explained in the documentation. In particular, the below diagram is important:

The example above is for Azure SQL Database but it is similar to our Cosmos DB example. In essence, you need the following:

• DNS forwarder in the VNET (above, that is 10.5.0.254): this is an extra (!!!) Windows or Linux VM configured as a DNS forwarder; it should forward to 168.63.129.16 which points to the Azure-provided DNS servers; if the virtual network of the VM is integrated with the private DNS zone that hosts privatelink.documents.azure.com, the A records in that zone can be resolved properly
• To allow the on-premises server to return the privatelink A records, setup conditional forwarding for documents.azure.com to the DNS forwarder in the virtual network

What should you do?

That’s always difficult to answer but most customers I work with initially tend to go for option 1. They want to create a zone for privatelink.x.y.z and register the records manually. Although that could be automated, it’s often a manual step. In general, I do not recommend using it.

I prefer the private DNS method because of the automatic registration of the records and the use of conditional forwarding. Although I don’t like the extra DNS servers, they will not be needed most of the time because customers tend to work with the hub/spoke model and the hub already contains DNS servers. Those DNS servers can then be configured to enable the resolution of the privatelink zones via forwarding.