Draft 2 and Ingress with Web Application Routing

If you read the previous article on Draft 2, we went from source code to deployed application in a few steps:

  • az aks draft create: creates a Dockerfile and Kubernetes manifests (deployment and service manifests)
  • az aks draft setup-gh: setup GitHub OIDC
  • az aks draft generate-workflow: create a GitHub workflow that builds and pushes the container image and deploys the application to Kubernetes

If you answer the questions from the commands above correctly, you should be up and running fairly quickly! 🚀

The manifests default to a Kubernetes service that uses the type LoadBalancer to configure an Azure public load balancer to access your app. But maybe you want to test your app with TLS and you do not want to configure a certificate in your container image? That is where the ingress configuration comes in.

You will need to do two things:

  • Configure web application routing: configures Ingress Nginx Controller and relies on Open Service Mesh (OSM) and the Secret Store CSI Driver for Azure Key Vault. That way, you are shielded from having to do all that yourself. I did have some issues with web application routing as described below.
  • Use az aks draft update to configure the your service to work with web application routing; this command will ask you for two things:
    • the hostname for your service: you decide this but the name should resolve to the public IP of the Nginx Ingress Controller installed by web application routing
    • a URI to a certificate on Azure Key Vault: you will need to deploy a Key Vault and upload or create the certificate

Configure web application routing

Although it should be supported, I could not enable the add-on on one of my existing clusters. On another one, it did work. I decided to create a new cluster with the add-on by running the following command:

az aks create --resource-group myResourceGroup --name myAKSCluster --enable-addons web_application_routing

⚠️ Make sure you use the most recent version of the Azure CLI aks-preview extension.

On my cluster, that gave me a namespace app-routing-system with two pods:

Nginx in app-routing-system

Although the add-on should also install Secrets Store CSI Driver, Open Service Mesh, and External DNS, that did not happen in my case. I installed the first two from the portal. I did not bother installing External DNS.

Enabling OSM
Enabling secret store CSI driver

Create a certificate

I created a Key Vault in the same resource group as my AKS cluster. I configured the access policies to use Azure RBAC (role-based access control). It did not work with the traditional access policies. I granted myself and the identity used by web application routing full access:

Key Vault Administrator for myself and the user-assigned managed id of web app routing add-on

You need to grant the user-assigned managed identity of web application routing access because a SecretProviderClass will be created automatically for that identity. The Secret Store CSI Driver uses that SecretProviderClass to grab a certificate from Key Vault and generate a Kubernetes secret for it. The secret will later be used by the Kubernetes Ingress resource to encrypt HTTP traffic. How you link the Ingress resource to the certificate is for a later step.

Now, in Key Vault, generate a certificate:

In Key Vault, click Certificates and create a new one

Above, I use nip.io with the IP address of the Ingress Controller to generate a name that resolves to the IP. For example, 10.2.3.4.nip.io will resolve to 10.2.3.4. Try it with ping. It’s truly a handy service. Use kubectl get svc -n app-routing-system to find the Ingress Controller public (external) IP.

Now we have everything in place for draft to modify our Kubernetes service to use the ingress controller and certificate.

Using az aks draft update

Back on your machine, in the repo that you used in the previous article, run az aks draft update. You will be asked two questions:

  • Hostname: use <IP Address of Nginx>.nip.io (same as in the common name of the cert without CN=)
  • URI to the certificate in Key Vault: you can find the URI in the properties of the certificate
There will be a copy button at the right of the certificate identifier

Draft will now update your service to something like:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubernetes.azure.com/ingress-host: IPADDRESS.nip.io
    kubernetes.azure.com/tls-cert-keyvault-uri: https://kvdraft.vault.azure.net/certificates/mycert/IDENTIFIER
  creationTimestamp: null
  name: super-api
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: super-api
  type: ClusterIP
status:
  loadBalancer: {}

The service type is now ClusterIP. The annotations will be used for several things:

  • to create a placeholder deployment that mounts the certificate from Key Vault in a volume AND creates a secret from the certificate; the Secret Store CSI Driver always needs to mount secrets and certs in a volume; rather than using your application pod, they use a placeholder pod to create the secret
  • to create an Ingress resource that routes to the service and uses the certificate in the secret created via the placeholder pod
  • to create an IngressBackend resource in Open Service Mesh

In my default namespace, I see two pods after deployment:

the placeholder pod starts with keyvault and creates the secret; the other pod is my app

Note that above, I actually used a Helm deployment instead of a manifest-based deployment. That’s why you see release-name in the pod names.

The placeholder pod creates a csi volume that uses a SecretProviderClass to mount the certificate:

SecretProviderClass

The SecretProviderClass references your Key Vault and managed identity to access the Key Vault:

spec of SecretProviderClass

If you have not assigned the correct access policy on Key Vault for the userAssignedIdentityID, the certificate cannot be retrieved and the pod will not start. The secret will not be created either.

I also have a secret with the cert inside:

Secret created by Secret Store CSI Driver; referenced by the Ingress

And here is the Ingress:

Ingress; note it says 8080 instead of the service port 80; do not change it! Never mind the app. in front of the IP; your config will not have that if you followed the instructions

All of this gets created for you but only after running az aks draft update and when you commit the changes to GitHub, triggering the workflow.

Did all this work smoothly from the first time?

The short answer is NO! 😉At first I thought Draft would take care of installing the Ingress components for me. That is not the case. You need to install and configure web application routing on your cluster and configure the necessary access rights.

I also thought web application routing would install and configure Open Service Mesh and Secret Store CSI driver. That did not happen although that is easily fixed by installing them yourself.

I thought there would be some help with certificate generation. That is not the case. Generating a self-signed certificate with Key Vault is easy enough though.

Once you have web application routing installed and you have a Key Vault and certificate, it is simple to run az aks draft update. That changes your Kubernetes service definition. After pushing that change to your repo, the updated service with the web application routing annotations can be deployed.

I got some 502 Bad Gateway errors from Nginx at first. I removed the OSM-related annotations from the Ingress object and tried some other things. Finally, I just redeployed the entire app and then it just started working. I did not spend more time trying to find out why it did not work from the start. The fact that Open Service Mesh is used, which has extra configuration like IngressBackends, will complicate troubleshooting somewhat. Especially if you have never worked with OSM, which is what I expect for most people.

Conclusion

Although this looks promising, it’s all still a bit rough around the edges. Adding OSM to the mix makes things somewhat more complicated.

Remember that all of this is in preview and we are meant to test drive it and provide feedback. However, I fear that, because of the complexity of Kubernetes, these tools will never truly make it super simple to get started as a developer. It’s just a tough nut to crack!

My own point of view here is that Draft v2 without az aks draft update is very useful. In most cases though, it’s enough to use standard Kubernetes services. And if you do need an ingress controller, most are easy to install and configure, even with TLS.

Trying out Draft 2 on AKS

Sadly no post about good Belgian beer 🍺.

Draft 2 is an open-source project that aims 🎯 to make things easier for developers that build Kubernetes applications. It can improve the inner dev loop, where the developers code and test their apps, in the following ways:

  • Automate the creation of a Dockerfile
  • Automate the creation of Kubernetes manifests, Helm charts, or Kustomize configs
  • Generate a GitHub Action workflow to build and deploy the application when you push changes

I have worked with Draft 1 in the past, and it worked quite well. Now Microsoft has integrated Draft 2 in the Azure CLI to make it part of the Kubernetes on Azure experience. A big difference with Draft 1 is that Draft 2 makes use of GitHub Actions (Wait? No Azure DevOps? 😲) to build and push your images to the development cluster. It uses GitHub OpenID Connect (OIDC) for Azure authentication.

That is quite a change and lots of bits and pieces that have to be just right. Make sure you know about Azure AD App Registrations, GitHub, GitHub Actions, Docker, etc… when the time comes to troubleshoot.

Let’s see what we can do? 👀

Prerequisites

At this point in time (June 2022), Draft for Azure Kubernetes Service is in preview. Draft itself can be found here: https://github.com/Azure/draft

The only thing you need to do is to install or upgrade the aks-preview extension:

az extension add --name aks-preview --upgrade

Next, type az aks draft -h to check if the command is available. You should see the following options:

create           
generate-workflow
setup-gh
up
update

We will look at the first four commands in this post.

Running draft create

With az aks draft create, you can generate a Dockerfile for your app, Kubernetes manifests, Helm charts, and Kustomize configurations. You should fork the following repository and clone it to your machine: https://github.com/gbaeke/draft-super

After cloning it, cd into draft-super and run the following command (requires go version 1.16.4 or higher):

CGO_ENABLED=0 go build -installsuffix 'static' -o app cmd/app/*
./app

The executable runs a web server on port 8080 by default. If that conflicts with another app on your system set the port with the port environment variable: run PORT=9999 ./app instead of just ./app. Now we know the app works, we need a Dockerfile to containerize it.

You will notice that there is no Dockerfile. Although you could create one manually, you can use draft for this. Draft will try to recognize your code and generate the Dockerfile. We will keep it simple and just create Kubernetes manifests. When you run draft without parameters, it will ask you what you want to create. You can also use parameters to specify what you want, like a Helm chart or Kustomize configs. Run the command below:

az aks draft create

The above command will download the draft CLI for your platform and run it for you. It will ask several questions and display what it is doing.

[Draft] --- Detecting Language ---
✔ yes
[Draft] --> Draft detected Go Checksums (72.289458%)

[Draft] --> Could not find a pack for Go Checksums. Trying to find the next likely language match...
[Draft] --> Draft detected Go (23.101180%)

[Draft] --- Dockerfile Creation ---
Please Enter the port exposed in the application: 8080
[Draft] --> Creating Dockerfile...

[Draft] --- Deployment File Creation ---
✔ manifests
Please Enter the port exposed in the application: 8080
Please Enter the name of the application: super-api
[Draft] --> Creating manifests Kubernetes resources...

[Draft] Draft has successfully created deployment resources for your project 😃
[Draft] Use 'draft setup-gh' to set up Github OIDC.

In your folder you will now see extra files and folders:

  • A manifests folder with two files: deployment.yaml and service.yaml
  • A Dockerfile

The manifests are pretty basic and just get things done:

  • create a Kubernetes deployment that deploys 1 pod
  • create a Kubernetes service of type LoadBalancer; that gives you a public IP to reach the app

The app name and port you specified after running az aks draft create is used to create the deployment and service.

The Dockerfile looks like the one below:

FROM golang
ENV PORT 8080
EXPOSE 8080

WORKDIR /go/src/app
COPY . .

RUN go mod vendor
RUN go build -v -o app  
RUN mv ./app /go/bin/

CMD ["app"]

This is not terribly optimized but it gets the job done. I would highly recommend using a two-stage Dockerfile that results in a much smaller image based on alpine, scratch, or distroless (depending on your programming language).

For my code, the Dockerfile will not work because the source files are not in the root of the repo. Draft cannot know everything. Replace the line that says RUN go build -v -o app with RUN CGO_ENABLED=0 go build -installsuffix 'static' -o app cmd/app/*

To check that the Dockerfile works, if you have Docker installed, run docker build -t draft-super . It will take some time for the base Golang image to be pulled and to download all the dependencies of the app.

When the build is finished, run docker run draft-super to check. The container should run properly.

The az aks draft create command did a pretty good job detecting the programming language and creating the Dockerfile. As we have seen, minor adjustments might be required and the Dockerfile will probably not be production-level quality.

GitHub OIDC setup

At the end of the create command, draft suggested using setup-gh to setup GitHub. Let’s run that command:

az aks draft setup-gh

Draft will ask for the name of an Azure AD app registration to create. Make sure you are allowed to create those. I used draft-super for the name. Draft will also ask you to confirm the Azure subscription ID and a name of a resource group.

⚠️Although not entirely clear from the question, use the resource group of your AKS cluster (not the MC_ group that contains your nodes!). The setup-gh command will grant the service principal that it creates the Contributor role on the group. This ensures that the GitHub Action azure/aks-set-context@v2.0 works.

Next, draft will ask for the GitHub organization and repo. In my case, that was gbaeke/draft-super. Make sure you have admin access to the repo. GitHub secrets will need to be created. When completed, you should see something like below:

Enter app registration name: draft-super
✔ <YOUR SUB ID>
Enter resource group name: rg-aks
✔ Enter github organization and repo (organization/repoName): gbaeke/draft-super█
[Draft] Draft has successfully set up Github OIDC for your project 😃
[Draft] Use 'draft generate-workflow' to generate a Github workflow to build and deploy an application on AKS.

Draft has done several things:

  • created an app registration (check Azure AD)
  • the app registration has federated credentials configured to allow a GitHub workflow to request an Azure AD token when you do pull requests, or push to main or master
  • secrets in your GitHub repo:AZURE_CLIENT_ID, AZURE_SUBSCRIPTION_ID,AZURE_TENANT_ID; these secrets are used by the workflow to request a token from Azure AD using federated credentials
  • granted the app registration contributor role on the resource group that you specified; that is why you should use the resource group of AKS!

The GitHub workflow you will create in the next step will use the OIDC configuration to request an Azure AD token. The main advantage of this is that you do not need to store Azure secrets in GitHub. The action that does the OIDC-based login is azure/login@v1.4.3.

Draft is now ready to create a GitHub workflow.

Creating the GitHub workflow

Use az aks draft generate-workflow to create the workflow file. This workflow needs the following information as shown below:

Please enter container registry name: draftsuper767
✔ Please enter container name: draft-super█
Please enter cluster resource group name: rg-aks
Please enter AKS cluster name: clu-git
Please enter name of the repository branch to deploy from, usually main: master
[Draft] --> Generating Github workflow
[Draft] Draft has successfully generated a Github workflow for your project 😃

⚠️ Important: use the short name of ACR. Do not append azurecr.io!

⚠️ The container registry needs to be created. Draft does not do that. For best results, create the ACR in the resource group of the AKS cluster because that ensures the service principal created earlier has access to ACR to build images and to enable admin access.

Draft has now created the workflow. As expected, it lives in the .github/workflows local folder.

The workflow runs the following actions:

  • Login to Azure using only the client, subscription, and tenant id. No secrets required! 👏 OIDC in action here!
  • Run az acr build to build the container image. The image is not built on the GitHub runner. The workflow expects ACR to be in the AKS resource group.
  • Get a Kubernetes context to our AKS cluster and create a secret to allow pulling from ACR; it will also enable the admin user on ACR
  • Deploy the application with the Azure/k8s-deploy@v3.1 action. It uses the manifests that were generated with az aks draft create but modifies the image and tag to match the newly built image.

Now it is time to commit our code and check the workflow result:

Looks fine at first glance…

Houston, we have a problem 🚀

For this blog post, I was working in a branch called draft, not main or master. I also changed the workflow file to run on pushes to the draft branch. Of course, the federated tokens in our app registration are not configured for that branch, only master and main. You have to be specific here or you will not get a token. This is the error on GitHub:

Oops

To fix this, just modify the app registration and run the workflow again:

Quick and dirty fix: update mainfic with a subject identifier for draft; you can also add a new credential

After running the workflow again, if buildImage fails, check that ACR is in the AKS resource group and that the service principal has Contributor access to the group. I ran az role assignment list -g rg-aks to see the directly assigned roles and checked that the principalName matched the client ID (application ID) of the draft-super app registration.

If you used the FQDN of ACR instead of just the short name. you can update the workflow environment variable accordingly:

ACR name should be the short name

After this change, the image build should be successful.

Looking better

If you used the wrong ACR name, the deploy step will fail. The image property in deployment.yaml will be wrong. Make the following change in deployment.yaml:

- image: draftsuper767.azurecr.io.azurecr.io/draft-super

to

- image: draftsuper767.azurecr.io/draft-super

Commit to re-run the workflow. You might need to cancel the previous one because it uses kubectl rollout to check the health of the deployment.

And finally, we have a winner…

🍾🍾🍾

In k9s:

super-api deployed to default namespace

You can now make changes to your app and commit your changes to GitHub to deploy new versions or iterations of your app. Note that any change will result in a new image build.

What about the az aks draft up command? It simply combines the setup of GitHub OIDC and the creation of the workflow. So basically, all you ever need to do is:

  • create a resource group
  • deploy AKS to the resource group
  • deploy ACR to the resource group
  • Optionally run az aks update -n -g --attach-acr (this gives the kubelet on each node access to ACR; as we have seen, draft can also create a pull secret)
  • run az aks draft create followed by az aks draft up

Conclusion

When working with Draft 2, ensure you first deploy an AKS cluster and Azure Container Registry in the same resource group. You need the Owner role because you will change role-based access control settings.

During OIDC setup, when asked for a resource group, type the AKS resource group. Draft will ensure the service principal it creates, has proper access to the resource group. With that access, it will interact with ACR and log on to AKS.

When asked for the ACR name, use the short name. Do not append azurecr.io! From that point on, it should be smooth sailing! ⛵

In a follow-up post, we will take a look at the draft update command.

From MQTT to InfluxDB with Dapr

In a previous post, we looked at using the Dapr InfluxDB component to write data to InfluxDB Cloud. In this post, we will take a look at reading data from an MQTT topic and storing it in InfluxDB. We will use Dapr 0.10, which includes both components.

To get up to speed with Dapr, please read the previous post and make sure you have an InfluxDB instance up and running in the cloud.

If you want to see a video instead:

MQTT to Influx with Dapr

Note that the video sends output to both InfluxDB and Azure SignalR. In addition, the video uses Dapr 0.8 with a custom compiled Dapr because I was still developing and testing the InfluxDB component.

MQTT Server

Although there are cloud-based MQTT servers you can use, let’s mix it up a little and run the MQTT server from Docker. If you have Docker installed, type the following:

docker run -it -p 1883:1883 -p 9001:9001 eclipse-mosquitto

The above command runs Mosquitto and exposes port 1883 on your local machine. You can use a tool such as MQTT Explorer to send data. Install MQTT Explorer on your local machine and run it. Create a connection like in the below screenshot:

MQTT Explorer connection

Now, click Connect to connect to Mosquitto. With MQTT, you send data to topics of your choice. Publish a json message to a topic called test as shown below:

Publish json data to the test topic

You can now click the topic in the list of topics and see its most recent value:

Subscribing to the test topic

Using MQTT with Dapr

You are now ready to read data from an MQTT topic with Dapr. If you have Dapr installed, you can run the following code to read from the test topic and store the data in InfluxDB:

const express = require('express');
const bodyParser = require('body-parser');

const app = express();
app.use(bodyParser.json());

const port = 3000;

// mqtt component will post messages from influx topic here
app.post('/mqtt', (req, res) => {
    console.log("MQTT Binding Trigger");
    console.log(req.body)

    // body is expected to contain room and temperature
    room = req.body.room
    temperature = req.body.temperature

    // room should not contain spaces
    room = room.split(" ").join("_")

    // create message for influx component
    message = {
        "measurement": "stat",
        "tags": `room=${room}`,
        "values": `temperature=${temperature}`
    };
    
    // send the message to influx output binding
    res.send({
        "to": ["influx"],
        "data": message
    });
});

app.listen(port, () => console.log(`Node App listening on port ${port}!`));

In this example, we use Node.js instead of Python to illustrate that Dapr works with any language. You will also need this package.json and run npm install:

{
  "name": "mqttapp",
  "version": "1.0.0",
  "description": "",
  "main": "app.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "body-parser": "^1.18.3",
    "express": "^4.16.4"
  }
}

In the previous post about InfluxDB, we used an output binding. You use an output binding by posting data to a Dapr HTTP URI.

To use an input binding like MQTT, you will need to create an HTTP server. Above, we create an HTTP server with Express, and listen on port 3000 for incoming requests. Later, we will instruct Dapr to listen for messages on an MQTT topic and, when a message arrives, post it to our server. We can then retrieve the message from the request body.

To tell Dapr what to do, we’ll create a components folder in the same folder that holds the Node.js code. Put a file in that folder with the following contents:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: mqtt
spec:
  type: bindings.mqtt
  metadata:
  - name: url
    value: mqtt://localhost:1883
  - name: topic
    value: test

Above, we configure the MQTT component to list to topic test on mqtt://localhost:1883. The name we use (in metadata) is important because that needs to correspond to our HTTP handler (/mqtt).

Like in the previous post, there’s another file that configures the InfluxDB component:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: influx
spec:
  type: bindings.influx
  metadata:
  - name: Url
    value: http://localhost:9999
  - name: Token
    value: ""
  - name: Org
    value: ""
  - name: Bucket
    value: ""

Replace the parameters in the file above with your own.

Saving the MQTT request body to InfluxDB

If you look at the Node.js code, you have probably noticed that we send a response body in the /mqtt handler:

res.send({
        "to": ["influx"],
        "data": message
    });

Dapr is written to accept responses that include a to and a data field in the JSON response. The above response simply tells Dapr to send the message in the data field to the configured influx component.

Does it work?

Let’s run the code with Dapr to see if it works:

dapr run --app-id mqqtinflux --app-port 3000 --components-path=./components node app.js

In dapr run, we also need to specify the port our app uses. Remember that Dapr will post JSON data to our /mqtt handler!

Let’s post some JSON with the expected fields of temperature and room to our MQTT server:

Posting data to the test topic

The Dapr logs show the following:

Logs from the APP (appear alongside the Dapr logs)

In InfluxDB Cloud table view:

Data stored in InfluxDB Cloud (posted some other data points before)

Conclusion

Dapr makes it really easy to retrieve data with input bindings and send that data somewhere else with output bindings. There are many other input and output bindings so make sure you check them out on GitHub!

%d bloggers like this: