Image Source: crossplane.io
Although Crossplane has been around for a while, I never got around to trying it. Crossplane has many capabilities. However, in this post, I will focus on the following aspects:
- Installing Crossplane on a Kubernetes cluster (AKS); you can install on a local cluster as well (e.g., k3s, kind, minikube, …) but then you would need Azure Arc for Kubernetes to install the microsoft.flux extension (I will be using GitOps with Flux via that extension)
- Adding and configuring providers for Azure and Kubernetes: providers allow you to deploy to Azure and Kubernetes (and much more) from Crossplane
- Deploying Azure infrastructure with Crossplane using a fully declarative GitOps approach
Introduction
Crossplane basically allows you to build a control plane that you or your teams can use to deploy infrastructure and applications. This control plane is built on Kubernetes. In short, suppose I want to deploy an Azure resource group with Crossplane, I would create the below YAML file and apply it with kubectl apply -f filename.yaml.
This is, in essence, a fully declarative approach to deploying Azure infrastructure using Kubernetes. There are other projects, such as the Azure Service Operator v2, that do something similar.
apiVersion: azure.jet.crossplane.io/v1alpha2
kind: ResourceGroup
metadata:
name: rg-crossplane
spec:
forProvider:
location: "westeurope"
tags:
provisioner: crossplane
providerConfigRef:
name: default
In order to enable this functionality, you need the following:
- Install Crossplane on your Kubernetes cluster
- Add a provider that can create Azure resources; above the jet provider for Azure is used; more about providers later
- Configure the provider with credentials; in this case Azure credentials
In a diagram:
Combination with GitOps
Although you can install and configure Crossplane manually and just use kubectl to add custom resources, I wanted to add Crossplane and custom resources using GitOps. To that end, I am using Azure Kubernetes Service (AKS) with the microsoft.flux extension. For more information to enable and install the extension, see my Flux v2 quick guide.
⚠️ The git repository I am using with Flux v2 and Crossplane is here: https://github.com/gbaeke/crossplane/tree/blogpost. This refers to the blogpost branch, which should match the content of this post. Tbe main branch might be different.
The repo contains several folders that match Flux kustomizations:
- infra folder: installs Crossplane and Azure Key Vault to Kubernetes; an infra kustomization will point to this folder
- secrets folder: creates a secret with Azure Key Vault to Kubernetes from Azure Key Vault; the secrets kustomization will point to this folder
- crossplane-apps folder: installs Azure resources and Kubernetes resources with the respective Crossplane providers; the apps kustomization will point to this folder
Note: if you do not know what Flux kustomizations are and how Flux works, do check my Flux playlist: https://www.youtube.com/playlist?list=PLG9qZAczREKmCq6on_LG8D0uiHMx1h3yn. The videos look at the open source version of Flux and not the microsoft.flux extension. To learn more about that extension, see https://www.youtube.com/watch?v=w_eoJbgDs3g.
Installing Crossplane
The infra customization installs Crossplane and Azure Key Vault to Kubernetes. The latter is used to sync a secret from Key Vault that contains credentials for the Crossplane Azure provider. More details are in the diagram below:
As noted above, the installation of Crossplane is done with Flux. First, there is the HelmRepository resource that adds the Crossplane Helm repository to Flux.
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
namespace: config-infra
name: crossplane
spec:
interval: 1m0s
url: https://charts.crossplane.io/stable
Next, there is the HelmRelease that installs Crossplane. Important: target namespace is crossplane-system (bottom line):
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: crossplane
namespace: config-infra
spec:
chart:
spec:
chart: crossplane
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: crossplane
namespace: config-infra
install:
createNamespace: true
interval: 1m0s
targetNamespace: crossplane-system
For best results, in the YAML above, set the namespace of the resource to the namespace you use with the AKS k8s-configuration. The resources to install Azure Key Vault to Kubernetes are similar.
To install the Crossplane jet provider for Azure:
---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
name: jet-azure-config
labels:
app: crossplane-provider-jet-azure
spec:
image: crossplane/provider-jet-azure-controller:v0.9.0
args: ["-d"]
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: crossplane-provider-jet-azure
spec:
package: crossplane/provider-jet-azure:v0.9.0
controllerConfigRef:
name: jet-azure-config
Above, debugging is turned on for the provider. This is optional. The provider actually runs in the crossplane-system namespace:
The provider is added via the Provider resource (second resource in the YAML manifest).
We can now create the AKS k8s-configuration, which creates a Flux source and a kustomization:
RG=your AKS resource group
CLUSTER=your AKS cluster name (to install Crossplane to)
az k8s-configuration flux create -g $RG -c $CLUSTER \
-n cluster-config --namespace config-infra -t managedClusters \
--scope cluster -u https://github.com/gbaeke/crossplane \
--branch main \
--kustomization name=infra path=./infra prune=true
The Flux source will be the repo specified with -u. There is one kustomization: infra. Pruning is turned on. With pruning, removing manifests from the repo results is removing them from Kubernetes.
The k8s-configuration should result in:
Crossplane is now installed with two providers. We can now configure the Azure provider with credentials.
Configuring Azure Credentials
You need to create a service principal by following the steps in https://crossplane.io/docs/v1.9/cloud-providers/azure/azure-provider.html. I compacted the resulting JSON with:
cat <path-to-JSON> | jq -c
The output of the above command was added to Key Vault:
The Key Vault I am using uses the Azure RBAC permission model. Ensure that the AKS cluster’s kubelet identity has at least the Key Vault Secrets User role. It is a user-assigned managed identity with a name like clustername-agentpool.
To actually create a Kubernetes secret from this Key Vault secret, the secrets folder in the git repo contains the manifest below:
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
name: azure-creds
namespace: crossplane-system
spec:
vault:
name: kvgebadefault # name of key vault
object:
name: azure-creds # name of the akv object
type: secret # akv object type
output:
secret:
name: azure-creds # kubernetes secret name
dataKey: creds # key to store object value in kubernetes secret
This creates a Kubernetes secret in the crossplane-system namespace with name azure-creds and a key creds that holds the credentials JSON.
To add the secret(s) as an extra kustomization, run:
RG=your AKS resource group
CLUSTER=your AKS cluster name
az k8s-configuration flux create -g $RG -c $CLUSTER \
-n cluster-config --namespace config-infra -t managedClusters \
--scope cluster -u https://github.com/gbaeke/crossplane \
--branch main \
--kustomization name=infra path=./infra prune=true \
--kustomization name=secrets path=./secrets prune=true dependsOn=["infra"]
Note that the secrets kustomization is dependent on the infra kustomization. After running this command, ensure the secret is in the crossplane-system namespace. The k8s-configuration uses the same source but now has two kustomizations.
Deploying resources with the Jet provider for Azure
Before explaining how to create Azure resources, a note on providers. As a novice Crossplane user, I started with the following Azure provider: https://github.com/crossplane-contrib/provider-azure. This works well but it is not so simple for contributors to ensure the provider is up-to-date with the latest and greatest Azure features. For example, if you deploy AKS, you cannot use managed identity, the cluster uses availability sets etc…
To improve this, Terrajet was created. It is a code generation framework that can generate Crossplane CRDs (custom resource definitions) and sets up the provider to use Terraform. Building on top of Terraform is an advantage because it is more up-to-date with new cloud features. That is the reason why this post uses the jet provider. When we later create an AKS cluster, it will take advantage of managed identity and other newer features.
Note: there is also a Terraform provider that can take Terraform HCL to do anything you want; we are not using that in this post
Ok, let’s create a resource group and deploy AKS. First, we have to configure the provider with Azure credentials. The crossplane-apps folder contains a file called jet-provider-config.yaml:
apiVersion: azure.jet.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: azure-creds
key: creds
The above ProviderConfig tells the provider to use the credentials in the Kubernetes secret we created earlier. We know we are configuring the jet provider from the apiVersion: azure.jet.crossplane.io/v1alpha1.
With that out of the way, we can create the resource group and AKS cluster. Earlier in this post, the YAML to create the resource group was already shown. To create a basic AKS cluster called clu-cp in this group, aks.yaml is used:
apiVersion: containerservice.azure.jet.crossplane.io/v1alpha2
kind: KubernetesCluster
metadata:
name: clu-cp
spec:
writeConnectionSecretToRef:
name: example-kubeconfig
namespace: crossplane-system
forProvider:
location: "westeurope"
resourceGroupNameRef:
name: rg-crossplane
dnsPrefix: "clu-cp"
defaultNodePool:
- name: default
nodeCount: 1
vmSize: "Standard_D2_v2"
identity:
- type: "SystemAssigned"
tags:
environment: dev
providerConfigRef:
name: default
Above, we refer to our resource group by name (resourceGroupNameRef) and we write the credentials to our cluster to a secret (writeConnectionSecretToRef). That secret will contain keys with the certificate and private key, but also a kubeconfig key with a valid kubeconfig file. We can use that later to connect and deploy to the cluster.
To see an example of connecting to the deployed cluster and creating a namespace, see k8s-provider-config.yaml and k8s-namespace.yaml in the repo. The resource k8s-provider-config.yaml will use the example-kubeconfig secret created above to connect to the AKS cluster that we created in the previous steps.
To create a kustomization for the crossplane-apps folder, run the following command:
RG=your AKS resource group
CLUSTER=your AKS cluster name
az k8s-configuration flux create -g $RG -c $CLUSTER \
-n cluster-config --namespace config-infra -t managedClusters \
--scope cluster -u https://github.com/gbaeke/crossplane \
--branch main \
--kustomization name=infra path=./infra prune=true \
--kustomization name=secrets path=./secrets prune=true dependsOn=["infra"] \
--kustomization name=apps path=./crossplane-apps prune=true dependsOn=["secrets"]
This folder does not contain a kustomization.yaml file. Any manifest you drop in it will be applied to the cluster! The k8s-kustomization now has the same source but three kustomizations:
After a while, an AKS cluster clu-cp should be deployed to resource group rg-crossplane:
To play around with this, I recommend using Visual Studio Code and the GitOps extension. When you make a change locally and push to main, to speed things up, you can reconcile the git repository and the apps kustomization manually:
Conclusion
In this post, we looked at installing and configuring Crossplane on AKS via GitOps and the microsoft.flux extension. In addition, we deployed a few Azure resources with Crossplane and its jet provider for Azure. We only scratched the surface here but I hope this gets you started quickly when evaluating Crossplane for yourself.
baeke.info 