Yesterday, I posted the Quick Guide to Kubernetes Workload Identity on AKS. This post contains a similar guide to enabling and using the Secret Store CSI driver for Azure Key Vault on AKS.
All commands assume bash. You should have the Azure CLI installed and logged in to the subscription as the owner (because you need to configure RBAC in the scripts below).
Step 1: Enable the driver
The command to enable the driver on an existing cluster is below. Please set the variables to point to your cluster and resource group:
RG=YOUR_RESOURCE_GROUP
CLUSTER=YOUR_CLUSTER_NAME
az aks enable-addons --addons=azure-keyvault-secrets-provider --name=$CLUSTER --resource-group=$RG
If the driver is already enabled, you will simply get a message stating that.
Step 2: Create a Key Vault
In this step, we create a Key Vault and configure RBAC. We will also add a sample secret.
# replace <SOMETHING> with a value like your initials for example
KV=<SOMETHING>$RANDOM
# name of the key vault secret
SECRET=demosecret
# value of the secret
VALUE=demovalue
# create the key vault and turn on Azure RBAC; we will grant a managed identity access to this key vault below
az keyvault create --name $KV --resource-group $RG --location westeurope --enable-rbac-authorization true
# get the subscription id
SUBSCRIPTION_ID=$(az account show --query id -o tsv)
# get your user object id
USER_OBJECT_ID=$(az ad signed-in-user show --query objectId -o tsv)
# grant yourself access to key vault
az role assignment create --assignee-object-id $USER_OBJECT_ID --role "Key Vault Administrator" --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$KV
# add a secret to the key vault
az keyvault secret set --vault-name $KV --name $SECRET --value $VALUE
You can use the portal to check the Key Vault and see the secret:
If you go to Access Policies, you will notice that the Key Vault uses Azure RBAC:
Step 3: Grant a managed identity access to Key Vault
In the previous step, your account was granted access to Key Vault. In this step, we will grant the same access to the managed identity that the secret store csi provider will use. We will need to configure the managed identity we want to use in later steps.
This guide uses the managed identity created by the secret store provider. It lives in the resource group associated with your cluster. By default, that group starts with MC_. The account is called azurekeyvaultsecretsprovider-<CLUSTER-NAME>.
# grab the managed identity principalId assuming it is in the default
# MC_ group for your cluster and resource group
IDENTITY_ID=$(az identity show -g MC\_$RG\_$CLUSTER\_westeurope --name azurekeyvaultsecretsprovider-$CLUSTER --query principalId -o tsv)
# grant access rights on Key Vault
az role assignment create --assignee-object-id $IDENTITY_ID --role "Key Vault Administrator" --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$KV
Above, we grant the Key Vault Administrator role. In production, that should be a role with less privileges.
Step 4: Create a SecretProviderClass
Let’s create and apply the SecretProviderClass in one step.
AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
CLIENT_ID=$(az aks show -g $RG -n $CLUSTER --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv)
cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: demo-secret
namespace: default
spec:
provider: azure
secretObjects:
- secretName: demosecret
type: Opaque
data:
- objectName: "demosecret"
key: demosecret
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "$CLIENT_ID"
keyvaultName: "$KV"
objects: |
array:
- |
objectName: "demosecret"
objectType: secret
tenantId: "$AZURE_TENANT_ID"
EOF
After retrieving the Azure AD tenant Id and managed identity client Id, the SecretProviderClass is created. Pay special attention to the following fields:
- userAssignedIdentityID: the clientId (⚠️ not the principalId we retrieved earlier) of the managed identity used by the secret store provider; you can use other user-assigned managed identities or even a system-assigned managed identity assigned to the virtual machine scale set that runs your agent pool; I recommend using user-assigned identity
- above, the clientId is retrieved via the
az akscommand
- above, the clientId is retrieved via the
- keyvaultName: the name you assigned your Key Vault
- tenantId: the Azure AD tenant Id where your identities live
- usePodIdentity: not recommended because pod identity will be replaced by workload identity
- useVMManagedIdentity: set to true even if you use user-assigned managed identity
Step 5: Mount the secrets in pods
Create pods that use the secrets.
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: secretpods
name: secretpods
spec:
replicas: 1
selector:
matchLabels:
app: secretpods
template:
metadata:
labels:
app: secretpods
spec:
containers:
- image: nginx
name: nginx
env:
- name: demosecret
valueFrom:
secretKeyRef:
name: demosecret
key: demosecret
volumeMounts:
- name: secret-store
mountPath: "mnt/secret-store"
readOnly: true
volumes:
- name: secret-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "demo-secret"
EOF
The above command creates a deployment that runs nginx. The Key Vault secrets are mounted in a volume that is mounted at mnt/secret-store. The Key Vault secret is also available as an environment variable demosecret.
Step 6: Verify
Issue the commands below to get a shell to the pods of the nginx deployment and check the mount path and environment variable:
export POD_NAME=$(kubectl get pods -l "app=secretpods" -o jsonpath="{.items[0].metadata.name}")
# if this does not work, check the status of the pod
# if still in ContainerCreating there might be an issue
kubectl exec -it $POD_NAME -- sh
cd /mnt/secret-store
ls # the file containing the secret is listed
cat demosecret; echo # demovalue is revealed
# echo the value of the environment variable
echo $demosecret # demovalue is revealed
Important: the secret store CSI provider always mounts secrets in a volume. A Kubernetes secret (here used to populate the environment variable) is not created by default. It is created here because of the secretObjects field in the SecretProviderClass.
Conclusion
The above commands should make it relatively straightforward to try the secret store CSI provider and understand what it does. It works especially well in GitOps scenarios where you cannot store secrets in Git and you do not have pipelines that can retrieve Azure Key Vault secrets at deploy time.
If you spot errors in the above commands, please let me know!
baeke.info 