Recently, a customer inquired if there is a way to leverage Azure Key Vault on on-prem/non-Azure k8s clusters. In this blog post, I will show the steps we followed to demonstrate the process of mounting Azure Key Vault secrets inside an on-prem Kubernetes cluster.
- Azure Key Vault is one of several key management solutions in Azure, and can be used to Securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets. Centralizing storage of application secrets in Azure Key Vault allows you to control their distribution. Key Vault greatly reduces the chances that secrets may be accidentally leaked. When using Key Vault, application developers no longer need to store security information in their application. Not having to store security information in applications eliminates the need to make this information part of the code.
- Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem and focuses on the application workloads, not the underlying infrastructure components.
- Kubernetes Secrets Store CSI Driver integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume. It allows Kubernetes to mount multiple secrets, keys, and certificates stored in enterprise-grade external secrets stores, such as Azure Key Vault, into pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.
The Secrets Store CSI Driver on Azure Kubernetes Service (AKS) provides the following methods of identity-based access to your Azure key vault.
- An Azure Active Directory pod identity (preview)
- An Azure Active Directory workload identity (preview)
- A user-assigned or system-assigned managed identity
- An Azure Active Directory (AD) service principal
NOTE: Service principals eventually expire and must be renewed to keep the cluster working. In addition, managing service principals adds complexity, thus it’s recommended to use managed identities when working with Azure resources. Managed identities are the default authentication method for an AKS cluster.
Given that the use-case in question is not AKS and the cluster is not hosted in Azure, we are left with no other option but to use service principals to access Azure Key Vault.
Create a new Kubernetes Cluster
If you do not have an on-prem Kubernetes cluster, you can follow instructions at How to Install Kubernetes Cluster on Debian 11 with Kubeadm to create a cluster on Hyper-V.
Create a new Azure key vault
In addition to a Kubernetes cluster, you’ll need an Azure key vault resource that stores the secret content in the cloud.
Create a new Resource Group
To create a new resource group, execute the following command:
|
|
Create a new Azure Key Vault resource
Execute the following command to create a new Azure Key vault resource:
|
|
NOTE: The key vault’s name must be globally unique.
Create Key Vault secrets
In this example, you’ll create two plain-text secrets in our Key vault as follows:
|
|
Create a Service Principal
Next, create a service principal with the following commands:
|
|
The command should output a JSON object similar to this::
|
|
You will need all four values in later steps, so it’s important to note them down at this point.
Assign Permissions to Key vault
Access to a key vault is controlled through two interfaces: the management plane and the data plane. The management plane is where you manage Key Vault itself. Operations in this plane include creating and deleting key vaults, retrieving Key Vault properties, and updating access policies. The data plane is where you work with the data stored in a key vault. You can add, delete, and modify keys, secrets, and certificates. To grant our service principal permissions to read keys, secrets, and certificates, we execute the following commands:
|
|
The rest of this process is completed in our on-prem Kubernetes cluster.
Install the Secrets Store CSI Driver
To install the Secrets Store CSI Driver on a fresh Kubernetes installation, execute the following script:
|
|
This will produce the following output:
|
|
Install the Azure Key Vault Provider for Secrets Store CSI Driver
Azure Key Vault provider for Secrets Store CSI Driver allows you to get secret contents stored in an Azure Key Vault instance and use the Secrets Store CSI driver interface to mount them into Kubernetes pods. To install the Azure Key Vault provider for Secrets Store CSI Driver, execute the following command on the cluster:
|
|
This will produce output as follows:
|
|
Create Service Principal Kubernetes Secret
In order for your on-prem Kubernetes cluster to connect to Azure Key Vault, you will need to create Kubernetes secrets accessible to the Secret Store CSI Driver. To create the secrets, execute the following commands against the cluster:
|
|
NOTE:
Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.In order to restrict access to Kubernetes secrets, consider enabling or configuring RBAC rules with least-privilege access to Secrets.
See Information security for Secrets for more details.
Deploy the Secret Provider Class
Create a YAML file:
|
|
Paste the following contents and save the file:
|
|
Deploy the Secret Provider Class:
|
|
Create a Pod to consume Azure Key Vault Secrets
Create a YAML file:
|
|
Paste the following contents and save the file:
|
|
Deploy the pod:
|
|
The command should produce the following output:
|
|
Verify Azure Key Vault Secrets in container
To verify that your secrets are accessible inside our container, you execute the following commands:
|
|
Important:
You can reduce the exposure of your vaults by specifying which IP addresses have access to them. The Virtual network service endpoints for Azure Key Vault allow you to restrict access to a list of IPv4 (internet protocol version 4) address ranges. Any user connecting to your key vault from outside those sources is denied access.For more information, see Configure Azure Key Vault firewalls and virtual networks
And there you have it! We managed to retrieve Azure Key Vault secrets from an on-prem Kubernetes cluster using a Service Principal. Hope the proves valuable to you, dear reader.
Credits:
Thanks to my colleagues Ravi Yadav and Hammad Aslam for the important pointers.
References:
Use Azure KeyVaults in BC OnPrem
Service Principal | Azure Key Vault Provider for Secrets Store CSI Driver
Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster
Kubernetes Secrets Store CSI Driver
Installation - Secrets Store CSI Driver
Installation | Azure Key Vault Provider for Secrets Store CSI Driver
Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster
Service Principal Examples for Azure Key Vault Provider for Secrets Store CSI Driver
Azure Key Vault security