How To: integrate Vault as External Root CA with cert-manager, Istio-CSR and Istio

Use case

This documentation will help you improve your Kubernetes Cluster security.

First we will enable Istio Mutual TLS (mTLS), so pods in the cluster will use TLS communication. By default Istio will issue it’s own Certificate, using istiod Self-Sign as Certificate Authority (CA), which considered to be less secure than Vault.

Therefore we will further improve the cluster security by setting External Vault server as the Root CA for the mTLS.

Versions:

istio: 1.11.1

istio-csr: 0.3.0

cert-manager: 1.5.0, 1.4.0

What will be discussed

In this documentation we will see how to set up Vault as an External CA for Kubernetes Cluster using Cert Manager.

Next we will integrate Istio with Cert Manager using Istio-CSR.

Then we will deploy an application and check that the application is running with the RootCA that we issued from vault.

Install cert-manager

When installing cert manager make sure the installCRDs is set to ‘true’.

helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.5.0 \
--set installCRDs=true

Enable Vault Public Key Infrastructure and Intermediate CA

Connect to Vault server from cli

$ export VAULT_TOKEN=<VAULT_TOKEN>
$ export VAULT_ADDR=https://<VAULT_ADDR>

Enable Vault PKI

STEP 1: Enable the pki secrets engine at the pki path.

$ vault secrets enable pki

STEP 2: Tune the pki secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.

  • Note that individual roles can restrict this value to be shorter on a per-certificate basis. This just configures the global maximum for this secrets engine.
$ vault secrets tune -max-lease-ttl=87600h pki

STEP 3: Generate the root certificate and save the certificate in CA_cert.crt

  • In order to issue certificate for Kubernetes Cluster we will set the Common Name to ‘svc
  • This generates a new self-signed CA certificate and private key. Vault will automatically revoke the generated root at the end of its lease period (TTL); the CA certificate will sign its own Certificate Revocation List (CRL).
$ vault write -field=certificate pki/root/generate/internal common_name="svc" ttl=87600h > CA_cert.crt

STEP 4: Configure the ‘CA’ and ‘CRL’ URLs

$ vault write pki/config/urls issuing_certificates="$VAULT_ADDR/v1/pki/ca"
crl_distribution_points="$VAULT_ADDR/v1/pki/crl"

Generate intermediate CA

STEP 1: Enable the pki secrets engine at the pki_int path.

$ vault secrets enable -path=pki_int pki

STEP 2: Tune the pki_int secrets engine to issue certificates with a maximum time-to-live (TTL) of 43800 hours.

$ vault secrets tune -max-lease-ttl=43800h pki_int

STEP 3: Execute the following command to generate an intermediate and save the CSR as pki_intermediate.csr .

$ vault write -format=json pki_int/intermediate/generate/internal common_name="svc Intermediate Authority" | jq -r '.data.csr' > pki_intermediate.csr

STEP 4: Sign the intermediate certificate with the root CA private key, and save the generated certificate as intermediate.cert.pem .

$ vault write -format=json pki/root/sign-intermediate csr=@pki_intermediate.csr format=pem_bundle ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem

STEP 5: Once the CSR is signed and the root CA returns a certificate, it can be imported back into Vault.

$ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Create Vault role

A role is a logical name that maps to a policy used to generate those credentials. It allows configuration parameters to control certificate common names, alternate names, the key uses that they are valid for, and more.

STEP: Create a role named cluster-dot-local

  • the role sets allowed _domains to ‘svc’ for Kubernetes internal use.
  • require_cn ‘false’ .
  • allowed_uri_sans for the whole cluster.
$ vault write pki_int/roles/cluster-dot-local allowed_domains="svc" allow_subdomains=true max_ttl="720h" require_cn=false allowed_uri_sans="spiffe://cluster.local/*"

Create Vault AppRole

STEP 1: Enable approle auth method by executing the following command.

$ vault auth enable approle 

STEP 2: Create policy by the name cert-manager

$ vault policy write cert-manager -<<EOF path "pki_int/sign/cluster-dot-local" { capabilities = ["update"] } EOF

STEP 3: Creates a role named cert-manager with cert-manager policy attached.

  • The generated token's time-to-live (TTL) is set to 1 hour and can be renewed for up to 4 hours of its first creation. (NOTE: This example creates a role which operates in pull mode.)
$ vault write auth/approle/role/cert-manager token_policies="cert-manager " token_ttl=1h token_max_ttl=4h

STEP 4: Get role-id and secret-id

$ vault read auth/approle/role/cert-manager/role-id
$ vault write -force auth/approle/role/cert-manager/secret-id

Integrate Vault and Cert Manager

First make sure that you have a connection from the Kubernetes Cluster to the Vault server.

  • In order to test the connectivity from the Cluster to Vault Server, we use the following set up.
  • If you know that the Cluster can connect to Vault server, you can skip to the next part (Integrate Cert Manager and Vault).

Connectivity test from within the pod to Vault server

STEP 1: Crate pod with security context

  • vault command will not work from within the pod, without the cap-add=IPC_LOCK capability enabled.
  • The ‘NET_ADMIN’ capability includes the IPC_LOCK.
apiVersion: v1
kind: Pod
metadata:
name: test1
spec:
serviceAccountName: cert-manager
automountServiceAccountToken: true
containers:
- name: sec-ctx-4
image: ubuntu:20.04
command: ["sleep"]
args: ["10000000000000000000000000000000"]
securityContext:
capabilities:
add: ["NET_ADMIN"]

STEP 2: Install vault cli in the pod and set capability

  • Run the following commands from within the pod.
$ apt update && apt-get install libcap2-bin sudo curl software-properties-common -y
$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install vault -y
$ setcap cap_ipc_lock= /usr/bin/vault
# set vault parameters for vault command usage
$ export VAULT_TOKEN=<VAULT_TOKEN>
$ export VAULT_ADDR=https://<VAULT_ADDR>
$ vault status
  • If the command ‘vault status’ retrieved your Vault server information you have connectivity from the pod.
  • Otherwise there might be a connectivity issue from the pod to your Vault server.

Integrate Cert Manager and Vault server

STEP 1: Create Kubernetes secret using for Vault AppRole authentication

  • Firstly, the ‘secretId’ must be stored inside a Kubernetes Secret on the same namespace as the Issuer .
  • In our case we will use the ‘istio-system’ namespace for future integration with Istio.
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: cert-manager-vault-approle
namespace: istio-system
data:
secretId: <APP_ROLE_SECRET_ID> # insert secretId base64 encoded

STEP 2: Create Issuer

  • Once submitted, the Vault issuer is able to be authenticate, using ‘approle’ auth.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: vault-issuer
namespace: istio-system
spec:
vault:
path: pki_int/sign/cluster-dot-local
server: https://<VAULT_ADDR> # insert vault url
auth:
appRole:
path: approle
roleId: <ROLE_ID> # base64 encoded not needed
secretRef:
name: cert-manager-vault-approle
key: secretId

STEP 3: Check that the issuer is verified

$ kubectl get issuers vault-issuer -n istio-system -o wide

NAME READY STATUS AGE
vault-issuer True Vault verified 1m
  • If you get an error, check the connectivity to Vault from the cluster as discussed earlier.

Install istio-csr

  • istio-csr is an agent that allows for Istio workload and control plane components to be secured using cert-manager. Certificates facilitating mTLS, inter and intra cluster, will be signed, delivered and renewed using cert-manager issuers.

Create secret from pem file

$ kubectl create secret generic istio-root-ca --from-file=ca.cert.pem=<PATH_OF_INTERMIDATE_PEM_FILE_FROM_VAULT> -n cert-manager

Install istio-csr

  • This agent also creates a secret, istiod-tls , which holds the tls cert/key for istiod to serve.
$ helm upgrade -i -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr \
--set "app.certmanager.issuer.name=vault-issuer" \
--set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.cert.pem" \
--set "volumeMounts[0].name=root-ca" \
--set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \
--set "volumes[0].name=root-ca" \
--set "volumes[0].secret.secretName=istio-root-ca"
  • Verify that istio-csr is running, and that the istiod Certificate is in a ready state
$ kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-756bb56c5-cdvln 1/1 Running 0 111s
cert-manager-cainjector-86bc6dc648-bjnrp 1/1 Running 0 111s
cert-manager-istio-csr-5b9cd98696-v4f6j 1/1 Running 0 12s
cert-manager-webhook-66b555bb5-4s7qb 1/1 Running 0 111s

$ kubectl get certificates -n istio-system
NAME READY SECRET AGE
istiod True istiod-tls 28s

Install IstioOperator and set Istio configuration

Install IstioOperator Controler

$ istioctl operator init

Set Istio to use mTLS

  • By default, the sidecar will be configured to accept both mTLS and non-mTLS traffic, known as PERMISSIVE mode. The mode can alternatively be configured to STRICT, where traffic must be mTLS, or DISABLE, where traffic must be plaintext. The mTLS mode is configured using a PeerAuthentication resource.
  • When setting the PeerAuthentication resource set to ‘istio-system’ Namespace, mTLS will be enabled in the whole cluster.
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "mtls"
namespace: istio-system
spec:
mtls:
mode: STRICT
  • Check that all Namespaces are set to mTLS from Kiali Dashboard

Configure IstioOperator Deployment

  • Deploy Istio profile: Demo
  • Configure External CA address for workloads
  • Disable istiod as the CA Server
  • Provide TLS certs for istiod from cert-manager
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istio-operator-csr
spec:
profile: "demo"
values:
global:
caAddress: cert-manager-istio-csr.cert-manager.svc:443
components:
pilot:
k8s:
env:
# Disable istiod CA Sever functionality
- name: ENABLE_CA_SERVER
value: "false"
overlays:
- apiVersion: apps/v1
kind: Deployment
name: istiod
patches:
# Mount istiod serving and webhook certificate from Secret mount
- path: spec.template.spec.containers.[name:discovery].args[7]
value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
- path: spec.template.spec.containers.[name:discovery].args[8]
value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
- path: spec.template.spec.containers.[name:discovery].args[9]
value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
- path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
value:
name: cert-manager
mountPath: "/etc/cert-manager/tls"
readOnly: true
- path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
value:
name: ca-root-cert
mountPath: "/etc/cert-manager/ca"
readOnly: true
- path: spec.template.spec.volumes[6]
value:
name: cert-manager
secret:
secretName: istiod-tls
- path: spec.template.spec.volumes[7]
value:
name: ca-root-cert
configMap:
defaultMode: 420
name: istio-ca-root-cert

Test the RootCA of a new application

  • Deploy the sleep application.
  • The app is located at the istioctl folder installation (in our demo — istio-1.11.1)
kubectl create ns foo
kubectl label ns foo istio-injection=enabled
kubectl apply -f samples/sleep/sleep.yaml -n foo

kubectl get pods -n foo
NAME READY STATUS RESTARTS AGE
sleep-8f795f47d-vv6hx 2/2 Running 0 42s
  • install the getmesh command
  • get the app RootCA:
$ getmesh istioctl pc secret <POD_NAME>.foo -o json > proxy_secret
  • Check the proxy_secret. There should be a field named ‘ROOTCA’ and in ‘ROOTCA’ there should be a field ‘trustedCA’ which is signed by cert-manager ‘vault-issuer’.
  • Compare the value of ‘trustedCA’, from the proxy_secret, with the value of intermidate.cert.pem that we got from vault. Don’t forget to encode the pem value in base64.
  • Those two should be the same, which means the application is using the RootCA that was issued from Vault.

References:


How To: integrate Vault as External Root CA with cert-manager, Istio-CSR and Istio was originally published in Everything Full Stack on Medium, where people are continuing the conversation by highlighting and responding to this story.

DevOps Engineer

DevOps Group

Thank you for your interest!

We will contact you as soon as possible.

Want to Know More?

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com
Thank you for your interest!

We will contact you as soon as possible.

Let's talk

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com