How To: Implement AWS KMS as Root CA using for Kubernetes

Use case

If you are looking to implement an external Certificate Authority for your Kubernetes cluster, and your Cloud Provider is AWS, you should consider using AWS Key Management Service.

In order to integrate AWS KMS to our Kubernetes Cluster we will use SkyScanner KMS-Issuer. KMS issuer is a cert-manager Certificate Request controller that uses AWS KMS to sign the certificate request.

Last but not least, we will showcase a dome of how to integrate AWS KMS as Root CA for Istio using Istio-CSR.

versions

  • SkyScanner kms-issuer : v1.0.0
  • Cert-Manager : v1.5.* , v1.4.*
  • istio: 1.11.*
  • istio-csr: 0.3.0

What will be discussed:

  1. What is AWS Key Management Service
  2. Benefits of using AWS KMS
  3. Applying AWS KMS and KMS-Controller
  4. How To: Set KMS-Issuer as Root CA for Istio using Istio-CSR Integrated with KMS-Issuer

What is AWS Key Management Service

AWS Key Management Service (KMS) makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and in your applications.

KMS works both with a symmetric key and an asymmetric key and supports asymmetric keys that represent a mathematically related RSA or elliptic curve (ECC) public and private key pair. You can download the public key for distribution and use outside of AWS. You can create asymmetric KMS keys for encryption and decryption, or signing and verification, but not both.

Benefits of using AWS KMS Asymmetric Key

https://aws.amazon.com/kms/features/#Overview

  1. Centralized Key Management: AWS KMS provides you with centralized control over the lifecycle and permissions of your keys. You can create new keys whenever you wish, and you can control who can manage keys separately from who can use them.
  2. AWS Service Integration: The service is integrated with other AWS services making it easy to encrypt data you store in these services and control access to the keys that decrypt it.
  3. Secure: AWS KMS is designed so that no one, including AWS employees, can retrieve your plaintext keys from the service. The service uses hardware security modules (HSMs) that have been validated under FIPS 140–2, or are in the process of being validated, to protect the confidentiality and integrity of your keys. Your plaintext keys are never written to disk and only ever used in volatile memory of the HSMs for the time needed to perform your requested cryptographic operation.
  4. Low Costs: $0.03 per 10,000 requests involving RSA 2048 keys. for more information regarding pricing: https://aws.amazon.com/kms/pricing/

Applying AWS KMS and KMS-Controller

When deploying KMS-Issuer v1.0.0 manually make sure to fix the known issues described later on.

STEP 1: Install Cert-Manager

  • When installing cert manager make sure the installCRDs is set to ‘true’.
  • Installing Cert-Manager is a prerequisite for using KMS-Issuer.
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.5.0 \
--set installCRDs=true

STEP 2: Fork Skyscanner/kms-issuer git repository

STEP 3: Install KMS-Issuer CRD’s

kubectl apply -k config/crd

STEP 4: Create AWS-KMS asymmetric key with RSA_2048

  • Copy the KMS ARN for the use of AWS Policy.
Note: KMS key can be created from the Kubernetes Cluster, as described: https://github.com/Skyscanner/kms-issuer.
In order to do that more permissions needs to be added to the Assume Role’s Policy.

STEP 5: Create AWS Policy and Role with Web Identity

  • The Role Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowUseOfTheKey",
"Effect": "Allow",
"Action": [
"kms:DescribeKey",
"kms:GetPublicKey",
"kms:Sign",
"kms:Verify"
],
"Resource": "<KMS_KEY_ARN>"
},
{
"Sid": "AllowAttachmentOfPersistentResources",
"Effect": "Allow",
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "<KMS_KEY_ARN>",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
}

STEP 6: Deploy KMS-Controller

Note: before running the deployment file make sure to fix the known issues for kms-issuer v1.0.0

Fix known issues for kms-issuer v1.0.0

  1. Wrong image: the deployment.yaml stated that the image is skyscanner/kms-issuer:dev, but when deploying we got an error “the image was not found”.
    The correct image can be found in Skyscanner/Kms-issuer repository.
    Correct image:
ghcr.io/skyscanner/kms-issuer:sha-3ee48e5

2. Kubernetes Service Account RBAC: was missing permission to the resource ‘leases’. Therefore, we added permissions to this resource. The permission was added to the ‘Role’ — ‘kms-issuer-leader-election-role’:

- apiGroups:
- "coordination.k8s.io"
resources:
- leases
verbs:
- "*"
  • Both fixes can be found in the:
deploy/kubernetes/kms-issuer.yaml
  • After fixing the known issue, you can run the deployment file
kubectl apply -f deploy/kubernetes/kms-issuer.yaml

STEP 7: Add Assume Role ARN to Kubernetes Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: kms-issuer-system
annotations:
eks.amazonaws.com/role-arn: <ARN_ROLE>

STEP 8: Deploy issuer

  • KMS-Issuer should run on the same namespace of your application.
apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSIssuer
metadata:
name: kms-issuer
namespace: <APP_NAMESPACE>
spec:
keyId: alias/<example_kms_key> # same name as AWS kms key
commonName: My Root CA # The common name for the root certificate
duration: 87600h # 10 years

STEP 9: Test that the certificate was issued:

kubectl get kmsissuer kms-issuer -n <NAMESPACE> -o json | jq -r ".status.certificate" |  base64 --decode  | openssl x509 -noout -text

Applying with Terraform

Example for Terraform deployment is in the following git repository:

kms-issuer/chart at main · guysaarTikal/kms-issuer

How To: Set KMS-Issuer as Root CA for Istio using Istio-CSR Integrated with KMS-Issuer

Deploying istio-csr

Set istio-csr to sign the certificate using the kms-issuer that we deployed earlier:

helm upgrade -i -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr --set "app.certmanager.issuer.name=kms-issuer" --set "app.logLevel=5" --set "app.certmanager.issuer.kind=KMSIssuer" --set "app.certmanager.issuer.group=cert-manager.skyscanner.net"
  • istio-csr will create a secret named ‘istiod-tls’ that holds the tls.crt and tls.key. It will also create a configmap named ‘istio-ca-root-cert’ that holed the root-ca.pem.
  • We will map them later on when deploying istio.

Deploy Istio-Operator

For more info regarding Istio-Operator installation: https://istio.io/latest/docs/setup/install/operator/

istioctl operator init

Deploy Istiod integrated with Istio-CSR

Configure IstioOperator Deployment for istiod

  • Deploy Istio profile: minimal
  • Configure External CA address for workloads
  • Disable istiod as the CA Server
  • Provide TLS certs for istiod from cert-manager
  • map configmap ‘istio-ca-root-cert’ and secret ‘istiod-tls’.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-controlplane
namespace: istio-system
spec:
profile: "minimal"
values:
global:
caAddress: cert-manager-istio-csr.cert-manager.svc:443
components:
pilot:
k8s:
env:
# Disable istiod CA Sever functionality
- name: ENABLE_CA_SERVER
value: "false"
overlays:
- apiVersion: apps/v1
kind: Deployment
name: istiod
patches:
# Mount istiod serving and webhook certificate from Secret mount
- path: spec.template.spec.containers.[name:discovery].args[7]
value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
- path: spec.template.spec.containers.[name:discovery].args[8]
value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
- path: spec.template.spec.containers.[name:discovery].args[9]
value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
- path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
value:
name: cert-manager
mountPath: "/etc/cert-manager/tls"
readOnly: true
- path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
value:
name: ca-root-cert
mountPath: "/etc/cert-manager/ca"
readOnly: true
- path: spec.template.spec.volumes[6]
value:
name: cert-manager
secret:
secretName: istiod-tls
- path: spec.template.spec.volumes[7]
value:
name: ca-root-cert
configMap:
defaultMode: 420
name: istio-ca-root-cert

Deploy Istio-Ingress

Configure IstioOperator Deployment for Istio-Ingress

  • It is recommended by Istio to deploy Istio-Ingress on a different namespace than istiod.
https://istio.io/latest/docs/setup/additional-setup/gateway/
As a security best practice, it is recommended to deploy the gateway in a different namespace from the control plane.
  • Set pilot Cert Provider: istiod
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-ingress
namespace: istio-system
spec:
profile: empty # Do not install CRDs or the control plane
components:
ingressGateways:
- name: ingressgateway
namespace: istio-ingress
enabled: true
label:
# Set a unique label for the gateway. This is required to ensure Gateways
# can select this workload
istio: ingressgateway
values:
gateways:
istio-ingressgateway:
# Enable gateway injection
injectionTemplate: gateway
global:
pilotCertProvider: istiod

References:


How To: Implement AWS KMS as Root CA using for Kubernetes was originally published in Everything Full Stack on Medium, where people are continuing the conversation by highlighting and responding to this story.

DevOps Engineer

DevOps Group

Thank you for your interest!

We will contact you as soon as possible.

Want to Know More?

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com
Thank you for your interest!

We will contact you as soon as possible.

Let's talk

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com