Kubernetes via Terraform on DigitalOcean
TL;DR
The following post will show how to spin-up a Kuberentes cluster on Digital Ocean in under 10 min providing you have an account an a spare 30-40$ per month, and all this with 31 lines of Terraform code …
I wouldn’t “go production” but it’s a great experiment with good value for money (IMHO).
So What lays here, well…
This setup will have a managed Kuberentes control plane which is 100% free + 2 instances in a kubernetes node_pool which is digital oceans way of calling workers or minions.
These 2 droplets (a.k.a instances) are 15$ per month, and if you decide to set up an ingress with a load balancer (probably should) that would be an extra 10$ …
Please note As you add more Droplets or node pool’s / storage your costs will rise of course …
Prerequisites
doctl- digital ocean cli- A digital ocean project + authentication token
terraform- for provisioning the cluster an worker nodeskubectl- for managing the cluster
Setting up the cluster
1. Configuring doctl
- Install
doctlby following the instruction here -
Once installed
doctl auth initprovide your auth and you should find aconfig.ymlunder$HOME/.config/doctl/config.ymlwhich is basically the digital ocean token and a bunch of other stuff related to it.We will use this file later on.
2. Review Terraform code
-
In order to keep
terraformpassword free, I use the following command to extract the token and populate thedo_configvariable, by settingTF_VAR_do_tokenenv var. And thus my code is clean and only the operator/system with this token can operate the cluster. -
Terraform’s
main.tfThis module is pretty simple (I didn’t bother with the terraform state ATM) …
variable "do_token" {}This allows us to poplate the var via environment variable with the
TF_VAR_prefix like so:export TF_VAR_do_token=$(cat ${HOME}/.config/doctl/config.yaml | grep access-token | awk '{print $NF}')Then setup the provider in our case
digitalocean(+ utlize the do_config var):provider "digitalocean" { token = "${var.do_token}" }And finally setting up the cluster is the following block:
resource "digitalocean_kubernetes_cluster" "tikal-cloud" { name = "tikal-do-cloud" region = "lon1" version = "1.12.1-do.2" tags = ["tikal-do-cloud"] node_pool { name = "standard-pool" size = "s-2vcpu-2gb" node_count = 2 } }Now, in order to export a
kubeconfigfile I can use to do the rest of my work outside (or maybe inside with the helm provider - I haven’t decided quite yet) I use the following resource:resource "local_file" "kubeconfig" { filename = "./kube_config_cluster.yml" content = "${digitalocean_kubernetes_cluster.tikal-cloud.kube_config.0.raw_config}" }So at the end of the execution, you will find your
kubeconfigin the current working directory under the namekube_config_cluster.yml.The full module’s code:
variable "do_token" {} provider "digitalocean" { token = "${var.do_token}" } resource "digitalocean_kubernetes_cluster" "tikal-cloud" { name = "tikal-cloud" region = "lon1" version = "1.12.1-do.2" tags = ["tikal-cloud"] node_pool { name = "standard-pool" size = "s-2vcpu-2gb" node_count = 2 } } provider "kubernetes" { host = "${digitalocean_kubernetes_cluster.tikal-cloud.endpoint}" client_certificate = "${base64decode(digitalocean_kubernetes_cluster.tikal-cloud.kube_config.0.client_certificate)}" client_key = "${base64decode(digitalocean_kubernetes_cluster.tikal-cloud.kube_config.0.client_key)}" cluster_ca_certificate = "${base64decode(digitalocean_kubernetes_cluster.tikal-cloud.kube_config.0.cluster_ca_certificate)}" } resource "local_file" "kubeconfig" { filename = "./kube_config_cluster.yml" content = "${digitalocean_kubernetes_cluster.tikal-cloud.kube_config.0.raw_config}" }
Please note: digital ocean certificates renew every 7 days so you probably will use this small resource quite a lot (or use digital ocean web to export it if you prefer…).
3. Running Terraform
```sh
$ terraform init
Initializing provider plugins...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.digitalocean: version = "~> 1.1"
* provider.kubernetes: version = "~> 1.5"
* provider.local: version = "~> 1.1"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
$ terraform plan -out `basename $PWD`.out
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ digitalocean_kubernetes_cluster.tikal-cloud
id: <computed>
cluster_subnet: <computed>
created_at: <computed>
endpoint: <computed>
ipv4_address: <computed>
kube_config.#: <computed>
name: "tikal-cloud"
node_pool.#: "1"
node_pool.0.id: <computed>
node_pool.0.name: "standard-pool"
node_pool.0.node_count: "2"
node_pool.0.nodes.#: <computed>
node_pool.0.size: "s-2vcpu-2gb"
region: "lon1"
service_subnet: <computed>
status: <computed>
tags.#: "1"
tags.3382956491: "tikal-cloud"
updated_at: <computed>
version: "1.12.1-do.2"
+ local_file.kubeconfig
id: <computed>
content: "${digitalocean_kubernetes_cluster.tikal-cloud.kube_config.0.raw_config}"
filename: "./kube_config_cluster.yml"
Plan: 2 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
This plan was saved to: digitalocean.out
To perform exactly these actions, run the following command to apply:
terraform apply "digitalocean.out"
$ terraform apply `basename $PWD`.out
digitalocean_kubernetes_cluster.tikal-cloud: Creating...
cluster_subnet: "" => "<computed>"
created_at: "" => "<computed>"
endpoint: "" => "<computed>"
ipv4_address: "" => "<computed>"
kube_config.#: "" => "<computed>"
name: "" => "tikal-cloud"
node_pool.#: "" => "1"
node_pool.0.id: "" => "<computed>"
node_pool.0.name: "" => "standard-pool"
node_pool.0.node_count: "" => "2"
node_pool.0.nodes.#: "" => "<computed>"
node_pool.0.size: "" => "s-2vcpu-2gb"
region: "" => "lon1"
service_subnet: "" => "<computed>"
status: "" => "<computed>"
tags.#: "" => "1"
tags.3382956491: "" => "tikal-cloud"
updated_at: "" => "<computed>"
version: "" => "1.12.1-do.2"
digitalocean_kubernetes_cluster.tikal-cloud: Still creating... (10s elapsed)
...
digitalocean_kubernetes_cluster.tikal-cloud: Still creating... (3m10s elapsed)
digitalocean_kubernetes_cluster.tikal-cloud: Creation complete after 3m12s (ID: 4d35b716-532c-4210-b0a8-07e224a52f5a)
local_file.kubeconfig: Creating...
(redacted)
iMGZEcE9sdTlUSmN0MnIvalNUaTJObk45R1B1Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==\n"
filename: "" => "./kube_config_cluster.yml"
local_file.kubeconfig: Creation complete after 0s (ID: a813e11a1d8cc4c03f463b3386ed80fa4da9131d)
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
```
4. Get up n running with kubernetes:
- setup kubeconfig:
export KUBECFG=$PWD/kube_config_cluster.yml - Get
nodes…kubectl get nodes NAME STATUS ROLES AGE VERSION wonderful-bhaskara-umf4 Ready <none> 3m48s v1.12.1 wonderful-bhaskara-umfh Ready <none> 3m47s v1.12.1
A little peak “Under the Hood”
A reltively short kubectl command ca reveal the pattern used by digital ocean kubectl get po --all-namespaces as you will see below:
kubectl get po --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system csi-do-controller-0 4/4 Running 2 15m 10.244.6.2 wonderful-bhaskara-umf4 <none>
kube-system csi-do-node-grmz8 2/2 Running 0 10m 10.131.7.185 wonderful-bhaskara-umfh <none>
kube-system csi-do-node-tjbgg 2/2 Running 0 10m 10.131.14.234 wonderful-bhaskara-umf4 <none>
kube-system kube-dns-55cf9576c4-mj98r 3/3 Running 0 15m 10.244.6.3 wonderful-bhaskara-umf4 <none>
kube-system kube-proxy-wonderful-bhaskara-umf4 1/1 Running 1 10m 10.131.14.234 wonderful-bhaskara-umf4 <none>
kube-system kube-proxy-wonderful-bhaskara-umfh 1/1 Running 1 10m 10.131.7.185 wonderful-bhaskara-umfh <none>
each droplet has a csi-do-node-$id which is tied to the “control plane as a service” of course the kube-proxy so traffic could be routed to the cluster and kube-dns for service discovery and dns for the cluster.
In addition (Thank you Rafi as always for sliver lineing …) Digital Ocean comes pre-packed with their do-storageclass which means you cn utlize persistance volumes & claims to persist your data if needed.
k get storageclasses.storage.k8s.io
NAME PROVISIONER AGE
do-block-storage (default) dobs.csi.digitalocean.com 13h
Conclution
All Code in this post can be found in this repo.
It’s been a long 3 years since spinning up a cluster is a piece of cake such as this … Hope you find this post informative and useful as a starting point.
Yours Sincerely, Haggai Philip Zagury, Group & Tech Lead, Tikal DevOps team, and FullStack Developers Israel Contributor.