Baking a production ready Kubernetes cluster with "kops & friends"

Intro

So without introduction to the tools used in this blog post (maybe another time …) I thought I would share with you something i’ve prepared in one of my projects, Lets start by stating what I am aiming for …

I want every micro-service we launch in our cluster to be monitored, logged, scalable, routable, accessible, authorized and more …

We already have a manual process of instantiating our cluster using kops with a set of constraints out of the scope of this post, but what I want to achieve is that when we rollout a cluster for testing / ci / new environment all these “Basic” needs are covered.

You will see below a list of completed which was the cause to blog about it, and the Still in the works and the Post installation is still manual :( which we will be working on …

All these should be found under tikalk/dev.hub.tikal.io github repo, I assume you can follow along there for any additions if I don’t blog about them. So lets get to it !!!

Our desired cluster state:

Complete (specified in this post):

  • Simple 1 master 3 nodes cluster (extend after)
  • Horizontal Auto Scaling enabled which requires heapster
  • Elasticsearch Fluentd Kibana for logging
  • Kubernetes dashboard

Still in the works:

  • Prometheus Operator
  • Istio
  • A demo app

Post installation:

  • Add spinnaker helm chart
  • Customer Applications as addons ?! / Spnnaker pipelines

Step by Step:

  1. Intro
  2. Our Desired cluster state
  3. Requirements
  4. Setup the environment
  5. Create a cluster with kops
  6. Review Cluster

    6.1 Cluster spec

    6.2 Instance Group spec

  7. Add kops addons
  8. Replace cluster config spec
  9. Update the cluster
  10. Wrapping up

    10.1 View cluster logs in kibana

Requirements:

Setup the environment

This will set AWS_PROFILE, AWS_ACCESS_KEY, AWS_SECRET_KEY, KOPS_STATE_STORE and KOPS_CLUSTER_NAME

source setenv.sh

The setenv.sh file:

Create a cluster with kops

Review Cluster

Which should yield:

Get Cluster Config

We want the spec to keep in source control + edit it before we actually provision the cluster.

Cluster spec

Our cluster spec should look like this:

Instancegroup spec

Our Instance Groups should look like this:

Add kops addons

Let’s add some out of the box to our ${KOPS_CLUSTER_NAME}-cluster.yaml cluster spec.

Add the following to your spec file:

See -> kops/issues/3554

This will result in a cluster with logging and autoscaling enabled … I am planning on adding some of my own addons but didn’t get around to it yet (hope to do in a separate post).

Replace cluster config spec

Using kops replace like so:

Update the cluster

Using kops update cluster like so:

If your like me and you added the addon’s after the cluster is created you also need to kops rolling-update cluster --yes considering addons are basically Kubernetes deployments / configMaps / Pod’s … so the kubernetes API should be able to pull them from your s3Bucket/addons folder.

Wrapping up

At this point you should have a cluster up and running with the following kubectl cluster-info result:

View cluster logs in kibana

Kibana log

As always hope you found this blog post useful, feel free to drop me a line …

Yours, HP

Thank you for your interest!

We will contact you as soon as possible.

Send us a message

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com