Kubernetes Prometheus Operator Hands-On
At first glance, Prometheus may seem quite a complicated product, but like any well-designed system, it consists of clearly expressed functional components and, in fact, does only three things:
- collects metrics
- executes rules
- saves results to the database time series data.
The article is devoted not so much to Prometheus itself, as to the integration of this system with Kubernetes, for which we actively use an auxiliary tool called Prometheus Operator. But it’s still necessary to begin with Prometheus …
First of all prom8s has 2 main functions:
- Collect metrics from each monitoring target with defined scrape interval.
- With defined evaluation_interval based on recording rules metrics are recording as a new metrics, based on alerting rules are firing alerts.
How to configure Prometheus?
Prometheus server has config and rule files (file with rules).
In config defined following sections:
scrape_configs- settings for searching monitoring targets
rule_files- list of rules files or directories where rules files located
rule_files: - /etc/prometheus/rules/rules-0/\* - /etc/prometheus/rules/rules-1/\*
alerting- settings for searching alertmanagers to where alerts are sending.
Where is the list of targets coming from?
Algorithm of Prometheus looks like:
- Prom8s reads section from scrape_config and based on that he tweaks his own mechanism of Service Discovery (SD).
- Mechanism of SD is communicating with K8s API to discover service endpoints.
- Based on inputs from K8s SD mechanism prometheus updates the list of targets.
In scrape_configs defined list of scrape jobs, each of them has following definition:
scrape_configs: - job_name: kube-prometheus/custom/0 # Name of scrape job'а # shown in section Service Discovery scrape_interval:30s # How often metrics are scraping scrape_timeout:10s # Timeout on request metrics_path: /metrics # path to metrics scheme: http # http or https # Service Discovery settings kubernetes_sd_configs: # means that we are getting targets from Kubernetes - api_server:null # Use address of API-server from env # variables role:endpoints # targets are getting from endpoints namespaces: names: # search endpoints only in following namespaces - foo - baz # Filtering settings what endpoints to add # What labels to add and remove for collected metrics relabel_configs: # Filtering based on value of label prometheus_custom_target # received from service binded with endpoint - source_labels: [__meta_kubernetes_service_label_prometheus_custom_target] regex:.+ # fit any not empty label action: keep # Filtering based on port name - source_labels: [__meta_kubernetes_endpoint_port_name] regex:http-metrics # fit if port named http-metrics action: keep # Adding job label with using value of label prometheus_custom_target # from service, and after that adding prefix "custom-" to value of label # job label using for grouping in Prometheus. He defines name of group, # which will shown as a target in targets page, also he will added to each metric # scraped from this target for ability to filter in rules and dashboards - source_labels: [__meta_kubernetes_service_label_prometheus_custom_target] regex: (.*) target_label: job replacement: custom-$1 action: replace # Adding namespace label - source_labels: [__meta_kubernetes_namespace] regex: (.*) target_label: namespace replacement: $1 action: replace # Adding service label - source_labels: [__meta_kubernetes_service_name] regex: (.*) target_label: service replacement: $1 action: replace # Adding instance label there we will see pod name - source_labels: [__meta_kubernetes_pod_name] regex: (.*) target_label: instance replacement: $1 action: replace
By this method Prometheus automatically tracks:
- Adding and removing pods (When K8s adding/removing pods endpoints are changing and Prom8s see that and adding/removing targets)
- Adding and removing services (targets) in defined namespaces.
Configuration changes are required in following cases:
- Addition of new scrape config (new kind of service that should be monitored)
- Namespace changing
With basic knowledge of Prometheus we can move to Prometheus Operator - Operators in general are design pattern by coreos built for Kubernetes in the remaining part of this article we will see how it makes the management of Prometheus in Kubernetes a breeze.
Prometheus Operator: what is it?
To simplify let’s descrive in highlevel what the promethues operator offers:
Firstly as many opertators the Prometheus Operator introduces CRD’s (Custom Resource Definitions) the three CRD’s are:
- prometheus - define installation of Prometheus cluster.
- servicemonitor - define how to monitor set of services.
- alertmanager - define alertmanagers cluster.
Secondndly, the operator tracks resources of prometheus and create for each of them following resources:
StatefulSet(with Prom8s itself)
- A Kubernetes
Secretwith the pormetheus.yaml (config of Prometheus) and configmaps.json (config for prometheus-config-reloader).
And finally, the operator tracks resources of servicemonitor and ConfigMaps with rules and based on those rules updates the prometheus.yaml and configmaps.json configurations (eventhough they are keeped in “secrets”).
So what happens with Prometheus in the POD?
Pod consist from two containers:
prometheus- with prometheus binary
prometheus-config-reloader- a side-car container which tracks for changes in
prometheus.yamland if necessary calls reload configuration of Prometheus whilst tracking ConfigMap changes according to certain conditions, then if necessary updates them and restart Prometheus.
Pod uses three volumes:
Config- mounted secret (two files: prometheus.yaml and configmap.json). Binded to both containers.
Rules- emptyDir to where prometheus-config-reloader writes files and prometheus reads. Binded to both but Prom8s only in read mode
Data- prometheus tsdb mounted only to Prometheus container.
How Service Monitor works?
- Prometheus Operator checks Service Monitors (and track their adding/removing/changes). Service monitor selector defined in prometheus resource. (see details in docs)
- For each Service Monitor, if namespaces is not defined then Prometheus Operator identifies (with calls to k8s API ) list of namespaces where services exists with selected labels in Service Monitor.
- Prometheus Operator reads service monitor resources and identified namespaces for generating part of configuration (section scrape_configs) and save configuration to secret.
- Data comes in pod from secrets rely on Kubernetes functionality and prometheus.yaml file is updating.
- When prometheus.yaml file is changed prometheus-config-reloader send http request to prometheus for.
- Prometheus reads configuration file and applies changes in scrape_configs section.
How is handling ConfigMaps with rules?
- Prometheus Operator tracks ConfigMaps, matched by ruleSelector, defined in prometheus resource.
- If ConfigMap is created/deleted, Prometheus Operator updates prometheus.yaml and following Service Monitors logic starts.
- If content of ConfigMap is changing then Prometheus Operator updates file configmap.json (in this file defined list of ConfiMaps and their checksum).
- Data comes in pod from secrets rely on Kubernetes functionality and configmap.json file is updating.
- When configmap.json file is changed prometheus-config-reloader is downloading ConfigMaps in rules directory(emptyDir) and send http request to prometheus for reloading.
- Prometheus reads configuration file and applies changes.
In this part we will focus on how easy it is to add new targets to prometheus managed by the prometheus operator and make those targets discoverable by matching kubernetes labels.
In order to add new scrape target for prometheus first of all we should have a pod with an exporter that will collect metrics from some source (mongodb, rabbitmq, zookeeper or application endpoint, etc).
In kubernetes we can use a deployment, or a daemonset, depending on your use case, and type of exporter. In most cases you will use a deployment for collecting metrics from various endpoints accessible via kube-dns within the internal network of the cluster. In some usecases cases a daemonset should be used, as an example the node-exporter collects operational node metrics and it should be located on each node in the cluster.
In the following example we are going to collect metrics from rabbitmq:
Deployment of rabbitmq exporter:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: rabbitmq-exporter namespace: monitoring spec: replicas: 1 template: metadata: labels: k8s-app: rabbitmq-exporter spec: containers: - env: - name: RABBIT_URL value: http://rabbitmq-management.tikal-io-minikube:15672 - name: PUBLISH_PORT value: "9419" name: rabbitmq-exporter image: kbudde/rabbitmq-exporter ports: - containerPor
In order to take advantage of Kubernetes built-in loadbalancing mechanisem we will create a service which “exposes” the exporter via labels to prometheus to scrapre, the service Resource Definition will remain between restarts of the deployment / pod’s hance in Prometheus the name will always remain the same
rabbitmq-exporter and the ip might vary due to outages / rolling updates / rollbacks etc.
So let’s create a service to serve the rabbitmq-exporter:
Service of rabbitmq exporter:
apiVersion: v1 kind: Service metadata: labels: k8s-app: rabbitmq-exporter name: rabbitmq-exporter namespace: monitoring spec: ports: - name: mqex port: 9419 protocol: TCP targetPort: mqex selector: k8s-app: rabbitmq-exporter
When we have service we can create ServiceMonitor CRD that will select services by matching labels and will add matched services to prometheus target list Automatically.
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: rabbitmq-exporter name: rabbitmq-exporter namespace: monitoring spec: endpoints: - interval: 30s port: mqex namespaceSelector: matchNames: - monitoring selector: matchLabels: k8s-app: rabbitmq-exporter
The above ServiceMonitor will render in the ui to somthing simalr to the following:
Prometheus Operator can help boost your monitoring capabilities in your Kubernetes environment. Comments / Finding are welcome in the form of comments below.
Thank you for reading Stas
We will contact you as soon as possible.