OpenTelemetry - Basic Kubernetes Setup
This article describes how to deploy a basic setup of OpenTelemetry-supported monitoring and tracing tools on Kubernetes.
OpenTelemetry offers a Kubernetes Operator and Helm charts which can be used to deploy various tools. Since Prometheus is not one of them, we have opted for using the Prometheus Operator to deploy Prometheus, and to deploy Grafana and Jaeger using regular Kubernetes Deployments.
All workloads created in this article, are created in a separate ‘monitoring’ namespace.
Prometheus
To deploy Prometheus, you can use the Prometheus Operator. This can be done in various ways, but in this article, we use the regular, Kubernetes manifests, method.
Before Prometheus Operator can be deployed, we need to install the Custom Resource Definitions.
The latest version can be found in their GitHub releases at https://github.com/prometheus-operator/prometheus-operator/releases/.
Once the Custom Resource Definitions have been created, you can deploy the Prometheus Operator using the following manifest
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
name: prometheus-operator
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
name: prometheus-operator
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- alertmanagers/finalizers
- alertmanagerconfigs
- prometheuses
- prometheuses/finalizers
- thanosrulers
- thanosrulers/finalizers
- servicemonitors
- podmonitors
- prometheusrules
- probes
verbs:
- '*'
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- '*'
- apiGroups:
- ""
resources:
- configmaps
- secrets
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- delete
- apiGroups:
- ""
resources:
- services
- services/finalizers
- endpoints
verbs:
- get
- create
- update
- delete
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
name: prometheus-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-operator
subjects:
- kind: ServiceAccount
name: prometheus-operator
namespace: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
name: prometheus-operator
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
spec:
containers:
- args:
- --kubelet-service=kube-system/kubelet
- --logtostderr=true
- --config-reloader-image=docker.io/jimmidyson/configmap-reload:v0.4.0
- --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.42.1
image: "quay.io/coreos/prometheus-operator:v0.42.1"
name: prometheus-operator
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
allowPrivilegeEscalation: false
ports:
- containerPort: 8080
name: http
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 200Mi
cpu: 200m
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: prometheus-operator
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
name: prometheus-operator
namespace: monitoring
spec:
clusterIP: None
ports:
- name: http
port: 8080
targetPort: http
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
The Operator will watch for Prometheus workloads to be defined before creating Prometheus Services.
Below manifest can be used to deploy a simple Prometheus Service including an Azure Disk for its data storage.
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
namespace: monitoring
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
namespace: monitoring
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: prometheus
allowVolumeExpansion: true
provisioner: kubernetes.io/azure-disk
parameters:
storageaccounttype: Standard_LRS
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus-servicemonitors
namespace: monitoring
spec:
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsUser: 0
serviceAccountName: prometheus
alerting:
alertmanagers:
- namespace: monitoring
name: alertmanager-service
port: alertmanager
serviceMonitorSelector:
matchLabels:
type: servicemonitor
ruleSelector:
matchLabels:
role: alert-rules
prometheus: prometheus-service
resources:
requests:
memory: 400Mi
cpu: 128m
limits:
memory: 800Mi
cpu: 256m
enableAdminAPI: false
logLevel: warn
retention: 15d
storage:
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
storageClassName: prometheus
resources:
requests:
storage: 8Gi
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: monitoring
spec:
ports:
- name: http
port: 9090
protocol: TCP
selector:
prometheus: prometheus-servicemonitors
The Prometheus Service will watch any ServiceMonitor definitions created and ingest data from the defined endpoints. By default, it will make requests to the /metrics path and ingest the data outputted there.
You will want to create a ServiceMonitor for any application you want to monitor.
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: microservice-service-monitor
namespace: monitoring
labels:
type: servicemonitor
spec:
namespaceSelector:
any: true
selector:
matchLabels:
name: microservice
endpoints:
- targetPort: 80
A ServiceMonitor will expose any application it finds based on the defined selectors. In above example, a selector is used to find workloads in any namespace with the name label set to microservice.
Jaeger
The easiest way to deploy Jaeger is through a Kubernetes manifest which creates a Deployment and Service definition.
Before creating the Deployment, we need to create a ConfigMap with the necessary Jaeger configuration.
apiVersion: v1
kind: ConfigMap
metadata:
name: jaeger-collector
namespace: monitoring
data:
collector.yaml: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
jaeger_storage_exporter:
trace_storage: memstore
extensions:
jaeger_query:
storage:
traces: memstore
jaeger_storage:
backends:
memstore:
memory:
max_traces: 100000
service:
extensions:
- jaeger_storage
- jaeger_query
telemetry:
metrics:
address: 0.0.0.0:8888
pipelines:
traces:
exporters:
- jaeger_storage_exporter
receivers:
- otlp
This ConfigMap defines what ports Jaeger should listen on to retrieve traces.
With the ConfigMap in place, we can create the Deployment and Service definitions.
apiVersion: v1
kind: Service
metadata:
name: jaeger-service
namespace: monitoring
spec:
ports:
- name: jaeger
port: 16686
targetPort: jaeger
- name: metrics
port: 8888
targetPort: metrics
- name: otlp-grpc
port: 4317
targetPort: otlp-grpc
- name: otlp-http
port: 4318
targetPort: otlp-http
selector:
app.kubernetes.io/name: jaeger-deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: jaeger-deployment
name: jaeger-deployment
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: jaeger-deployment
template:
metadata:
labels:
app.kubernetes.io/name: jaeger-deployment
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8888"
prometheus.io/scrape: "true"
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- args:
- --config=/conf/collector.yaml
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: jaegertracing/jaeger:latest
imagePullPolicy: Always
name: jaeger-deployment
ports:
- containerPort: 16686
name: jaeger
protocol: TCP
- containerPort: 16686
name: jaeger-query
protocol: TCP
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 4317
name: otlp-grpc
protocol: TCP
- containerPort: 4318
name: otlp-http
protocol: TCP
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 500Mi
cpu: 500m
volumeMounts:
- mountPath: /conf
name: otc-internal
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: collector.yaml
path: collector.yaml
name: jaeger-collector
name: otc-internal
This will deploy Jaeger in the most simplistic way, using memory for storage. Note that this does mean that the data in Jaeger will disappear and reset whenever the application is restarted.
With above configuration, OpenTelemetry traces can be sent to jaeger-service over port 4317 for GRPC or 4318 for HTTP.
Grafana
Grafana can be deployed in various ways. In this example we deploy it as a simple manifest to deploy ConfigMaps, a Deployment and a Service definition.
First, you will want to create a ConfigMap to define the data sources for Grafana:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasource
namespace: monitoring
labels:
grafana_datasource: '1'
data:
datasource.yaml: |-
apiVersion: 1
datasources:
- name: Prometheus
uid: webstore-metrics
type: prometheus
access: proxy
url: http://prometheus-service.monitoring:9090/
isDefault: true
- name: Jaeger
uid: webstore-traces
type: jaeger
url: http://jaeger:16686/jaeger/ui
editable: true
isDefault: false
In above example, two data sources will be created for Prometheus and Jaeger.
Additionally, we want to create a ConfigMap to define Grafana configuration regarding where to locate dashboards.
apiVersion: v1
data:
dashboards.yaml: |-
apiVersion: 1
providers:
- name: '0'
orgId: 1
folder: ''
folderUid: ''
type: file
disableDeletion: false
editable: true
updateIntervalSeconds: 30
options:
path: /etc/grafana/provisioning/dashboards
kind: ConfigMap
metadata:
name: grafana-dashboards
namespace: monitoring
labels:
app: grafana
Having created the necessary ConfigMaps, we can now deploy the Deployment and Service configurations.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: grafana-deployment
name: grafana-deployment
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: grafana-deployment
template:
metadata:
labels:
app.kubernetes.io/name: grafana-deployment
spec:
containers:
- image: docker.io/grafana/grafana:latest
name: grafana-deployment
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
ports:
- containerPort: 3000
name: grafana
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 2500Mi
cpu: 500m
volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-storage
- mountPath: /etc/grafana/provisioning/datasources
name: grafana-datasources
readOnly: true
- mountPath: /etc/grafana/provisioning/dashboards
name: grafana-dashboards
readOnly: true
- mountPath: /etc/grafana/provisioning/dashboards/rabbitmq-overview-dashboard
name: rabbitmq-overview-dashboard
readOnly: true
- mountPath: /etc/grafana/provisioning/dashboards/dotnet-overview-dashboard
name: dotnet-overview-dashboard
readOnly: true
- mountPath: /etc/grafana/provisioning/dashboards/redis-overview-dashboard
name: redis-overview-dashboard
readOnly: true
livenessProbe:
httpGet:
path: /api/health
port: 3000
httpHeaders:
- name: X-Kubernetes-Probe
value: Liveness
timeoutSeconds: 300
periodSeconds: 30
failureThreshold: 3
readinessProbe:
httpGet:
path: /api/health
port: 3000
httpHeaders:
- name: X-Kubernetes-Probe
value: Liveness
timeoutSeconds: 300
periodSeconds: 30
failureThreshold: 3
volumes:
- name: grafana-storage
emptyDir: {}
- name: grafana-datasources
configMap:
name: grafana-datasource
- configMap:
name: grafana-dashboards
name: grafana-dashboards
- name: rabbitmq-overview-dashboard
configMap:
name: rabbitmq-overview-dashboard
- name: dotnet-overview-dashboard
configMap:
name: dotnet-overview-dashboard
- name: redis-overview-dashboard
configMap:
name: redis-overview-dashboard
nodeSelector:
kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
name: grafana-service
namespace: monitoring
spec:
ports:
- port: 3000
protocol: TCP
selector:
app.kubernetes.io/name: grafana-deployment
You may have noticed that, in above example, we specified three volumes and volumeMounts for dashboards.
These dashboards have also been created as ConfigMaps, where the JSON schema of a Grafana dashboard is stored in the below way.
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-overview-dashboard
namespace: monitoring
labels:
app: grafana
data:
kubernetes.json: |-
{
... JSON here
}