Configure Prometheus
This page shows how you configure LWS to use prometheus metrics.
Before you begin
Make sure you the following conditions are set:
- A Kubernetes cluster is running.
- The kubectl command-line tool has communication with your cluster.
- Prometheus is installed
- Cert Manager can be optionally installed
LWS supports either Kustomize or installation via a Helm chart.
Kustomize Installation
- Enable
prometheusinconfig/default/kustomization.yamland uncomment all sections with ‘PROMETHEUS’.
Kustomize Prometheus with certificates
If you want to enable TLS verification for the metrics endpoint, follow the directions below.
- Set
internalCertManagement.enabletofalsein the LWS configuration. - Comment out the
internalcertfolder inconfig/default/kustomization.yaml. - Enable
cert-managerinconfig/default/kustomization.yamland uncomment all sections with ‘CERTMANAGER’. - To enable secure metrics with TLS protection, uncomment all sections with ‘PROMETHEUS-WITH-CERTS’.
Helm Installation
Prometheus installation
LWS can also supports helm deployment for Prometheus.
- Set
enablePrometheusin your values.yaml file to true.
Helm Prometheus with certificates
If you want to secure the metrics endpoints with external certificates:
- Set
internalCertManagement.enabletofalsein the LWS configuration. - Set both
enableCertManagerandenablePrometheusto true. - Provide values for the tlsConfig, see the example below:
An example for your tlsConfig in the helm chart could be as follows:
...
metrics:
prometheusNamespace: monitoring
# tls configs for serviceMonitor
serviceMonitor:
tlsConfig:
serverName: lws-controller-manager-metrics-service.lws-system.svc
ca:
secret:
name: lws-metrics-server-cert
key: ca.crt
The secrets must reference the cert manager generated secrets.
LWS metrics
Besides the controller-runtime metrics, the LWS controller manager serves these counters on the same endpoint:
| Metric | Labels | Description |
|---|---|---|
lws_group_recreations_total | namespace, name, reason | Groups the pod controller recreated by deleting their leader pod. reason is PodRestart when the restart policy recreated the group after a container of the group restarted or a pod of the group was deleted, and UpgradeSplit when an LWS upgrade split the group across PodGroups. |
lws_group_restart_budget_exhausted_total | namespace, name | Groups terminated because they used up maxGroupRestarts. Such a group stays down until it is recovered explicitly. |
lws_group_restart_backoffs_total | namespace, name | Group recreations delayed by restartBackoff. |
namespace and name identify the LeaderWorkerSet. Its series start at 0 when the controller first reconciles it, so that rate() and increase() see the first event, and are removed when the LeaderWorkerSet is deleted.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.