Note
To monitor a Pachyderm cluster with Prometheus, a Pachyderm Enterprise License is required.
Monitor your Pachyderm cluster with Prometheus¶
Pachyderm's deployment manifest exposes Prometheus metrics, allowing an easy set up of the monitoring of your cluster. Only available for self-managed deployments today.
Warning
These installation steps are for Informational Purposes ONLY. Please refer to your full Prometheus documentation for further installation details and any troubleshooting advice.
Prometheus installation and Service Monitor creation¶
-
Helm install kube-prometheus-stack, Prometheus' Kubernetes cluster monitoring using the Prometheus Operator:
-
Get Repo Info
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update
-
Install the Prometheus-operator helm chart
$ helm install <a-release-name> prometheus-community/kube-prometheus-stack
-
-
Create a ServiceMonitor for Pachyderm in Kubernetes:
- Create a myprometheusservice.yaml
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: pachyderm-scraper labels: release: <a-release-name> spec: selector: matchLabels: suite: pachyderm namespaceSelector: matchNames: - default endpoints: - port: prom-metrics interval: 30s
-
Create a ServiceMonitor looking to scrape metrics from
suite: pachyderm
:The prometheus-operator will search for the pods based on the label selector$ kubectl create -f myprometheusservice.yaml
<a-release-name>
and creates a prometheus target so prometheus will scrape the metrics endpointprom-metrics
.In this case, it looks for anything with the label
suite: pachyderm
- which is by default associated with all Pachyderm resources.
Note
Our Service Monitor
pachyderm-scraper
above maps the endpoint portprom-metrics
to a correspondingprom-metrics
port described in Pachyderm's deployment manifest. Let's take a quick look at this file:In the json file, find:$ kubectl -o json get service/pachd
{ "name": "prom-metrics", "port": 1656, "protocol": "TCP", "targetPort": "prom-metrics" }
- Create a myprometheusservice.yaml
Port-Forward¶
One last step before you can collect your metrics: If you followed the instruction above, you can connect to Prometheus by using kubectl port-forward.
$ kubectl port-forward pod/prometheus-<a-release-name>-kube-prometheus-stack-prometheus-0 9090
Browse¶
You can now browse your targets (http://localhost:9090/targets). Run a pipeline of your choice. The pachyderm-scraper
should be visible:
In the ClassicUI tab, you should be able to see the new pachyderm metrics.
References¶
- Kube Prometheus Stack documentation.
- Prometheus documentation.