Monitor with Prometheus

Learn how to monitor a Pachyderm cluster using Prometheus.

February 8, 2023

Pachyderm’s deployment manifest exposes Prometheus metrics, allowing an easy set up of the monitoring of your cluster. Only available for self-managed deployments today.


These installation steps are for Informational Purposes ONLY. Please refer to your full Prometheus documentation for further installation details and any troubleshooting advice.

Prometheus installation and Service Monitor creation #

  1. Helm install kube-prometheus-stack, Prometheus’ Kubernetes cluster monitoring using the Prometheus Operator:
helm repo add prometheus-community
helm repo update
helm install <a-release-name> prometheus-community/kube-prometheus-stack
  1. Create a ServiceMonitor for Pachyderm in Kubernetes:

Our Service Monitor pachyderm-scraper above maps the endpoint port prom-metrics to a corresponding prom-metrics port described in Pachyderm’s deployment manifest. Let’s take a quick look at this file:

kubectl -o json get service/pachd

In the json file, find:

    "name": "prom-metrics",
    "port": 1656,
    "protocol": "TCP",
    "targetPort": "prom-metrics"

Port-Forward #

One last step before you can collect your metrics: If you followed the instruction above, you can connect to Prometheus by using kubectl port-forward.

  kubectl port-forward pod/prometheus-<a-release-name>-kube-prometheus-stack-prometheus-0 9090

If you have an existing Prometheus deployment, please navigate to your Prometheus GUI.

Browse #

You can now browse your targets (http://localhost:9090/targets). Run a pipeline of your choice. The pachyderm-scraper should be visible:

pachyderm scraper target

In the ClassicUI tab, you should be able to see the new pachyderm metrics.

References #

<! –* Find the full list of Pachyderm metrics here: - Pachd metrics - Pipeline metrics ->