Monitor with Prometheus

Learn how to monitor a Pachyderm cluster using Prometheus.

February 8, 2023

Pachyderm’s deployment manifest exposes Prometheus metrics, allowing an easy set up of the monitoring of your cluster. Only available for self-managed deployments today.

⚠️

These installation steps are for Informational Purposes ONLY. Please refer to your full Prometheus documentation for further installation details and any troubleshooting advice.

Prometheus installation and Service Monitor creation #

  1. Helm install kube-prometheus-stack, Prometheus’ Kubernetes cluster monitoring using the Prometheus Operator:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install <a-release-name> prometheus-community/kube-prometheus-stack
  1. Create a ServiceMonitor for Pachyderm in Kubernetes:
ℹ️

Our Service Monitor pachyderm-scraper above maps the endpoint port prom-metrics to a corresponding prom-metrics port described in Pachyderm’s deployment manifest. Let’s take a quick look at this file:

kubectl -o json get service/pachd

In the json file, find:

    {
    "name": "prom-metrics",
    "port": 1656,
    "protocol": "TCP",
    "targetPort": "prom-metrics"
    }

Port-Forward #

One last step before you can collect your metrics: If you followed the instruction above, you can connect to Prometheus by using kubectl port-forward.

  kubectl port-forward pod/prometheus-<a-release-name>-kube-prometheus-stack-prometheus-0 9090

If you have an existing Prometheus deployment, please navigate to your Prometheus GUI.

Browse #

You can now browse your targets (http://localhost:9090/targets). Run a pipeline of your choice. The pachyderm-scraper should be visible:

pachyderm scraper target

In the ClassicUI tab, you should be able to see the new pachyderm metrics.

References #

<! –* Find the full list of Pachyderm metrics here: - Pachd metrics - Pipeline metrics ->