Monitor your Pachyderm cluster with Prometheus¶

Pachyderm's deployment manifest exposes Prometheus metrics, allowing an easy set up of the monitoring of your cluster. Only available for self-managed deployments today.

Prometheus installation and Service Monitor creation¶

1. Helm install kube-prometheus-stack, Prometheus' Kubernetes cluster monitoring using the Prometheus Operator:

   • Get Repo Info

     helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
     helm repo update
     

   • Install the Prometheus-operator helm chart

     $ helm install <a-release-name> prometheus-community/kube-prometheus-stack
     

2. Create a ServiceMonitor for Pachyderm in Kubernetes:

   • Create a myprometheusservice.yaml

     apiVersion: monitoring.coreos.com/v1
     kind: ServiceMonitor
     metadata:
     name: pachyderm-scraper
     labels:
         release: <a-release-name>
     spec:
         selector:
             matchLabels:
             suite: pachyderm
         namespaceSelector:
             matchNames:
             - default
         endpoints:
         - port: prometheus-metrics
             interval: 30s
     

   • Create a ServiceMonitor looking to scrape metrics from suite: pachyderm:

     $ kubectl create -f myprometheusservice.yaml
     

     The prometheus-operator will search for the pods based on the label selector <a-release-name> and creates a prometheus target so prometheus will scrape the metrics endpoint prometheus-metrics.

     In this case, it looks for anything with the label suite: pachyderm - which is by default associated with all Pachyderm resources.

   Note

   Our Service Monitor pachyderm-scraper above maps the endpoint port prometheus-metrics to a corresponding prometheus-metrics port described in Pachyderm's deployment manifest. Let's take a quick look at this file:

   $ pachctl deploy local --dry-run > pachd.json
   

   In pachd.json, find:

   - name: prometheus-metrics
       nodePort: 30656
       port: 656
       protocol: TCP
       targetPort: 656
   

Port-Forward¶

One last step before you can collect your metrics: If you followed the instruction above, you can connect to Prometheus by using kubectl port-forward.

$   kubectl port-forward pod/prometheus-<a-release-name>-kube-prometheus-stack-prometheus-0 9090

Browse¶

You can now browse your targets (http://localhost:9090/targets). Run a pipeline of your choice. The pachyderm-scraper should be visible:

In the ClassicUI tab, you should be able to see the new pachyderm metrics.

References¶

• Find the full list of Pachyderm metrics here:
  • Pachd metrics
  • Pipeline metrics
• Kube Prometheus Stack documentation.
• Prometheus documentation.