Skip to content

Enable logs aggregation with Loki

Note

To deploy and configure a Pachyderm cluster to read logs from Loki, a Pachyderm Enterprise License is required.

Shipping logs to Loki

Loki retrieves logs from pods in Kubernetes through an agent service called Promtail. Promtail runs on each node and sends logs from Kubernetes pods to the Loki API Server, tagging each log entry with information about the pod that produced it.

You need to configure Promtail for your environment to ship logs to your Loki instance. If you are running multiple nodes, then you will need to install and configure Promtail for each node shipping logs to Loki.

Fetching logs

While installing Loki will enable the collection of logs, commands such as pachctl logs will not fetch logs directly from Loki until the LOKI_LOGGING environment variable on the pachd container is true.

This is controlled by the helm value pachd.lokiLogging, which can be set by adding the following to your values.yaml file:

    pachd:
        lokiLogging: true

Pachyderm reads logs from the Loki API Server with a particular set of tags. The URI at which Pachyderm reads from the Loki API Server is determined by the LOKI_SERVICE_HOST and LOKI_SERVICE_PORT environment values automatically added by Loki Kubernetes service.

If Loki is deployed after the pachd container, the pachd container will need to be redeployed to receive these connection parameters.

Note

If you are not running Promtail on the node where your Pachyderm pods are located, you will be unable to get logs for pipelines running on that node via pachctl logs -p pipelineName.

Default Loki Bundle

Per default, Pachyderm ships with an embedded version of Loki that can be deployed by adding the lokiDeploy: true next to the existing lokiLogging: true.

pachd:
  lokiDeploy: true
  lokiLogging: true

In such case, add the following section to your value.yaml:

loki-stack:
  loki:
    persistence:
      enabled: true
      accessModes:
      - ReadWriteOnce
      size: 5Gi
      storageClassName: standard
      annotations: {}
  grafana:
    enabled: true

Grafana Users

  • To use Grafana, deploy with loki-stack.grafana.enabled: true.
  • To access Grafana, run port-forward with kubectl port-forward svc/pachyderm-grafana 4001:80. Change the port 4001 to what suits you best.
  • Login to localhost:4001 with the username admin, and the password found with running kubectl get secret pachyderm-grafana -o jsonpath="{.data.admin-password}" | base64 -d. If enterprise is activated, you will be able to inspect containers logs in your console.

    Container logs

References


Last update: May 19, 2022
Does this page need fixing? Edit me on GitHub