Unified Deployment

In this guide, we’ll show you how to update your cluster configuration to support a unified installation and deployment of Pachyderm + .

Before You Start #

This guide assumes that you have already completed all of the following:

1. Deployed Pachyderm using one of the cloud deployment guides ( AWS, GCP, or Azure).
2. Added an active Enterprise License Key.
3. Set up TLS (SSL, HTTPS) for your Pachyderm cluster.
4. Set up an OIDC connector for your Pachyderm cluster.

Self-Signed Certificates #

If you are using a self-signed certificate for your implementation, you must update the Determined Helm values.yaml file at .Values.externalCaCertSecretName to include the name of the secret containing the root certificate.

How to Configure a Unified Setup #

1. Create Necessary Secrets #

You will need to create two secrets for Determined:

1. A Determined Enterprise docker image credentials secret (e.g., det-image)

   kubectl create secret docker-registry det-image \
   --docker-server=https://index.docker.io/v1/ \
   --docker-username=<username> \
   --docker-password=<password> \
   --docker-email=<email> \
   --output=json > det-image-secret.json

   Copy

2. A Determined Enterprise admin credentials secret (e.g., det-creds)

   kubectl create secret generic det-creds \
   --from-literal=determined-username=admin \
   --from-literal=determined-password="" \
   --output=json > det-creds-secret.json

   Copy

2. Update the Pachyderm Helm Chart #

1. Open your values.yaml file or generate a local copy using the following command:

   helm get values pachyderm > values.yaml

   Copy
2. Update the determined.enabled field in values.yaml file to true.
3. Update the determined.oidc section of your values.yaml file to include the OIDC provider, client ID, and URLs:

   determined:
     oidc:
       enabled: true
       provider: "" # your oidc.upstreamIDPs.config.id; e.g., Auth0 or Okta
       idpRecipientUrl: "" # https://<proxy.host.value.com>:8080 
       idpSsoUrl: "" # https://<proxy.host.value.com>/dex
       clientId: "determined"
       clientSecretKey: ""
       clientSecretName: ""
       authenticationClaim: ""
       scimAuthenticationAttribute: ""
       autoProvisionUsers: false
       groupsAttributeName: ""
       displayNameAttributeName: ""

   Copy
4. Provide a determined.tlsSecret if applicable.
5. Update the pachd section of your values.yaml file to include the full endpoint address and the name of the Determined admin credentials secret:

   pachd:
     determined:
       apiEndpoint: # https://determined-master-service-internal-<HELM RELEASE NAME>:8082
       credentialsSecretName: det-creds 
     activateEnterprise: true

   Copy

Adding Users to Pipelines #

You can add a determined section to your pipeline specification file and make use of a user via the $DET_USER and $DET_PASS environment variables. This can be used by the user code that run determined work to talk back to Pachyderm and can be used with the Pachyderm SDK.

{
    "pipeline": {
      "name": "<PIPELINE NAME>"
    },
    "description": "<PIPELINE DESCRIPTION>",
    "input": {
      "pfs": {
        "name": "data",
        "repo": "input",
        "branch": "master",
        "glob": "/",
        "emptyFiles": true
      }
    },
    "transform": {
      "cmd": ["/bin/sh"],
      "stdin": ["pip install determined && echo $DET_PASS | det user login $DET_USER && det model list -w WORKSPACE-NAME  > /pfs/out/WORKSPACE-NAME.txt"],
      "image": "python:3.8"
    },
    "determined": {
      "workspaces": ["WORKSPACE-NAME"]
    }
  }

Note

These tokens life cycles are scoped to the jobs and are revoked after the job ends.