Skip to content

Upgrade Pachyderm

Upgrades between minor releases or point releases, such as 2.1.0 to version 2.2.0, should be seamless. Therefore, the upgrade procedure is simple and requires little to no downtime. As a good practice, we recommend that you check the release notes before an upgrade to get an understanding of the changes introduced between your current version and your target.

Warning

Do not use these steps to upgrade between major versions as it might result in data corruption.

Complete the following steps to upgrade Pachyderm from one minor release to another.

Backup your cluster

As a general good practice, start with the backup of your cluster as described in the Backup and Restore section of this documentation.

Update your helm values

When left blank, some values are automatically generated during the installation. If you do not retrieve, then include these values back into your original values.yaml, the upgrade will likely fail.

Warning

You must add a secret to your values.yaml file or supply the generated values back via --set on the command line during the upgrade. Check the troubleshooting section at the end of this page for details on how to retrieve those generated values.

The settings that are autogenerated if they are left blank during an initial installation are:

global:
      postgresql:
            postgresqlPassword
pachd:
      clusterDeploymentID
      rootToken
      enterpriseSecret
      oauthClientSecret

CARRY YOUR CLUSTERID OVER

Aside from retrieving any generated values and inserting them back as credentials or as secrets in your values.yaml, you will need to retrieve your clusterID and carry it over to your new deployment by setting the field pachd.clusterDeploymentID with your ID. Note that your Cluster ID is located in your ~/.pachyderm/config.json. You can also find it by running pachctl config get context <your-context-name>' (runpachctl list context` to list out all contexts).

Upgrade pachctl version

  • To update to the latest version of Pachyderm, run the steps below depending on your operating system:

    • For macOS, run:
    brew tap pachyderm/tap && brew install pachyderm/tap/pachctl@2.2  
    
    • For a Debian-based Linux 64-bit or Windows 10 or later running on
      WSL:
    curl -o /tmp/pachctl.deb -L https://github.com/pachyderm/pachyderm/releases/download/v2.2.0/pachctl_2.2.0_amd64.deb && sudo dpkg -i /tmp/pachctl.deb  
    
    • For all other Linux flavors:
    curl -o /tmp/pachctl.tar.gz -L https://github.com/pachyderm/pachyderm/releases/download/v2.2.0/pachctl_2.2.0_linux_amd64.tar.gz && tar -xvf /tmp/pachctl.tar.gz -C /tmp && sudo cp /tmp/pachctl_2.2.0_linux_amd64/pachctl /usr/local/bin  
    

Note

For a specific target release, specify the targeted major/minor version of pachctl for brew and major/minor/point release for curl in the commands above.

  • Verify that the installation was successful by running pachctl version --client-only:

    pachctl version --client-only  
    

    System Response:

    COMPONENT           VERSION  
    pachctl             <This should display the version you installed>  
    

Helm upgrade

  • Redeploy Pachyderm by running the helm upgrade command with your updated values.yaml:

    helm repo add pach https://helm.pachyderm.com
    helm repo update
    helm upgrade pachd -f my_pachyderm_values.yaml pach/pachyderm --version <your_chart_version>
    

Note

Each chart version is associated with a given version of Pachyderm. You will find the list of all available chart versions and their associated version of Pachyderm on Artifacthub.

  • The upgrade can take some time. You can run kubectl get pods periodically to check the status of the deployment. When Pachyderm is deployed, the command shows all pods as READY:

    kubectl get pods
    
    Once the pods are up, you should see a pod for pachd running (alongside etcd, pg-bouncer, postgres, console etc... depending on your installation).

    System response:

    NAME                     READY     STATUS    RESTARTS   AGE
    pachd-3677268306-9sqm0   1/1       Running   0          4m
    ...
    
  • Verify that the new version has been deployed:

    pachctl version
    

    System response:

    COMPONENT           VERSION
    pachctl             2.2.0
    pachd               2.2.0
    

    The pachd and pachctl versions must both match the new version.

Troubleshoot upgrades

Most of the issues you might run into when upgrading Pachyderm are related to the failure to retrieve then re-set default values that were generated during the first installation.

Here is a list of all the values used by Pachyderm when installing/upgrading a cluster. Some are optional and dependent on your installation choices.

In general, these values can be provided in three different ways:

  • A - (RECOMMENDED in production) Create your secret(s) ahead of your cluster creation

    Create your secret(s) ahead of your cluster creation and provide each secret name in your values.yaml at the time of the first installation.

    Find the list of Secret Keys for each setting and their corresponding secret name field in your values.yaml (Column A) in the table below.

  • B - Provide credentials directly in your values.yaml

    You do not want to create your secrets right away, but provide specific secret credentials directly in your values.yaml or by setting them with a --set argument during the upgrade.

    In this case, (see column B), we will read then insert those values in a default secret pachyderm-bootstrap-config for you. The associated keys in pachyderm-bootstrap-config are listed in column C of the following table.

  • C - Neither A nor B

    You have not created secrets ahead of your installation, nor did you provide values directly in the values.yaml; we created default values for you in the default secret pachyderm-bootstrap-config (See column C below for their key).

Important

It is important to note that if no secret name is provided for the fields mentioned in A, Pachyderm will retrieve the dedicated plain-text secret values in the helm values (B) and populate a generic, default, auto-generated secret (pachyderm-bootstrap-config) at the time of the installation (see keys in C). If no value is found in either one of those two cases, default values are used in pachyderm-bootstrap-config. This default secret is reset at each upgrade, and new default values are created causing the upgrade to fail unless they are retrieved and set back into their appropriate fields.

Secret KEY name
Description
A - Create your secrets ahead
of your cluster creation
B - Pass credentials in values.yaml
C - Neither A nor B - KEY name in default pachyderm-bootstrap-config secret
root-token Root clusterAdmin pachd.rootTokenSecretName pachd.rootToken rootToken
root-token Root clusterAdmin of the enterprise server pachd.enterpriseRootTokenSecretName pachd.enterpriseRootToken enterpriseRootToken
postgresql-password Password to your database global.postgresql.postgresqlExistingSecretName
global.postgresql.postgresqlExistingSecretKey
global.postgresql.postgresqlPassword postgresql-password * in separate secret called postgres
OAUTH_CLIENT_SECRET Oauth client secret for Console
Required if you set Console
console.config.oauthClientSecretSecretName console.config.oauthClientSecret oidcClients[1].secret
enterprise-license-key Your enterprise license pachd.enterpriseLicenseKeySecretName pachd.enterpriseLicenseKey license
pachd-oauth-client-secret Oauth client secret for pachd pachd.oauthClientSecretSecretName pachd.oauthClientSecret oidcClients[0].secret
enterprise-secret Needed if you connect to an enterprise server pachd.enterpriseSecretSecretName pachd.enterpriseSecret enterpriseSecret
upstream-idps The list of dex connectors, each containing Oauth client info connecting to an upstream IDP oidc.upstreamIDPsSecretName oidc.upstreamIDPs idps

Tip

Run kubectl get secret pachyderm-bootstrap-config -o go-template='{{.data.rootToken | base64decode }}' to retrieve the value corresponding to the key rootToken in the default secret pachyderm-bootstrap-config.

Attention users of Pachyderm bundled PostgreSQL

If you are using the built in postgreSQL, you cannot use an existing secret to pass postgreSQL credentials. You need to retrieve then pass your auto-generated credentials in your values (global.postgresql.postgresqlPassword).


Last update: May 18, 2022
Does this page need fixing? Edit me on GitHub