Upgrade Pachyderm

If you need to upgrade Pachyderm from one major version to another, such as from 1.8.x to 1.9.x, follow the instructions in the Migrate between major versions.

Upgrades from one minor version to another, such as from version 1.9.0 to version 1.9.2 do not introduce breaking changes. Therefore, the upgrade procedure is simple and requires little to no downtime.

Warning

Do not use these steps to upgrade between major versions because it might result in data corruption.

To upgrade Pachyderm to a minor version, complete the following steps:

  1. Back up your cluster as described in the Backup and Restore section.

  2. Destroy your Pachyderm cluster:

    pachctl undeploy

  3. Upgrade pachctl by using brew for macOS or apt for Linux:

    Example:

    $ brew upgrade pachyderm/tap/pachctl@1.9
==> Upgrading 1 outdated package:
pachyderm/tap/pachctl@1.9
==> Upgrading pachyderm/tap/pachctl@1.9
...

    Note: You need to specify the version of pachctl to which you want to upgrade. For example, if you want to upgrade 1.9.0 to 1.9.2, add @1.9 at the end of the upgrade path.

  4. Confirm that the new version has been successfully installed by running the following command:

    $ pachctl version --client-only
COMPONENT           VERSION
pachctl             1.9.2

  5. Redeploy Pachyderm by running the pachctl deploy command with the same arguments, fields, and storage resources that you specified when you deployed the previous version of Pachyderm:

    $ pachctl deploy <args>
serviceaccount "pachyderm" created
storageclass "etcd-storage-class" created
service "etcd-headless" created
statefulset "etcd" created
service "etcd" created
service "pachd" created
deployment "pachd" created
service "dash" created
deployment "dash" created
secret "pachyderm-storage-secret" created

Pachyderm is launching. Check its status with "kubectl get all"
Once launched, access the dashboard by running "pachctl port-forward"

    The deployment takes some time. You can run kubectl get pods periodically to check the status of the deployment. When Pachyderm is deployed, the command shows all pods as READY:

    $ kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
dash-482120938-np8cc     2/2       Running   0          4m
etcd-0                   1/1       Running   0          4m
pachd-3677268306-9sqm0   1/1       Running   0          4m

  6. Verify that the new version has been deployed:

    pachctl version
COMPONENT           VERSION
pachctl             1.9.2
pachd               1.9.2

    The pachd and pachctl versions must both match the new version.

Troubleshooting Minor Upgrades

This section describes issues that you might run into when upgrading Pachyderm and provides guidelines on how to resolve them.

StatefulSets vs static persistent volumes

StatefulSets are a mechanism provided in Kubernetes 1.9 and newer to manage the deployment and scaling of applications. It can use Persistent Volume Provisioning or pre-provisioned PV’s, both of which are dynamically allocated from Pachyderm's point of view. Thus, the --dynamic-etcd-nodes flag to pachctl deploy is used to deploy Pachyderm using StatefulSets.

It is recommended that you deploy Pachyderm using StatefulSets when possible. All of the instructions for cloud provider deployments do this by default. We also provide instructions for on-premises deployments using StatefulSets.

If you have deployed Pachyderm using StatefulSets, you can still use the same deploy command to re-deploy Pachyderm. Kubernetes is smart enough to see the previously utilized volumes and re-use them.

etcd re-deploy problems

Depending on the cloud you are deploying to and the previous deployment configuration, we have seen certain cases in which volumes don't get attached to the right nodes on re-deploy (especially when using AWS). In these scenarios, you may see the etcd pod stuck in a Pending, CrashLoopBackoff, or other failed state. Most often, deleting the corresponding etcd pod(s) or nodes to redeploy them or re-deploying all of Pachyderm again will fix the issue.

AlreadyExists errors on re-deploy

Occasionally, you might see errors similar to the following:

Error from server (AlreadyExists): error when creating "STDIN": secrets "pachyderm-storage-secret" already exists

This might happen when re-deploying the enterprise dashboard, for example. These warning are benign.

pachctl connnection problems

When you upgrade Pachyderm versions, you may lose your local port-forward to connect pachctl to your cluster. If you are not using port-forward and you are instead setting pachd address config value to connect pachctl to your cluster, the IP address for Pachyderm may have changed.

To fix problems with connections to pachd after upgrading, you can perform the appropriate remedy for your situation:

  • Re-run pachctl port-forward &, or
  • Set the pachd address config value to the updated value, e.g.: pachctl config update context `pachctl config get active-context` --pachd-address=<cluster ip>:30650