If you need to upgrade Pachyderm from one major version to another, such as from
1.9.x, follow the instructions in the Migrate between major versions.
Upgrades from one minor version to another, such as from version
1.9.0 to version
1.9.2 do not introduce breaking changes. Therefore, the upgrade procedure is simple and requires little to no downtime.
Do not use these steps to upgrade between major versions because it might result in data corruption.
To upgrade Pachyderm to a minor version, complete the following steps:
Back up your cluster as described in the Backup and Restore section.
Destroy your Pachyderm cluster:
brewfor macOS or
$ brew upgrade firstname.lastname@example.org ==> Upgrading 1 outdated package: email@example.com ==> Upgrading firstname.lastname@example.org ...
Note: You need to specify the version of
pachctlto which you want to upgrade. For example, if you want to upgrade
@1.9at the end of the upgrade path.
Confirm that the new version has been successfully installed by running the following command:
$ pachctl version --client-only COMPONENT VERSION pachctl 1.9.2
Redeploy Pachyderm by running the
pachctl deploycommand with the same arguments, fields, and storage resources that you specified when you deployed the previous version of Pachyderm:
$ pachctl deploy <args> serviceaccount "pachyderm" created storageclass "etcd-storage-class" created service "etcd-headless" created statefulset "etcd" created service "etcd" created service "pachd" created deployment "pachd" created service "dash" created deployment "dash" created secret "pachyderm-storage-secret" created Pachyderm is launching. Check its status with "kubectl get all" Once launched, access the dashboard by running "pachctl port-forward"
The deployment takes some time. You can run
kubectl get podsperiodically to check the status of the deployment. When Pachyderm is deployed, the command shows all pods as
$ kubectl get pods NAME READY STATUS RESTARTS AGE dash-482120938-np8cc 2/2 Running 0 4m etcd-0 1/1 Running 0 4m pachd-3677268306-9sqm0 1/1 Running 0 4m
Verify that the new version has been deployed:
pachctl version COMPONENT VERSION pachctl 1.9.2 pachd 1.9.2
pachctlversions must both match the new version.
Troubleshooting Minor Upgrades¶
This section describes issues that you might run into when upgrading Pachyderm and provides guidelines on how to resolve them.
StatefulSets vs static persistent volumes¶
StatefulSets are a mechanism provided in Kubernetes 1.9 and newer to manage the deployment and scaling of applications. It can use Persistent Volume Provisioning or pre-provisioned PV’s, both of which are dynamically allocated from Pachyderm's point of view. Thus, the
--dynamic-etcd-nodes flag to
pachctl deploy is used to deploy Pachyderm using StatefulSets.
It is recommended that you deploy Pachyderm using StatefulSets when possible. All of the instructions for cloud provider deployments do this by default. We also provide instructions for on-premises deployments using StatefulSets.
If you have deployed Pachyderm using StatefulSets, you can still use the same deploy command to re-deploy Pachyderm. Kubernetes is smart enough to see the previously utilized volumes and re-use them.
etcd re-deploy problems¶
Depending on the cloud you are deploying to and the previous deployment configuration, we have seen certain cases in which volumes don't get attached to the right nodes on re-deploy (especially when using AWS). In these scenarios, you may see the
etcd pod stuck in a
CrashLoopBackoff, or other failed state. Most often, deleting the corresponding
etcd pod(s) or nodes to redeploy them or re-deploying all of Pachyderm again will fix the issue.
AlreadyExists errors on re-deploy¶
Occasionally, you might see errors similar to the following:
Error from server (AlreadyExists): error when creating "STDIN": secrets "pachyderm-storage-secret" already exists
This might happen when re-deploying the enterprise dashboard, for example. These warning are benign.
pachctl connnection problems¶
When you upgrade Pachyderm versions, you may lose your local
port-forward to connect
pachctl to your cluster. If you are not using
port-forward and you are instead setting pachd address config value to connect
pachctl to your cluster, the IP address for Pachyderm may have changed.
To fix problems with connections to
pachd after upgrading, you can perform the appropriate remedy for your situation:
pachctl port-forward &, or
- Set the pachd address config value to the updated value, e.g.:
pachctl config update context `pachctl config get active-context` --pachd-address=<cluster ip>:30650