Upgrade Pachyderm¶
Info
- If you need to upgrade Pachyderm between major versions, such as from
1.12.2
to2.0.0
, follow the instructions in the Migrate between major versions. - Prior to 1.11, minor releases required a migration. This is no longer the case.
Upgrades between minor releases or point releases, such as from version 1.12.5
to version 1.13.0
do not introduce breaking changes. Therefore, the upgrade procedure is simple and requires little to no downtime.
Warning
Do not use these steps to upgrade between major versions because it might result in data corruption.
To upgrade Pachyderm from one minor release to another, complete the following steps:
-
Back up your cluster as described in the Backup and Restore section.
-
Destroy your Pachyderm cluster:
pachctl undeploy
-
Upgrade
pachctl
by usingbrew
for macOS orapt
for Linux:Example:
brew upgrade pachyderm/tap/pachctl@1.13
System response:
==> Upgrading 1 outdated package: pachyderm/tap/pachctl@1.13 ==> Upgrading pachyderm/tap/pachctl@1.13 ...
Note: You need to specify the major/minor version of
pachctl
to which you want to upgrade. For example, if you want to upgrade1.12.0
to the latest point release of the 1.12, add@1.12
at the end of the upgrade path. -
Confirm that the new version has been successfully installed by running the following command:
pachctl version --client-only
System response:
COMPONENT VERSION pachctl 1.13.0
-
Redeploy Pachyderm by running the
pachctl deploy
command with the same arguments, fields, and storage resources that you specified when you deployed the previous version of Pachyderm:pachctl deploy <args>
System response:
serviceaccount "pachyderm" created storageclass "etcd-storage-class" created service "etcd-headless" created statefulset "etcd" created service "etcd" created service "pachd" created deployment "pachd" created service "dash" created deployment "dash" created secret "pachyderm-storage-secret" created Pachyderm is launching. Check its status with "kubectl get all" Once launched, access the dashboard by running "pachctl port-forward"
The deployment takes some time. You can run
kubectl get pods
periodically to check the status of the deployment. When Pachyderm is deployed, the command shows all pods asREADY
:kubectl get pods
System response:
NAME READY STATUS RESTARTS AGE dash-482120938-np8cc 2/2 Running 0 4m etcd-0 1/1 Running 0 4m pachd-3677268306-9sqm0 1/1 Running 0 4m
-
Verify that the new version has been deployed:
pachctl version
System response:
COMPONENT VERSION pachctl 1.13.0 pachd 1.13.0
The
pachd
andpachctl
versions must both match the new version.
Troubleshooting point release Upgrades¶
This section describes issues that you might run into when upgrading Pachyderm and provides guidelines on how to resolve them.
StatefulSets vs static persistent volumes¶
StatefulSets are a mechanism provided in Kubernetes 1.9 and newer to manage the deployment and scaling of applications. It can use Persistent Volume Provisioning or pre-provisioned PV’s, both of which are dynamically allocated from Pachyderm's point of view. Thus, the --dynamic-etcd-nodes
flag to pachctl deploy
is used to deploy Pachyderm using StatefulSets.
Tip
It is recommended that you deploy Pachyderm using StatefulSets when possible. All of the instructions for cloud provider deployments do this by default. We also provide instructions for on-premises deployments using StatefulSets.
If you have deployed Pachyderm using StatefulSets, you can still use the same deploy command to re-deploy Pachyderm. Kubernetes is smart enough to see the previously utilized volumes and re-use them.
etcd
re-deploy problems¶
Depending on the cloud you are deploying to and the previous deployment configuration, we have seen certain cases in which volumes don't get attached to the right nodes on re-deploy (especially when using AWS). In these scenarios, you may see the etcd
pod stuck in a Pending
, CrashLoopBackoff
, or other failed state. Most often, deleting the corresponding etcd
pod(s) or nodes to redeploy them or re-deploying all of Pachyderm again will fix the issue.
AlreadyExists
errors on re-deploy¶
Occasionally, you might see errors similar to the following:
Error from server (AlreadyExists): error when creating "STDIN": secrets "pachyderm-storage-secret" already exists
This might happen when re-deploying the enterprise dashboard, for example. These warning are benign.
pachctl
connnection problems¶
When you upgrade Pachyderm versions, you may lose your local port-forward
to connect pachctl
to your cluster. If you are not using port-forward
and you are instead setting pachd address config value to connect pachctl
to your cluster, the IP address for Pachyderm may have changed.
To fix problems with connections to pachd
after upgrading, you can perform the appropriate remedy for your situation:
- Re-run
pachctl port-forward
, or - Set the pachd address config value to the updated value, e.g.:
pachctl config update context `pachctl config get active-context` --pachd-address=<cluster ip>:30650