AWS + Pachyderm

Learn how to deploy to Pachyderm to the cloud with AWS.

December 5, 2022

Before You Start #

This guide assumes that you have already tried Pachyderm locally and have all of the following installed:


1. Create an EKS Cluster #

  1. Use the eksctl tool to deploy an EKS Cluster:
eksctl create cluster --name pachyderm-cluster --region <region> -profile <your named profile>
  1. Verify deployment:
kubectl get all

2. Create an S3 Bucket #

  1. Run the following command:
aws s3api create-bucket --bucket ${BUCKET_NAME} --region ${AWS_REGION}
  1. Verify.
aws s3 ls

3. Create a Values.yaml #

Version:

4. Configure Helm #

Run the following to add the Pachyderm repo to Helm:

helm repo add pach https://helm.pachyderm.com
helm repo update
helm install pachd pach/pachyderm -f my_pachyderm_values.yaml 

5. Verify Installation #

  1. In a new terminal, run the following command to check the status of your pods:
kubectl get pods
NAME                                           READY   STATUS      RESTARTS   AGE
pod/console-5b67678df6-s4d8c                   1/1     Running     0          2m8s
pod/etcd-0                                     1/1     Running     0          2m8s
pod/pachd-c5848b5c7-zwb8p                      1/1     Running     0          2m8s
pod/pg-bouncer-7b855cb797-jqqpx                1/1     Running     0          2m8s
pod/postgres-0                                 1/1     Running     0          2m8s
  1. Re-run this command after a few minutes if pachd is not ready.

6. Connect to Cluster #

pachctl config import-kube local --overwrite
pachctl config set active-context local
pachctl port-forward
ℹ️

If the connection commands did not work together, run each separately.

Optionally open your browser and navigate to the Console UI.

💡

You can check your Pachyderm version and connection to pachd at any time with the following command:

pachctl version
COMPONENT           VERSION  

pachctl             2.4.1  
pachd               2.4.1