Deploy Kubernetes with kops¶

kops is one of the most popular open-source tools that enable you to deploy, manage, and upgrade a Kubernetes cluster in the cloud. By using kops you can quickly spin-up a highly-available Kubernetes cluster in a supported cloud platform.

Prerequisites¶

Before you can deploy Pachyderm on Amazon AWS with kops, you must have the following components configured:

• Install AWS CLI
• Install kubectl
• Install kops
• Install pachctl
• Install jq
• Install uuid

Configure kops¶

kops, which stands for Kubernetes Operations, is an open-source tool that deploys a production-grade Kubernetes cluster on a cloud environment of choice. You need to have access to the AWS Management console to add an Identity and Access Management (IAM) user for kops.

For more information about kops, see kops AWS documentation. These instructions provide more details about configuring additional cluster parameters, such as enabling version control or encryption on your S3 bucket, and so on.

To configure kops, complete the following steps:

1. In the IAM console or by using the command line, create a kops group with the following permissions:

   • AmazonEC2FullAccess
   • AmazonRoute53FullAccess
   • AmazonS3FullAccess
   • IAMFullAccess
   • AmazonVPCFullAccess

2. Add a user that will create a Kubernetes cluster to that group.

3. In the list of users, select that user and navigate to the Security credentials tab.
4. Create an access key and save the access and secret keys in a location on your computer.

5. Configure an AWS CLI client:

   aws configure
   

6. Use the access and secret keys to configure the AWSL client.

7. Create an S3 bucket for your cluster:

   aws s3api create-bucket --bucket <name> --region <region>
   

   Example:

   aws s3api create-bucket --bucket test-pachyderm --region us-east-1
   

   System Response:

   {
        "Location": "/test-pachyderm"
   }
   

8. Optionally, configure DNS as described in Configure DNS. In this example, a gossip-based cluster that ends with k8s.local is deployed.

9. Export the name of your cluster and the S3 bucket for the Kubernetes cluster as variables.

   Example:

   export NAME=test-pachyderm.k8s.local
   export KOPS_STATE_STORE=s3://test-pachyderm
   

10. Create the cluster configuration:

    kops create cluster --zones <region> ${NAME}
    

11. Optionally, edit your cluster:

    kops edit cluster ${NAME}
    

12. Build and deploy the cluster:

    kops update cluster ${NAME} --yes
    

    The deployment might take some time.

13. Run kops cluster validate periodically to monitor cluster deployment. When kops finishes deploying the cluster, you should see the output similar to the following:

    kops validate cluster
    

    System Response:

    Using cluster from kubectl context: test-pachyderm.k8s.local
    
    Validating cluster svetkars.k8s.local
    
    INSTANCE          GROUPS
    NAME              ROLE     MACHINETYPE MIN MAX SUBNETS
    master-us-west-2a Master   m3.medium   1   1   us-west-2a
    nodes             Node     t2.medium   2   2   us-west-2a
    
    NODE                                                   STATUS
    NAME                                           ROLE    READY
    ip-172-20-45-231.us-west-2.compute.internal    node    True
    ip-172-20-50-8.us-west-2.compute.internal      master  True
    ip-172-20-58-132.us-west-2.compute.internal    node    True
    

14. Proceed to Deploy Pachyderm on AWS.