Deploy Kubernetes with kops
¶
kops
is one of the most popular open-source tools that enable you to deploy, manage, and upgrade a Kubernetes cluster in the cloud. By using kops
you can quickly spin-up a highly-available Kubernetes cluster in a supported cloud platform.
Prerequisites¶
Before you can deploy Pachyderm on Amazon AWS with kops
, you must have the following components configured:
Configure kops
¶
kops
, which stands for Kubernetes Operations, is an open-source tool that deploys a production-grade Kubernetes cluster on a cloud environment of choice. You need to have access to the AWS Management console to add an Identity and Access Management (IAM) user for kops
.
For more information about kops
, see kops AWS documentation. These instructions provide more details about configuring additional cluster parameters, such as enabling version control or encryption on your S3 bucket, and so on.
To configure kops
, complete the following steps:
-
In the IAM console or by using the command line, create a
kops
group with the following permissions: -
AmazonEC2FullAccess
AmazonRoute53FullAccess
AmazonS3FullAccess
IAMFullAccess
-
AmazonVPCFullAccess
-
Add a user that will create a Kubernetes cluster to that group.
- In the list of users, select that user and navigate to the Security credentials tab.
- Create an access key and save the access and secret keys in a location on your computer.
- Configure an AWS CLI client:
aws configure
-
Use the access and secret keys to configure the AWSL client.
-
Create an S3 bucket for your cluster:
aws s3api create-bucket --bucket <name> --region <region>
Example:
aws s3api create-bucket --bucket test-pachyderm --region us-east-1
System Response:
{
"Location": "/test-pachyderm"
}
-
Optionally, configure DNS as described in Configure DNS. In this example, a gossip-based cluster that ends with
k8s.local
is deployed. -
Export the name of your cluster and the S3 bucket for the Kubernetes cluster as variables.
Example:
export NAME=test-pachyderm.k8s.local
export KOPS_STATE_STORE=s3://test-pachyderm
- Create the cluster configuration:
kops create cluster --zones <region> ${NAME}
- Optionally, edit your cluster:
kops edit cluster ${NAME}
- Build and deploy the cluster:
kops update cluster ${NAME} --yes
The deployment might take some time.
- Run
kops cluster validate
periodically to monitor cluster deployment. Whenkops
finishes deploying the cluster, you should see the output similar to the following:
kops validate cluster
System Response:
Using cluster from kubectl context: test-pachyderm.k8s.local
Validating cluster svetkars.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-us-west-2a Master m3.medium 1 1 us-west-2a
nodes Node t2.medium 2 2 us-west-2a
NODE STATUS
NAME ROLE READY
ip-172-20-45-231.us-west-2.compute.internal node True
ip-172-20-50-8.us-west-2.compute.internal master True
ip-172-20-58-132.us-west-2.compute.internal node True
- Proceed to Deploy Pachyderm on AWS.