Overview
This post will walk through the steps on how we provisioned our production Kubernetes cluster @mobingi. Some of the bits here are already automated in our case but I will try to include as much details as I can.
Our goals would be the following:
- Provision a Kubernetes cluster on AWS using kops.
- The cluster will have two autoscaling groups: one for on-demand, one for spot instances.
- It’s going to be a gossip-based cluster.
- RBAC is enabled in the cluster.
There is no need to rewrite the otherwise excellent installation instructions written in the kops wiki. Basically, we can just follow the setup instructions there. The instructions in this post will be specific to our goals above.
Gossip-based cluster
We will skip the DNS configuration section as we are provisioning a gossip-based cluster, meaning, the cluster name will be something like "<some-name>.k8s.local". We will be using "mycluster.k8s.local" as our cluster name.
SSH keypair
We will create the keypair in AWS under EC2 -> Key Pairs -> Create Key Pair. We need this keypair when we need to ssh to our cluster nodes. After saving the keypair somewhere, we will generate the public key using the following command:
$ ssh-keygen -y -f mycluster.pem > mycluster.pem.pub
Create the cluster
At this point, we should already have our environment variables set, mainly NAME and KOPS_STATE_STORE. To create the cluster:
# I'm from Japan so I'm using ap-northeast-1 (Tokyo)
$ kops create cluster \
--ssh-public-key mycluster.pem.pub
--zones ap-northeast-1a,ap-northeast-1c \
--authorization RBAC \
--cloud aws ${NAME} \
--yes
It will take some time before the cluster will be ready. To validate the cluster:
$ kops validate cluster
Using cluster from kubectl context: mycluster.k8s.local
Validating cluster mycluster.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-ap-northeast-1a Master m3.medium 1 1 ...
nodes Node t2.medium 2 2 ...
NODE STATUS
NAME ROLE READY
ip-x-x-x-x.ap-northeast-1.compute.internal master True
ip-x-x-x-x.ap-northeast-1.compute.internal node True
ip-x-x-x-x.ap-northeast-1.compute.internal node True
...
Your cluster mycluster.k8s.local is ready
Notice that we only have one instance for our master. We can also opt to have a highly available master using these options, which is generally recommended for production clusters. Based on our experience though, this single master instance setup is good enough for development and/or staging clusters. There’s going to be downtime if master goes down in which case the duration will depend on how long AWS autoscaling group kicks in. During that window, k8s API won’t be accessible but the nodes will continue to work, including our deployed applications.
Spot instance autoscaling group
Once the cluster is ready, we will add another instance group for spot instances. The default instance group created in the previous command, named “nodes”, will be our on-demand group. To add:
$ kops create ig nodes-spots -subnet ap-northeast-1a,ap-northeast-1c
You can then edit using the following contents (modify values as needed):
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: <some-datetime-value-here>
labels:
kops.k8s.io/cluster: mycluster.k8s.local
name: nodes-spot
spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
machineType: t2.medium
maxPrice: "0.02"
maxSize: 10
minSize: 2
nodeLabels:
kops.k8s.io/instancegroup: nodes
spot: "true"
role: Node
subnets:
- ap-northeast-1a
- ap-northeast-1c
We can now update our cluster with the following commands:
# [optional] update our on-demand group's max size to some number
$ kops edit ig nodes
# [optional] preview the changes to be applied
$ kops update cluster ${NAME}
# actual cluster update
$ kops update cluster ${NAME} --yes
# [optional] check if we need rolling update
$ kops rolling-update cluster
# if so, add --yes option
$ kops rolling-update cluster --yes
We can now validate the cluster to see our changes:
$ kops validate cluster
Validating cluster mycluster.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-ap-northeast-1a Master m3.medium 1 1 ...
nodes Node t2.medium 2 4 ...
nodes-spot Node t2.medium 2 10 ...
NODE STATUS
NAME ROLE READY
ip-x-x-x-x.ap-northeast-1.compute.internal master True
ip-x-x-x-x.ap-northeast-1.compute.internal node True
ip-x-x-x-x.ap-northeast-1.compute.internal node True
...
Your cluster mycluster.k8s.local is ready
When our cluster was created, kops also has automatically generated our config file for kubectl. To verify:
$ kubectl cluster-info
Kubernetes master is running at https...
KubeDNS is running at https...
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Setup cluster autoscaler
Clusters created using kops use autoscaling groups, but without scaling policies (at the time of writing). To enable dynamic scaling of our cluster, we will use cluster autoscaler. Before cluster autoscaler deployment, we need to setup some prerequisites.
First, we need to attach the following permissions to master and nodes IAM role. Go to IAM roles console and add an inline policy to the roles created by kops. Role names would be something like:
- master.mycluster.k8s.local
- nodes.mycluster.k8s.local
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
The latest installation instructions can be found here. The general idea is to choose the latest yaml file, update it with your own values, and apply the file using kubectl. The readme file also provides a script that does the download and edit for us. We will be using the following command line arguments for our autoscaler:
command:
- ./cluster-autoscaler
- --cloud-provider=aws
- --v=4
- --stderrthreshold=info
- --scale-down-delay=5m
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --nodes=2:4:nodes.mycluster.k8s.local
- --nodes=2:10:nodes-spot.mycluster.k8s.local
You should update the last two line with your own autoscaling group min/max values. Finally, we deploy our autoscaler with:
$ kubectl create -f cluster-autoscaler.yml
deployment "cluster-autoscaler" created
That’s it. You may also want to install these addons if you like.
SSH to a node
# sample ip's only
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE ...
ip-x.x.x.x.ap-northeast-1... Ready master 1d v1.8.6 1.2.3.4 Debian GNU/Linux 8 (jessie) ...
ip-x.x.x.x.ap-northeast-1... Ready node 1d v1.8.6 1.2.3.5 Debian GNU/Linux 8 (jessie) ...
ip-x.x.x.x.ap-northeast-1... Ready node 1d v1.8.6 1.2.3.6 Debian GNU/Linux 8 (jessie) ...
ip-x.x.x.x.ap-northeast-1... Ready node 1d v1.8.6 1.2.3.7 Debian GNU/Linux 8 (jessie) ...
ip-x.x.x.x.ap-northeast-1... Ready node 1d v1.8.6 1.2.3.8 Debian GNU/Linux 8 (jessie) ...
# default username is `admin`
$ ssh -i mycluster.pem admin@1.2.3.4