Running Kubernetes on AWS using kops

Overview

This post will walk through the steps on how we provisioned our production Kubernetes cluster @mobingi. Some of the bits here are already automated in our case but I will try to include as much details as I can.

Our goals would be the following:

There is no need to rewrite the otherwise excellent installation instructions written in the kops wiki. Basically, we can just follow the setup instructions there. The instructions in this post will be specific to our goals above.

Gossip-based cluster

We will skip the DNS configuration section as we are provisioning a gossip-based cluster, meaning, the cluster name will be something like "<some-name>.k8s.local". We will be using "mycluster.k8s.local" as our cluster name.

SSH keypair

We will create the keypair in AWS under EC2 -> Key Pairs -> Create Key Pair. We need this keypair when we need to ssh to our cluster nodes. After saving the keypair somewhere, we will generate the public key using the following command:

$ ssh-keygen -y -f mycluster.pem > mycluster.pem.pub

Create the cluster

At this point, we should already have our environment variables set, mainly NAME and KOPS_STATE_STORE. To create the cluster:

# I'm from Japan so I'm using ap-northeast-1 (Tokyo)
$ kops create cluster \
      --ssh-public-key mycluster.pem.pub
      --zones ap-northeast-1a,ap-northeast-1c \
      --authorization RBAC \
      --cloud aws ${NAME} \
      --yes

It will take some time before the cluster will be ready. To validate the cluster:

$ kops validate cluster
Using cluster from kubectl context: mycluster.k8s.local

Validating cluster mycluster.k8s.local

INSTANCE GROUPS
NAME                    ROLE    MACHINETYPE     MIN     MAX     SUBNETS
master-ap-northeast-1a  Master  m3.medium       1       1       ...
nodes                   Node    t2.medium       2       2       ...

NODE STATUS
NAME                                                    ROLE    READY
ip-x-x-x-x.ap-northeast-1.compute.internal              master  True
ip-x-x-x-x.ap-northeast-1.compute.internal              node    True
ip-x-x-x-x.ap-northeast-1.compute.internal              node    True
...

Your cluster mycluster.k8s.local is ready

Notice that we only have one instance for our master. We can also opt to have a highly available master using these options, which is generally recommended for production clusters. Based on our experience though, this single master instance setup is good enough for development and/or staging clusters. There’s going to be downtime if master goes down in which case the duration will depend on how long AWS autoscaling group kicks in. During that window, k8s API won’t be accessible but the nodes will continue to work, including our deployed applications.

Spot instance autoscaling group

Once the cluster is ready, we will add another instance group for spot instances. The default instance group created in the previous command, named “nodes”, will be our on-demand group. To add:

$ kops create ig nodes-spots -subnet ap-northeast-1a,ap-northeast-1c

You can then edit using the following contents (modify values as needed):

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: <some-datetime-value-here>
  labels:
    kops.k8s.io/cluster: mycluster.k8s.local
  name: nodes-spot
spec:
  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
  machineType: t2.medium
  maxPrice: "0.02"
  maxSize: 10
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
    spot: "true"
  role: Node
  subnets:
  - ap-northeast-1a
  - ap-northeast-1c

We can now update our cluster with the following commands:

# [optional] update our on-demand group's max size to some number
$ kops edit ig nodes

# [optional] preview the changes to be applied
$ kops update cluster ${NAME}

# actual cluster update
$ kops update cluster ${NAME} --yes

# [optional] check if we need rolling update
$ kops rolling-update cluster

# if so, add --yes option
$ kops rolling-update cluster --yes

We can now validate the cluster to see our changes:

$ kops validate cluster
Validating cluster mycluster.k8s.local

INSTANCE GROUPS
NAME                    ROLE    MACHINETYPE     MIN     MAX     SUBNETS
master-ap-northeast-1a  Master  m3.medium       1       1       ...
nodes                   Node    t2.medium       2       4       ...
nodes-spot              Node    t2.medium       2       10      ...

NODE STATUS
NAME                                                    ROLE    READY
ip-x-x-x-x.ap-northeast-1.compute.internal              master  True
ip-x-x-x-x.ap-northeast-1.compute.internal              node    True
ip-x-x-x-x.ap-northeast-1.compute.internal              node    True
...

Your cluster mycluster.k8s.local is ready

When our cluster was created, kops also has automatically generated our config file for kubectl. To verify:

$ kubectl cluster-info
Kubernetes master is running at https...
KubeDNS is running at https...

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Setup cluster autoscaler

Clusters created using kops use autoscaling groups, but without scaling policies (at the time of writing). To enable dynamic scaling of our cluster, we will use cluster autoscaler. Before cluster autoscaler deployment, we need to setup some prerequisites.

First, we need to attach the following permissions to master and nodes IAM role. Go to IAM roles console and add an inline policy to the roles created by kops. Role names would be something like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

The latest installation instructions can be found here. The general idea is to choose the latest yaml file, update it with your own values, and apply the file using kubectl. The readme file also provides a script that does the download and edit for us. We will be using the following command line arguments for our autoscaler:

command:
  - ./cluster-autoscaler
  - --cloud-provider=aws
  - --v=4
  - --stderrthreshold=info
  - --scale-down-delay=5m
  - --skip-nodes-with-local-storage=false
  - --expander=least-waste
  - --nodes=2:4:nodes.mycluster.k8s.local
  - --nodes=2:10:nodes-spot.mycluster.k8s.local 

You should update the last two line with your own autoscaling group min/max values. Finally, we deploy our autoscaler with:

$ kubectl create -f cluster-autoscaler.yml
deployment "cluster-autoscaler" created

That’s it. You may also want to install these addons if you like.

SSH to a node

# sample ip's only
$ kubectl get node -o wide
NAME                           STATUS    ROLES     AGE   VERSION   EXTERNAL-IP  OS-IMAGE                      ...
ip-x.x.x.x.ap-northeast-1...   Ready     master    1d    v1.8.6    1.2.3.4      Debian GNU/Linux 8 (jessie)   ...
ip-x.x.x.x.ap-northeast-1...   Ready     node      1d    v1.8.6    1.2.3.5      Debian GNU/Linux 8 (jessie)   ...
ip-x.x.x.x.ap-northeast-1...   Ready     node      1d    v1.8.6    1.2.3.6      Debian GNU/Linux 8 (jessie)   ...
ip-x.x.x.x.ap-northeast-1...   Ready     node      1d    v1.8.6    1.2.3.7      Debian GNU/Linux 8 (jessie)   ...
ip-x.x.x.x.ap-northeast-1...   Ready     node      1d    v1.8.6    1.2.3.8      Debian GNU/Linux 8 (jessie)   ...

# default username is `admin`
$ ssh -i mycluster.pem admin@1.2.3.4
---
If you have any questions or feedback, please reach out @flowerinthenyt.