Disclaimer: The contents of this post were first delivered by me in a talk for Macquarie Cloud Services at VMWare vForum Sydney in November 2017.

Shaun Domingo at vForum Sydney in November 2017 for Macquarie Cloud Services

In 2001 VMWare launched the first commercial server virtualisation products VMWare GSX and ESX which paved the way for the standardisation of virtualisation in every enterprise across the globe. Virtualisation became all pervasive, and by the beginning of the public cloud era the perception was if you weren’t virtualising your data centre you were missing significant cost and efficiency benefits.

Fast forward 17 years and we’re still on the search for cost savings and greater scalability. And why wouldn’t we? Virtualisation is great however it hasn’t allowed us to fully utilise our compute resources.

Containers provide a solution to that issue. Containers are light-weight, standalone, portable units. Think of them as isolated application processes running in user space. Containers should only package the code and application dependencies required to run that process … nothing more.

With containers you achieve greater density because the bloat that comes from installing and running an operating system in line with your application is removed. Instead, containers share the host’s operating system kernel, start instantly and overall use less compute and storage.

Containers take the guess work out of deploying applications. Containers run based on images defined by … you. If you are a developer or engineer, you construct a descriptor that declares what a container image should look like. In the instance of Docker, that’s a Dockerfile. That container is typically based on some other container where most of the heavy lifting has been done for you. As you add instructions to the descriptor, a layered filesystem stores the files that correspond to that instruction and records the diff between itself and the previous instruction, allowing you to uniquely address any layer across the persisted container image history.

There are a couple of types of containers, CoreOS’ rkt (CoreOS now acquired by RedHat) and the most popular option, Docker.

So who’s the target audience here? The truth is, everyone. The power of containers is truly build once, run anywhere. If you’re running Docker, that’s anywhere there is a Docker daemon with access to dependent build images. The old developer mantra “but it works on my machine!” should now actually be true … if it works locally it will work in every single place that container image has been deployed.

Containers are critical in the deployment of automated continuous integration workflows. They ensure consistency between environments, ensure workloads fire up fast and provide us with higher density across our compute and storage resources.

Software Delivery - the last 50 years

The process of shipping software has undergone serious transformation in the last 50 years. In the early days we were bound by limitations in computer hardware, but with the advent of the internet, ecommerce and soon after, the cloud, software delivery has increasingly become an concern about automation. The more things are automated, the greater the saving.

The undeniable future.

So if the deployment of software is already driven by software, where exactly are we going? In many ways the future is shaped by our past.

One thing is certain, “Software is eating the world” (Marc Andreesen). And we’re getting better at creating that software. As humans we are inherently lazy. That plays a big role in our desire for self-improvement and has compelled us to invent ways of generating code and automatically deploy it. It would be hard to deny that software will eventually write software.

But until that day we have a challenge, don’t we? We’re on a constant drive to ship code quickly. Why? Because the faster we release to customers the more responsive we are perceived to be. That drives significantly increased customer satisfaction.

But how do we continuously deploy code without risk or fear? There will always be resistance to consistent change however containerisation and the continuous integration, delivery and deployment model gives a guaranteed way of building, testing, packaging and deploying code across environments in an idempotent way. More about this shortly.

For the first time, containers also allow us to keep an infrastructural building block that’s storable, archiveable and transferrable between environments. This is a win-win for dev and ops, and this allows us to design for platform independence.

Containers offer platform independence, resource density, isolation, speed and operational simplicity. They are portable and run on any modern operating system.

I find this evolution fascinating, because our advancement in the container space runs strong parallels to the advancements in tech in the shipping industry over 200 years.

The nature of what we ship doesn’t change, the way we ship does.

It’s no wonder Docker has built it’s brand and nomenclature around the shipping industry. We’ve had thousands of years to learn from a now well-refined industry. So here’s 3 quick facts about the shipping industry you may not have known.

Software Delivery - the last 50 years

References 1, 2, 3.

Take a look at these fine vessels. They are all marvels of their time. Did you know that containerisation of cargo has been around since the 18th century? They became popular because containers could be transferred from one mode of transport to another, including horse drawn carts!

Although relatively slow in today’s terms, technological innovation around the burning of fossil fuels and the improvement in boiler technology meant that steam engines became bigger, faster and more efficient at moving cargo.

Can you draw any parallels to our recent journey through server, storage and networking technology?

Intermodalism revolutionised the shipping industry.

Intermodalism, a system that is based on the theory that efficiency is vastly improved when the same container can be transported with minimum interruption via different transport modes from an initial place of recept to a final delivery point many kilometres away revolutionised the shipping industry. They could move seamlessly between ships, trucks and trains.

Intermodalism revolutionised the shipping industry.

References: 4 (CC BY-SA 3.0), 5

Did you know until the 1970s virtually all goods were shipped around the world loose? Crammed into the holds of old fashioned cargo ships?

Can you relate to that in your environments? If you’re like me you’re reminiscing, oh the days of FTPing files into random cPanel environments.

Again, does this draw any parallels to your journey?

Globalisation could not have existed without containerisation.

Globalisation could not have existed without containerisation. The advancement in shipping technology brought about worldwide interconnectivity and now 90% of what we buy arrives by ship. And ecommerce has directly played a central role in that development. Studies online say that we’re now shipping in excess of 10 billion tons of cargo a year and that will expand to 23 billion tons by 2060.

Containerisation enabled globalisation

References: 6, 7

As we look more and more to containerisation in the tech world, the parallels are striking, aren’t they? Containers grant so much power. The intermodal nature of Docker and rkt containers across any type of platform is supremely powerful, particularly in this day of Hybrid IT and Cloud. Containers fire up instantaneously, so the power of being able to scale horizontally in seconds is a huge advantage when you need to respond to traffic demands quickly.

The parallels to our development in software delivery is uncanny. Evidently we’ve learned a lot from the advancements in the shipping industry.

The nature of what we ship doesn’t have to change, the way we ship should.

Microservices and Containers

In software, we’ve been breaking down and compartmentalising our applications for years. There’s nothing new here. However, containers have given rise to a new pattern, Microservices. Microservices are effectively self-contained applications that are responsible for a single purpose. They are great because they can maintain their own interface contracts and take on their own release pipeline. This brings challenges too. How will you tag your source and build releases such that they are truly independent? What happens with shared libraries and services, where do they get sourced from?

Regardless of what we’re shipping, containers give us the ability to be much more efficient in deploying our workloads. Pushing to prod shouldn’t be a pain. In fact as much as possible it shouldn’t even be thought of.

The real value of containerisation is realised through true platform independence. The ability to push a workload to any underlying cloud provider. Deploying an App to Macquarie Cloud Services will look and feel similar to deploying to another provider.

Intermodalism is revolutionising the way we deploy workloads.

Intermodalism revolutionising workload deployment

Intermodalism has revolutionised the way we deploy workloads. Continuous Integration, i.e. the ability to constantly integrate with a centralised source code management tool, pull some code, automatically run tests thus verifying the build and automatically shipping across multiple environments truly realises the value of containers. You can fire up build slaves in containers, spawn builds from those containers, then ship a container to a registry and automatically deploy out to a container delivery platform of choice. And your choices here are currently quite large – Kubernetes, Docker Swarm, and Rancher.

Containerisation is enabling standarisation and portability at global-scale.

Containers enable standardisation and portability

Reference: 8

While containers in the shipping industry were a massive enabler for globalisation, containers in the IT industry are evidently enabling agility in small business all the way through to the enterprise. Data dog HQ a company that provides monitoring and analytics, crunches the numbers over their installbase every year. They’ve seen incredible growth over the last 24 months. And for the first time they’re seeing that larger companies are leading the adoption. This means that more workloads are going to be shipped in the enterprise than ever before.

Microservices hierarchy of needs.

Bilgin Ibryam's microservices hierarchy of needs

The benefits of containers are self-evident: fast start-up speed, compact and nimble. They are not magical though! How you tie containers together, monitor them and otherwise operationalise them requires thought.

You’re probably familiar with psychologist Albert Maslo’s hierarchy of needs. In early 2017, a guy by the name of Bilgin Ibryam had a go at defining what our microservices hierarchy of needs might look like. Outside of the infrastructural delivery of containers, how are we going to manage capacity in the cluster? How do we run health checks and auto-heal services that are running slow or are simply unresponsive? How are you going to do service discovery? How will you automatically inject environment configuration into your running containers?

Thankfully the opensource community with the help of some amazing companies have been working on container orchestration platforms: Kubernetes, Swarm, Mesos … to provide the framework required to look after these characteristics. And the most mature enterprise option, Openshift adds a nice PaaS layer on top of Kubernetes.

Where are you going to host your containers? Is it going to be in a secure private cloud, or public? Where are you going to run dev? Locally, or in the cloud? How are you going to scale your virtualisation or compute to adopt?

As you can see there are non-tech considerations to think through also … how will you train your organisation to leverage containers, to automatically test and continuously deliver then deploy, to deliver services that are anti-fragile?

10 years of application deployment has taught us a lot.

10 years of Application Deployment

We’ve come a long way in 10 years. We’ve established that virtualisation is a fundamental building block now, and into the future. It’s not going away. However, the way we leverage that building block will. A lot. Consider Function-as-a-Service and the additional delivery power that provides.

We’ve also learned that:

  • Deploying software by hand is wasteful and error-prone.
  • Testing software manually is expensive and ineffective.
  • Idempotent automation is crucial, and automation is key.
  • Services are best built stateless, immutable.
  • Upgrading systems is laborious and torturous (even when automated).
  • Configuration management (vRealise, Ansible, Chef, Puppet … ) tooling is powerful because it facilitates idempotent automation.
  • Performance matters.

The great advantages sometimes obfuscate some of the difficulties of building and managing containers. For example, how are you going to ensure your workloads are sufficiently firewalled away from each another. How are you going to meet your compliance and regulatory requirements, do containers change that? The truth is that dependency management is hard, and mastering the art of managing microservice dependencies is key to success in this space.

The great news is that the container community have been developing solutions for years now, and while container foundations have nearly completely settled there has still been a lot of movement on the orchestration platform side. Now that the community is pretty firmly settled on Kubernetes as the winning orchestration layer formula, the ecosystem is really starting to stabilise.

All good things to those who wait. Malcolm McLean didn’t build the intermodal shipping container in a day!

In Conclusion.

Having used containers for a few years now, here are my top 10 containerisation tips:

  1. Start with something achievable. Just one thing.
  2. Define your DevOps goals before embarking on transformation.
  3. Stick to the patterns defined by the 12-factor app.
  4. Flock to Kubernetes as it’s the most popular with the best community support. Use RedHat Openshift if you want Enterprise Support and PaaS on top.
  5. Keep the process of continuously integrating as similar as possible to the Development environment. No surprises.
  6. Ensure you have a platform for centralised logging and monitoring. Check out the ELK stack.
  7. If you’re using Docker, ensure you read about Dockerfile best practices.
  8. Always clean your container image of package cruft after you’re done installing images to keep your images small.
  9. If using Docker, make use of multi-stage builds (build vs run).
  10. Docker doesn’t replace configuration management. You still need a way to manage your secrets, config, information across your docker hosts. Even if that is Kubernetes.

There is a lot of buzz regarding containers, but the DevOps community has moved beyond it. I remember visiting the inaugral Docker meetup in Sydney, Australia in June 2014. “Who’s using Docker in Production”, was one of the questions. 2 hands went up. For a couple of years the same question was asked. Thankfully we’ve moved well beyond the concerns around adoption, maturity and security and Docker is now prevalent in continuous delivery pipelines everywhere.

Containers are the new go-to for app deployment and delivery in medium to large enterprises. The journey to containers isn’t as simple as it might first seem. Many organisations I’ve spoken to and worked with are trying to puzzle together how to manage containers and best leverage them to drive agility within their respective organisations.

We’ve now hit a pivotal point in the container journey. Scheduling and managing container workloads is easier than ever before, and we’re now properly thinking about security and what this tech means for our business.

Comment and share

Scene setting

Shifting from a great dev environment to a poor one can be particularly infuriating. I wanted to discuss some frustrations I’ve had shifting a dev environment from Linux/OSX (+ Vagrant + Chef) to Windows.

First up, context. I have a small dev team. They code, they don’t dev the ops. They will, just not yet.

So I wanted to make it as simple as possible for them to get going. Trick is, like many big companies the SOE is Windows and the budget for alternate hardware & software is lean.

In open source we trust. Should be easy right, even if it is Windows.

The easiest and obvious solution would be to run a guest Linux VM of choice and have vagrant fire from within. Due to the cost constraint we’re not buying VMWare workstation and my Devs don’t have a masters in KVM or Xen. Apparently Hyper-V supports nested virt in early preview in 2016 but our Windows aren’t that shiny yet. To VirtualBox we turn.

Except, VirtualBox doesn’t support nested virtualisation either. Well it sort of does if you run a 32 bit OS on a single CPU without VT-x, but damn, that’s slow.

What are we left with? Well we need to break the layers of nesting by shifting the dev environment up a level to the native OS. Argh Windows. Ok if we have to. Again most open source implementations worth their salt are Windows compatible, right? In this case, it was hard going.

Command line git = git scm / Git bash

Cmd.exe looks like a virus, and as far as I care, it is. So git-scm wraps a mingw64 abstraction to provide Git Bash. Surprisingly it does an ok job. You at least get some history and can change the size of the terminal easily.

ChefDK

Installs gracefully, works successfully. We just wanted it so we could use knife and berkshelf, the rest was superfluous.

Vagrant and VirtualBox

Both install perfectly fine. However my first instinct was to use cygwin with vagrant and chef instead of Git scm. In summary, don’t. Git-scm seems to set up paths correctly (unlike Cygwin). Embedded ruby gets set up correctly too.

Git Deploy Keys

A piece of cake on *nix, but Windows? Puttygen? Damn, is that still around? Rather, use Git bash & ssh-keygen to do it a better way.

Summary

It took me close to a day to figure out how to replicate a simple, semi-sensible development environment in Windows.

Why didn’t I just get the Devs to install Ubuntu? Well they probably should, but the corporate world is still bound to MS Office. Furthermore, since they are web Devs they need access to a local version of the Adobe Creative Cloud suite. They can’t fire up windows VMs from *nix because they don’t have access to those images.

(F) Would not recommend.

Comment and share

In a previous post I unearthed how I put together a distributed wordpress cluster running in a multi-node CoreOS cluster.

In this post I’ll explore what it takes to get the same topology running on Kubernetes and CoreOS with help from VirtualBox and Vagrant. Vagrant provides a way to emulate the setup of a Production environment, locally - which is very handy. I started setting up my own cluster in the cloud, but I got stuck with the maze of TLS steps, so I’ll make a todo to figure that out and blog about that another time. I found the Kubernetes guides to be a bit spaghetti at this time.

Fortunately CoreOS have done most of the hard work for us. CoreOS have a starter repo that sets up a single or multi-node CoreOS cluster with Kubernetes installed and ready to go. The repository has scripts to build a Kubernetes bound CoreOS cluster on top of Vagrant, AWS or more generically across self-controlled VMs or bare metal. These scripts configure all the services, TLS certificates and more. If you’ve read the Kubernetes set up documentation you’ll know this kick-starter coreos-kubernetes repo is a massive time saver.

In the remainder of this post I’ll show you how you can get your own multi-node Kubernetes + CoreOS environment running distributed wordpress.

Install VirtualBox and Vagrant

See this link for details.

You’ll need a database

I’m still not sold on persistence / databases in containers. So for the purpose of this demo we’ll set up a standard Ubuntu VM with Vagrant and Virtualbox.

Clone my mysql-vagrant repo locally:

1
2
3
git clone git@github.com:shaundomingo/mysql-vagrant.git
cd mysql-vagrant
vagrant up

In a few minutes you’ll have a VM running with MySQL installed. Have a look here for the database access credentials: https://github.com/shaundomingo/mysql-vagrant.

If you want to SSH onto your database server:

1
vagrant ssh

Clone the CoreOS-Kubernetes GitHub repo

1
git clone https://github.com/coreos/coreos-kubernetes.git

Deploy some VirtualBox VMs for the Kubernetes stack using Vagrant

Determine your topology

The repo comes with everything you need to either set up a single-node or multi-node topology. For this example, we’ll take the multi-node approach. In the multi-node topology, your VMs are split into 3 components:

  • Kubernetes Controller
  • Kubernetes Workers
  • CoreOS etcd

Having these components separated out is a good idea … any one of the components can be updated or taken out in isolation. You wouldn’t want your workers going down if the Kubernetes Controller went under the knife. Although you’d probably want at least 2 instances of each component (particularly etcd!).

1
cd coreos-kubernetes/multi-node/vagrant

Notice in the vagrant directory a config.rb.sample file. For fun, let’s cp this file and edit the file leaving a single controller, 3 workers and one etcd.

1
2
cp config.rb.sample config.rb
vim config.rb

Change config.rb so that you’re left with the following, save and exit. Oh, and make sure you have about 2.5Gb of RAM free or your system could start weeping uncontrollably.

1
2
3
4
5
6
7
8
9
10
$update_channel="alpha"
$controller_count=1
$controller_vm_memory=512
$worker_count=3
$worker_vm_memory=512
$etcd_count=1
$etcd_vm_memory=512

Now we make a couple of minor changes to the Vagrantfile so we can access the Kubernetes deployed app from all worker nodes at the end:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
diff --git a/multi-node/vagrant/Vagrantfile b/multi-node/vagrant/Vagrantfile
index d86efa7..4020ba9 100644
--- a/multi-node/vagrant/Vagrantfile
+++ b/multi-node/vagrant/Vagrantfile
@@ -15,6 +15,7 @@ $worker_count = 1
$worker_vm_memory = 512
$etcd_count = 1
$etcd_vm_memory = 512
+$port_forward_range_start = 30100
CONFIG = File.expand_path("config.rb")
if File.exist?(CONFIG)
@@ -181,7 +182,9 @@ Vagrant.configure("2") do |config|
end
workerIP = workerIP(i)
+ hostPort = ($port_forward_range_start + i) - 1
worker.vm.network :private_network, ip: workerIP
+ worker.vm.network :forwarded_port, guest: $port_forward_range_start, host: hostPort, protocol: 'tcp'
provisionMachineSSL(worker,"worker","kube-worker-#{workerIP}",[workerIP])

Deploy your stack

I hate to say it, but it’s as easy as:

1
vagrant up

Why is it easy? Well the CoreOS team have done the hard work for you. Take a look at the Vagrantfile. The Vagrantfile does the following:

  • Dynamically determines IPs for the CoreOS cluster
  • Sets up your TLS config (see ../ssl/init-ssl-ca) for magic
  • Provisions each etcd (needs to be up first), controller, then worker VMs bundling the ssl files as a tar in each VM, ready for consumption

Assuming you’ve run vagrant up, watch it do all the things. When it’s finished you should be able to view the state of the cluster and ssh into every VM:

1
2
3
4
vagrant status
vagrant ssh e[1]
vagrant ssh c[1]
vagrant ssh w[1|2|3]

Kubernetes is using etcd to store configuration and data relating to the Kubernetes registry and scheduler. You can be fairly certain that if that guy goes down, it’s going to cause some issues.

SSH into all of your nodes and check on their state. If you’re on a Macbook Air, you could feel some heat and hear some serious RPMs from your CPU fan. That’s ok, all in good fun.

Log onto the etcd VM (e1) and take a look at the values in etcd.

1
2
vagrant ssh e1
etcdctl ls --recursive

Later you’ll be able to check etcd to see how kubernetes stores info about replication controllers, services, endpoints, namespaces and events:

1
2
3
vagrant ssh e1
etcdctl get /registry/controllers/default/wordpress
{"kind":"ReplicationController","apiVersion":"v1","metadata":{"name":"wordpress","namespace":"default","selfLink":"/api/v1/namespaces/default/replicationcontrollers/wordpress","uid":"7df15e33-ba51-11e5-a115-0800272b2ba2","resourceVersion":"11740","generation":1,"creationTimestamp":"2016-01-13T23:58:07Z","labels":{"app":"wordpress"}},"spec":{"replicas":3,"selector":{"app":"wordpress"},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"wordpress"}},"spec":{"containers":[{"name":"wordpress","image":"sdomsta/wp","ports":[{"name":"http-server","containerPort":80,"protocol":"TCP"}],"env":[{"name":"WORDPRESS_DB_NAME","value":"app"},{"name":"WORDPRESS_DB_HOST","value":"33.33.33.33:3306"},{"name":"WORDPRESS_DB_USER","value":"app"},{"name":"WORDPRESS_DB_PASSWORD","value":"app"},{"name":"WORDPRESS_CACHE_S3_KEY","value":"akey"},{"name":"WORDPRESS_CACHE_S3_SECRET","value":"asecret"},{"name":"WORDPRESS_CACHE_S3_BUCKET","value":"coreos-wordpress"},{"name":"WORDPRESS_CACHE_HOME","value":"http://wordpress.external"},{"name":"WORDPRESS_ADMIN_ENABLED","value":"true"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst"}}},"status":{"replicas":3,"observedGeneration":1}}

Configure your shell to talk to your new kube cluster

You’ll need to download the kubernetes kubectl utility (go binary) for your operating system.

Take a look at the kubectl releases page to gather the latest version number then plug it into the following curl commands (s/VERSION/actual version/).

First, download the latest kubernetes binary for *nix (go Go!):

1
curl -O https://storage.googleapis.com/kubernetes-release/release/vVERSION/bin/linux/amd64/kubectl

Or an OS X workstation:

1
curl -O https://storage.googleapis.com/kubernetes-release/release/vVERSION/bin/darwin/amd64/kubectl

Put the kubectl binary somewhere on your path and/or change your path in your .zshrc or .bashrc so that it’s accessible each time your terminal loads. The easiest place to put this is /usr/local/bin.

Now we need to set up the configuration so your client can talk to your new multi-node cluster:

1
2
3
4
5
cd coreos-kubernetes/multi-node/vagrant
kubectl config set-cluster vagrant-multi-cluster --server=https://172.17.4.101:443 --certificate-authority=${PWD}/ssl/ca.pem
kubectl config set-credentials vagrant-multi-admin --certificate-authority=${PWD}/ssl/ca.pem --client-key=${PWD}/ssl/admin-key.pem --client-certificate=${PWD}/ssl/admin.pem
kubectl config set-context vagrant-multi --cluster=vagrant-multi-cluster --user=vagrant-multi-admin
kubectl config use-context vagrant-multi

Test to see your Kubernetes cluster is running

1
2
3
4
5
6
7
8
$ kubectl get nodes
NAME LABELS STATUS AGE
172.17.4.201 kubernetes.io/hostname=172.17.4.201 Ready 51m
172.17.4.202 kubernetes.io/hostname=172.17.4.202 Ready 51m
172.17.4.203 kubernetes.io/hostname=172.17.4.203 Ready 51m
$ kubectl cluster-info
Kubernetes master is running at https://172.17.4.101:443

If you see what I see, you’re heading in the right direction. Hoorah!

Let’s deploy wordpress

In a previous post I tried to get 10s of wordpress instances firing up through CoreOS’ native scheduler fleet. The challenge is that CoreOS does not provide native load balancing across workloads. This is where Kubernetes comes in. Kubernetes has “replication controllers” and “services”. Replication controllers set up a strategy for deploying your services into “pods” as containers across your cluster. Services abstract access to Kubernetes pods via endpoints. A pod is a colocated group of applications running with a shared context. You never deal directly with pods, instead, replication controllers which manage the lifecycle of pods.

In my previous CoreOS exercise I used vulcand + etcd as the mechanism for service discovery and load balancing. I did this by using fleet unit files which would poll and listen for new containers as they were fired up and add the relevant service endpoint into vulcand when ready. While the process worked, there were many problems:

  1. Vulcand is still new and documentation is understandably minimal
  2. So much escaping in unit files, it called out for a DSL
  3. Sticky-sessions was a to-do feature
  4. Deploying unit files felt repetitive, boring and hacky. It just didn’t feel right.

So with that, it’s time to look to Kubernetes, today’s gold class scheduling standard.

We take the CoreOS + Fleet + Etcd + Vulcand + Docker architecture …

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
+---------80----------+
| |
| External LB |
| |
+--------8888---------+
/ | \
/ | \
+-------8888---------+ +--------8888--------+ +-------8888---------+
| | | | | |
| core-01 | | core-02 | | core-03 |
| | | | | |
+--------------------+ +--------------------+ +--------------------+
| vulcand | | vulcand | | vulcand |
+--------------------+ +--------------------+ +--------------------+
| discovery sidekick | | discovery sidekick | | discovery sidekick |
+--------------------+---+--------------------+--+--------------------+
| wp wp wp wp wp wpn | | wp wp wp wp wp wpn | | wp wp wp wp wp wpn |
+--------------------+ +--------------------+ +--------------------+

NB: EtcD on each node in this example.

… and turn it into the Kubernetes architecture (CoreOS + Kubernetes + Etcd + Docker):

NB: See this link for a full breakdown on the Kubernetes architecture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
+---------80----------+
| |
| External LB / FW |
| |
+--------30100--------+
/ | \
/ |
+-------30100-------+ +-------30100-------+ +-------30100-------+
| * /| | * \ | | * |
| +-------+ +-----+ | | +-------+ +-----+ | | +-----+ +-------+ |
| |kubelet| |Proxy| | | |kubelet| |Proxy| | | |Proxy| |kubelet| |
| +-------+/+-----+ | | +-------+/+-----+ | | +-----+ +-------+ |
| | /\ | | | | /\ | | | | /\ | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
| | Pod | | Pod | | | | Pod | | Pod | | | | Pod | | Pod | |
| | c1 | | c(l) | | | | c(m) | | c(n) | | | | c(o) | | c(p) | |
| +------+ +------+ | | +------+ +------+ | | +------+ +------+ |
+--w1---------------+ +--w2---------------+ +--w3---------------+

NB: c(l,m,n,o,p) are containers. For our exercise we’ll deploy one Wordpress container per pod, but you don’t have to. I like to think of pods as a great way to isolate your application components. Tightly coupled would work as peas in a pod, ha!

You may recall that we deployed two more Virtual Machines, c1 (the controller) and e1 (etcd). Your kubectl binary makes a connection to the master components (authz, REST api) on the controller (c1) virtual machine. The Controller manages workloads on all your worker nodes.

1
2
3
4
5
6
+--------------------+ +--------------------+
| CoreOS | | Kubernetes |
| etcd |<->| controller node |<-> Worker Cluster kubelets
| (distributed | | (Authz | REST API |
| storage) | | Scheduler / RC) |
+--------------------+ +--------------------+

Note on our local machine Kubernetes does not support load balancers across multiple worker nodes (how could it?). So we use a NodePort service type in our replication controller (RC) which exposes our Kube proxy service (which runs on every worker node) on an externalised port 30100 which we port forward to from our local machine as 30100, 30101, 30102 per worker node w1,2,3 in our Vagrantfile (these were the changes we made earlier to get access).

Since we’ve already set up a mysql database, deploying wordpress is particularly simple. We just pass in the required database connection details as environment variables in the replication controller container spec.

We’ll get 3 pods running, each running a single Wordpress docker container.

Download my sample repo containing this simple Kubernetes example

Clone my repo, take a look at the files in the repo and then run the following:

1
2
git clone git@github.com:shaundomingo/kubernetes-examples.git
cd kubernetes-examples/clusterable-wordpress

Since I’ve previously built a wordpress container, I’m going to leverage that container image sdomsta/wp, available on Dockerhub.

You should now be in the clusterable-wordpress directory, ready to get some wordpress pods running.

Get wordpress scheduled

Before you do this, ensure you have your mysql-vagrant VM running in VirtualBox! The wordpress container will need it to fire up.

From here, create a replication controller that will ensure 3 replicas of your wordpress container spec is up and running. If an error occurs within any pod, the kubernetes scheduler will try to revive them by restarting any failed container. You’ll create the replication controller from a spec (wordpress-controller.json) that gets written into etcd for persistence via the kubernetes controller services.

1
$ kubectl create -f wordpress-controller.json

The wordpress app is now live. But you won’t yet be able to hit it. This is where we come to understand the kubernetes proxy.

Create a service

We now need a service that will expose your pods to the outside world. There are a number of ways to do this including type=LoadBalancer and type=NodePort.

Since we don’t have a load balancer on our machine that can talk through to our host-only interface, we’ll create a simple NodePort service that will listen on a port 30100 on every worker node, via the kube proxy.

We’ll then be able to hit the service on each worker node’s VirtualBox host-only interface (http://172.17.4.201:30100, http://172.17.4.202:30100, http://172.17.4.203:30100), or on the port forward port on your local machine (http://127.0.0.1:30100, http://127.0.0.1:30101, http://127.0.0.1:30102 respectively).

1
2
3
4
5
6
7
$ kubectl create -f wordpress-service.json
You have exposed your service on an external port on all nodes in your
cluster. If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:30100) to serve traffic.
See http://releases.k8s.io/release-1.1/docs/user-guide/services-firewalls.md for more details.
service "wpfrontend" created

Up, up and away. Try and hit your wordpress instance: http://172.17.4.201:30100/.

If all is well you should now see the wordpress installation screen.

Successful deployment of distributed Wordpress via Kubernetes

How can I validate it’s all working successfully?

Use kubectl to query the kubernetes API to ensure all the components are up and running and published correctly.

1
$ kubectl get rc,pods,services -o wide

You’ll note a single wordpress replication controller, 3 pods (running your wordpress containers) and a wpfrontend service of type NodePort (that is exposing 30100). You know so because you saw so.

1
2
3
4
5
6
7
8
9
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
wordpress wordpress sdomsta/wp app=wordpress 3 2h
NAME READY STATUS RESTARTS AGE NODE
wordpress-cyphn 1/1 Running 0 2h 172.17.4.203
wordpress-e5lb9 1/1 Running 0 2h 172.17.4.202
wordpress-xuxon 1/1 Running 0 2h 172.17.4.201
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
kubernetes 10.3.0.1 <none> 443/TCP <none> 2h
wpfrontend 10.3.0.6 nodes 80/TCP app=wordpress 2h

Each pod fires up a wordpress container using the specified sdomsta/wp docker image. Keep running the following command until all pods are ‘Running’:

1
kubectl get pods

You can check that your wpfrontend service is connected to ports exposed by your pods by looking at your endpoints:

1
2
3
4
$ kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 172.17.4.101:443 2h
wpfrontend 10.2.59.2:80,10.2.60.2:80,10.2.65.3:80 2h

That command is interesting because it demonstrates the magic of Kubernetes. Each pod gets allocated its own private endpoint IP. The kube proxy magically knows how to connect to it and balance traffic across each of the endpoints. Very cool.

For fun you should try and break things. Try and scale up until your machine dies. Turn off your database VM, turn off a CoreOS worker VM. Did Kubernetes reschedule your pods? I had some issues when my machine was heavily resource constrained, do you?

How do I scale?

The moment you’ve surely been waiting for. Well good news, it’s very simple, and super fast.

1
2
3
$ kubectl scale rc wordpress --replicas=30
replicationcontroller "wordpress" scaled
$ kubectl get pods

Watch the pods as they move from Pending to Running (if you’re quick enough!). When you’re done playing, scale back down, noticing how Kubernetes selects the pods to scale back down.

1
$ kubectl scale rc wordpress --replicas=3

Do you get the same three containers after scaling back down? Maybe, but unlikely.

Some other things I’ve learned

Debugging containers in pods that won’t start

If your pod won’t Run successfully, it may be very difficult to diagnose the actual problem. I found that entrypoint.sh logs didn’t filter their way through to the kubectl logs. To get around this I ssh’d onto the coreos worker in question (vagrant ssh w(n)) and took a look at /var/log/containers. That directory contains a log per launched container, which is mighty helpful to examine faults.

Interestingly journalctl didn’t list the errors either.

Easy access to the kubernetes API

There is an authorization layer in front of all Kubernetes services. Getting access to the kubernetes controller API is best done via kubectl proxy which will proxy the remote API through a port (default 8001) on your local machine.

This is advantageous because other Kubernetes services like the kube-ui and kubedash are published from the controller node, and most of those utilities access the kubernetes API on it’s internally published port 443 which is not available to you outside of the guest on 443 (unless you port forward 443 which would be strange)…

If you need to regenerate your CoreOS/Kubernetes cluster

1
2
cd coreos-kubernetes/multi-node/vagrant
vagrant destroy --force; vagrant up

I’ve done this about 50 times I’d say.

My host system is crashing when deploying replication controllers / services

I think it might be VirtualBox. Make sure you’re on the latest version of VirtualBox and try again.

More iceberg to come

That’s just the tip of the iceberg. There is so much more to show and tell regarding Kubernetes. I’m most interested in utilities like the kube-ui / kubedash and usage monitoring and how kubernetes deals with high workloads, resource management and exception scenarios. In my next post I’ll talk about how you monitor the health of your replicated pods.

Comment and share

You have a couple of choices when it comes to using CoreOS on Cloudstack and XenServer. You can always follow the CoreOS tutorial which gives you the simplest way to get going.

If you want to build a template from the latest (or particular) alpha, beta or stable CoreOS version , follow these simple steps:

  • Copy the CoreOS ISO URL of the required type (Alpha / Beta / Stable) from: https://coreos.com/docs/running-coreos/platforms/iso/. You’ll use this URL in Cloudstack shortly.
  • Upload the ISO into Cloudstack -> Templates -> Select view: ISO -> Register ISO
    • Name and Description: coreos-(alpha|beta|stable)-version
    • URL: The Cloudstack Management Server will download the file from the specified URL.
    • Zone: Choose the zone where you want the ISO to be available, or All Zones to make it available everywhere.
    • Bootable: Checked.
    • OS Type: Other 64-bit.
    • extractable: Your call. This determines whether users can generate a download URL for your ISO.
    • Public: Checked.
    • Featured: Your call. If featured it will be shown in the featured ISOs list of ISOs in the Instance deployment wizard.
  • Create an instance from the ISO choosing an appropriately sized disk offering. I choose 50gb as this is more than enough for my needs. You’ll want at least 2GB of RAM (selecting 512mb causes panic on startup, so aim higher).
  • Once the VM is created, navigate to the instance and click to view the console
  • Install CoreOS onto xvda:
    • sudo coreos-install -d /dev/xvda -C stable -o cloudstack
      Image of Instance Console installing CoreOS
  • Stop the VM and template.
    • Take a snapshot of your instance’s ROOT Storage volume
    • Once the snapshot is complete, create a template from the snapshot, ensuring the template’s OS Type is set to ‘Other 64-bit’
      • Name and Description. coreos-(alpha|beta|stable)-version-template
      • OS Type. ‘Other 64-bit’
      • Public. Checked.
      • Password Enabled: Checked. Although, you won’t use a password, you’ll use SSH keys.
      • Dynamically Scalable: Unchecked.
    • Wait a while for the template to be created.
    • Copy the template to all appropriate zones

You now have a clean XenServer CoreOS template that is ready for all your distributed computing needs.

Repeat this process when you want to build an alternate CoreOS template.

You will quickly find the need to pass userdata to your VM to configure your SSH keys. This can be done by using the Cloudstack API.

Comment and share

Over the last four years I have been fortunate to be part of a cloud computing company that’s thrown the kitchen sink at cloud innovation. For us the differentiator was making it crazy easy to deploy and maintain an app. This article is a personal reflection on the consideration and use of Chef in a Platform-as-a-Service (PaaS) to which I was a key contributor.

Identifying the key functional and non-functional requirements is the first step in the journey of product development. To formulate this list, you need to talk to lots of people. How will your customer deploy, update, rollback and debug their service? For a PaaS this forms the key interaction contract between you and a user.

Waiting for anything in this game is pain. One fundamental Chef feature that will hold you back in building your PaaS: idempotency. Don’t get me wrong idempotency is a necessity. You need the confidence that you’ll get the same resultant state each time you apply a set of resources based on a given configuration. Chef’s done a great job at ensuring you have access to pre and post-processors to help skip as much processing as possible (c.f. not_ifs and only_ifs).

If idempotency is so good, why is it so bad? When you’re talking about software, applying the same instructions over and over again is costly. I witnessed the evolution of a primary wrapper cookbook that leveraged over 20 community cookbooks to do its thing. It would take a convergence 30 seconds to figure out that there was nothing at all to do. This is bad. If a convergence is triggered with the view of deploying a service the application LWRP isn’t engaged until the end of the run … you have to wait at least 25 to 30 seconds for the activity you care about to occur. Time is money.

Why does it take that long? To answer that question you need to consider the componentary required to drive a PaaS. How would you leverage Chef to perform the following functions? Would you at all?

  • Provisioning / Virtualisation
  • Orchestration
  • Multi-tenancy / isolation
  • Package management
  • OS Updates / Patching
  • Management (Credentials / logging)
  • Monitoring
  • User Workflow and integration into 3rd party ecosystems
  • Application deployment & database migrations

So what are the options? You could carefully craft your cookbooks to only execute the bits you care about at any point in time, but that goes against the grain of Chef. You could update the runlist of a node or nodes prior to executing the bit you care about, but at some point you’re going to hit an exception and the node won’t return to the starting state.

The answer in my view is fairly simple but in reality difficult to tackle. Choose the right tool for the right job. Provisioning is best handled by platform specific APIs. Orchestration can be handled by any number of distributed, coordination systems. And of course you should consider containers, they take away the pain of OS-level package management.

Comment and share

CoreOS can seem daunting at first. This tutorial is built to encourage you in your journey and to demonstrate the power of this minimalistic operating system.

Exercise 1: Boot up a 3 machine coreos cluster

1a: Install prerequisite software

1b: Clone the coreos-vagrant repo

1
2
mkdir ~/coreos; cd ~/coreos
git clone https://github.com/coreos/coreos-vagrant.git

1c: modify your config.rb

1
2
3
4
cd coreos-vagrant
cp config.rb.sample config.rb
vim config.rb

change $num_instances:

1
$num_instances=3

change $update_channel:

1
$update_channel=‘beta’

change $forwarded_ports:

1
$forwarded_ports = {8000 => 8888}

You’ll see why later.

1d: get yourself a discovery token

Visit https://discovery.etcd.io/new?size=3

You’ll need a new one every time you rebuild a cluster.

Copy the entire URL provided in the body of the response somewhere safe.

1e: modify your user-data and turn on etcd2

etcd2 is a complete reimplementation based on research and learning about Raft. While backwards compatibility exists, we’d be silly not to go for the latest and greatest.

Update coreos-vagrant/user-data by changing the following.

Set the reboot strategy on CoreOS update:

1
2
update:
reboot-strategy: etcd-lock

Disable etcd properties in favour of etcd2:

1
2
3
#etcd:
# addr: $public_ipv4:4001
# peer-addr: $public_ipv4:7001

Add your discovery token from 1D (warning: you need to do this each time):

1
discovery: https://discovery.etcd.io/YOUR_DISCOVERY_TOKEN_HERE_FROM_1D

Stop etcd.service from starting in favour of etcd2:

1
2
#- name: etcd.service
# command: start

Enable etcd2:

1
2
- name: etcd2.service
command: start

1f: magic

In the coreos-vagrant directory run:

1
2
vagrant up
vagrant status

Within a few minutes, you’ll have 3 CoreOS machines, ready to go all based on the beta channel.

Exercise 2: Interact with your cluster

2a: Download fleetctl

1
PATH=$PATH:/path/to/fleetctl
  • Restart your terminal

2b: Start your engines

1
2
cd /path/to/coreos-vagrant
eval $(ssh-agent)

Add the correct vagrant .ssh identify

1
vagrant ssh-config | sed -n "s/IdentityFile//gp" | uniq | xargs ssh-add

2c: Get ready for fleet

1
export FLEETCTL_TUNNEL="127.0.0.1:$(vagrant ssh-config | grep -i 'port' | awk '{print $2; exit}')"

Remove your fleet known hosts

Fortunately there is a specific known_hosts file to clean up. Do this after every cluster rebuild.

1
rm ~/.fleetctl/known_hosts

2e: List your machines via fleet

You should see your 3 CoreOS machines listed.

1
fleetctl list-machines

SSH onto one of your CoreOS hosts. exit when done.

1
fleetctl ssh (machine-id)

Exercise 3: Use CoreOS with scale

3a: Initial setup and play

1
mkdir ~/coreos; cd ~/coreos

Download some sample unit files:

1
git clone git@github.com:shaundomingo/coreos-units.git

Hello unit

Submit the hello unit and watch the containers say hello.

1
2
3
4
5
cd ~/coreos/coreos-units/hello
fleetctl submit hello.service; fleetctl start hello.service
fleetctl journal -f hello.service

Once you’re done greeting yourself, stop the unit and destroy it.

1
fleetctl destroy hello.service

3b: Deploy at scale - add config

Hello worlds suck. Let’s deploy something that’s interesting with scale!

Let’s fire up lots of Wordpress instances. We won’t go into detail in this tutorial, but this is what we’ll build (minus the top Load Balancer - but expect it’s easy).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
+---------80----------+
| |
| Load Balancer |
| |
+--------8888---------+
/ | \
/ | \
+-------8888---------+ +--------8888--------+ +-------8888---------+
| | | | | |
| core-01 | | core-02 | | core-03 |
| | | | | |
+--------------------+ +--------------------+ +--------------------+
| vulcand | | vulcand | | vulcand |
+--------------------+ +--------------------+ +--------------------+
| discovery sidekick | | discovery sidekick | | discovery sidekick |
+--------------------+---+--------------------+--+--------------------+
| wp wp wp wp wp wpn | | wp wp wp wp wp wpn | | wp wp wp wp wp wpn |
+--------------------+ +--------------------+ +--------------------+

NOTE: The wp instances will talk through to a single database server. While databases on CoreOS and containers is entirely possible, and done, I prefer to exclude the database from this setup. Set yourself up with a VM, install mariadb (/mysql) and follow the wordpress database installation instructions.

Edit the unit files for wordpress@.service and wordpress-admin.service.

Edit wordpress@.service …

1
vim coreos-units/clusterable-wordpress/wordpress/wordpress@.service

… and change line 12 to incorporate your database and S3 creds:

1
ExecStart=/usr/bin/docker run -rm --name wordpress-%i -e WORDPRESS_DB_NAME=dbname -e WORDPRESS_DB_HOST=dbhost:dbport -e WORDPRESS_DB_USER=dbuser -e WORDPRESS_DB_PASSWORD=dbpassword -e WORDPRESS_CACHE_S3_KEY=s3key -e WORDPRESS_CACHE_S3_SECRET=s3secret -e WORDPRESS_CACHE_S3_BUCKET=s3bucket -e WORDPRESS_CACHE_HOME="http://wordpress.local:8888" -p 80 sdomsta/clusterable-wordpress

Edit wordpress-admin.service …

1
vim coreos-units/clusterable-wordpress/wordpress/wordpress-admin.service

… and change line 12 to this (only difference is this: -e WORDPRESS_ADMIN_ENABLED=true):

1
ExecStart=/usr/bin/docker run -rm --name wordpress-%i -e WORDPRESS_DB_NAME=dbname -e WORDPRESS_DB_HOST=dbhost:dbport -e WORDPRESS_DB_USER=dbuser -e WORDPRESS_DB_PASSWORD=dbpassword -e WORDPRESS_CACHE_S3_KEY=s3key -e WORDPRESS_CACHE_S3_SECRET=s3secret -e WORDPRESS_CACHE_S3_BUCKET=s3bucket -e WORDPRESS_CACHE_HOME="http://wordpress.local:8888" -e WORDPRESS_ADMIN_ENABLED=true -p 80 sdomsta/clusterable-wordpress

3c: Run up your cluster

Either follow clusterable-wordpress/README.md for instructions on how to run, or if you’re like me and like to cheat:

1
2
cd ~/coreos/coreos-units/clusterable-wordpress
./wordpress-up.sh

3d: Observe your mighty cluster firing up

The cluster deploys the following:

  1. A discovery sidekick unit that watches for wordpress units that start and get destroyed. This particular unit is unique in that it can detect the type of wordpress container firing up, and configures the vulcand locations and paths as required.
  2. vulcand, a “programmatic load balancer backed by etcd” that listens for etcd entries and reconfigures itself instantly as config changes.
  3. wpadmin (the wordpress control panel) as a separate container. After much trial and error vulcand doesn’t support sticky session persistence and wp-admin isn’t written to deal with sessions moving all over the place.
  4. customised wordpress containers, using the w3 total cache plugin, with wp-admin disabled by a custom .htaccess denying access.

The vulcan@.service, wordpress@.service and wordpress.service units all depend on docker images, which take a while to download the first time. As your cluster fires up watch the status of all units until all are up and running:

1
fleetctl list-units

Since you’re firing these containers up at the same time, your containers may take some time to start and be in running state.

3e: Check and discover

1
2
3
fleetctl journal -f discovery@1.service
fleetctl journal -f wordpress@1.service
fleetctl journal -f wordpress-admin.service

Ctrl+c at any time.

3f: Play with your brand new wordpress site

Add an entry to your hostfile:

1
sudo vim /etc/hosts

Add the following line:

1
127.0.0.1 wordpress.local

Note, you must call the host wordpress.local because this is the value configured in our sample wordpress database.

And finally… click on this magic link:

http://wordpress.local:8888

Refresh the page over and over again and watch the container id change.

3g: clean up

Oh, don’t forget to shut everything down! Repeat until everything is destroyed:

1
2
fleetctl destroy (name)(@number).service
fleetctl list-units

If you don’t want this coreos cluster anymore, just blow it away (in fact you don’t really have to manually clean up each unit, as above … just destroy your CoreOS cluster)!

1
2
cd ~/coreos/coreos-vagrant
vagrant destroy

Celebrate good times

You’ve deployed a scalable Wordpress site.

Grab yourself some french toast, paw paw juice and toffee apple and chill out in celebration.

If you need some help with CoreOS, ping me on twitter. I think this OS brings great promise and I’m keen to understand how you’ll use it.

Comment and share

  • page 1 of 1

Shaun Domingo

Experienced technical leader, architect and ops-focused developer.


Head of Technology & Operations


Sydney, Australia