Investigating Docker Swarm
Creating a Multi-host Docker Application
I have recently become very interested in Docker Swarm, in particular with Swarm mode, and I wanted to investigate its capabilities further in order to understand how I might justify its use in a production environment (if you’re unsure exactly what Swarm is I wrote a post about it that might help). I even went so far as to create my own Raspberry Pi Cluster in order to experiment with Swarm. A search for Docker Swarm will turn up many useful articles explaining how to create a simple swarm cluster but there seem to be relatively few articles that investigate it in greater detail. I decided to create an application that allowed me to exercise all the available features of Swarm in order to become familiar enough with it to use it with confidence in a production context.
This article isn’t about Swarm itself but is about the preparatory process of turning a Node.js application into a set of containerised processes that can be distributed across a number of Docker hosts. Once I have achieved this goal I can begin my investigations.
The Application
I have created an application that I will use to investigate Swarm features. It is written in Node.js and consists of separate processes that communicate over sockets and tcp using message queues. It isn’t intended to solve a particular problem but simply to allow me to learn. There are 3 main elements to this application which I will briefly explain.
1. The Web API
The first part of the application is the web API, shown below:
This consists of a server written in Express that listens on port 3000. The server creates a Socket client that sends and receives messages over the network.
2. The Request Cluster
The Socket connected to the web API communicates with the Request Cluster which uses Node.js cluster to create multiple child processes. The master process runs a Socket Server which listens for messages from the web API. The master process is connected to a number of worker processes, each of which runs a ZeroMQ requester. ZeroMQ is a highly-scalable, low-latency messaging service. I used it simply because I wanted to learn more about it and because it allows me to create tcp messages. The ZeroMQ Requester uses the REQ/REP (request/reply) pattern commonly used in networked programming. In this pattern a request comes in then a reply goes out. Additional incoming requests are queued and later dispatched by ZeroMQ. The application is, however, only aware of one request at a time.
When the Socket Server receives a message it triggers the requester in each worker process to send a message over tcp. One of the features of Node.js clusters is that child processes share the server ports, meaning that each of the workers can send a request on the same port. These requesters communicate with the last element of the application, the Response Cluster.
3. The Response Cluster
Like the Request Cluster, the Response Cluster consitsts of a Node.js cluster. In this case, the master process runs a ZeroMQ ROUTER/DEALER pair while the worker processes run ZeroMQ Responsers. The standard REQ/REP socket pair operates sequentially, meaning that a given reqeuster or responder will only ever be aware of one process at a time. What I want is parallel message processing for which I require a ROUTER/DEALER pair.
The ROUTER can be thought of as a parallel REP socket. Where REPs can only reply to one message at a time the ROUTER can handle many requests simultaneously. It remembers which connection each request came in on and will route reply messages accordingly.
If a ROUTER socket is a parallel REP socket then the DEALER is a parallel REQ socket. It can send multiple requests in parallel. Incoming requests to the ROUTER will be passed off to the DEALER to send out to its connections. In this case the DEALER is connected to multiple worker processes each running a ZeroMQ Responder.
These responders can be made to complete asynchronous tasks and report their success. In the first instance, the responders in the Response Cluster are reading the text from a file and sending it back to the requester.
Putting it Together
What I have ended up with is an application in which a web server listens to requests. When a request is received it sends a message over a socket to a client which triggers multiple requesters to send tcp messages. These messages are received by a ROUTER/DEALER pair which forwards them to multiple responders. In the first iteration of this application all the responders do is to read the text from a file and return it. In order to simulate latency each responder implements a random delay of between 1 and 3 seconds before responding. The application is entirely artificial and only exists for the purposes of exercising Docker Swarm. The number of worker processes in both the request and response clusters is configurable, as is the delay in responding. Ultimately I’ll configure the responders to do more advanced things like write do a database but I’m keeping it simple for now.
I created a very simple web page to run the application. It consists of two buttons, one for connecting the web server to the request cluster socket and one for making a request. When a request is triggered the page displays information about the messages sent and received.
I used Node.js clusters to send requests in parallel and to respond to requests in parallel. I could have done this in other ways but I want to compare the behaviour of Node.js clusters to Swarm clusters.
Transition from single host to multiple hosts
In its initial incarnation my application is running as 3 separate Node.js processes communicating over the localhost network.
I can run the 3 separate processes in my application on a single host (my Mac) using NPM scripts to run Gulp tasks:
$ npm run gulp:start:requester
$ npm run gulp:start:responder
$ npm run gulp:start:serverWhat I want to do is to run each process in its own container, each on a separate Docker host. I’ll do it in two stages.
Stage 1: Multiple Containers, Single Docker Host
The first stage is to create a container for each process and host them on the same machine, in the same Docker host as shown below.
The first thing to do is to create a Dockerfile that will generate the containers:
# Version: 0.0.2
FROM risingstack/alpine:3.4-v6.9.1-4.1.0
MAINTAINER Paul Nebel "[email protected]"
ENV REFRESHED_AT 2016_07_23
LABEL name="Base dev image for zeromq"
LABEL version="1.1"
# Add the relevant packages so that we can 'npm install zmq'
RUN apk update -q &&\
apk add --no-cache make gcc g++ python zeromq zeromq-dev &&\
apk upgrade
# Create "dogfish" user
RUN addgroup appuser &&\
adduser -G appuser -g "App User" -h /home/dogfish -s /bin/ash -D dogfish &&\
chown -R dogfish:appuser /usr/local
# Set up some semblance of an environment
WORKDIR /home/dogfish
ENV HOME /home/dogfish
RUN mkdir /home/dogfish/app &&\
mkdir /home/dogfish/.npm-global &&\
chown -R dogfish:appuser /home/dogfish &&\
echo "export PATH=~/.npm-global/bin:$PATH" >> /home/dogfish/.profile
USER dogfish
RUN npm config set prefix '~/.npm-global' &&\
source /home/dogfish/.profile &&\
npm install -g npm &&\
npm install -g nodemon &&\
npm config set python /usr/bin/python &&\
npm cache clear
WORKDIR /home/dogfish/app
VOLUME /home/dogfish/appI used the excellent Node.js image created by Rising Stack and based on Alpine Docker as the base for my own image. This uses the ash shell rather than bash, which requires a little more juggling to get what I wanted. As an example, ash comes without a sudo command so I had to change NPMs default directory to one local to the user I created in order to avoid getting EACCESS errors when installing modules.
The next thing to do is to create a docker-compose.yml file to create and run my containers:
version: '2'
services:
requester:
build: .
volumes:
- $PWD/..:/home/dogfish/app
networks:
- zerobridge
command: sh ./shell/scripts/startup.sh ${NODE_ENV} ${NODE_PATH} ./services/requesterManager.js target.txt
responder:
extends:
service: requester
command: sh ./shell/scripts/startup.sh ${NODE_ENV} ${NODE_PATH} ./services/responderManager.js
server:
extends:
service: requester
ports:
- "3000:3000"
depends_on:
- responder
- requester
command: sh ./shell/scripts/startup.sh ${NODE_ENV} ${NODE_PATH} ./services/server.js
networks:
zerobridge:When developing in Docker I map the local directory containing the code to be run to a volume in the container. The Node.js processes run using Gulp communicated over the localhost network using an IP address of 127.0.0.1. In order for the processes running in the containers to communicate with each other I created a bridge network called zerobridge to achieve the same effect. Containers running on the same bridge network can communicate between themselves directly using the service names they have been created with. This is accounted for in the configuration file. Thus the web server now connects directly to the requester service (using host=requester, port=5678) while the requester connects directly to the responder (using tcp://responder:5432). The responder uses a wildcard domain (tcp://*:5432).
Note that the version of ZeroMQ installed in the containers is not compatible with the version of ZeroMQ installed on my Mac, so to avoid issues I had to delete my node_modules directory before running the services in the containers. This is because of the volume mapping explained above - the containers will install their modules on my local machine in this configuration but if the modules are already installed they will use whatever is there.
Once again, I used NPM scripts to run Gulp tasks to start the containers:
$ npm run docker:up:requester
$ npm run docker:up:responder
$ npm run docker:up:serverI can see that the containers are running and the app behaves as expected:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
552110a9dfa5 dockerzmq_responder "sh ./shell/script..." 8 minutes ago Up 7 minutes dockerzmq_responder_1
fcfc2c0801c3 dockerzmq_server "sh ./shell/script..." 12 minutes ago Up 6 minutes 0.0.0.0:3000->3000/tcp dockerzmq_server_1
f74e9dc68252 dockerzmq_requester "sh ./shell/script..." 17 minutes ago Up 7 minutes dockerzmq_requester_1Stage 2: Multiple Containers, Multiple Docker Hosts
The next stage is to create a container for each process and create a separate Docker host for each container on the same machine as shown below.
In production mode this is the point where I would create a base image from my Dockerfile and add it to Docker Hub. I would configure an automated build triggered by a Github webhook to re-build the image every time I commit changes to the master branch of the repository.
The only difference between my current Dockerfile and that for creating images would be that the new Dockerfile would use git to checkout the repository locally to the container, rather than using a mapped volume as in development. This would have the additional advantage that it would not be necessary to delete the node_modules directory as described above.
However, I’m still in development mode so I’m going to map the directory containing my code to a volume on each of the Docker hosts and continue to create containers directly using my local Dockerfile.
In order for the containers running in each host to communicate I now need to create a different kind of network. Whereas containers on the same host can communicate over a bridge network, containers in different hosts need an overlay network to communicate. Since I am not yet using Docker Swarm mode, I will have to create an overlay network using a key-value store. Once I have created an overlay network any containers in any host that are connected to that network will be able to communicate with each other using their service names, as for the bridge network above.
Even though I just said I will not be using Swarm mode, creating this overlay network will involve creating a Swarm. This is because Docker Swarm and Swarm mode are not the same thing, somewhat confusingly! Swarm mode was created in Docker version 1.12 and allows for natively managing a cluster of Docker Engines. Swarm is what was available before Docker version 1.12 and is what I am using in this instance (one step at a time).
The first thing I need to do is to set up a key-value store which will hold information about the network state including discovery, networks, endpoints, IP addresses and more. Docker supports Consul, Etcd, and ZooKeeper key-value stores. This example uses Consul. Let’s provision a VirtualBox machine called mh-keystore:
$ docker-machine create -d virtualbox mh-keystoreThis will create a new Docker Machine on my Mac:
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
mh-keystore - virtualbox Running tcp://192.168.99.100:2376 v1.13.1I now need to set the environment to the local mh-keystore machine so that I can create a Consul container:
$ eval "$(docker-machine env mh-keystore)"In order to create the Consul container I’ll use the official Consul image from Docker Hub:
$ docker run -d -p "8500:8500" -h "consul" progrium/consul -server -bootstrapI now have a progrium/consul image running in the mh-keystore machine. The server is called consul and is listening on port 8500:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fda22e403a53 progrium/consul "/bin/start -serve..." 17 hours ago Up 17 hours 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp kickass_bartikThis image must be running before creating any other Docker hosts otherwise Consul won’t be able to register them. The next thing to do is to create a Swarm cluster. I’ll use docker-machine to provision the hosts for my network before creating the network itself. I’ll create several machines in VirtualBox, the first of which will act as the swarm master. As each host is created it will be passed the options that are needed by the overlay network driver. Let’s create the Swarm master:
$ docker-machine create \
-d virtualbox \
--swarm --swarm-master \
--swarm-discovery="consul://$(docker-machine ip mh-keystore):8500" \
--engine-opt="cluster-store=consul://$(docker-machine ip mh-keystore):8500" \
--engine-opt="cluster-advertise=eth1:2376" \
mac-miniThe --cluster-store option tells this Engine the location of the key-value store for the overlay network. The bash expansion $(docker-machine ip mh-keystore) resolves to the IP address of the Consul created earlier. The --cluster-advertise option advertises the machine on the network. I now need to create a further two Docker hosts and add them to the Swarm cluster:
$ docker-machine create -d virtualbox \
--swarm \
--swarm-discovery="consul://$(docker-machine ip mh-keystore):8500" \
--engine-opt="cluster-store=consul://$(docker-machine ip mh-keystore):8500" \
--engine-opt="cluster-advertise=eth1:2376" \
demo1
$ docker-machine create -d virtualbox \
--swarm \
--swarm-discovery="consul://$(docker-machine ip mh-keystore):8500" \
--engine-opt="cluster-store=consul://$(docker-machine ip mh-keystore):8500" \
--engine-opt="cluster-advertise=eth1:2376" \
demo2I can now list the machines to see that they’re all up and running:
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
demo1 - virtualbox Running tcp://192.168.99.104:2376 mac-mini v1.13.1
demo2 - virtualbox Running tcp://192.168.99.105:2376 mac-mini v1.13.1
mac-mini - virtualbox Running tcp://192.168.99.101:2376 mac-mini (master) v1.13.1
mh-keystore * virtualbox Running tcp://192.168.99.100:2376 v1.13.1Now I’m ready to create the overlay network. The usual way to do this is to set the environment to the Swarm master and create the network on the command-line. However, I’m using docker-compose to create my containers so I’ll use that to create the network by expanding the networks definition a follows:
networks:
zerobridge:
my-overlay:
driver: overlayNow, when I run docker-compose, the overlay network my-overlay will be created. If I also update the definition of the requester service (which is extended by the server and responder services) to use this network then communication between the containers in different Docker hosts should be possible:
services:
requester:
build: .
volumes:
- $PWD/..:/home/dogfish/app
networks:
- my-overlay
command: sh ./shell/scripts/startup.sh ${NODE_ENV} ${NODE_PATH} ./services/requesterManager.js target.txtIf I set my environment to the Swarm master and start a container I shold see that the network is created:
$ eval "$(docker-machine env mac-mini)"
$ npm run docker:up:serverThis starts the server service in the foreground, so I have to open another terminal window and set my environment to the Swarm master to see the created network:
$ eval "$(docker-machine env mac-mini)"
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
74a3d55b2f21 bridge bridge local
41703c3c00c1 docker_gwbridge bridge local
ba3945515268 dockerzmq_my-overlay overlay global
1af37dd52b9c host host local
bd4d3618c2f0 none null localThe new network my-overlay has been created. It has been prefixed with the name of the directory containing my docker-compose file. I can now set my environment to each Swarm of the other Docker hosts, demo1 and demo2, and start the services requester and receiver on them. I can see that each of them also has access to the overlay network:
$ eval "$(docker-machine env demo1)"
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
0b3f0666bf85 bridge bridge local
0fe2171eb4a6 docker_gwbridge bridge local
ba3945515268 dockerzmq_my-overlay overlay global
6ffe5df43bbc host host local
59066dba92f3 none null local
$ eval "$(docker-machine env demo2)"
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
d2886247ddd2 bridge bridge local
0b4bfe957e43 docker_gwbridge bridge local
ba3945515268 dockerzmq_my-overlay overlay global
4a7aec486c48 host host local
006946905d07 none null localConclusion
I have successfully run my application 3 different ways on a single machine:
- As 3 Node.js processes
- As 3 Docker containers running in a single Docker host with a bridge network
- As 3 Docker containers each running in a different Docker host with an overlay network
The next step is to run 3 different Docker hosts on 3 different physical machines on my Raspberry Pi cluster, when I can really start playing with Swarm mode.
Leave a comment