Flannel and Docker on Fedora - Getting Started

Lets set up 3 Fedora servers for the purposes of testing flannel on Fedora. These can be bare metal, VMs (on KVM, VMware, RHEV, etc...). Why do we want to test this? This is to demonstrate setting up the flannel overlay network and confirming connectivity. Specifically, I want to test container connectivity across hosts.  I'd like to make sure that container A on host A can talk to container B on host B. I received quite a bit of guidance from Jeremy Eder of breakage.org - Thanks for the tips!

Our 3 Flannel hosts:

fed-master 192.168.121.105
fed-minion1 192.168.121.166
fed-minion2 192.168.121.108

A few setup notes: I haven't looked at this on GCE or AWS. It helps to add the hosts to /etc/hosts, or have some other DNS solution. In my case, I set up these VM's in Vagrant on my laptop and modified /etc/hosts.

Software used on these Fedora hosts.

       
# rpm -qa | egrep "etc|docker|flannel"
flannel-0.2.0-1.fc21.x86_64
docker-io-1.4.0-1.fc21.x86_64
etcd-0.4.6-6.fc21.x86_64

On fed-master:

Look at networking before flannel configuration.

       
# ip a

Start etcd on fed-master.

       
# systemctl start etcd; systemctl status etcd

Configure Flannel by creating a flannel-config.json in your current directory.  The contents should be:

       
{
"Network": "10.0.0.0/16",
"SubnetLen": 24,
"Backend": {
"Type": "vxlan",
"VNI": 1
     }
}

Upload the configuration to the etcd server.

       
# curl -L http://x.x.x.x:4001/v2/keys/coreos.com/network/config -XPUT --data-urlencode value@flannel-config.json

Verify the key exists.

       
# curl -L http://x.x.x.x:4001/v2/keys/coreos.com/network/config

Backup the flannel configuration file.

       
# cp /etc/sysconfig/flanneld{,.orig}

Configure flannel, use your interface on your system.  Mine is eth0.

       
# sed -i 's/#FLANNEL_OPTIONS=""/FLANNEL_OPTIONS="eth0"/g' /etc/sysconfig/flanneld

The /etc/sysconfig/flanneld should look like this (sub your IP for the FLANNEL_ETCD key).

       
# grep -v ^\# /etc/sysconfig/flanneld

FLANNEL_ETCD="http://192.168.121.105:4001"
FLANNEL_ETCD_KEY="/coreos.com/network"
FLANNEL_OPTIONS="--iface=eth0"

Start up the flanneld service.

       
# systemctl restart flanneld
# systemctl status flanneld

Check the interfaces on the host now. Notice there is now a flannel.1 interface.

       
# ip a

Now that fed-master is configured, let's configure the minions (fed-minion{1,2}).

From the minions:

Use curl to check firewall settings from the minion to the master.  We need to ensure connectivity to the etcd service.

       
curl -L http://fed-master:4001/v2/keys/coreos.com/network/config

From the fed-master:

 Copy over flannel configuration to the minions.

       
# for i in 1 2; do scp /etc/sysconfig/flanneld fed-minion$i:/etc/sysconfig/.; done

From master, restart services on the minions.

       
# for i in 1 2; do ssh root@fed-minion$i systemctl restart flanneld; done
# for i in 1 2; do ssh root@fed-minion$i systemctl enable flanneld; done

From master, check the new interface on the minions.

       
# for i in 1 2; do ssh root@fed-minion$i ip a l flannel.1; done

From any node in the cluster, check the cluster members by issuing a query to etcd via curl.  You should see that three servers have consumed subnets.  You can associate those subnets to each server by the MAC address that is listed in the output.

       
# curl -L http://fed-master:4001/v2/keys/coreos.com/network/subnets | python -mjson.tool

From all nodes, review the /run/flannel/subnet.env file.  This file was generated automatically by flannel.

       
# cat /run/flannel/subnet.env

Configure Docker:

Configure the Docker daemon on each minion. The /usr/lib/systemd/system/docker.service unit file on each minion should look as follows, pay special attention to the items in bold.  We are instructing systemd to import and read the /run/flannel/subnet.env file to set up the variables used in the ExecStart key below.  Specifically setting the Docker bridge IP and the MTU for flannel.

       
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
EnvironmentFile=-/run/flannel/subnet.env
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
ExecStart=/usr/bin/docker -d -H fd:// $OPTIONS $DOCKER_STORAGE_OPTIONS --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
LimitNOFILE=1048576
LimitNPROC=1048576

[Install]
WantedBy=multi-user.target

Remember to issue on each minion.

       
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
systemctl status docker

Check the network on the minion. If Docker fails to load, or the flannel IP is not set correctly, reboot the system. A functioning configuration should look like the following; notice the docker0 and flannel.1 interfaces.

       
# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

2: eth0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:15:9f:89 brd ff:ff:ff:ff:ff:ff
inet 192.168.121.166/24 brd 192.168.121.255 scope global dynamic eth0
valid_lft 3349sec preferred_lft 3349sec
inet6 fe80::5054:ff:fe15:9f89/64 scope link
valid_lft forever preferred_lft forever

3: flannel.1:  mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 82:73:b8:b2:2b:fe brd ff:ff:ff:ff:ff:ff
inet 10.0.81.0/16 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::8073:b8ff:feb2:2bfe/64 scope link
valid_lft forever preferred_lft forever

4: docker0:  mtu 1500 qdisc noqueue state DOWN group default
link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
inet 10.0.81.1/24 scope global docker0
valid_lft forever preferred_lft forever

At this point the flannel cluster is set up and we can test it. We have etcd running on the fed-master node and flannel / Docker running on fed-minion{1,2} minions. Next steps are for testing cross-host container communication which will confirm that Docker and flannel are configured properly.

From each minion, pull a Docker image for testing. In our case, we'll use fedora:20.

Issue the following on fed-minion1.

       
# docker run -it fedora:20 bash

This will place you inside the container. Check the IP address.

       
# ip a l eth0
5: eth0:  mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:00:51:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.81.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:aff:fe00:5102/64 scope link
valid_lft forever preferred_lft forever

You can see here that the IP address is on the flannel network.

Issue the following commands on fed-minion2:

       
# docker run -it fedora:20 bash
# ip a l eth0
5: eth0:  mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:00:45:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.69.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:aff:fe00:4502/64 scope link
valid_lft forever preferred_lft forever

Now, from the container running on fed-minion2, ping the container running on fed-minion1:

       
# ping 10.0.81.2
PING 10.0.81.2 (10.0.81.2) 56(84) bytes of data.
64 bytes from 10.0.81.2: icmp_seq=2 ttl=62 time=2.93 ms
64 bytes from 10.0.81.2: icmp_seq=3 ttl=62 time=0.376 ms
64 bytes from 10.0.81.2: icmp_seq=4 ttl=62 time=0.306 ms

You should have received a reply. That's it. flannel is set up on the two minions and you have cross host communication. Etcd is set up on the master node. Next step is to overlay the cluster with kubernetes.

Important links:

Flannel
etcd
Fedora
breakage 
Docker