Running Kubernetes in Offline Mode

Here I'll talk about how to run kubernetes on a flight that doesn't have wifi... or, Red Hat Summit hands on lab that is completely disconnected.  In either case, to set some context, this is useful for me while I'm running on a single host kubernetes configuration for a lab or development where network access is limited or non-existent.

The issue is that K8s tries to pull the pause container whenever it launches a pod.  As such, it tries to connect to gcr.io and make a connection to download the pause image. The gcr.io is the Google Container Registry.  When you are in a disconnected environment this will cause the pod to enter a state of pending until it can pull down the pause container. 

Here's what you can do to bypass that - at least the only thing I know you can do: pull the pause container ahead of time.  It helps if you know you'll be in an environment with limited access ahead of time. 

       
# docker pull gcr.io/google_containers/pause
Trying to pull repository gcr.io/google_containers/pause ...
6c4579af347b: Download complete 
511136ea3c5a: Download complete 
e244e638e26e: Download complete 
Status: Downloaded newer image for gcr.io/google_containers/pause:latest




# docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
fedora/apache                    latest              1eff270e703a        7 days ago          649.7 MB
gcr.io/google_containers/pause   1.0                 6c4579af347b        11 months ago       239.8 kB
gcr.io/google_containers/pause   go                  6c4579af347b        11 months ago       239.8 kB
gcr.io/google_containers/pause   latest              6c4579af347b        11 months ago       239.8 kB

Now try to launch a pod:

       
# kubectl create -f apache.json


# kubectl get pods
POD                 IP                  CONTAINER(S)        IMAGE(S)            HOST                LABELS              STATUS
apache                                  my-fedora-apache    fedora/apache       127.0.0.1/          name=apache         Pending

The pod is in pending state.  You will see the following error if you check the log files.

       
# journalctl -fl -u kube-apiserver.service -u kube-controller-manager.service -u kube-proxy.service -u kube-scheduler.service -u kubelet.service -u etcd -u docker


<snip>
Jun 30 17:29:11 localhost.localdomain docker[978]: time="2015-06-30T17:29:11Z" level="info" msg="-job pull(docker.io/kubernetes/pause, latest) = ERR (1)"
Jun 30 17:29:11 localhost.localdomain kubelet[1544]: E0630 17:29:11.946950    1544 kubelet.go:1002] Failed to introspect network container: Get https://index.docker.io/v1/repositories/kubernetes/pause/images: dial tcp: lookup index.docker.io: no such host; Skipping pod "apache.default.etcd"
<snip>

You'll now need to tag it such that kubernetes realizes that it's local and is able to pull it.

       
# docker tag gcr.io/google_containers/pause docker.io/kubernetes/pause



# docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
fedora/apache                    latest              1eff270e703a        7 days ago          649.7 MB
gcr.io/google_containers/pause   1.0                 6c4579af347b        11 months ago       239.8 kB
gcr.io/google_containers/pause   go                  6c4579af347b        11 months ago       239.8 kB
gcr.io/google_containers/pause   latest              6c4579af347b        11 months ago       239.8 kB
kubernetes/pause                 latest              6c4579af347b        11 months ago       239.8 kB

At this point, you should be funtional. 

       
# kubectl get pods
POD                 IP                  CONTAINER(S)        IMAGE(S)            HOST                LABELS              STATUS
apache              172.17.0.2          my-fedora-apache    fedora/apache       127.0.0.1/          name=apache         Running

You don't need to re-deploy the pod.  K8s will pick up on the available pause image and launch the contianer correctly.