Putting Jenkins in a Docker Container

When I first started learning about Docker a year ago and exploring its use I had trouble finding great documentation and examples—even today many describe simple use cases that ultimately aren’t production ready. Productionizing applications with Docker containers requires adjusting to their ephemeral nature and single-process focus. This presents challenges to applications with data persistence needs or multi-process architectures.

As I mentioned in my last post, we use Jenkins as a foundational piece of open source software on top of which we build our automation. Jenkins is also a great application to demonstrate one way to think about “Dockerizing” your applications. We deploy Jenkins with these architectural components in mind:

  • Jenkins master server (Java process)
  • Jenkins master data (Plugins, Job Definitions, etc)
  • NGINX web proxy (we use SSL certs etc, with NGINX is an easy choice here)
  • Build slave agents (machines either being SSH’d into, or JNLP connecting to, Jenkins Master)

This is a good place to start. Over this series of blog posts I’ll be covering ways to think about all of the above as containers and finish with an advanced look at ways to use Docker containers as build slaves. For starters, we’ll create a Jenkins master server in a Docker container. Then we’ll move on to dealing with data persistence and adding a web proxy with NGINX.

This entire blog series will cover the following Docker concepts:

  • Making your own Dockerfiles
  • Minimizing image dependencies on public images
  • Creating and using Data-Volumes, including backups
  • Creating containerized “Build Environments” using containers
  • Handling “secret” data with images and Jenkins

If you haven’t taken a look at the Cloudbees Jenkins Docker image, start there as it’s really quite good. This was my reference point when first thinking about running Jenkins in a Docker container and for many people this might be sufficient. You can find their documentation here and their Git repo/Dockerfile here.

This first blog is split into two lesson plans. Each is sized to take about 30 minutes to complete. First up, part one is getting your development environment ready and learning to work with the default Jenkins Docker container that Cloudbees offers. Part two is about laying the groundwork to wrap this image in your own Dockerfile and taking more elegant control of the image. Together they are designed to get you started, especially if you’ve never worked with Docker before or are relatively new to Docker—although they assume you already know and understand how to work with Jenkins.  If you’re experienced with Docker, some of the material in Lesson 1 will be a bit of a rehash of things you probably already know.

Lesson 1: Set Up and Run Your First Image

Get Your Dev Environment Ready

Let’s get started by getting you ready to roll. Here at Riot we work with Docker (and Jenkins) on Windows, Mac OSX, and Linux. I prefer to work with Docker on OSX, though it’s perfectly functional on Windows since Docker 1.6. Either way your Docker Host server will be running in a Linux Virtual Machine.  One of my favorite tools, Docker-Compose, is not yet Windows compatible (as of Docker 1.8 and Compose 1.4).  In either case (OSX or Windows) you will be installing Docker Toolbox (formerly Boot2Docker).

As a side note, Microsoft is planning to release a native Docker server for Windows sometime in 2016 and we’re eagerly looking forward to Windows Docker Containers.  For now, we’ll focus on what is available today.

Pre-Requirements:

  1. You’ll need Windows 7 (or later) or Mac OSX 10.7 (or later).
  2. You’ll need a machine that’s able to run VirtualBox—you may need to enable virtualization in your BIOS on some PCs.
  3. If you already have Virtualbox and Docker Toolbox installed, you can skip Step 1 below. This blog was written using Docker Toolbox 1.8 and Virtualbox 5.0.
    1. Please note: when installing Docker Toolbox on a system with a pre-existing Virtualbox install you may run into some interesting challenges. When I did it, Virtualbox didn’t fully upgrade and corrupted existing image configurations. I recommend uninstalling Virtualbox and wiping all previous images—back them up to OVA files or ISO’s if you need to.
    2. If you're installing a new version of Docker Toolbox over a very old version of boot2docker you may run into issues. It’s best to fully wipe boot2docker and its ISO images before proceeding.

A Quick Note on Kitematic

With Docker 1.8 and the release of Docker Toolbox, Docker now includes “Kitematic,” a nifty GUI tool to help manage and visualize what’s happening with your Docker images and containers.  Most of this tutorial focuses on using command-line arguments and working with Docker without the Kitematic GUI. This is to better expose the reader to the underlying mechanisms. Likewise, in later blogs I start covering the use of Compose to start and stop multiple containers at once. Still, Kitematic is a cool tool and I will focus a blog in the future on its use in the development life cycle.

Step 1: Install Docker Toolbox

  1. Go to: https://www.docker.com/toolbox 
  2. Download and install Docker Toolbox for your operating system. Please keep in mind that behind the scenes this is installing VirtualBox and a host of other tools (Compose, Kitematic, Docker Machine).
  3. Follow all setup instructions.
  4. Verify your installation is working by running “Docker Quickstart Terminal.”  On Windows, this will be a shortcut on your desktop; and, on Mac OSX, in your applications/Docker folder. Then check the following commands and make sure you don’t get any errors:​
    • docker ps
    • docker info

Step 2: Find Your IP Address

We’ll want the ability to hit the Jenkins web server (and, eventually, our NGINX one). The easiest way to do that is to just point your browser to the IP address of your new Virtual Machine Docker host running on Virtualbox.

You could monkey around with ifconfig or ipconfig to figure it out, but thankfully Docker Toolbox comes with a handy command line option by utilizing docker-machine:

  • docker-machine ip default

That’s the IP of your host and where your web services will be listening! Granted this IP is on your local machine and not externally reachable. If you want outside services to hit your machine, you’ll need to set up port forwarding in Virtualbox. I’ll leave that as a separate exercise for you, as Virtualbox is well documented.

Step 3: Pull and Run the Cloudbees Jenkins Container

  1. Stay in your Docker terminal window.
  2. Pull Jenkins from the public repo by running:
    • docker pull jenkins
    • docker run -p 8080:8080 --name=jenkins-master jenkins
  3. Open your favorite browser and point it to: http://yourdockermachineiphere:8080

If Jenkins doesn’t show up in your browser but the container is running, double check you did Step 2 correctly and have the right IP address.

Please note, you’ll see that I use the docker “--name” flag and name the container “jenkins-master”—this is a convention I’ll use throughout this blog. Naming your containers is a handy best practice with three benefits:

  1. It makes them easy to remember and interact with.
  2. Docker doesn't allow two containers to have the same name, which prevents mistakes like accidentally starting two identical ones.
  3. Many common Docker tools (like docker-compose) work by using specific container names, so getting used to your containers being named is a good best practice.

Step 4: Making This a Little More Practical

The previous step starts Jenkins in its most basic start up settings. If you’re like Riot, it’s unlikely you run Jenkins on default settings. Let’s walk through adding some useful options and why.

Step 4a: Daemonizing

You probably don’t want to see Jenkins logs spew to standard out most of the time. So use the Docker daemon flag to start the container (-d).

  1. Ctrl-c on the terminal window running your container to stop it
  2. Run the following commands:
    • docker rm jenkins-master
    • docker run -p 8080:8080 --name=jenkins-master -d jenkins

Now you should just get a hash string displayed and be returned to your terminal. If you’re new to Docker, that hash string is actually the unique ID of your container (useful if you start automating these commands).

Step 4b: Memory Settings

We tend to run Jenkins with some beefy settings at Riot. Here’s a basic start, run the following commands:

  • docker stop jenkins-master
  • docker rm jenkins-master
  • docker run -p 8080:8080 --name=jenkins-master -d --env JAVA_OPTS="-Xmx8192m" jenkins

That should give Jenkins a nice 8 GB memory pool and room to handle garbage collection. Please note if you’re using Java 1.7 or previous I recommend using this instead:

  • docker run -p 8080:8080 --name=jenkins-master -d --env JAVA_OPTS=”-Xmx8192m -XX:PermSize=256m -XX:MaxPermSize=1024m” jenkins

Given that the Cloudbees container uses Java 1.8 we don’t need the PermSize settings, because Java 1.8 no longer has PermGen.

Step 4c: Upping the Connection Pool

At Riot, we get a lot of traffic to our Jenkins server, so we’ve learned to give Jenkins a bit more breathing room.  Run the following:

  • docker stop jenkins-master
  • docker rm jenkins-master
  • docker run -p 8080:8080 --name=jenkins-master -d --env JAVA_OPTS="-Xmx8192m" --env JENKINS_OPTS="--handlerCountStartup=100 --handlerCountMax=300" jenkins

That’ll give Jenkins a nice base pool of handlers and a cap. Coincidently you’ve now learned how to use both JAVA OPTS and JENKINS OPTS as environment variables in a meaningful way. Please note, this works because of how Cloudbees conveniently structured their Dockerfile.

Step 5: Putting It All Together

I placed everything we learned here into a simple makefile so you can use make commands to control running your Jenkins docker container. You can find it here:

You can use the following commands:

  • make build    - Pulls the Jenkins image
  • make run      - Runs the container
  • make stop    - Stops the container
  • make clean  - Stops and deletes the existing container

You don’t have to use the makefile, I just find it easier than typing out the whole run command. You could easily put these in a script of your choosing instead. 

Comments

Hopefully you see how easy it is to get up and running with Docker and Jenkins. I’ve tried to give you some basic options to take the default Cloudbees Jenkins container and make it a little more useable. Cloudbees has many useful recommendations for running their container, such as how to pre-install plugins and store Jenkins data.

That brings me to future posts on the subject. This container/image is useful but has a few drawbacks: no consistent logging, no persistence, no web server proxy in front of it, and no clean way to guarantee you use the version of Jenkins you want. This brings up questions like: what if you want to stick to an older version? Or what if you want to use the latest available release period?

In the next lesson I’ll cover making this container a little more robust. In particular:

  • Creating your own Dockerfile to wrap the Cloudbees base.
  • Moving some of the environment variables into this new image.
  • Creating a log folder, setting permissions and other useful Jenkins directories, as well as how to retrieve the Jenkins logs while it’s running.

Lesson 2 - A Jenkins Base Image Wrapper

In the previous lesson I discussed getting a development environment set up to run Docker and experimented with the Jenkins Docker image provided by Cloudbees. We found that it was simple and easy to use, with some great features out of the box. To make progress in the improvement areas we identified, the Docker concepts covered in this lesson are:

  • Making your own Dockerfile
  • Setting environment variables in a Dockerfile
  • Creating folders and permissions in a Dockerfile
  • Using docker exec to run commands against a running container

Making your Base Dockerfile

We want to make some changes to how Jenkins starts by default. In the last blog we handled this by creating a makefile that passed in arguments as environment variables. Given that we want to do this every time, we can just move those into our own Dockerfile. Additionally, our own Dockerfile will let us lock the version of Jenkins we’re using in case Cloudbees updates theirs and we’re not ready to upgrade.

We do this through four steps:

  1. Create a working directory.
  2. In your favorite text editor, create a new file called “Dockerfile.”
  3. Add the following to the file and save it:
    • FROM jenkins:1.609.1
      MAINTAINER yourname
  4. Then at the command line enter:
    • docker build -t myjenkins .

What we did here was pull a particular version of the Jenkins image from the public docker repo. You’ll find all the available versions here:

You can always set the FROM clause to be whatever version of the images available. However, you can’t just set the version to whatever version of Jenkins you want. This is the image version “tag” or “label,” and Cloudbees is nice enough to make it match the Jenkins version inside the image.  Cloudbees doesn’t make images for every version of Jenkins they release, however—usually only their long term support builds.  If you want to use a specific version, checkout a future blog post where I talk about how to stop using the Cloudbees image and make your own.

Testing the New Dockerfile

We can switch over to the image very easily by modifying our docker run command to the following:

  • docker run -p 8080:8080 --name=jenkins-master -d --env JAVA_OPTS="-Xmx8192m" --env JENKINS_OPTS="--handlerCountStartup=100 --handlerCountMax=300" myjenkins

Now we can cleanup those environment variables by placing them in our own Dockerfile.

Adding Environment Variables to our Dockerfile

It’s easy to add default settings for things like environment variables to Dockerfiles. This also provides a nice piece of self-documentation. And, because you can always overwrite these when running the Docker container, there’s really no downside.

  1. In your Dockerfile add the following lines after the “MAINTAINER” line:

    • ENV JAVA_OPTS="-Xmx8192m"
      ENV JENKINS_OPTS="--handlerCountStartup=100 --handlerCountMax=300"
  2. Save and rebuild your image: 
    • docker build -t myjenkins .

Pretty simple! You can test that it still works by entering the following three commands:

  • docker stop jenkins-master
  • docker rm jenkins-master
  • docker run -p 8080:8080 --name=jenkins-master -d myjenkins

Your image should start right up! But how do you know the environment variables worked? Simple: the same way you’d check to see what arguments started your Jenkins app normally using ps. And this works even if you’re developing on Windows.

Running a Basic Command Against your Container

To confirm the Java and Jenkins options are set we can run ps in our container and see the running Jenkins Java process by using Docker exec:

  • docker exec jenkins-master ps -ef | grep java

You should see something similar to this come back:

  • jenkins  1  0 99 21:28 ?  00:00:35 java -Xmx8192m -jar /usr/share/jenkins/jenkins.war --handlerCountStartup=100 --handlerCountMax=300

From this, you can easily see our settings have stuck. docker exec is a simple way to execute shell commands inside your container and also an incredibly simple way to inspect them.  This even works on Windows because, remember, the command after “exec” is being run inside your container and thus is based on whatever base image your container uses.

Setting up a Log Folder

In the previous blog we noted that we lost visibility into the Jenkins logs when running our container with the daemonize flag (-d). We want to use the built in Jenkins feature to set a log folder. We’re going to need to do this in our Dockerfile and then pass in the logging option to Jenkins.

Let’s edit our Dockerfile again. Between the “MAINTAINER” and first ENV line we’re going to add the following:

  • RUN mkdir /var/log/jenkins

We place the command at this location in the file to follow best practices. It is more likely we will change the environment variables than these setup directories, and each command line in a Dockerfile essentially becomes its own image layer. You maximize layer re-use by putting frequently changed items near the bottom.

Now build your image again:

  • docker build -t myjenkins .

You’ll get an error that looks like:

  • ---> Running in 0b5ac2bce13b
    mkdir: cannot create directory ‘/var/log/jenkins’: Permission denied
    

No worries. This is because the default Cloudbees container sets the running user to the “Jenkins” user. If you look at their Dockerfile (found here: https://github.com/ Jenkinsci/docker/blob/master/Dockerfile) you should see, near the bottom:

  • USER jenkins

This is inconvenient for what we’re trying to do—in normal Linux you’d just use SUDO or some other means to create the folder (/var/log is owned by root). Fortunately for us Docker lets us switch users. Add the following to your Dockerfile:

  1. Before your RUN mkdir line add:
    • USER root
  2. ​After your RUN mkdir line add: 
    • RUN chown -R  jenkins:jenkins /var/log/jenkins
  3. After your RUN chown line add: 
    • USER jenkins

Note that we had to also add a chown command because we want the Jenkins user to be able to write the folder. Next, we set root and then reset Jenkins so that the Dockerfile’s behavior is preserved.

Now build your image again:

  • docker build -t myjenkins .

And, your errors should be gone.

With the log directory set (please note: you can place this folder wherever you’d like, we use /var/log for consistency) we can now tell Jenkins to write to that folder on startup by modifying the JENKINS_OPTS environment variables.

In your Dockerfile edit the JENKINS_OPTS line to look like this:

  • ENV JENKINS_OPTS="--handlerCountStartup=100 --handlerCountMax=300 --logfile=/var/log/jenkins/jenkins.log"

Now build your image one more time:

  • docker build -t myjenkins .

Let’s test our new image by seeing if we can tail the log file! Try the following commands:

  • docker stop jenkins-master
  • docker rm jenkins-master
  • docker run -p 8080:8080 --name=jenkins-master -d myjenkins

With the container running we can tail the log file if everything worked:

  • docker exec jenkins-master tail -f /var/log/jenkins/jenkins.log

Retrieve Logs if Jenkins Crashes

Time for a bonus round! As long as we’re discussing logs, Docker presents an interesting problem if Jenkins crashes. The container will stop running and docker exec will no longer work. So what to do?

We’ll discuss more advanced ways of persisting the log file later. For now, because the container is stopped we can copy files out of it using the docker cp command.  Let’s simulate a crash by stopping the container, then retrieving the logs:

  1. ctrl-c to exit out of the log file tail
  2. Run the following commands:
    • docker stop jenkins-master
    • docker cp jenkins-master:/var/log/jenkins/jenkins.log jenkins.log
    • cat jenkins.log

Concluding Thoughts

You can find all the work in my tutorial Git repo (and the updated convenience makefile) here:

By making our own Dockerfile that wraps the Cloudbees one we were able to make life a little easier for ourselves. We set up a convenient place to store the logs and learned how to look at them with the docker exec command. We moved our default settings into the Dockerfile and now we can store this in source control as a good piece of self-documentation.

We still have a data persistence challenge. We’ve learned how to pull log files out of a stopped container (handy when Jenkins crashes). But in general if our container stops we’re still losing all the jobs we created. So without persistence, this Jenkins image is only useful for local development and testing.

That leaves us with the next blog. With our foundations in place — our own Dockerfile wrapper, locked to a handy version of Jenkins — we can solve the persistence problem. The next blog article will explore these concepts:

  • Preserving Jenkins Job and Plugin data
  • Docker Data Persistence with Volumes
  • Making a Data-Volume container
  • Sharing data in volumes with other containers

For more information, check out the rest of this series:

Part I: Thinking Inside the Container 
Part II: Putting Jenkins in a Docker Container (this article)
Part III: Docker & Jenkins: Data That Persists
Part IV: Jenkins, Docker, Proxies, and Compose
Part V: Taking Control of Your Docker Image
Part VI: Building with Jenkins Inside an Ephemeral Docker Container
Part VII: Tutorial: Building with Jenkins Inside an Ephemeral Docker Container
Part VIII: DockerCon Talk and the Story So Far

    Posted by Maxfield F Stewart