Developing At Speed With Jenkins

BayLisa Sort But Cool

June 14, 2012

Robert Harker

Sr. Service Engineer, Yahoo! Sports

(Up arrow, down arrow to move between slides)

This is a a Presentation about the Jenkins build automation system works and how Yahoo! Sports uses it.


Robert Harker

Currently a Sr. Service Engineer at Yahoo! Sports

On a team of 4 SEs managing ~3,000 hosts

Heavily involved with Yahoo! Sports move to a DevOps team
and continuous integration with Jenkins
Working on migrating Yahoo! Sports to new hardware
into two consolidated data centers

Active in the OpenStack and cloud computing community


Yahoo! Sports Uses Jenkins For Agile Development

Yahoo! Sports has adopted an agile development environment:

Sports bases our work around a scrum team that includes Devel, QE and SE

Sports works on weekly sprints which include new features and bug fixes

Sports pushes a new release to production every week

Sports does daily pushes for minor bug fixes
and product driven changes like ad campaigns

Jenkins has been key for build, test and deploy automation


What is Jenkins

Jenkins is an automated continuous integration system
Similar to CruiseControl and DamageControl

Jenkins can automatically:

Trigger builds on events like SVN check-ins

Trigger software tests when build completes

Update system configuration information

Automatically deploy to integration and test environments

Monitor external jobs

Notify via email, IRC or IM job status and results

Manually deploy to production environments


Jenkins Features

Jenkins Supports:
Change sets: a list of changes made by the build
and checked into a version control system

Tight integration with SVN/git for traditional builds

Tight integration with Mavin for Java builds

JUnit/TestNG test reporting: tabulated, summarized, and historical

Distributed builds: build slaves to speed up builds

Plugins for different build, deploy, reporting and 3rd party tools


Jenkins Is A Java Based Application

Can be run with:
java -jar jenkins.war

Can be deployed in a servlet container like Tomcat

No database or external applications required

Can be configured:

Entirely using its web GUI with error checking

By editing its XML config files

Via its RESTfull API


Typical Jenkins Build Job Flow

Jenkins Flow Diagram
Jenkins Flow

Configure a Jenkins Job

Jenkins Conf1 Diagram
Jenkins Conf1

Configure Jenkins Upstream/Downstream jobs

Jenkins Conf2 Diagram
Jenkins Conf2

Jenkins Deploy

Jenkins Deploy Diagram
Jenkins Deploy

Build Automation

Sports builds new packages and deploys to develoment integration hosts
within 10 minutes of a SVN check in
Each time a change is made, Jenkins builds and deploys changes
For all classes of hosts: FE, API, cache

Because of this Sports has a 3 step build process:

A Jenkins job to discover if changes have been made,
Recomple the code and build the new packages

A Jenkins job to install the packages in generic chroot environments

A Jenkins job to update the "roles" and create a deployment tag


Test Automation

Sports uses Jenkins to automate our test suite:

Daily regression tests

Periodic and on demand smoke tests

Jenkins automatically runs smoke tests as part of automated deploy jobs

Smoke tests allow Ops to validate basic functionality after a deploy

Jenkins keeps track of the number of test errors
Provides a web based interface to drill down into the test failures

Deploy Automation

Sports uses Jenkins to deploy to all of our environments:
Devel integration, QA, stage and production

Jenkins uses the same deploy process for:

A full install to reimaged host

Updates of farms of hosts

Jenkins applies a "tag" to all the hosts being upgraded

Jenkins then does a generic update of that tag on all the hosts


Yahoo! Sports Deploy Procedure

Our update procedure is based on a Yahoo! tool called Pogo
Soon to be released as OpenSource

Jenkins uses the same deploy procedure across all of our environments

Avoids problems with differences between the environments

Jenkins uses the same deploy procedure for bare metal restores and daily pushes

Jenkins can restore a host or an environment quickly and consistently



Agile development built around scrum teams, weekly releases
and Jenkins automation has been very good for Yahoo! Sports
The DevOps model of Development, QE and SE working together
has lead to cooperation across functional groups
The weekly sprint keeps Development and product happy
with new features introduced quickly
The daily push keeps Development and SE happy with no break fixes
If we find a problem, we do a push. Up to 3 or 4 a day sometimes
The automated build, test and deploy features of Jenkins have significantly
reduced our outages and impressively reduced our outage minutes


Robert Harker

Sr. Service Engineer

Yahoo! Sports