{"id":2880,"date":"2016-04-29T10:20:38","date_gmt":"2016-04-29T16:20:38","guid":{"rendered":"http:\/\/dev.iachieved.it\/iachievedit\/?p=2880"},"modified":"2020-07-14T22:08:35","modified_gmt":"2020-07-15T03:08:35","slug":"an-arm-build-farm-with-jenkins","status":"publish","type":"post","link":"https:\/\/dev.iachieved.it\/iachievedit\/an-arm-build-farm-with-jenkins\/","title":{"rendered":"An ARM Build Farm with Jenkins"},"content":{"rendered":"<p>There was a time when setting up a Continuous Integration server took a lot of work.  I personally have spent several days wrestling with getting <a href=\"http:\/\/cruisecontrol.sourceforge.net\">CruiseControl<\/a> installed, configured, and working just right.  Today it is much more straightforward and, for the most part, a simple <code>apt-get install jenkins<\/code> is all it takes to get a functional <a href=\"https:\/\/jenkins.io\">Jenkins CI server<\/a> up and running.<\/p>\n<p>In this tutorial we&#8217;re going to look at using Jenkins to set up a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Compile_farm\">build farm<\/a> of ARM systems.  My personal interest in doing so is to support the <a href=\"https:\/\/dev.iachieved.it\/iachievedit\/swift-for-arm-systems\/\">Swift on ARM<\/a> group of folks working to get Swift 3.0 support for ARMv7 devices such as the Raspberry Pi 2, BeagleBone Black, etc.  While support for cross-compiling Swift is maturing there is still a desire to natively compile on an ARM system.<\/p>\n<h2>The Jenkins Build Server<\/h2>\n<p>To make things simple we&#8217;re going to focus on installing Jenkins on an Ubuntu 14.04 server.  Using the instructions on the <a href=\"https:\/\/wiki.jenkins-ci.org\/display\/JENKINS\/Installing+Jenkins+on+Ubuntu\">Jenkins Wiki<\/a> for an Ubuntu 14.04 system:<\/p>\n<pre>\nwget -q -O - https:\/\/jenkins-ci.org\/debian\/jenkins-ci.org.key | sudo apt-key add -\nsudo sh -c 'echo deb http:\/\/pkg.jenkins-ci.org\/debian binary\/ > \/etc\/apt\/sources.list.d\/jenkins.list'\nsudo apt-get update\nsudo apt-get install jenkins\n<\/pre>\n<p>The Jenkins daemon will start automatically upon installation.  Once it does, open a browser and go to http:\/\/YOUR_HOST:8080\/, where YOUR_HOST is the server you just installed Jenkins on.<\/p>\n<figure id=\"attachment_2895\" aria-describedby=\"caption-attachment-2895\" style=\"width: 953px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins__Jenkins__and_AppDelegate_swift_and_Add_New_Post_\u2039_iAchieved_it_\u2014_WordPress.png\" rel=\"attachment wp-att-2895\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins__Jenkins__and_AppDelegate_swift_and_Add_New_Post_\u2039_iAchieved_it_\u2014_WordPress.png\" alt=\"Unlock!\" width=\"953\" height=\"541\" class=\"size-full wp-image-2895\" \/><\/a><figcaption id=\"caption-attachment-2895\" class=\"wp-caption-text\">Unlock!<\/figcaption><\/figure>\n<p>The initial password can be found on the Jenkins server with <code>sudo cat \/var\/lib\/jenkins\/secrets\/initialAdminPassword<\/code>, and will be something like &#8220;54d5f68be4554b5c8316689728721b37&#8221;.  Paste it in and click <b>Continue<\/b>.<\/p>\n<p>You&#8217;ll be prompted to choose whether or not to install suggested plugins or specifically select plugins.  Go with <b>Install suggested plugins<\/b> for now.  The next screen will be all of the plugins being loaded in.  Once complete you&#8217;ll move on to creating the first admin user:<\/p>\n<figure id=\"attachment_2898\" aria-describedby=\"caption-attachment-2898\" style=\"width: 958px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins_and_Add_New_Post_\u2039_iAchieved_it_\u2014_WordPress.png\" rel=\"attachment wp-att-2898\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins_and_Add_New_Post_\u2039_iAchieved_it_\u2014_WordPress.png\" alt=\"Create an Admin User\" width=\"958\" height=\"542\" class=\"size-full wp-image-2898\" \/><\/a><figcaption id=\"caption-attachment-2898\" class=\"wp-caption-text\">Create an Admin User<\/figcaption><\/figure>\n<p>Once entering the admin user information, you&#8217;ll see Jenkins is ready!<\/p>\n<figure id=\"attachment_2900\" aria-describedby=\"caption-attachment-2900\" style=\"width: 391px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins.png\" rel=\"attachment wp-att-2900\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins.png\" alt=\"Yes!\" width=\"391\" height=\"163\" class=\"size-full wp-image-2900\" \/><\/a><figcaption id=\"caption-attachment-2900\" class=\"wp-caption-text\">Yes!<\/figcaption><\/figure>\n<p>Click that <b>Start using Jenkins<\/b> button and (wait for it), start using Jenkins.<\/p>\n<h2>Build Agents<\/h2>\n<p>We&#8217;re interested in using Jenkins to perform native builds on ARM systems.  To that end we&#8217;ll add slave nodes to our build server.<\/p>\n<p>First, let&#8217;s make sure we have an ARM system to compile on.  I&#8217;ve become a big fan of using <a href=\"https:\/\/www.scaleway.com\">Scaleway<\/a> for spinning up ARM systems.  If you have your own ARM device such as BeagleBoard or Raspberry Pi, those will work as well, just be aware that compiling on single-core ARM devices with limited RAM can be, well, painful.  With Scaleway you can spin up quad-core ARM systems with 2G of RAM for about $3.50 a month.<\/p>\n<p>I&#8217;m going to assume you have an ARM system (either from Scaleway or a physical device that your Jenkins host can communicate with).  Let&#8217;s look at what it takes to get things configured.  There&#8217;s a bit of a dance to do to ensure that the master and slave can communicate with each other.  In short we need to:<\/p>\n<ul>\n<li>Install Java on the ARM system so the <i>slave agent<\/i> can run\n<li>Create a build user on the ARM system\n<li>Create a public\/private key pair on the build server\n<li>Add the build server&#8217;s public key to the ARM build user&#8217;s `authorized_keys` file\n<\/ul>\n<p><b>1.<\/b> Install Java and create a build user<\/p>\n<p>Your ARM device is going to need Java to run the Jenkins slave agent, so on the ARM system run <code>sudo apt-get install openjdk-7-jre-headless<\/code>.<\/p>\n<p>Let&#8217;s also set up the build user while we&#8217;re here with <code>sudo adduser build<\/code>.  Follow the prompts to create the user appropriately.<\/p>\n<p><b>2.<\/b> Create a public\/private key pair on the build server<\/p>\n<p>On the build server, <code>su<\/code> to the <code>jenkins<\/code> user and <code>cd<\/code> to its home directory.  Run <code>ssh-keygen<\/code> to create the key pair.<\/p>\n<p><b>3.<\/b> Add the public key to the <code>authorized_keys<\/code> file<\/p>\n<p>Now, add the <code>~\/.ssh\/id_rsa.pub<\/code> contents <i>of the Jenkins server <code>jenkins<\/code> user<\/i> to the <code>\/home\/build\/.ssh\/authorized_keys<\/code> file <i>of the ARM system build user<\/i>.<\/p>\n<h3>Adding the Build Slave Node<\/h3>\n<p>Now, let&#8217;s add the ARM device as a build slave.  Under <b>Manage Jenkins<\/b> go to the <b>Manage Nodes<\/b> link.<\/p>\n<figure id=\"attachment_2903\" aria-describedby=\"caption-attachment-2903\" style=\"width: 527px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Manage_Jenkins__Jenkins_.png\" rel=\"attachment wp-att-2903\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Manage_Jenkins__Jenkins_.png\" alt=\"Manage Nodes\" width=\"527\" height=\"53\" class=\"size-full wp-image-2903\" \/><\/a><figcaption id=\"caption-attachment-2903\" class=\"wp-caption-text\">Manage Nodes<\/figcaption><\/figure>\n<p>Click on <b>New Node<\/b> and we&#8217;ll give it a name like <code>arm1<\/code>.  Select the <b>Permanent Agent<\/b> radio button and then click <b>OK<\/b>.<\/p>\n<p>You&#8217;ll move on to a screen that requires some more information (typical!)  We&#8217;re interested in:<\/p>\n<ul>\n<li>Remote root directory\n<li>Labels\n<li>Launch method\n<\/ul>\n<p>For <b>Remote root directory<\/b> we&#8217;ll use the home directory of our new build user, so <code>\/home\/build<\/code>.  For the label, type in <code>arm<\/code>.  It will become apparent as to why in a bit.<\/p>\n<p>For <b>Launch method<\/b> choose <i>Launch slave agents on Unix machines via SSH<\/i>.  In the host field enter your ARM system&#8217;s hostname or IP address <i>that is accessible from your Jenkins build server<\/i>.<\/p>\n<figure id=\"attachment_2904\" aria-describedby=\"caption-attachment-2904\" style=\"width: 606px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins-1.png\" rel=\"attachment wp-att-2904\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins-1.png\" alt=\"Launch method\" width=\"606\" height=\"128\" class=\"size-full wp-image-2904\" \/><\/a><figcaption id=\"caption-attachment-2904\" class=\"wp-caption-text\">Launch method<\/figcaption><\/figure>\n<p>Now, click on the <b>Add<\/b> button (the one with the key) to go to the <b>Add Credentials<\/b> configuration page.  For the <b>Kind<\/b> choose <b>SSH Username with private key<\/b>.  For the <b>Username<\/b> we&#8217;ll use <code>build<\/code>.  For the <b>Private Key<\/b> select <b>From the Jenkins master ~\/.ssh\/<\/b>.  Recall earlier when we had you create a public\/private key pair for the <code>jenkins<\/code> user on the master?  This is why.  When trying to contact the slave agent the Jenkins system will load the private key from the <code>~\/.ssh\/<\/code> directory of the <code>jenkins<\/code> user.  The slave agent will be ready with the public key in the <code>authorized_keys<\/code> file.<\/p>\n<figure id=\"attachment_2909\" aria-describedby=\"caption-attachment-2909\" style=\"width: 718px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins-2.png\" rel=\"attachment wp-att-2909\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Jenkins-2.png\" alt=\"Adding Credentials\" width=\"718\" height=\"481\" class=\"size-full wp-image-2909\" \/><\/a><figcaption id=\"caption-attachment-2909\" class=\"wp-caption-text\">Adding Credentials<\/figcaption><\/figure>\n<p>Click <b>Add<\/b> to add the credentials, and then <b>Save<\/b> to save the new node configuration.<\/p>\n<p>You should be taken back to the <b>Node Management<\/b> page where you&#8217;ll see your new node (probably in an offline state):<\/p>\n<figure id=\"attachment_2911\" aria-describedby=\"caption-attachment-2911\" style=\"width: 909px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Nodes__Jenkins_.png\" rel=\"attachment wp-att-2911\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/Nodes__Jenkins_.png\" alt=\"arm1 Offline\" width=\"909\" height=\"206\" class=\"size-full wp-image-2911\" \/><\/a><figcaption id=\"caption-attachment-2911\" class=\"wp-caption-text\">arm1 Offline<\/figcaption><\/figure>\n<p>You can either click <b>Refresh status<\/b> or go to the node&#8217;s logs by clicking on the nodename <code>arm1<\/code> and then selecting <b>Log<\/b>.  If everything was configured properly you&#8217;ll see:<\/p>\n<pre class=\"crayon:false\">\n[04\/27\/16 17:01:42] [SSH] Starting slave process: cd \"\/home\/build\" && java  -jar slave.jar\n<===[JENKINS REMOTING CAPACITY]===>channel started\nSlave.jar version: 2.56\nThis is a Unix agent\nEvacuated stdout\nAgent successfully connected and online\n<\/pre>\n<p>Now let&#8217;s create a new job and use our ARM agent to build opencv.<\/p>\n<h2>Building OpenCV<\/h2>\n<p>We know we&#8217;re going to be using <code>git<\/code> to check out the OpenCV source tree from Github, and there are a number of other dependencies required for building, so we need to install all of them on our ARM system:<\/p>\n<pre class=\"crayon:false\">\nsudo apt-get install build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev\n<\/pre>\n<p>OpenCV is traditionally built with the following commands:<\/p>\n<pre>\nmkdir release && cd release\ncmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=\/usr\/local ..\nmake\n<\/pre>\n<p>These will be our basic build steps, with the exception we&#8217;ll use <code>make -j4<\/code> to take advantage of our 4 core ARM build agent.<\/p>\n<h3>Creating the Build Project<\/h3>\n<p>To create a new build project, click on <b>Create New Item<\/b> in the main Jenkins menu, and choose <b>Freestyle project<\/b>.  We&#8217;ll name our project <b>OpenCV 3.0<\/b>.  Click <b>OK<\/b> at the bottom of the options.<\/p>\n<figure id=\"attachment_2914\" aria-describedby=\"caption-attachment-2914\" style=\"width: 1156px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/New_Item__Jenkins_.png\" rel=\"attachment wp-att-2914\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/New_Item__Jenkins_.png\" alt=\"Creating a new Freestyle Project\" width=\"1156\" height=\"315\" class=\"size-full wp-image-2914\" \/><\/a><figcaption id=\"caption-attachment-2914\" class=\"wp-caption-text\">Creating a new Freestyle Project<\/figcaption><\/figure>\n<p>You&#8217;ll be on the <b>General<\/b> tab to get started.  For now we&#8217;re interested in the general project options, <b>Source Code Management<\/b> section, and <b>Build<\/b> section.<\/p>\n<p>Since we want to build the OpenCV project on the ARM system we are going to <i>restrict<\/i> the project to only build on those nodes matching our <b>arm<\/b> label.<\/p>\n<figure id=\"attachment_2892\" aria-describedby=\"caption-attachment-2892\" style=\"width: 1026px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/OpenCV_3_0_2_Config__Jenkins_.png\" rel=\"attachment wp-att-2892\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/OpenCV_3_0_2_Config__Jenkins_.png\" alt=\"Restrict Project\" width=\"1026\" height=\"89\" class=\"size-full wp-image-2892\" \/><\/a><figcaption id=\"caption-attachment-2892\" class=\"wp-caption-text\">Restrict Project<\/figcaption><\/figure>\n<p>For OpenCV 3.0 we&#8217;ll choose <b>Git<\/b> as the SCM type, and then enter the URL to the OpenCV source, <code>https:\/\/github.com\/itseez\/opencv\/<\/code>.  Jenkins allows you to specify the branch to build on, and we&#8217;ll leave the default <code>master<\/code>.<\/p>\n<figure id=\"attachment_2887\" aria-describedby=\"caption-attachment-2887\" style=\"width: 680px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/scmgit.jpg\" rel=\"attachment wp-att-2887\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/scmgit.jpg\" alt=\"GIT for OpenCV\" width=\"680\" height=\"238\" class=\"size-full wp-image-2887\" \/><\/a><figcaption id=\"caption-attachment-2887\" class=\"wp-caption-text\">GIT for OpenCV<\/figcaption><\/figure>\n<p>For the actual build we will enter a single <b>Execute shell<\/b> build step.  It should be noted that this is <i>an example only<\/i>.  There are a number of ways to configure and script out Jenkins project; each project will have different build steps to fit the needs of the underlying task at hand.<\/p>\n<figure id=\"attachment_2890\" aria-describedby=\"caption-attachment-2890\" style=\"width: 1011px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/buildscript.png\" rel=\"attachment wp-att-2890\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/buildscript.png\" alt=\"Configure and Build OpenCV\" width=\"1011\" height=\"284\" class=\"size-full wp-image-2890\" \/><\/a><figcaption id=\"caption-attachment-2890\" class=\"wp-caption-text\">Configure and Build OpenCV<\/figcaption><\/figure>\n<p>Save your project, and then click <b>Build Now<\/b>!  In the <b>Build History<\/b> pane you&#8217;ll see a flashing blue dot (hopefully).  Click on the dot to go to the build page where you can select to look at the console output of the build.  If everything was set up correctly you should now see OpenCV building away on your ARM build agent!<\/p>\n<figure id=\"attachment_2916\" aria-describedby=\"caption-attachment-2916\" style=\"width: 976px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/OpenCV_3_0__9__Jenkins_.png\" rel=\"attachment wp-att-2916\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/OpenCV_3_0__9__Jenkins_.png\" alt=\"Building on arm1\" width=\"976\" height=\"217\" class=\"size-full wp-image-2916\" \/><\/a><figcaption id=\"caption-attachment-2916\" class=\"wp-caption-text\">Building on arm1<\/figcaption><\/figure>\n<figure id=\"attachment_2919\" aria-describedby=\"caption-attachment-2919\" style=\"width: 952px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/OpenCV_3_0__7_Console__Jenkins_.png\" rel=\"attachment wp-att-2919\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2016\/04\/OpenCV_3_0__7_Console__Jenkins_.png\" alt=\"OpenCV 3.0 Build Log\" width=\"952\" height=\"596\" class=\"size-full wp-image-2919\" \/><\/a><figcaption id=\"caption-attachment-2919\" class=\"wp-caption-text\">OpenCV 3.0 Build Log<\/figcaption><\/figure>\n<p>In our example OpenCV took 42 minutes to build on a quad-core ARM system from Scaleway.  Not bad.<\/p>\n<h2>Final Thoughts<\/h2>\n<p>This was clearly not a comprehensive Jenkins tutorial.  Folks familiar with the craft know that setting up and configuring continuous integration servers and build projects to produce traceable artificacts is a disclipline unto itself.  However, it should be clear that creating a compile farm consisting of slave build agents does not have to be complicated.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There was a time when setting up a Continuous Integration server took a lot of work. I personally have spent several days wrestling with getting CruiseControl installed, configured, and working just right. Today it is much more straightforward and, for the most part, a simple apt-get install jenkins is all it takes to get a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":2922,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[99,21],"tags":[],"class_list":["post-2880","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-arm","category-devops"],"_links":{"self":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/2880"}],"collection":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/comments?post=2880"}],"version-history":[{"count":30,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/2880\/revisions"}],"predecessor-version":[{"id":4172,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/2880\/revisions\/4172"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/media\/2922"}],"wp:attachment":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/media?parent=2880"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/categories?post=2880"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/tags?post=2880"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}