iAchieved.it

Software Development Tips and Tricks

By

Ansible and AWS – Part 2

In this post we’re going to look at Ansible variables and facts, but mostly at variables (because I haven’t worked with facts much to be honest). This is the second part of our series titled Ansible and AWS and adds to the first, so if you get lost make sure and have a look at Ansible and AWS – Part 1.

At some point you’re going to get to a point where you have two machines that mostly look like one another except for their environment, size, web URL, etc. I’ve come across this when having two servers in separate environments, say, development, staging, and production. They will have different URLs, different amounts of CPU or RAM (which can drive certain configuration values). Or, let’s say each machine backs up data to a given S3 bucket, and that backup script you wrote needs the bucket name. A perfect usecase for an Ansible variable.

So let’s quickly look at the four primary locations for a variable, and then I’ll share several rules of thumb I use as to where to put one:

  • in ansible_hosts
  • in group_vars
  • in host_vars
  • in the playbook

Now I did say primary locations because there are other places; for now we’re ignoring variables and defaults that are included in roles or provided on the commandline. For the canonical reference on variables, see the official documentation.

ansible_hosts

Don’t put variables here. Moving on.

I kid, somewhat. When first starting out you may see ansible_hosts files that look like this:

[all]
18.188.72.168 HOSTNAME=ansible-helloworld OPERATING_ENVIRONMENT=staging

Terrible form, but let’s try it out in our playbook (see Ansible and AWS – Part 1) and the sample Github repository. We’re going to add another task to our playbook that creates a file on the server based upon a template. The templating language is Jinja2 (don’t worry, subsequent posts will go into Jinja and Ansible templates in much greater detail). First, create a new directory (inside your ansible-helloworld directory) called templates. This is a specific name for Ansible, so don’t name it template or something else:

# cd ansible-helloworld
# mkdir templates

Inside of templates create a file called environment.j2 (.j2 is the extension used by Jinja2 templates) and populate it with the following content:

# Created by Ansible

OPERATING_ENVIRONMENT='{{ OPERATING_ENVIRONMENT }}'

Note! I personally prefer any file on a server that was created by Ansible to say as much right at the beginning. So many times people have gone on to a server and edited a file without realizing that it was generated and stands a good chance to be overwritten. We could do an entire article on what a good header for a such a file might look like. Hell, I might just do that!

Then, in your playbook add the following task to the end:

Remember that YAML is indentation sensitive, so if you paste this into your playbook, make sure it is properly aligned with the rest of your tasks.

One last step! Locate your hostname task and change the hardcoded ansible-helloworld (that’s our hostname), to "{{ HOSTNAME }}", like this:

If you’re quick on the uptake, you should already know what is going to happen when this playbook is executed. The variable HOSTNAME in the ansible_hosts file is going to be applied in the hostname task, and the OPERATING_ENVIRONMENT variable will be applied in the template task.

Go ahead and run the playbook:

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [18.188.72.168]

TASK [Set our hostname] ********************************************************
ok: [18.188.72.168]

TASK [Install base packages] ***************************************************
ok: [18.188.72.168] => (item=[u'htop', u'zsh'])

TASK [Create /etc/environment] *************************************************
changed: [18.188.72.168]

PLAY RECAP *********************************************************************
18.188.72.168              : ok=4    changed=1    unreachable=0    failed=0

Because there is a new task (the template task), and our hostname didn’t change, we see changed=1.

If you look at /etc/environment on the server, you should see:

cat /etc/environment
# Created by Ansible

OPERATING_ENVIRONMENT='staging'

Nice.

Let’s change our hostname in ansible_hosts to simply helloworld:

[all]
18.188.72.168 HOSTNAME=helloworld OPERATING_ENVIRONMENT=staging

Rerunning the playbook will change the hostname to helloworld, since that is the new value of the HOSTNAME variable.

Group Variables

Notice in our ansible_hosts file there is the [all] tag? Well, we can change to that split hosts up. Let’s rewrite our ansible_hosts file to look like this:

[staging]
18.188.72.168 HOSTNAME=helloworld

and then create a directory (inside your ansible-helloworld directory) called group_vars. Then in group_vars create a file called staging.yml and in it put:

---
OPERATING_ENVIRONMENT:  staging

and run your playbook. If you’re following along verbatim nothing should happen. All we’ve done is extracted variables common to staging servers (like our OPERATING_ENVIRONMENT variable) into a single file that will apply to all of the hosts in the staging group.

Try renaming staging.yml to somethingelse.yml and rerunning. You should get an error regarding an undefined variable, since Ansible wasn’t able to find OPERATING_ENVIRONMENT. When you supplied a properly named group_vars file (staging.yml) it is able to look it up.

Finally, for our group variables, notice the syntax changed from the ansible_hosts “ini-style” syntax to a YAML file. This is important to note!

Host Variables

Now let’s take a look at host variables and how to use them. Create a directory called host_vars, again, inside the ansible-helloworld directory. In it create a directory named the same as how your host is defined in ansible_hosts. My server is being referenced as 18.188.72.168, but since AWS provides an FQDN, I’ll switch to that to demonstrate how what is in ansible_hosts can be an IP address, alias in /etc/hosts, or an FQDN resolvable by DNS. I’m going to change my ansible_hosts to this:

[staging]
ec2-18-188-72-168.us-east-2.compute.amazonaws.com

and then create a file called vars.yml in host_vars/ec2-18-188-72-168.us-east-2.compute.amazonaws.com and place the following content:

---
HOSTNAME:  helloworld

Try the playbook out!

Take Note

Have you noticed that we’ve eliminated the variables from our ansible_hosts file? You may feel otherwise, but I’m a strong proponent of ansible_hosts files that consist of nothing more than groups of hostnames. You might think at first that you’ll have just a couple of variables per host, but odds are this will grow to a dozen or more quickly! Think about the various things that are configurable on an environment or host basis:

  • what monitoring environment will this host report to?
  • where are syslogs being sent?
  • what S3 bucket is used for backing up configuration files (that aren’t in Ansible)?
  • what are the credentials to that bucket?
  • does the host (or group) have custom nameservers?
  • how is application-specific configuration handled?

And so on. Trust me. Get into the good habit of creating a group_vars and host_vars directories and start putting your variables there.

Variables In Playbooks (and Other Places)

You can also put variables in your playbooks, but I rarely do. If it is in the playbook it’s really less of a variable and more of a strict setting, since it will override anything from your ansible_hosts file, group_vars or host_vars. If you’re set on it, try this out. In your playbook, add vars: like this:

That is, nestle it in between the become_method and tasks. Run it and you’ll see that both your hostname and /etc/environment file changes.

Facts

Ansible also provides the ability to reference facts that it has gathered. To be sure, I don’t use this feature as often, but sometimes I’ll need the host’s IP address (perhaps to put it in a configuration file), or how many CPUs it has. Seriously, check out the documentation of all of the awesome facts you can reference in your playbook. Here are some that you may find yourself using, especially if your playbooks require subtle tweaks to support different distributions, amount of memory, CPU, etc.:

  • ansible_distribution_name
  • ansible_distribution_release
  • ansible_distribution_version
  • ansible_eth0 (and properties of it)
  • ansible_memtotal_mb
  • ansible_processor_cores
  • ansible_processor_count

You can use these in your playbook in the same manner as variables:

This is a debug task will just print out a message:

TASK [Total Server RAM (MB)] ***************************************************
ok: [ec2-18-188-72-168.us-east-2.compute.amazonaws.com] => {
    "msg": "Total Server RAM:  990 MB"
}

Let’s make use of facts in our /etc/environment template:

templates/environment.j2:

# Created by Ansible ({{ ansible_date_time.date }} {{ ansible_date_time.time }} {{ ansible_date_time.tz }})

OPERATING_ENVIRONMENT='{{ OPERATING_ENVIRONMENT }}'

Notice here we’re using the ansible_date_time fact to include in our generated /etc/environment file when the file was created. After running:

ubuntu@helloworld:~$ cat /etc/environment
# Created by Ansible (2018-05-11 11:56:48 UTC)

OPERATING_ENVIRONMENT='staging'

Final Remarks

This post might seem at first glance to be a little less “meaty” than Part 1, but variables and facts will comprise a large part of your future playbooks (we didn’t even talk about how they are used in roles). As promised here are some guidelines I use when deciding where to put variable definitions:

If the hosts for a given Ansible playbook can be organized into logical groups (such as staging and production), and there are a set of variables that will be common to all of the staging servers, and likewise common to all of the production servers, these variables are a good candidate to put into group_vars. Examples here might be:

  • the endpoints for log shipping
  • the IP addresses for name servers
  • the IP of a monitoring server
  • AWS region information (for example, if staging is in one region and production is in another)

Or, let’s say you run some type of monitor traps that send e-mails on certain events. You might want to send staging alerts to staging-alerts@iachieved.it vs. production-alerts@iachieved.it. Ansible group variables might come in handy here.

If the variable is specific to a host, then obviously you’d put the information in host_vars. I prefer to explicitly set server hostnames, and so HOSTNAME goes into the vars.yml for the host. Application-specific information for that specific host, put it into host_vars.

I’m hard pressed to think of when I’d want to explicitly put a variable in either ansible_hosts or the playbook itself; in ansible_hosts it just clutters up the file and in the playbook it’s effectively a constant.

Now, make no mistake: you will, over time, refactor your playbooks and the locations and groupings of variables. If you’re a perfectionist and lose sleep over whether you’ve spent enough time in discussions over how to architect your playbooks, well, I feel sad for you. It’ll never be right the first time, so make the best decision you can and sleep well knowing it can be refactored as your environment or needs change.

Getting the Code

You can find the finished playbook for this article on the part2 branch of this Github repository.

By

Ansible and AWS – Part 1

It’s been some time since I’ve posted to this blog, which is a shame, because I do indeed enjoy writing as well as sharing what I’ve learned with the hope it helps someone else. The truth is, however, that writing quality articles takes a lot of time. I also suffer from that thought that surely someone else out there has written on a given topic and done a much better job that I could do. And the reality is, someone probably has, but that shouldn’t stop me from doing something I enjoy.

So with that, here’s the first of what I hope to be several more articles on how to harness the power of Ansible with AWS. I firmly believe in learning by doing and I also believe that at first, you should take few shortcuts. What I mean by that is this: are you the type of person that’s saddened by the fact that some schools allow calculators in grade school? Or, even better, not bothering teaching someone how to use a library? And by that I mean how to get off of your ass, go to the library, navigate the stacks, and find a wealth of information you just might not find on Google? To be fair, I’m not really the Get off my lawn! type, but it does irk me somewhat when I see people automating things they don’t have a fundamental understanding of in the first place.

With that, my approach will be to start off with doing some things manually. Then I’ll illustrate how to take the manual steps and automate them. Sometimes it isn’t worth automating something; usually it’s something you do infrequently enough and the energy required to automate it outweighs the benefit. I usually use the DevOps equivalent of the three times rule. If I’ve done something by hand three times in a row, it’s a good candidate for automation, but not necessarily before then.

Ansible

Ansible is just so wonderful. If you’ve never heard of it, it falls into the general class of automation tools that allow you to codify a set of instructions for doing or building something. I tried to make that as abstract as possible, because the something could be:

  • installing a web server application
  • updating a configuration file
  • adding a new user to a set of servers
  • applying a new software patch
  • building a virtual machine

and so on. It’s almost as if these sets of instructions are like playbooks or recipes or cookbooks. So much so many of the popular tools call their files variations of these phrases. Ansible uses tasks, roles, and playbooks.

Installing Ansible

To use a tool you’ll have to install it. I develop on a Mac and can tell you that brew install ansible works great (if you don’t have Brew installed on your Mac you should be ashamed of yourself). As of this writing I’m using Ansible version 2.5.2, as that is what brew installed.

Using Ansible

Ansible is easy to use, though at first it may not seem like it. I know you want to cut to the chase and just figure out how to automate that thing and you want it now. Why do I have to do all this stuff just to do this thing I think should be easy! I can hear you say it because I say it all the time. That initial learning curve is always a pain in the ass.

Here are some simple hints to get started as quickly as possible. First, create a folder called ansible-helloworld and use it as home base. Then, you’re going to create two files: ansible.cfg and ansible_hosts.

ansible.cfg:

[defaults]
inventory=ansible_hosts
host_key_checking=False

[ssh_connection]
pipelining=True

Now, for ansible_hosts, this is where you put the names of your servers that you’re going to be applying your tasks to. We don’t have any hosts yet, so we need to create one. You can, of course, apply Ansible tasks to your own Mac, but we won’t do that. You could also install VirtualBox and create a virtual machine to play with, but we’ll skip that too and go straight to AWS.

Amazon Web Services

Like Ansible, AWS is awesome. I have on my bookshelf a copy of The Cloud at Your Service, where I was first introduced to AWS. This is awesome, I thought. Rather than hosting a webserver from this old Sony Vaio (whatever happened to that thing anyway?), now I can create a VM in the cloud. Forgive those that are middle-aged and still look back at where computing was, and where it is today, and think wow.

Since AWS was my first love, I’ve tended to stick with it, and to be honest, for good reason. The product has become cheaper over the years even while adding more services such as S3, Route 53, VPCs (there was a time VPCs didn’t exist), and now things like RDS and much more. Yes, there are other cloud computing platforms out there (Azure, Google, Linode, Digital Ocean, Scaleway, and many more), and given the need I could go and probably master them all, but for now I’m quite content with AWS. If you find yourself using another platform, Ansible will still work, you just may need to tweak some of the techniques and steps presented here.

Finally, this isn’t an AWS tutorial by any means, though we will get into some handy Ansible modules that take some of the drudgery out of creating instances and databases (in which case if you’re on another platform you will definitely need to find some other module that does the equivalent). So if you don’t know how to create an EC2 instance and set some basic security groups, you might start here.

One last note before we get back to Ansible, and that’s I’ll be using Ubuntu Server 16.04, also known as Xenial Xerus. You may prefer to work with CentOS, Red Hat Enterprise, or plain Debian. While Ansible is generally distribution agnostic, some things may not work exactly the same.

Back to Ansible

Alright, I hope you created an EC2 instance in AWS. I created one in the Ohio region (us-east-2), and as promised used Ubuntu 16.04 LTS (ami-916f59f4). For basic tutorials go ahead and use the t2.micro instance, it’s a lot cheaper. You should also have ensured that you have access to this instance, that is, your public SSH key was installed on this new instance and your security groups permit you to access it (protecting your EC2 infrastructure with security groups is another great topic that I hope to get to one day).

My instance was given a public IP of 18.188.72.168 and my key was installed, so ssh ubuntu@18.188.72.168 logged me right in:

ssh ubuntu@18.188.72.168
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-1052-aws x86_64)

Hot damn.

One of the unfortunate things about the default AMI for Ubuntu 16.04 is that it doesn’t include python on it. Now granted, Python has only been around since 1991, but still, you’d think it was worth including as a default package in an Ubuntu instance. This is important because Ansible requires python to be on the target instance (we’re talking specifically about Linux instances here). So, until we show you how to create your own AMI, we need to install Python on this VM. Easy enough:

ubuntu@ip-172-31-22-141:~$ sudo apt-get install python
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib python-minimal
  python2.7 python2.7-minimal
Suggested packages:
  python-doc python-tk python2.7-doc binutils binfmt-support
The following NEW packages will be installed:
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib python
  python-minimal python2.7 python2.7-minimal
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,877 kB of archives.
After this operation, 16.6 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Of course we entered Y.

Now for the fun part, some Ansible!

Your ansible_hosts File

In your ansible_hosts file just put this:

[all]
18.188.72.168

Now, that’s the public IP address of my instance, and you aren’t going to use that, you’ll use your instance’s IP. The point is your ansible_hosts file needs either an IP or FQDN of the machine(s) that you’re going to be working with. It’s that simple really.

Your First Playbook

You’re ready for your first playbook. Call it whatever your like, we’ll just use playbook.yml for now. And here we go:

playbook.yml:

That’s it! See you in Part 2!

Ansible Modules

Just kidding. We’ll add a bit more to our first playbook, but first, let’s take a look at the syntax of the file. The extension .yml is on purpose as Ansible roles, tasks, variable configurations, playbooks, etc. are all written using YAML, a “human friendly data serialization standard for all programming languages.” YAML’s syntax is indentation-driven (like Python), so you need to make sure everything is properly indented or you’ll get incomprehensible errors. For a quick check you can install something like yamllint (availabe on the Mac with brew install yamllint).

For a brief moment we’ll ignore the hosts and remote_user tags and focus instead on the tasks section. We have one task listed here: hostname. Ansible provides great reference documentation to look up all of the things you can do with this module. This is a simple one as it only takes one parameter, name.

Now, don’t confuse the name parameter of the hostname module with that of the name parameter of the task. This used to trip me up. It’s easier to see task entries when there are more than one, say let’s do that. I’m going to be very specific on how this is written (and then tell you not to do it):

Notice how the apt module (which also has great reference documentation) is the second list entry under tasks; we know this because of the dash. But then there is some indentation (two spaces) and three key-value pairs (name, state, and update_cache). Then the indentation is backed out and we have name. The second name here (which has the value Install htop) is associated with the task at hand (in our case, apt).

If you’re confused, don’t worry. YAML (and any serialization syntax) can be baffling and bewildering. But if you follow some basic conventions and keep playbooks simple (which will involve breaking things down over time into roles and other tasks), it’ll be easy to read.

Now, what you aren’t supposed to do is put the name parameter for the task at the bottom. Clean that up!

Much better. Now, let’s run it!

ansible-playbook playbook.yml

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [18.188.72.168]

TASK [Set our hostname] ********************************************************
changed: [18.188.72.168]

TASK [Install htop] ************************************************************
changed: [18.188.72.168]

PLAY RECAP *********************************************************************
18.188.72.168              : ok=3    changed=2    unreachable=0    failed=0

Idempotency

In general, an operation is idempotent if it produces the same result even after being executed multiple times. This is an important property to strive for in Ansible playbooks, though my experience is that it isn’t always achievable. It may appear on the surface that our above playbook is idempotent though it isn’t. For argument’s sake let’s look at what we would desire from an idempotent playbook:

  • if the hostname isn’t set to ansible-helloworld, set it to ansible-helloworld
  • if htop isn’t installed, install it, otherwise do nothing

On the first pass of the playbook we see changed=2, indicating that two tasks changed something on the server. Indeed, we set the hostname and then installed htop. Let’s run it again:

PLAY RECAP *********************************************************************
18.188.72.168              : ok=3    changed=0    unreachable=0    failed=0

Now, nothing has changed. Run it again!

PLAY RECAP *********************************************************************
18.188.72.168              : ok=3    changed=0    unreachable=0    failed=0

Success! An idempotent playbook.

Until update_cache: true triggers an upgrade of the htop application from the underlying repository. Strictly speaking this one line prevents the playbook from being idempotent, and if it were critical that the server didn’t receive any apt-get updates, removing update_cache can help but would likely be insufficient.

Final Thoughts

We didn’t cover the remote_user, become, etc. directives in our playbook, but that’s okay. If you remove them the playbook won’t work at all. Let’s add one last interesting twist to the playbook, and that’s installing more than htop to the server. I happen to like Z Shell (and even more, Oh My Zsh) installed on my servers. So let’s install it.

We could create a separate apt task (separate from the one installing htop) to install zsh, but that seems a bit silly. Let’s use the with_items capability like this:

Confusing syntax alert! Believe it or not, it took me some time before this syntax felt natural, because it’s sort of like writing for loops in Apache Ant. It’s clunky, and namely because you’re taking a markup-type language and trying to create logical constructs in it. It just feels weird at first.

The first thing that’s weird is the "{{ item }}" syntax. What’s with the quotes? There weren’t quotes before. Why those braces? Two braces? Not one brace? What is item? Who sets that? Is it magic?

Try removing the quotes. Go ahead. You’ll be sorry you did when you’re greeted with We could be wrong, but this one looks like it might be an issue with missing quotes. Always quote template expression brackets when they start a value. Assholes. So the quotes need to stay there.

The braces mark off template interpolation. Whatever the variable item is set to will be used. The with_items parameter supplies, in turn, each of the items, substituting the items (no pun intended) in the list into the item variable. So in this way you can see how the playbook will run now:

# ansible-playbook playbook.yml

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [18.188.72.168]

TASK [Set our hostname] ********************************************************
ok: [18.188.72.168]

TASK [Install htop] ************************************************************
changed: [18.188.72.168] => (item=[u'htop', u'zsh'])

PLAY RECAP *********************************************************************
18.188.72.168              : ok=3    changed=1    unreachable=0    failed=0

That’s it for Part 1 of this series! If you’ve never worked with Ansible before, I hope this was helpful in getting you started; yes, there are other tutorials out there on Ansible but I wanted to lay the foundation for a series of articles that will walk you through how to go from a simple playbook that installs a couple of packages to roles that can install, configure, and back up MySQL databases and much, much more.

If you have any suggestions for things you’d like to see in an article, let me know by posting a comment or on Twitter @iachievedit.

Thanks!

By

SSH Hardening

DevOps ToolChain, WikiPedia, CC BY-SA 4.0

Why Hardening

Hardening, as I define it, is the process of applying best practices to make it harder for others to obtain unauthorized access to a given service or to the data transmitted by the service. In this post we’ll take a look at hardening SSH access to our server, as well as making it more difficult for others to potentially snoop our SSH traffic.

We’ll be using a fresh AWS EC2 instance running Ubuntu 16.04 for our examples. If you’re running a virtual server in Azure, Digital Ocean, or some other hosting provider, you’ll want to check out how the equivalent of AWS security groups are configured. And of course, these techniques can also be applied to non-virtual systems.

AWS Security Groups

The first step in hardening your SSH server is applying a more restrictive security group to your instance. Think of AWS security groups as custom firewalls you can apply to your instance. Even better, these custom firewalls can apply source-based filtering rules that only allow traffic from subnets or hosts you specify.

Subnet-based rules provides for rules like “Only allow SSH traffic from my development team’s network 10.90.4.0/24” If your internal network is segregated and configured such that developers must authenticate to receive a 10.90.4.0/24 IP address, an additional safeguard is added.

A host-based rule will only allow traffic from a given host (strictly speaking, a given IP address; if a host is behind a NAT then any hosts also behind that NAT will be allowed). This is what we’ll use.

private-ssh-sg

In the above example, we’ve created a security group private-ssh-sg and added a single Inbound rule that allows traffic on port 22 from a specific IP address. This will effectively only allow packets whose source IP is specified in that rule to reach port 22 of the instance.

SSH Cipher Strength

Another technique you can use to harden your SSH server is ensuring that the latest strong key exchange protocols, ciphers, and message authentication code (MAC) algorithms are utilized.

We’ve used several references as a guide to hardening SSH, including Mozilla’s OpenSSH Guidelines as well as ssh-audit, a nice tool designed specifically for this task. Using ssh-audit is as easy as

$ git clone https://github.com/arthepsy/ssh-audit
$ cd ssh-audit
$ ./ssh-audit.py your.ip.address.here

sshaudit

Our first pass uncovers a number of issues:

  • use of weak elliptic curves in the key exchange algorithms supported
  • use of weak elliptic curves in the host-key algorithms supported

ssh-audit goes on to recommend the key exchange, host key, and MAC algorithms to remove.

Let’s look at changes we want to make to our /etc/ssh/sshd_config file:

Restart your SSH daemon with sudo systemctl restart ssh. Note: It’s a good idea to have a second terminal logged in if you bork your SSH configuration and lock yourself out of your instance.

Once we’ve updated our sshd_config configuration, it’s time to run an audit against it.

Shell

Nice! Strong key exchange, encryption, and MAC algorithms all around.

If you’re looking for a simple SSH daemon configuration, look no further:

Two-Factor Authentication

There are different interpretations as to what constitutes two-factor or multi-factor authentication. Many believe that two-factor authentication implies an additional authentication code delivered via text message or provided by a key fob. Others may consider the steps taken to obtain access to a given computing resource as a part of the authentication steps (e.g., to obtain access to a given server you must get past the security guard, provide a retinal scan, and so on). In this example, we’re going to use the former interpretation.

Authy

We’ve chosen to use Authy in this example for two-factor authentication using a time-based one time password. To get started, install the Authy application on your phone (iOS or Android) and follow the quick-start prompts.

After you’ve successfully set up the application on your phone, you can download the app to your desktop or add it to Google Chrome.

Getting Your EC2 Instance Ready

To use the authentication code provided by Authy to add an additional authentication step for SSH logins requires installing the libpam-google-authenticator module and configuring both SSH and PAM.

Install the module with sudo apt-get install libpam-google-authenticator.

Now, as a user that needs to use two-factor authentication, run google-authenticator to get set up. The application will generate several prompts, the first of which is Do you want authentication tokens to be time-based to which you’ll answer “yes”

google-authenticator will then generate QR-code that you can scan with the Authy phone application, as well as a secret key that can be used with the phone, desktop, or browser application. When using the desktop application I prefer just copy-paste of the secret key.

There are additional prompts from the application to follow:

Do you want me to update your "/home/ubuntu/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) n

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

NB: It is a good idea to save your emergency scratch codes in the event you lose access to the devices that are generating your OTPs.

Now that you’ve configured the authenticator, it’s time to update sshd_config to consult the PAM Google Authenticator module when a user attempts to log in.

Open /etc/ssh/sshd_config as root, and set the following:

ChallengeResponseAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive

Restart ssh with sudo systemctl restart ssh.

Now, in /etc/pam.d/sshd replace the line @include common-auth with auth required pam_google_authenticator.so.

Once the sshd PAM module has been configured in this manner users will be challenged for a two-factor authentication code, so it’s important that every user on the system be configured with the google-authenticator application.

Test your login!

Try logging in via ssh with the user you’ve just configured for two-factor authentication. You should receive a prompt requesting a verification code (after your key is authenticated).

enter_code

Enter the code displayed on your Authy app (note that all of your Authy apps will display the same code for the same application configuration) to login.

code_entered

One Last Thought on Authy

While writing this post I was doing additional research on two-factor authentication implementations; while Authy supports time-based one time passwords, it supports additional methods that require access to their infrastructure. If you don’t care for providing your cellphone number (and a lot of people don’t), try out Authenticator, a Chrome plugin that doesn’t require any account setup.

You’ll notice if you try out different applications that you can use the same secret key and each application will generate the same code at the same time, hence the time-based one time password.

Recap

There is no such thing as perfect security save turning off the computer, disconnecting all of its cables, putting it in a trunk, filling that with cement, and tossing it into the Pacific. Even that might not be perfect. In the absence of perfection we can put as many barriers in place between our server and others that shouldn’t have access to it. In this post we’ve taken a look at several of those barriers:

  • source-based firewall rules that only allow access on port 22 from a specific IP or subnet
  • hardened key exchange algorithms, ciphers, and MACs for SSH
  • two-factor authentication that requires both public key authentication as well as an OTP code

We did not cover additional techniques such as configuring SSH to listen on a different port; I’ve found that despite explaining that this is done primarily to minimize port-scanning chatter (who wants to sift through auth or fail2ban logs with script kiddie traffic) it never fails to incite the crowd of folks who just learned the phrase security through obscurity to gather up their pitchforks.

If you have any additional recommendations regarding SSH security hardening, please leave a comment!

By

An ARM Build Farm with Jenkins

There was a time when setting up a Continuous Integration server took a lot of work. I personally have spent several days wrestling with getting CruiseControl installed, configured, and working just right. Today it is much more straightforward and, for the most part, a simple apt-get install jenkins is all it takes to get a functional Jenkins CI server up and running.

In this tutorial we’re going to look at using Jenkins to set up a build farm of ARM systems. My personal interest in doing so is to support the Swift on ARM group of folks working to get Swift 3.0 support for ARMv7 devices such as the Raspberry Pi 2, BeagleBone Black, etc. While support for cross-compiling Swift is maturing there is still a desire to natively compile on an ARM system.

The Jenkins Build Server

To make things simple we’re going to focus on installing Jenkins on an Ubuntu 14.04 server. Using the instructions on the Jenkins Wiki for an Ubuntu 14.04 system:

The Jenkins daemon will start automatically upon installation. Once it does, open a browser and go to http://YOUR_HOST:8080/, where YOUR_HOST is the server you just installed Jenkins on.

Unlock!

Unlock!

The initial password can be found on the Jenkins server with sudo cat /var/lib/jenkins/secrets/initialAdminPassword, and will be something like “54d5f68be4554b5c8316689728721b37”. Paste it in and click Continue.

You’ll be prompted to choose whether or not to install suggested plugins or specifically select plugins. Go with Install suggested plugins for now. The next screen will be all of the plugins being loaded in. Once complete you’ll move on to creating the first admin user:

Create an Admin User

Create an Admin User

Once entering the admin user information, you’ll see Jenkins is ready!

Yes!

Yes!

Click that Start using Jenkins button and (wait for it), start using Jenkins.

Build Agents

We’re interested in using Jenkins to perform native builds on ARM systems. To that end we’ll add slave nodes to our build server.

First, let’s make sure we have an ARM system to compile on. I’ve become a big fan of using Scaleway for spinning up ARM systems. If you have your own ARM device such as BeagleBoard or Raspberry Pi, those will work as well, just be aware that compiling on single-core ARM devices with limited RAM can be, well, painful. With Scaleway you can spin up quad-core ARM systems with 2G of RAM for about $3.50 a month.

I’m going to assume you have an ARM system (either from Scaleway or a physical device that your Jenkins host can communicate with). Let’s look at what it takes to get things configured. There’s a bit of a dance to do to ensure that the master and slave can communicate with each other. In short we need to:

  • Install Java on the ARM system so the slave agent can run
  • Create a build user on the ARM system
  • Create a public/private key pair on the build server
  • Add the build server’s public key to the ARM build user’s authorized_keys file

1. Install Java and create a build user

Your ARM device is going to need Java to run the Jenkins slave agent, so on the ARM system run sudo apt-get install openjdk-7-jre-headless.

Let’s also set up the build user while we’re here with sudo adduser build. Follow the prompts to create the user appropriately.

2. Create a public/private key pair on the build server

On the build server, su to the jenkins user and cd to its home directory. Run ssh-keygen to create the key pair.

3. Add the public key to the authorized_keys file

Now, add the ~/.ssh/id_rsa.pub contents of the Jenkins server jenkins user to the /home/build/.ssh/authorized_keys file of the ARM system build user.

Adding the Build Slave Node

Now, let’s add the ARM device as a build slave. Under Manage Jenkins go to the Manage Nodes link.

Manage Nodes

Manage Nodes

Click on New Node and we’ll give it a name like arm1. Select the Permanent Agent radio button and then click OK.

You’ll move on to a screen that requires some more information (typical!) We’re interested in:

  • Remote root directory
  • Labels
  • Launch method

For Remote root directory we’ll use the home directory of our new build user, so /home/build. For the label, type in arm. It will become apparent as to why in a bit.

For Launch method choose Launch slave agents on Unix machines via SSH. In the host field enter your ARM system’s hostname or IP address that is accessible from your Jenkins build server.

Launch method

Launch method

Now, click on the Add button (the one with the key) to go to the Add Credentials configuration page. For the Kind choose SSH Username with private key. For the Username we’ll use build. For the Private Key select From the Jenkins master ~/.ssh/. Recall earlier when we had you create a public/private key pair for the jenkins user on the master? This is why. When trying to contact the slave agent the Jenkins system will load the private key from the ~/.ssh/ directory of the jenkins user. The slave agent will be ready with the public key in the authorized_keys file.

Adding Credentials

Adding Credentials

Click Add to add the credentials, and then Save to save the new node configuration.

You should be taken back to the Node Management page where you’ll see your new node (probably in an offline state):

arm1 Offline

arm1 Offline

You can either click Refresh status or go to the node’s logs by clicking on the nodename arm1 and then selecting Log. If everything was configured properly you’ll see:

[04/27/16 17:01:42] [SSH] Starting slave process: cd "/home/build" && java  -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave.jar version: 2.56
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

Now let’s create a new job and use our ARM agent to build opencv.

Building OpenCV

We know we’re going to be using git to check out the OpenCV source tree from Github, and there are a number of other dependencies required for building, so we need to install all of them on our ARM system:

sudo apt-get install build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev

OpenCV is traditionally built with the following commands:

These will be our basic build steps, with the exception we’ll use make -j4 to take advantage of our 4 core ARM build agent.

Creating the Build Project

To create a new build project, click on Create New Item in the main Jenkins menu, and choose Freestyle project. We’ll name our project OpenCV 3.0. Click OK at the bottom of the options.

Creating a new Freestyle Project

Creating a new Freestyle Project

You’ll be on the General tab to get started. For now we’re interested in the general project options, Source Code Management section, and Build section.

Since we want to build the OpenCV project on the ARM system we are going to restrict the project to only build on those nodes matching our arm label.

Restrict Project

Restrict Project

For OpenCV 3.0 we’ll choose Git as the SCM type, and then enter the URL to the OpenCV source, https://github.com/itseez/opencv/. Jenkins allows you to specify the branch to build on, and we’ll leave the default master.

GIT for OpenCV

GIT for OpenCV

For the actual build we will enter a single Execute shell build step. It should be noted that this is an example only. There are a number of ways to configure and script out Jenkins project; each project will have different build steps to fit the needs of the underlying task at hand.

Configure and Build OpenCV

Configure and Build OpenCV

Save your project, and then click Build Now! In the Build History pane you’ll see a flashing blue dot (hopefully). Click on the dot to go to the build page where you can select to look at the console output of the build. If everything was set up correctly you should now see OpenCV building away on your ARM build agent!

Building on arm1

Building on arm1

OpenCV 3.0 Build Log

OpenCV 3.0 Build Log

In our example OpenCV took 42 minutes to build on a quad-core ARM system from Scaleway. Not bad.

Final Thoughts

This was clearly not a comprehensive Jenkins tutorial. Folks familiar with the craft know that setting up and configuring continuous integration servers and build projects to produce traceable artificacts is a disclipline unto itself. However, it should be clear that creating a compile farm consisting of slave build agents does not have to be complicated.

By

Using Monit with Node.js and PM2

As I’ve said before, I’m a bit of a whore when it comes to learning new languages and development frameworks. So it comes as no surprise to myself that at some point I’d start looking at Node.js and JavaScript.

I have another confession to make. I hate JavaScript, even more so than PHP. Nothing about the language is appealing to me, whether its the rules about scoping and the use of var or the bizarre mechanism by which classes are declared (not to mention there are several ways). Semicolons are optional, kind of, but not really. I know plenty of developers who enjoy programming in Objective-C, Python, Ruby, etc.; I have never met anyone who says “Me? I love JavaScript!” Well, perhaps a web UI developer whose only other “language” is CSS or HTML. In fact, a lot of people go out of their way to articulate why JavaScript sucks.

So along comes Node.js, which we can all agree is the new hotness. I’m not sure why it is so appealing. JavaScript on the server! Event-driven programming! Everything is asynchronous and nothing blocks! Okay, great. I didn’t really ask for JavaScript on the server, and event-driven programming is not new. When you develop iOS applications you’re developing in an event-driven environment. Python developers have had the Twisted framework for years. The venerable X system is built upon an event loop. Reading the Node.js hype online one would think event-driven callback execution was invented in the 21st century.

Of course, the Node.js community is also reinventing the wheel in other areas as well. What do the following have in common: brew, apt-get, rpm, gem, easy_install, pip. Every last one is a “package manager” of some sort, aimed at making your life easy, automagically downloading and installing software along with all of its various dependencies onto your system. A new framework is nothing without a package manager that it can call its own, thus the Node.js world gives us the Node Package Manager, or npm. That’s fine. I like to think of myself as a “full-stack developer”, so if I need to learn a new package manager and all of its quirks, so be it.

Unfortunately it didn’t stop there. Node.js has its own collection of application “management” utilities; you know, those helper utilities that aim to provide an “environment” in which to run your application. Apparently Forever was popular for some time until it was displaced by PM2, a “Production process manager for Node.js / io.js applications”

I’m not quite sure when it became en vogue to release version 0 software for production environments, but I suppose it’s all arbitrary (hell, Node.js is what, 0.12?) But true to a version 0 software release, PM2 has given me nothing but fits in creating a system that consistently brings up a Node.js application upon reboot. Particularly in machine-to-machine (M2M) applications this is important; there is frequently no opportunity to ssh into a device that’s on the cellular network and installed out in an oil field tank site. The system must be rock-solid and touch free once it’s installed in the field.

To date the most pernicious bug I’ve come across with PM2 is it completely eating the dump.pm2 file that it ostensibly uses to “resurrect” the environment that was operating. A number of people have reported this issue as well. If I can’t rely on PM2 to consistently restart my Node.js application, I need something to watch the watchers. So who watches the watchers? Monit of course.

Because PM2 refused to cooperate I decided to utilize monit to ensure my Node.js application process was up and running, even after a reboot. In this configuration file example I am checking my process pid (located in the /root/.pm2/pids directory) and then using pm2 start and pm2 stop as the start and stop actions.

NB: Monit executes its scripts with a bare bones environment. If you are ever stumped by why your actions “work on the command line but not with monit”, see this this Stack Overflow post. In the case of PM2, it is critical that the PM2_HOME environment variable be set prior to calling pm2.

The first iteration of my monit configuration looked like this:

Only if this were sufficient, but it isn’t.

For some reason PM2 insists on appending a process ID to the pidfile filename (perhaps for clustering where you need a bunch of processes of the same name), so a simple pidfile check won’t suffice. Other folks even went to the Monit lists looking for wildcard pidfile support and quoted PM2 as the reason why they felt they needed it.

So, now our monit configuration takes advantage of the matching directive and looks like this:

Granted, we should not be running as root here. Future iterations will move the applications to a non-privileged user, but for now this gives us a system that successfully restarts our Node.js applications after a reboot. PM2 is a promising tool and definitely takes care of a lot of mundane tasks we need to accomplish to daemonize an application; unfortunately it is a little rough around the edges when it comes to consistent actions surrounding the ability to survive system restarts. Don’t take my word for it: read the Github issues.

Conclusion

A rolling stone gathers no moss. The more things stay the same, the more things change (or is it the other way around?). I have nothing against new frameworks, but there are times when being an early adopter requires one to pull out a tried-and-true applications to get the job done. In this case our old friend Monit helps up fill in the gaps while Node.js and PM2 mature.