Ansible and AWS – Part 3

This is the third article in a series looking at utilizing Ansible with AWS. So far the focus has been Ansible itself, and we’ll continue that in this post and look at the Ansible Vault. The final version of the Ansible playbook are on the part3 branch of this repository.

You will frequently come across occasions where you want to submit something that needs to be kept secret to a code repository. API keys, passwords, credentials, or other sensitive information that really is a part of the configuration of a server or application, and should be revisioned controlled, but shouldn’t be exposed to “the world” are great candidates for things to lock up in an Ansible vault. I’ll illustrate several different applications of a vault, and then share a great technique for diffing vault files.

Vault It!

Any file can be encrypted (vaulted) with the ansible-vault command. now.txt contains:

[code lang=text]
Now is the time for all good men to come to the aid of their country.
[/code]

Let’s encrypt it with ansible-vault:

[code lang=text]
# ansible-vault encrypt now.txt
New Vault password:
[/code]

Type in a good password and you’ll be prompted to confirm it. If the passwords match you’ll see Encryption successful. Now, cat the file:

[code lang=text]
# cat now.txt
$ANSIBLE_VAULT;1.1;AES256
30323261363061623735613964633862613232393261323638396435373561396237313161336531
…
33363033636132353034306130383064306264613935353133386664613062613438383361663462
33393438396236383230
[/code]

You’ll notice that ansible-vault did not change the name of your file or add an extension. Thus from looking at the filename you can’t tell this file is a “vault” file.

To edit the file, use ansible-vault edit now.txt. You’ll be prompted for the password (I hope you remember it!), and if provided successfully ansible-vault will open the file with the default editor (on my Mac it’s vi but you can set the EDITOR shell variable explicitly). Make any changes you like and then save them in your editor. ansible-vault will reencrypt the file for you automatically.

ansible-vault view is another handy command; if you just want to look at the contents of the file and not edit them use ansible-vault view FILENAME and supply the password.

Practical Use

The most practical use of the vault capability is to protect sensitive information such as API keys, etc. We’re going to use it for two applications: encrypt AWS IAM credentials for accessing an S3 bucket, and an SSL certificate key.

First though, let’s talk about where to put the actual variables. We could put them in vars.yml and then encrypt the entire file. This is overkill. Better is to create a file called vault (so we know it’s a vault!) and put our values there. But, we’re going to use a little indirection technique like this:

vault:

By the way, to be clear, the vault file here is located in our host_vars directory for the target host, alongside our vars.yml for that host.

vars.yml:

Again, this is the vars.yml file we created previously for our host.

Our playbooks and template files will reference AWS_ACCESS_KEY and AWS_SECRET_KEY which in turn will get their values from the vault. In this manner you can quickly view your vars.yml and know which values you get from the vault (hence the prefix VAULT_). NB: This is purely a convention I’ve come to find to be extremely useful!

The Real Thing

To access S3 buckets we’re going to use the s3cmd package, so let’s add it to our playbook apt task:

Then, in our AWS console we’ll create a new IAM user named S3Reader and give it Programmatic access. For this user we can Attach existing policies directly and use the existing policy AmazonS3ReadOnlyAccess. You should be given both an access key and a secret key. We’ll plug these both in to our vault file and then encrypt it with ansible-vault encrypt.

Even though s3cmd is installed, it needs a configuration file .s3cfg. We’re going to create one with our credentials (the ones just obtained), again through the use of the template task. Create a file in the templates directory called s3cfg.j2 (make note we aren’t putting a dot in front of the file) with the following content:

[code lang=text]
[default]
access_key={{ AWS_ACCESS_KEY }}
secret_key={{ AWS_SECRET_KEY }}
[/code]

Now, add the template task that will create the .s3cfg file:

Note the use of the user and group parameters for this task. The file is being created in the /home/ubuntu/ directory and since our Ansible playbook is issuing commands as the root user (the become_user directive in the playbook we’ve glossed over thus far), it would have, by default, set the owner and group of the file to root. We don’t want that, so we explicitly call out that the user and group should be set to ubuntu.

Execute your playbook, this time including --ask-vault to ensure you get prompted for your vault password (we’ll show you in a minute how to set this in a configuration file).

[code lang=text]
# ansible-playbook playbook.yml –ask-vault
[/code]

If everything was done correctly you’ll have a new file .s3cfg in your /home/ubuntu directory that has your S3 credentials (the ones you placed in the vault file). Indeed, we can now run the s3cmd ls:

[code lang=text]
ubuntu@helloworld:~$ s3cmd ls
2016-10-01 14:06 s3://elasticbeanstalk-us-east-1-124186967073
2017-10-04 00:20 s3://iachievedit-cbbucket
2015-12-21 23:47 s3://iachievedit-repos
…
[/code]

Nice.

key.pem

If you run a website you’re undoubtedly familiar with SSL certificates and the need to have one (or more). You’re also familiar with the private key for your certificate (unless you started your career with Let’s Encrypt and it’s abstracted some of the “magic”). That private key is, strictly speaking, a part of the server configuration, and we want everything on our server to be revisioned controlled, including that private key. So let’s vault it.

I have a private key named key.pem and the contents look like what you’d expect:

[code lang=text]
# cat key.pem
—–BEGIN PRIVATE KEY—–
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDDXnM+DZ0tTn9M
38CRP2dwOuRJhH3xRc3WcTmL1S+EIxdFGIffy0+YVXU9Uzbn0XDl0TgtfZwBcMeU
…
wVumMxo0FpnH9Wb3QtVwlv1zxaW2VakRvdiZ31PkUWK7oCkFJPVj0yGe/MDsCfQX
lCDhi/ncV+DL/FWXgPafUIA=
—–END PRIVATE KEY—–
[/code]

We don’t want to submit this as-is to revision control, so let’s encrypt it with ansible-vault and put it in a directory called files.

[code lang=text]
# ansible-vault encrypt $HOME/key.pem
# cd ansible-helloworld
# mkdir files
# cp $HOME/key.pem files/
[/code]

Note here that you will want to use the same vault password as you did for vault. Also, the directory files (like templates) is a special name to Ansible. Don’t call it file or my_files, etc., just files!

Our key.pem will be copied to /etc/ssl/private, so let’s create a task for that in the playbook:

If you run the playbook now and then check /etc/ssl/private/key.pem on your server you’ll see that Ansible decrypted the file and stored the decrypted contents; precisely what we wanted!

Your Vault Password

If you grow tired of supplying the vault password to --ask-vault each time you run a playbook, or even when using ansible-vault edit you can save the password in plain text in your home directory or other secure location. This is an example of a file you would not submit to revision control!

[code lang=text]
# export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible_vault_pass.txt
# ansible-playbook playbook.yml
[/code]

The file ~/.ansible_vault_pass.txt contains the vault password in the clear. Add the environment variable ANSIBLE_VAULT_PASSWORD_FILE to your shellrc file (e.g., .zshrc, .bashrc, etc.) to make things even simpler.

Vault Diffs

I found this one weird trick just the other day and it’s a lifesaver if you are responsible for reviewing changes to API keys, passwords, etc. Courtesy of Mark Longair on Stack Overflow.

We have two “vaulted” files in our repository, a file named vault in our host_vars directory and key.pem. We’d like to easily diff them for changes, so following along with the Stack Overflow post, create a .gitattributes file with:

[code lang=text]
host_vars/*/vault diff=ansible-vault merge=binary
files/key.pem diff=ansible-vault merge=binary
[/code]

Then, set the diff driver for files with that diff=ansible-vault attribute:

[code lang=text]
# git config –global diff.ansible-vault.textconv "ansible-vault view"
[/code]

With this, if you make changes to your vaulted files, a git diff will automatically show you the actual diff vs. the encrypted diff (which is, obviously, unreadable).

Final Thoughts

Credentials, API keys, passwords, private keys. It’s been hammered in your head to not submit those to revision control systems, and rightfully so if they aren’t encrypted. But these pieces of information are critical to the successful build of a server or environment, and part of the charm that Ansible brings is the ability to rebuild a server (or sets of servers) and not worry about whether or not you missed a step. With the Ansible vault, however, you can encrypt this information and include it alongside the rest of your Ansible playbook; supply the vault password and watch the magic happen.

This Series

Each post in this series is building upon the last. If you missed something, here are the previous posts. We’ve also put everything on this Github repository with branches that contain all of the changes from one part to the next.

• Part 1
• Part 2