Software Development Tips and Tricks


SSH Hardening

DevOps ToolChain, WikiPedia, CC BY-SA 4.0

Why Hardening

Hardening, as I define it, is the process of applying best practices to make it harder for others to obtain unauthorized access to a given service or to the data transmitted by the service. In this post we’ll take a look at hardening SSH access to our server, as well as making it more difficult for others to potentially snoop our SSH traffic.

We’ll be using a fresh AWS EC2 instance running Ubuntu 16.04 for our examples. If you’re running a virtual server in Azure, Digital Ocean, or some other hosting provider, you’ll want to check out how the equivalent of AWS security groups are configured. And of course, these techniques can also be applied to non-virtual systems.

AWS Security Groups

The first step in hardening your SSH server is applying a more restrictive security group to your instance. Think of AWS security groups as custom firewalls you can apply to your instance. Even better, these custom firewalls can apply source-based filtering rules that only allow traffic from subnets or hosts you specify.

Subnet-based rules provides for rules like “Only allow SSH traffic from my development team’s network” If your internal network is segregated and configured such that developers must authenticate to receive a IP address, an additional safeguard is added.

A host-based rule will only allow traffic from a given host (strictly speaking, a given IP address; if a host is behind a NAT then any hosts also behind that NAT will be allowed). This is what we’ll use.


In the above example, we’ve created a security group private-ssh-sg and added a single Inbound rule that allows traffic on port 22 from a specific IP address. This will effectively only allow packets whose source IP is specified in that rule to reach port 22 of the instance.

SSH Cipher Strength

Another technique you can use to harden your SSH server is ensuring that the latest strong key exchange protocols, ciphers, and message authentication code (MAC) algorithms are utilized.

We’ve used several references as a guide to hardening SSH, including Mozilla’s OpenSSH Guidelines as well as ssh-audit, a nice tool designed specifically for this task. Using ssh-audit is as easy as

$ git clone https://github.com/arthepsy/ssh-audit
$ cd ssh-audit
$ ./ssh-audit.py your.ip.address.here


Our first pass uncovers a number of issues:

  • use of weak elliptic curves in the key exchange algorithms supported
  • use of weak elliptic curves in the host-key algorithms supported

ssh-audit goes on to recommend the key exchange, host key, and MAC algorithms to remove.

Let’s look at changes we want to make to our /etc/ssh/sshd_config file:

Restart your SSH daemon with sudo systemctl restart ssh. Note: It’s a good idea to have a second terminal logged in if you bork your SSH configuration and lock yourself out of your instance.

Once we’ve updated our sshd_config configuration, it’s time to run an audit against it.


Nice! Strong key exchange, encryption, and MAC algorithms all around.

If you’re looking for a simple SSH daemon configuration, look no further:

Two-Factor Authentication

There are different interpretations as to what constitutes two-factor or multi-factor authentication. Many believe that two-factor authentication implies an additional authentication code delivered via text message or provided by a key fob. Others may consider the steps taken to obtain access to a given computing resource as a part of the authentication steps (e.g., to obtain access to a given server you must get past the security guard, provide a retinal scan, and so on). In this example, we’re going to use the former interpretation.


We’ve chosen to use Authy in this example for two-factor authentication using a time-based one time password. To get started, install the Authy application on your phone (iOS or Android) and follow the quick-start prompts.

After you’ve successfully set up the application on your phone, you can download the app to your desktop or add it to Google Chrome.

Getting Your EC2 Instance Ready

To use the authentication code provided by Authy to add an additional authentication step for SSH logins requires installing the libpam-google-authenticator module and configuring both SSH and PAM.

Install the module with sudo apt-get install libpam-google-authenticator.

Now, as a user that needs to use two-factor authentication, run google-authenticator to get set up. The application will generate several prompts, the first of which is Do you want authentication tokens to be time-based to which you’ll answer “yes”

google-authenticator will then generate QR-code that you can scan with the Authy phone application, as well as a secret key that can be used with the phone, desktop, or browser application. When using the desktop application I prefer just copy-paste of the secret key.

There are additional prompts from the application to follow:

Do you want me to update your "/home/ubuntu/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) n

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

NB: It is a good idea to save your emergency scratch codes in the event you lose access to the devices that are generating your OTPs.

Now that you’ve configured the authenticator, it’s time to update sshd_config to consult the PAM Google Authenticator module when a user attempts to log in.

Open /etc/ssh/sshd_config as root, and set the following:

ChallengeResponseAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive

Restart ssh with sudo systemctl restart ssh.

Now, in /etc/pam.d/sshd replace the line @include common-auth with auth required pam_google_authenticator.so.

Once the sshd PAM module has been configured in this manner users will be challenged for a two-factor authentication code, so it’s important that every user on the system be configured with the google-authenticator application.

Test your login!

Try logging in via ssh with the user you’ve just configured for two-factor authentication. You should receive a prompt requesting a verification code (after your key is authenticated).


Enter the code displayed on your Authy app (note that all of your Authy apps will display the same code for the same application configuration) to login.


One Last Thought on Authy

While writing this post I was doing additional research on two-factor authentication implementations; while Authy supports time-based one time passwords, it supports additional methods that require access to their infrastructure. If you don’t care for providing your cellphone number (and a lot of people don’t), try out Authenticator, a Chrome plugin that doesn’t require any account setup.

You’ll notice if you try out different applications that you can use the same secret key and each application will generate the same code at the same time, hence the time-based one time password.


There is no such thing as perfect security save turning off the computer, disconnecting all of its cables, putting it in a trunk, filling that with cement, and tossing it into the Pacific. Even that might not be perfect. In the absence of perfection we can put as many barriers in place between our server and others that shouldn’t have access to it. In this post we’ve taken a look at several of those barriers:

  • source-based firewall rules that only allow access on port 22 from a specific IP or subnet
  • hardened key exchange algorithms, ciphers, and MACs for SSH
  • two-factor authentication that requires both public key authentication as well as an OTP code

We did not cover additional techniques such as configuring SSH to listen on a different port; I’ve found that despite explaining that this is done primarily to minimize port-scanning chatter (who wants to sift through auth or fail2ban logs with script kiddie traffic) it never fails to incite the crowd of folks who just learned the phrase security through obscurity to gather up their pitchforks.

If you have any additional recommendations regarding SSH security hardening, please leave a comment!


Encrypting and Obscuring API Keys

Keys, or “credentials”, are a fact of life when working with REST APIs. Sign up for Amplitude for mobile analytics and you get a key to identify yourself and the application you’re working with. If you are using Aeris Weather APIs you get both an ID and secret. If you’re going to access these APIs in your iOS application you have to put those keys somewhere. In a previous post we walked through how to store API keys in property list files. In this tutorial we’re going to encrypt those keys and look at ways on how to access them in your app.

Before we get started, however, I want to make a few comments about security. I am not a security expert, and in all likelihood, neither are you. The original title of this article was Securing API Keys, but on reflection I renamed it to Obscuring API Keys. Nothing in this post will make the claim that this method of obscuring your API keys is impossible to break, because the reality is that anyone intent on stealing your API keys is going to do it. This method will only make it a little harder.

Encrypting Your Keys

To add another level of protection to our API keys, we’re going to use Blowfish Cipher Feedback encryption on them. To use this method of encryption you will want to choose:

  • a secret
  • an initialization vector

The secret must be at least 8 characters in length with a maximum of 56 characters. Choose whatever secret you like; a phrase, jumble of characters, it doesn’t matter. The initialization vector should be a random vector of 8 bytes. Once you’ve chosen these two pieces of data, download our Key Encrypter. Enter the secret and the IV, and then the API key you want to encrypt. For example, let’s say the API key we want to encrypt is 6bbb3654679105f33cf8c491ed9b04df and we’ve chosen dontusethissecret as our secret and decafbadbaddecaf. Enter these values and click Encrypt.

API Key Encrypter

API Key Encrypter

The resulting output is a hexadecimal string that can be fed back through a decryption routine, along, of course, with the original secret and initialization vector.

Note: It is usually at this point someone objects and says, “but your secret and IV will be in your code.” That is true, they will. But your secret can be a substring of a text label in your application, your copyright notice, etc. Foolproof? No. Harder to disassemble. Yes.

Decrypting Your Key

The encryption and decryption routines make use of the Blowfish implementation included in mbed TLS. The routines are written in C and we leverage a bridging header to expose these routines in Swift.

Our decryption function in Swift 4.2 looks like this:

The question, now, is how to use the decryption method and where to put the secret and initialization vector in your code. This is up to you (and there are a variety of ways). Just remember that no matter what you do, if someone is determined to figure it out and extract it, they will. This is just another step they have to go through after decrypting and decompiling the app from a jailbroken phone, etc.

To test out decryption of the generated string, download this example iOS app, open in Xcode, and run it. You should see the original API key 6bbb3654679105f33cf8c491ed9b04df.

Getting the Source

The source code for both the Mac encryption application and an example of how to decrypt an API key is available on Bitbucket:

Feel free to change out the encryption routines; Blowfish CFB is just one of many options to choose from.


To be honest, I posted this with some trepidation. Security is a topic that draws a lot of attention and frequently, well, developers with an attitude. How often have you heard the phrases:

  • Security through obscurity isn’t!
  • Never write your own cryptographic routine!
  • If you don’t know what you’re doing then don’t!
  • You’re just adding another level of indirection!

and so on. Yet at the same time we’re told to be more security conscious and program with it in mind. REST API keys for accessing services are provided to “everyday developers” and most documentation glosses over how to (more) securely embed them in your mobile app.

In the end, the author of DexGuard for Android sums up storing API keys on the mobile client nicely in a Stack Overflow post:

In the end, it’s an economic trade-off that you have to make: how important are the keys, how much time or software can you afford, how sophisticated are the hackers who are interested in the keys, how much time will they want to spend, how much worth is a delay before the keys are hacked, on what scale will any successful hackers distribute the keys, etc. Small pieces of information like keys are more difficult to protect than entire applications. Intrinsically, nothing on the client-side is unbreakable, but you can certainly raise the bar.” – Eric LaFortune

These are good guidelines to apply when determining what level of security you want to apply to your mobile applications. At some point you have to balance the likelihood of your keys being discovered against what you stand to lose if they do, and upon that decide how much time and effort you’re going to invest in obscuring them in your application.