Auditing Shared Account Usage

Occasionally you find yourself in a situation where utilizing a shared account cannot be avoided. One such scenario is managing the deployment of NodeJS applications with Shipit and PM2. Here’s how the scenario typically works:

Alice, Bob, and Carol are three developers working on NodeJS applications that need to be deployed to their staging server. They’ve decided on the use of PM2 as their process manager, and ShipIt as their deployment tool. Their shipitfile.js file contains a block for the staging server, and it looks something like:

staging: {
      servers: [
        {
          host: 'apps.staging.iachieved.it',
          user: 'deployer',
        },
      ],
      environment:  'staging',
      branch: 'develop',
    },

As we can see the deployer user will be used to deploy our application, and by extension, will be the user that pm2 runs the application under. Alice, Bob, and Carol all have their SSH keys put in /home/deployer/.ssh/authorized_keys so they can deploy. Makes sense.

Unfortunately what this also means is Alice, Bob, or Carol can ssh to the staging server as the deployer user. Even though deployer is an unprivileged user, we really don’t want that. Moreover, by default, we can’t trace who deployed, or if someone is misusing the deployer user. Let’s take a look at how we can address this.

Creating a Deployment Group

The first thing we want to do is to create a security group for those that are authorized to perform deployments. I’ll be using an Active Directory security group in this example, but a standard Unix group would work as well. We can use getent to see the members of the group. getent will come in handy to help determine whether someone attempting to deploy is authorized.

# getent group "application deployment@iachieved.it"
application deployment@iachieved.it:*:1068601118:alice@iachieved.it,bob@iachieved.it

SSH authorized_keys command

Until I started researching this problem of auditing and restricting shared account usage I was unaware of the command option in the SSH authorized_keys file. One learns something new every day. What the command option provides for is executing a command immediately upon SSHing via a given key. Consider that we put the following entry in the deployer user ~/.ssh/authorized_keys file:

ssh-rsa AAAA...sCBR alice

and this is Alice’s public key. We would expect that Alice would be able to ssh deployer@apps.iachieved.it and get a shell. But what if we wanted to intercept this SSH and run a script instead? Let’s try it out:

deployer@apps.iachieved.it:~/.ssh/authorized_keys:

command="/usr/bin/logger -p auth.INFO Not permitted" ssh-rsa AAAA...sCBR alice

When Alice tries to ssh as the deployer user, we get an entry in auth.log:

Jul  5 22:30:58 apps deployer: Not permitted

and Alice sees Connection to apps closed..

Well that’s no good! We do want Alice to be able to use the deployer account to deploy code.

A Wrapper Script

First, we want Alice to be able to deploy code with the deployer user, but we also want to:

• know that it was Alice
• ensure Alice is an authorized deployer
• not allow Alice to get a shell

Let’s look at how we can create a script to execute each SSH invocation that will meet all of these criteria.

Step 1, let’s log and execute whatever Alice was attempting to do.

/usr/local/bin/deploy.sh:

SSH_ORIGINAL_COMMAND will be set automatically by sshd, but we need to provide SSH_REMOTE_USER, so in the authorized_keys file:

command="export SSH_REMOTE_USER=alice@iachieved.it;/usr/local/bin/deploy.sh" ssh-rsa AAAA...sCBR alice

Note that we explicitly set SSH_REMOTE_USER to alice@iachieved.it. The takeaway here is that it associates any attempt by Alice to use the deployer account to her userid. We then execute deploy.sh which logs the invocation. If Alice tries to ssh and get a shell with ssh deployer@apps the connection will still be closed, as SSH_ORIGINAL_COMMAND is null. But, let’s say she runs ssh deployer@apps ls /:

alice@iachieved.it@apps ~> ssh deployer@apps ls /
bin
boot
dev
etc

In /var/log/auth.log we see:

Jul  6 13:43:25 apps sshd[18554]: Accepted publickey for deployer from ::1 port 48832 ssh2: RSA SHA256:thZna7v6go5EzcZABkieCmaZzp+6WSlYx37a3uPOMSs
Jul  6 13:43:25 apps sshd[18554]: pam_unix(sshd:session): session opened for user deployer by (uid=0)
Jul  6 13:43:25 apps systemd-logind[945]: New session 54 of user deployer.
Jul  6 13:43:25 apps systemd: pam_unix(systemd-user:session): session opened for user deployer by (uid=0)
Jul  6 13:43:26 apps deployer: alice@iachieved.it executed ls /

What is important here is that we can trace what Alice is executing.

Managing a Deployment Security Group

Left as is this technique is much preferred to a free-for-all with the deployer user, but more can be done using security groups to have finer control of who can use the account at any given time. Let’s add an additional check in the /usr/local/bin/deploy.sh script with the uig function introduced in the last post.

The authorized_keys file gets updated, and let’s add Bob and Carol’s keys for our additional positive (Bob) and negative (Carol) test:

Since Bob is a member of the application deployment@iachieved.it group, he can proceed:

Jul  6 20:09:26 apps sshd[21886]: Accepted publickey for deployer from ::1 port 49148 ssh2: RSA SHA256:gs3j1xHvwJcSMBXxaqag6Pb7A595HVXIz2fMoCX2J/I
Jul  6 20:09:26 apps sshd[21886]: pam_unix(sshd:session): session opened for user deployer by (uid=0)
Jul  6 20:09:26 apps systemd-logind[945]: New session 79 of user deployer.
Jul  6 20:09:26 apps systemd: pam_unix(systemd-user:session): session opened for user deployer by (uid=0)
Jul  6 20:09:27 apps deployer: bob@iachieved.it is in application deployment@iachieved.it and executed ls /

Now, Carol’s turn to try ssh deployer@apps ls /:

Jul  6 20:15:37 apps deployer: carol@iachieved.it is not in application deployment@iachieved.it and is not authorized to execute ls /

Poor Carol.

Closing Thoughts

For some teams the idea of having to manage a deployment security group and bespoke authorized_keys file may be overkill. If you’re in an environment with enhanced audit controls and accountability the ability to implement safeguards and audits to code deployments may be a welcome addition.

A Script for Testing Membership in a Unix Group

Sometimes you just need a boolean test for a given question. In this post we’ll look at answering the question, “Is this user in a given group?” Seems simple enough.

It’s easy to see what groups a user is a member of in a shell:

% id
alice@iachieved.it@darthvader:~$ id
uid=1068601116(alice@iachieved.it) gid=1068601115(iachievedit@iachieved.it) groups=1068601115(iachievedit@iachieved.it),1068600513(domain users@iachieved.it),1068601109(linux ssh@iachieved.it),1068601118(application deployment@iachieved.it)

Note that Alice is an Active Directory domain user. We want to test whether or not she is a member of the application deployment@iachieved.it group. We can see this with our eyes in the terminal, but a little scripting is in order. We’ll skip error checking for this first example.

uig.sh:

Let’s take it out for a spin.

alice@iachieved.it@darthvader:~$ ./uig.sh `whoami` "linux administrators@iachieved.it"
User alice@iachieved.it IS NOT in group linux administrators@iachieved.it

Now let’s test whether or not Alice is in the application deployment@iachieved.it group:

alice@iachieved.it@darthvader:~$ ./uig.sh `whoami` "application deployment@iachieved.it"
User alice@iachieved.it IS in group application deployment@iachieved.it

Terrific. This will come in handy in the next blog post.

Let’s clean this up into a function that can be sourced in a script or shell:

alice@iachieved.it@darthvader:~$ uig `whoami` "linux ssh@iachieved.it"
alice@iachieved.it@darthvader:~$ echo $?
0

Or, the invocation we’re most likely to use: