{"id":3362,"date":"2018-05-13T16:12:35","date_gmt":"2018-05-13T21:12:35","guid":{"rendered":"https:\/\/dev.iachieved.it\/iachievedit\/?p=3362"},"modified":"2018-05-13T16:12:35","modified_gmt":"2018-05-13T21:12:35","slug":"ansible-and-aws-part-4","status":"publish","type":"post","link":"https:\/\/dev.iachieved.it\/iachievedit\/ansible-and-aws-part-4\/","title":{"rendered":"Ansible and AWS &#8211; Part 4"},"content":{"rendered":"<p>So far in our series we&#8217;ve covered some fundamental Ansible basics.  So fundamental, in fact, that we really haven&#8217;t shared anything that hasn&#8217;t been written or covered before.  In this post I hope to change that with an example of creating an <a href=\"https:\/\/aws.amazon.com\/rds\/\">AWS RDS database<\/a> (<a href=\"https:\/\/www.mysql.com\/\">MySQL<\/a>-powered) solely within an Ansible playbook.<\/p>\n<p>If you&#8217;re new to this series, we&#8217;re building up an Ansible playbook one step at a time, starting with <a href=\"https:\/\/dev.iachieved.it\/iachievedit\/ansible-and-aws-part-1\/\">Part 1<\/a>.  Check out the <a href=\"https:\/\/github.com\/iachievedit\/ansible-helloworld\">Github repository<\/a> that accompanies this series to come up to speed.  The final result of this post will be on the <code>part4<\/code> branch.<\/p>\n<h2>Prerequisites<\/h2>\n<p>First, some prerequisites, you&#8217;ll need to:<\/p>\n<ul>\n<li>generate a MySQL password<\/li>\n<li>pick a MySQL administrator username<\/li>\n<li>create an IAM user in S3 for RDS access<\/li>\n<\/ul>\n<p>I love <a href=\"https:\/\/www.howtogeek.com\/howto\/30184\/10-ways-to-generate-a-random-password-from-the-command-line\/\">this page<\/a> for one-liner password creation.  Here&#8217;s one that works nicely on macOS:<\/p>\n<p>[code lang=text]<br \/>\ndate +%s | shasum -a 256 | base64 | head -c 32 ; echo<br \/>\n[\/code]<\/p>\n<p>Whatever you generate will go in your <code>vault<\/code> file, and for this post we&#8217;ll create a new vault file for our <code>staging<\/code> group.<\/p>\n<h3>IAM User<\/h3>\n<p>We&#8217;re going to use an AWS IAM user with programmatic access to create the RDS database for us.  I&#8217;m just going to name mine <code>RDSAdministrator<\/code> and directly attach the policy <code>AmazonRDSFullAccess<\/code>.  Capture your access key and secret access key for use in the vault.<\/p>\n<h2>Reorganizing Our Variables<\/h2>\n<p>I mentioned in <a href=\"https:\/\/dev.iachieved.it\/iachievedit\/ansible-and-aws-part-2\/\">Part 2<\/a> that like code, playbooks will invariably be refactored.  We&#8217;re going to refactor some bits now!<\/p>\n<p>There&#8217;s a special <i>group<\/i> named <code>all<\/code> (we&#8217;ve used it before), and we&#8217;re going to use it with our RDS IAM credentials.  This is worth paying particularly close attention to.<\/p>\n<p>In <code>group_vars<\/code> we&#8217;ll create two directories, <code>all<\/code> and <code>staging<\/code>.  <code>staging.yml<\/code> that we had before will be renamed to <code>staging\/vars.yml<\/code>.  <code>all<\/code> will have a special file named <code>all.yml<\/code> and both <code>all<\/code> and <code>staging<\/code> directories will contain a <code>vault<\/code> file, so in the end we have something like this:<\/p>\n<p>[code lang=text]<br \/>\ngroup_vars\/<br \/>\n          |<br \/>\n          +-all\/<br \/>\n          |    |<br \/>\n          |    |-all.yml<br \/>\n          |    +-vault<br \/>\n          |<br \/>\n          +-staging\/<br \/>\n                   |<br \/>\n                   |-vars.yml<br \/>\n                   +-vault<br \/>\n[\/code]<\/p>\n<p>This is important.  The use of <code>all<\/code> and <code>all.yml<\/code> is another bit of magic.  Don&#8217;t rename <code>all.yml<\/code> to <code>vars.yml<\/code>, it will not work!<\/p>\n<p>Let&#8217;s recap what variables we&#8217;re placing in each file and why:<\/p>\n<table>\n<tr>\n<th>Variable<\/th>\n<th>Location<\/th>\n<th>Rationale<\/th>\n<\/tr>\n<tr>\n<td>`AWS_RDS_ACCESS_KEY`<\/td>\n<td>`group_vars\/all\/all.yml`<\/td>\n<td>IAM access credentials to RDS will apply to all groups (staging, production, etc.)<\/td>\n<\/tr>\n<tr>\n<td>`AWS_RDS_SECRET_KEY`<\/td>\n<td>`group_vars\/all\/all.yml`<\/td>\n<td>IAM access credentials to RDS will apply to all groups (staging, production, etc.)<\/td>\n<\/tr>\n<tr>\n<td>`AWS_RDS_SECURITY_GROUP`<\/td>\n<td>`group_vars\/all\/all.yml`<\/td>\n<td>Our security group allowing access to created MySQL databases will apply to all Ansible groups<\/td>\n<\/tr>\n<tr>\n<td>`OPERATING_ENVIRONMENT`<\/td>\n<td>`group_vars\/staging\/vars.yml`<\/td>\n<td>The `OPERATING_ENVIRONMENT` is set to `staging` for all servers in that group<\/td>\n<\/tr>\n<tr>\n<td>`MYSQL_ADMIN_USERNAME`<\/td>\n<td>`group_vars\/staging\/vars.yml`<\/td>\n<td>Staging servers will utilize the same MySQL credentials<\/td>\n<\/tr>\n<tr>\n<td>`MYSQL_ADMIN_PASSWORD`<\/td>\n<td>`group_vars\/staging\/vars.yml`<\/td>\n<td>Staging servers will utilize the same MySQL credentials<\/td>\n<\/tr>\n<tr>\n<td>`HOSTNAME`<\/td>\n<td>`host_vars\/<HOSTNAME>\/vars.yml`<\/td>\n<td>Each host has its own hostname, and thus this goes into the `host_vars`<\/td>\n<\/tr>\n<tr>\n<td>`AWS_S3_ACCESS_KEY`<\/td>\n<td>`host_vars\/<HOSTNAME>\/vars.yml`<\/td>\n<td>IAM access credentials to read S3 buckets is currently limited to a single host<\/td>\n<\/tr>\n<tr>\n<td>`AWS_S3_SECRET_KE`Y<\/td>\n<td>`host_vars\/<HOSTNAME>\/vars.yml`<\/td>\n<td>IAM access credentials to read S3 buckets is currently limited to a single host<\/td>\n<\/tr>\n<\/table>\n<p>This is our current organization.  Over time we may decide to limit the RDS IAM credentials to a specific host, or move the S3 IAM credentials to the <code>staging<\/code> group.<\/p>\n<h2>RDS<\/h2>\n<p>Amazon&#8217;s <a href=\"https:\/\/aws.amazon.com\/rds\/\">Relational Database Service<\/a> is a powerful tool for instantiating a hosted relational database.  With several different types of database engines to choose from I expect increased usage of RDS, especially by smaller shops that need to reserve their budget for focusing on applications and less on managing infrastructure items such as databases.<\/p>\n<p>If you&#8217;ve never used RDS before, I highly suggest you go through the AWS Management Console and create one <i>by hand<\/i> before using an Ansible playbook to create one.<\/p>\n<h3>MySQL Access Security Group<\/h3>\n<p>We&#8217;re going to cheat here a bit, only because I haven&#8217;t had the opportunity to work with EC2 security groups with Ansible.  But to continue on we&#8217;re going to need a security group that we&#8217;ll apply to our database instance to allow traffic on port 3306 from our EC2 instances.<\/p>\n<p>I&#8217;ve created a group called <code>mysql-sg<\/code> and set the Inbound rule to accept traffic on port 3306 from any IP in the 172.31.0.0\/16 subnet (which covers my account&#8217;s default VPC address range).<\/p>\n<p><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2018\/05\/mysql_security_group.png\" rel=\"attachment wp-att-3378\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2018\/05\/mysql_security_group.png\" alt=\"mysql_security_group\" width=\"669\" height=\"224\" class=\"aligncenter size-full wp-image-3378\" srcset=\"https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2018\/05\/mysql_security_group.png 669w, https:\/\/dev.iachieved.it\/iachievedit\/wp-content\/uploads\/2018\/05\/mysql_security_group-300x100.png 300w\" sizes=\"(max-width: 669px) 100vw, 669px\" \/><\/a><\/p>\n<p><b>Note:<\/b>  If you have abandoned the default VPC in favor of a highly organized set of VPCs for staging, production, etc. environments, you&#8217;ll want to adjust this.<\/p>\n<h3>Pip and Boto<\/h3>\n<p>The Ansible AWS tasks rely on the <a href=\"http:\/\/boto.cloudhackers.com\/en\/latest\/\">boto<\/a> package for interfacing with the AWS APIs (which are quite extensive).  We will want to utilize the latest <code>boto<\/code> package available and install it through <code>pip<\/code> rather than <code>apt-get install<\/code>.  The following tasks accomplish this for us:<\/p>\n<pre class=\"lang:yaml\">\n  # Install Python modules for AWS tasks\n  - name:  Install pip\n    apt:\n      name: python-pip\n      state: present\n  - name:  Install Python modules for AWS tasks\n    pip:\n      name:  boto\n      state: present\n<\/pre>\n<h3>`rds` Module<\/h3>\n<p>Finally, we&#8217;ve come to the all-important <a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/modules\/rds_module.html\">RDS module<\/a>!  Our previous Ansible modules have been straightforward with just a couple of parameters (think <code>apt<\/code>, <code>hostname<\/code>, etc.).  The <code>rds<\/code> module requires a bit more configuration.<\/p>\n<p>Here&#8217;s the task and then we&#8217;ll discuss each of the parameters.<\/p>\n<pre class=\"lang:yaml\">\n  # Create Amazon RDS database\n  - name:  Create AWS RDS database\n    rds:\n      aws_access_key:      \"{{ AWS_RDS_ACCESS_KEY }}\"\n      aws_secret_key:      \"{{ AWS_RDS_SECRET_KEY }}\"\n      command:             create\n      db_engine:           MySQL\n      region:              us-east-2\n      instance_name:       \"{{ OPERATING_ENVIRONMENT }}-mysql\"\n      instance_type:       db.t2.micro\n      username:            \"{{ MYSQL_ADMIN_USERNAME }}\"\n      password:            \"{{ MYSQL_ADMIN_PASSWORD }}\"\n      size:                10\n      publicly_accessible: no\n      vpc_security_groups: \"{{ AWS_RDS_SECURITY_GROUP }}\"\n      wait:                yes\n      wait_timeout:        900\n    register:  rds\n<\/pre>\n<p>This is minimal number of parameters I&#8217;ve found to be required to get a functioning RDS database up-and-running.<\/p>\n<p>The <code>aws_access_key<\/code> and <code>aws_secret_key<\/code> parameters are self-explanatory, and we provide the RDS IAM credentials from our encrypted vault.<\/p>\n<p>The <code>command<\/code> parameter used here is <code>create<\/code>, i.e., we are creating a new RDS database.  <code>db_engine<\/code> is set to <code>MySQL<\/code> since we&#8217;re going to be use MySQL as our RDS database engine.  Let&#8217;s take a look at the rest:<\/p>\n<ul>\n<li>`region` &#8211; like many AWS services, RDS databases are located in specific regions.  `us-east-2` is the region we&#8217;re using in this series.  For a complete list of regions available for RDS, see <a href=\"https:\/\/docs.aws.amazon.com\/AmazonRDS\/latest\/UserGuide\/Concepts.RegionsAndAvailabilityZones.html\">Amazon&#8217;s documentation<\/a>.\n<li>`instance_name` &#8211; our RDS instance needs a name\n<li>`instance_type` &#8211; the RDS instance type.  `db.t2.micro` is a small free-tier eligible type.  For a complete list of instance classes (types), see <a href=\"https:\/\/docs.aws.amazon.com\/AmazonRDS\/latest\/UserGuide\/Concepts.DBInstanceClass.html\">Amazon&#8217;s documentation<\/a>.\n<li>`username` &#8211; our MySQL administrator username\n<li>`password` &#8211; our MySQL administrator password\n<li>`size` &#8211; the size, in GB, of our database.  10GB is the smallest database size one can create (try going lower)\n<\/ul>\n<p>The next two parameters <code>publicly_accessible<\/code> and <code>vpc_security_groups<\/code> dictate who can connect to our RDS database instance.  We <i>don&#8217;t<\/i> want our database to be publicly accessible; only instances within our VPC should be able to communicate with this database, so we set <code>publicly_accessible<\/code> to <code>no<\/code>.  However, without specifying a security group for our instance, no one will be to talk to it.  Hence, we specify that our security group created above is to be applied with the <code>vpc_security_groups<\/code> parameter.<\/p>\n<p>Creating an RDS database takes time.  For our <code>db.t2.micro<\/code> instance, from initial creation to availability took about 10 minutes.  To prevent an error such as <code>Timeout waiting for RDS resource staging-mysql<\/code> we increased our <code>wait_timeout<\/code> from 300 seconds (5 minutes) to 900 seconds (15 minutes).<\/p>\n<h3>register<\/h3>\n<p>You&#8217;ll notice we snuck in a new parameter to our <code>rds<\/code> task called <code>register<\/code>.  <code>register<\/code> is the <a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/user_guide\/playbooks_conditionals.html#register-variables\">Ansible method<\/a> of storing the result of a task in order to use it later in the playbook.  The <code>rds<\/code> task provides a considerable amount of detailed output that is useful to us, the least of which is the <i>endpoint<\/i> name of the RDS database we just created.  That&#8217;s important to know!  We&#8217;ll use this to create a database in our new MySQL instance with the <code>mysql_db<\/code> task.<\/p>\n<p>Like the <code>rds<\/code> task, <code>mysql_db<\/code> has a few dependencies.  For it to function you&#8217;ll want to add the <code>mysql-python<\/code> pip package, which in turn requires <code>libmysqlclient-dev<\/code> (not including <code>libmysqlclient-dev<\/code> will throw a <code>mysql_config not found<\/code> error).  This can be accomplished with:<\/p>\n<pre class=\"lang:yaml\">\n  - name:  Install MySQL dependencies\n    apt:\n      name:  \"{{ item }}\"\n      state: present\n    with_items:\n      - libmysqlclient-dev\n      - mysql-client\n\n  - name:  Install Python modules for MySQL tasks\n    pip:\n      name:  mysql-python\n      state: present\n<\/pre>\n<p>Including <code>mysql-client<\/code> was not, strictly speaking, required, but it&#8217;s a handy package to have installed on a server that accesses a database.<\/p>\n<h3>Creating a MySQL database with Ansible<\/h3>\n<p>Now that we&#8217;ve created our RDS instance, let&#8217;s create a database in that instance with Ansible&#8217;s <code>mysql_db<\/code> task.<\/p>\n<pre class=\"lang:yaml\">\n  - name:  Create MySQL database\n    mysql_db:\n      login_host:      \"{{ rds.instance.endpoint }}\"\n      login_user:      \"{{ MYSQL_ADMIN_USERNAME }}\"\n      login_password:  \"{{ MYSQL_ADMIN_PASSWORD }}\"\n      name:            helloworld\n      state:           present\n<\/pre>\n<p>Remember that <code>register:  rds<\/code> on the <code>rds<\/code> task is what allows us to use <code>rds.instance.endpoint<\/code> (for grins you might put in a <code>debug<\/code> task to see all of the information the <code>rds<\/code> task returns).  We provide our MySQL username and password (this is to login to the RDS instance), and the name of the database we want to create.  <code>present<\/code> is the default for the <code>state<\/code> parameter, but I like to include it anyway to be explicit in the playbook as to what I&#8217;m trying to accomplish.<\/p>\n<p>There are a lot of parameters for the <code>mysql_db<\/code> module, so make sure and check out the <a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/modules\/mysql_db_module.html\">Ansible documentation<\/a>.<\/p>\n<h3>Configuration Information<\/h3>\n<p>Undoubtedly we&#8217;re going to provide our software with access to the database, and knowing it&#8217;s endpoint is important.  Let&#8217;s save this information to our <code>\/etc\/environment<\/code> by adding the following line to <code>templates\/environment.j2<\/code>:<\/p>\n<pre>\nMYSQL_ENDPOINT='{{ rds.instance.endpoint }}'\n<\/pre>\n<p>Note that since we&#8217;re going to be referencing the output of the <code>rds<\/code> task in this template we will want to move writing out <code>\/etc\/environment<\/code> to the end of the playbook.<\/p>\n<p>If desired, we can also create a <code>.my.cnf<\/code> file for the <code>ubuntu<\/code> user that provides automated access to the database.  Now, it&#8217;s up to you to decide whether or not you&#8217;re comfortable with the idea of the password included in <code>.my.cnf<\/code>; if you aren&#8217;t, simply remove it.  To create <code>.my.cnf<\/code> we&#8217;ll introduce one additional new module, <code>blockinfile<\/code>.<\/p>\n<pre class=\"lang:yaml\">\n  # Create .my.cnf\n  - name:  Create .my.cnf\n    blockinfile:\n      create:  yes\n      path:    \/home\/ubuntu\/.my.cnf\n      owner:   ubuntu\n      group:   ubuntu\n      block: |\n        [client]\n        host=\"{{ rds.instance.endpoint }}\"\n        user=\"{{ MYSQL_ADMIN_USERNAME }}\"\n        password=\"{{ MYSQL_ADMIN_PASSWORD }}\"\n<\/pre>\n<p><a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/modules\/blockinfile_module.html\"><code>blockinfile<\/code><\/a> allows you to add blocks of text to either an existing file, or if you specify <code>create: yes<\/code> as shown here, will create the file and add the provided content (given in the <code>block<\/code> parameter).<\/p>\n<p>You can see how running this task performs:<\/p>\n<p>[code lang=text]<br \/>\nubuntu@helloworld:~$ cat .my.cnf<br \/>\n# BEGIN ANSIBLE MANAGED BLOCK<br \/>\n[client]<br \/>\nhost=&quot;staging-mysql.c8tzlhizx66z.us-east-2.rds.amazonaws.com&quot;<br \/>\nuser=&quot;mysqladmin&quot;<br \/>\npassword=&quot;&lt;Our MySQL administrator password&gt;&quot;<br \/>\n# END ANSIBLE MANAGED BLOCK<\/p>\n<p>ubuntu@helloworld:~$ mysql<br \/>\nWelcome to the MySQL monitor.  Commands end with ; or \\g.<br \/>\nYour MySQL connection id is 83<br \/>\nServer version: 5.6.39-log MySQL Community Server (GPL)<br \/>\n[\/code]<\/p>\n<h2>Recap<\/h2>\n<p>We&#8217;ve covered a lot of ground in this post!  Several new modules, refactoring our variables, using Amazon&#8217;s RDS service.  Whew!  Here&#8217;s a quick recap of the new modules that you&#8217;ll want to study:<\/p>\n<ul>\n<li><a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/modules\/rds_module.html\">`rds`<\/a> &#8211; you can create, modify, and delete RDS instances in your AWS account through this powerful task\n<li><a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/user_guide\/playbooks_conditionals.html#register-variables\">`register`<\/a> &#8211; the `register` keyword allows you to capture the output a task and use that output in subsequent tasks in your playbook\n<li><a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/modules\/mysql_db_module.html\">`mysql_db`<\/a> &#8211; once your RDS instance is created, add a new database to it with the `mysql_db` task\n<li><a href=\"http:\/\/docs.ansible.com\/ansible\/latest\/modules\/blockinfile_module.html\">`blockinfile`<\/a> &#8211; another great task to have in your Ansible arsenal, `blockinfile` can add blocks of text to existing files or create new ones\n<\/ul>\n<h2>This Series<\/h2>\n<p>Each post in this series is building upon the last.  If you missed something, here are the previous posts.  We&#8217;ve also put everything on <a href=\"https:\/\/github.com\/iachievedit\/ansible-helloworld\">this Github repository<\/a> with branches that contain all of the changes from one part to the next.<\/p>\n<ul>\n<li><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/ansible-and-aws-part-1\/\">Part 1<\/a>\n<li><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/ansible-and-aws-part-2\/\">Part 2<\/a>\n<li><a href=\"https:\/\/dev.iachieved.it\/iachievedit\/ansible-and-aws-part-3\/\">Part 3<\/a>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>So far in our series we&#8217;ve covered some fundamental Ansible basics. So fundamental, in fact, that we really haven&#8217;t shared anything that hasn&#8217;t been written or covered before. In this post I hope to change that with an example of creating an AWS RDS database (MySQL-powered) solely within an Ansible playbook. If you&#8217;re new to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":3318,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[68,21],"tags":[],"class_list":["post-3362","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ansible","category-devops"],"_links":{"self":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/3362"}],"collection":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/comments?post=3362"}],"version-history":[{"count":23,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/3362\/revisions"}],"predecessor-version":[{"id":3386,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/posts\/3362\/revisions\/3386"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/media\/3318"}],"wp:attachment":[{"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/media?parent=3362"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/categories?post=3362"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.iachieved.it\/iachievedit\/wp-json\/wp\/v2\/tags?post=3362"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}