Building REST APIs with Zewo

2/20/2016 Update: Zewo has undergone a significant refactoring since this tutorial was originally written. The information below is no longer accurate and I will be working to provide a new tutorial as soon as possible.

Zewo

Zewo is new collection of modules designed to build web applications in Swift. Zewo provides:

  • an HTTP server (Epoch)
  • an HTTP router
  • HTTP request and response entity classes
  • an interface to a Mustache template responder (Sideburns)
  • PostgreSQL and MySQL database adapters

and much more. Although relatively new to the scene, Zewo already has enough functionality to build a complete REST-based web service application. In the past I would have first considered Ruby and Sinatra, but now with the arrival of Swift on Linux, Zewo provides another great option.

A Quick API Design

We’re going to build a REST API similar to that of AT&T’s M2X platform. M2X is self-described as providing time-series data storage, device management, message brokering, event triggering, alarming, and data visualization for your industrial Internet of Things (IOT) products and services. That’s a mouthful. Our API will be less ambitious and implement the time-series data storage piece, but that is enough to showcase some of the capabilities of Zewo.

In the IOT world we can think about devices and streams of data that are associated with the device. For example, a device could be a single-board computer like a BeagleBone or Raspberry Pi with several sensors attached to it. The sensor, in turn, would be providing a stream of datapoints, say temperature readings. These concepts can be modeled nicely in a relational database fronted by a REST API. And, if relational databases aren’t your thing, folks are working on NoSQL interfaces such as MongoDB.

Devices, Streams, and Datapoints

Our model will consist of Device, Stream, and Datapoint.

A Device has the following attributes, or properties:

  • name
  • serial
  • location
  • streams

A Stream has the following properties:

  • name
  • datapoints

And finally, a Datapoint (sometimes referred to as a reading) will be:

  • timestamp
  • value

Now, believe me, this is the not the end-all be-all data model for an IoT application, but it provides enough structure to make our REST API interesting.

The operations we will support are:

  • creating a new device
  • posting a value to a stream
  • retrieving the values of a stream

Of course, this is limited functionality, so it is left up to the reader to add things like:

  • updating the location of the device
  • adding a new stream to a device

So, let’s dive in!

First Things First

The example code provided at the end of the tutorial has only been tested on Linux, and everything done here is on an Ubuntu 14.04 system. To get up and running you will need to:

There are detailed instructions on these steps further below.

Once the prerequisites are installed we’re going to use the Swift Package Manager to bootstrap in our Zewo dependencies. In a directory named iotapp create your Package.swift file:

and then run swift build. You will see the Swift Package Manager download and compile all of the required packages:

Cloning Packages/Epoch
Using version 0.1.0 of package Epoch
Cloning Packages/HTTPParser
Using version 0.1.0 of package HTTPParser
Cloning Packages/CHTTPParser
Using version 0.1.0 of package CHTTPParser
Cloning Packages/HTTP
Using version 0.1.2 of package HTTP
Cloning Packages/Core
Using version 0.1.1 of package Core
Cloning Packages/CURIParser
Using version 0.1.0 of package CURIParser
Cloning Packages/Venice
Using version 0.1.0 of package Venice
Cloning Packages/CLibvenice
Using version 0.1.0 of package CLibvenice
Cloning Packages/Middleware
Using version 0.1.0 of package Middleware
Cloning Packages/Router
Using version 0.1.0 of package Router
Cloning Packages/PostgreSQL
Using version 0.1.0 of package PostgreSQL
Cloning Packages/SQL
Using version 0.1.0 of package SQL
Cloning Packages/CLibpq
Using version 0.1.0 of package CLibpq
Compiling Swift Module 'Core' (15 sources)
Linking Library:  .build/debug/Core.a
Compiling Swift Module 'HTTP' (10 sources)
Linking Library:  .build/debug/HTTP.a
Compiling Swift Module 'HTTPParser' (4 sources)
Linking Library:  .build/debug/HTTPParser.a
Compiling Swift Module 'Venice' (21 sources)
Linking Library:  .build/debug/Venice.a
Compiling Swift Module 'Epoch' (9 sources)
Linking Library:  .build/debug/Epoch.a
Compiling Swift Module 'Router' (1 sources)
Linking Library:  .build/debug/Router.a
Compiling Swift Module 'Middleware' (11 sources)
Linking Library:  .build/debug/Middleware.a
Compiling Swift Module 'SQL' (6 sources)
Linking Library:  .build/debug/SQL.a
Compiling Swift Module 'PostgreSQL' (5 sources)
Linking Library:  .build/debug/PostgreSQL.a

Note: If the above bombed out it is likely due to missing dependencies. You don’t need to perform the bootstrap steps just yet, but if you’re chomping at the bit run the following:

sudo apt-get install libpq-dev
echo "deb [trusted=yes] http://apt.zewo.io/deb ./" | sudo tee --append /etc/apt/sources.list
sudo apt-get update
sudo apt-get install uri-parser http-parser libvenice 

Anatomy of a Zewo REST Application

A Zewo REST application looks similar to other HTTP application stacks such as Rails, Sinatra, etc. At the top of the stack is the familiar HTTP Server. Zewo provides us with Epoch, described as a Venice based HTTP server for Swift 2.2 on Linux. A straightforward main.swift initializes our server:

Next down the stack is our routing functionality, which is an instance of Zewo Router. The router is initialized in Router.swift which looks like this:

The >>> admittedly looks like a little black magic. It is an overloaded operator defined in Zewo Middleware. In this example it provides for logging our API actions.

The route.router("/api", APIv1 >>> log) statement will anchor our web service at /api. Let’s look now at APIv1.swift:

APIv1 is where the real action is happening. Our URL anchor point is now extended to /api/v1 and we’ve specified a number of specific routes from there:

  • GET /api/v1/version
  • POST /api/v1/devices
  • GET /api/v1/devices/:serial
  • POST /api/v1/devices/:serial/streams/:name/value
  • GET /api/v1/devices/:serial/streams/:name

Again, while not the richest API, it serves as starting point.

If you’ve ever worked with Sinatra you will recognize the named parameters in the route patterns. These parameters, such as :serial and :name will be provided to us in our controllers, which we’ll see next.

Controllers

Zewo does all the heavy lifting of receiving an HTTP request and then routing the request to the appropriate controller. It is here though where we have to pick up the request and do something with it.

What is a controller again exactly? According to the Rails ActionController documentation a controller “is responsible for making sense of the request and producing the appropriate output”. This is what our DeviceController does for us:

Recall that it was the APIv1 router that calls deviceController.create when a POST /devices is received. The create function takes a Request object as its only argument, and then returns a Response. In other words, the function is making sense of the request and producing the appropriate output. Making sense of the request is broken down into distinct steps:

  • obtaining the JSON body of the request and extracting required fields name and serial
  • obtaining the location field if it is available, otherwise the location defaults to (0.0, 0.0)
  • extracting the required streams field from the JSON
  • inserting a new Device into the database
  • inserting a new Stream for each stream specified

The response to this request can be one of two types:

  • .BadRequest (HTTP Status 400)
  • .OK (HTTP Status 200)

In both cases we return a JSON body with additional information.

An example of a valid JSON request body for POST /devices looks like this:

If the request was successful we expect a response to look like:

The server returned an additional field id in the JSON response. Some APIs are designed such that the client use the id field for future references to the created object. Our API, on the other hand, will use the serial number to identify a given device.

Records

Observe that the controller does not interact directly with the database. That is the job of our record class, which is modeled after the concept of ActiveRecord in Rails. The DeviceRecord class knows about Postgres, what table is used to manage devices (in fact, it creates that table), and how to insert and find devices. In Rails ActiveRecord is smart enough to read the table schema and create methods such as find on the fly. Our DeviceRecord does not go that far; we aren’t building Rails here.

This is the content of our DeviceRecord.swift file:

At a fundamental level there really isn’t a whole lot going on in this file, with of course the exception that we are issuing Postgres SQL commands to the underlying database adapter and reading the result.

Before leaving the DeviceRecord, notice that insert returns Device. This is not a good idea, and we should change the implementation to return Device? in the event there is an issue with the SQL INSERT. The example code in Github will return Device? so we can catch these errors.

Combining all the components together, top-to-bottom, we get a view of what the “stack” looks like:

Basic Web Service Stack
Basic Web Service Stack

A Working Example

Now that we have a basic idea of how Zewo is used to build a REST web service, let’s run a working example.

First, install of the dependencies you’ll need, which includes libpq-dev for the Postgres headers, as well as some additional packages Zewo requires.

sudo apt-get install libpq-dev
echo "deb [trusted=yes] http://apt.zewo.io/deb ./" | sudo tee --append /etc/apt/sources.list
sudo apt-get update
sudo apt-get install uri-parser http-parser libvenice 

Now, get the code from Github and switch to the basic branch.

git clone https://github.com/iachievedit/iotapp
cd iotapp
git checkout basic

Build the application with swift build and run it:

swift build
.build/debug/iotapp

If Postgres isn’t running or cannot otherwise be contacted, you might see this error:

fatal error: 'try!' expression unexpectedly raised an error: PostgreSQL.Connection.Error.ConnectFailed("could not connect to server: Connection refused\n\tIs the server running on host \"localhost\" (::1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Connection refused\n\tIs the server running on host \"localhost\" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\n"): file swift/stdlib/public/core/ErrorType.swift, line 53
Illegal instruction (core dumped)

In this case, install Postgres and create your user and database:

$ sudo apt-get install postgresql
$ sudo su - postgres
$ psql template1
psql (9.4.5)
Type "help" for help.

template1=# CREATE USER iotuser WITH PASSWORD 'iotuser';
CREATE ROLE
template1=# CREATE DATABASE iot_staging;
CREATE DATABASE
template1=# GRANT ALL PRIVILEGES ON DATABASE "iot_staging" to iotuser;
GRANT
template1=# \q
$ exit

Once the application is up and running, we can use cURL or Postman to create a new device. Here we’ll use cURL:

$ curl -X POST -H "Content-Type: application/json" -d '{"name":"BeagleBone", "serial":"N9TT-8GXF-B7GF-RXNC", "location":{"latitude":-34.8836, "longitude":-56.1819}, "streams":[{"name":"cpu-occupancy"}, {"name":"free-memory"}]}' 'http://localhost:8080/api/v1/devices'

Our Zewo application processes the request, logging to STDOUT:

12/23/15, 8:11 PM /home/iotapp/iotapp/Sources/DeviceRecord.swift:23 insert(_:serial:location:): INSERT into devices (name,serial,location) VALUES('BeagleBone','N9TT-8GXF-B7GF-RXNC',POINT(-34.8836,-56.1819)) RETURNING id
POST /api/v1/devices HTTP/1.1
Content-Type: application/json
Accept: */*
User-Agent: curl/7.43.0
Host: localhost:8080
Content-Length: 200

{"name":"BeagleBone",
 "serial":"N9TT-8GXF-B7GF-RXNC",
 "location":{
  "latitude":-34.8836,
  "longitude":-56.1819
  },
  "streams":[
      {"name":"cpu-occupancy"},
      {"name":"free-memory"}
  ]
}
-
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
Content-Length: 128

{"id":"1","name":"BeagleBone","location":{"longitude":-56.1819,"latitude":-34.8836},"streams":[],"serial":"N9TT-8GXF-B7GF-RXNC"}
----------------------------------------

The response to our command is what we expect, a JSON string with our device!

{"id":"1","name":"BeagleBone","location":{"longitude":-56.1819,"latitude":-34.8836},"streams":[],"serial":"N9TT-8GXF-B7GF-RXNC"}

Try issuing the command again with the same serial number. You should see {"message":"error creating device"} thanks to the logic we added to return a .BadRequest if we couldn’t insert the device.

Creating Streams and Posting Datapoints

We now need to add the functionality that creates streams and datapoints. Unlike with Device we do not need a StreamController. Streams are created directly through a StreamRecord instance in the DeviceController.

Our StreamRecord code looks like this:

Note that our streams schema definition contains device_id, which is a reference to the device that “owns” the stream. Again, unlike with Rails, we are not specifying has_a or belongs_to relationships explicitly; we manage this through code.

Posting Datapoints

A stream is a collection of data points, or values. For example, let’s say our BeagleBone device had a temperature sensor attached to it. We would want to POST a temperature reading to the temperature stream, like this:

curl -X POST -H "Content-Type: application/json" -d '{"value":"77"}' 'http://localhost:8080/api/v1/devices/NFTT-8GXF-B7GF-RXNC/streams/temperature/value'

Our API responds with:

{
  "value": "77",
  "inserted_at": "2015-12-23 21:13:42.763442",
  "id": "1"
}

Some time later the temperature increases to 78 degrees, so another value is posted. Later on, it increases to 80 degrees, and so on. Then, we can retrieve our datapoints via a GET request:

curl -X GET -H "Content-Type: application/json" 'http://localhost:8080/api/v1/devices/NFTT-8GXF-B7GF-RXNC/streams/temperature'

The following data is returned:

{
  "datapoints": [
    {
      "value": "77",
      "inserted_at": "2015-12-23 21:13:42.763442",
      "id": "1"
    },
    {
      "value": "78",
      "inserted_at": "2015-12-23 21:15:30.536468",
      "id": "2"
    },
    {
      "value": "80",
      "inserted_at": "2015-12-23 21:15:54.651966",
      "id": "3"
    }
  ]
}

Here is our implementation of DatapointController. Note how the methods DeviceRecord.findBySerial(serial:) and StreamRecord.findByName(name:,deviceId:) are used to associate the datapoint with the correct device and stream.

Our DatapointRecord implementation looks like:

An exercise for the reader: The DatapointRecord.findByStream(streamId:count:) function is called with a hardcoded value of 10. Extend the API to allow for the number of datapoints to be specified either by a JSON structure in the POST body, or through a URI query parameter. Consider also adding inserted_before and inserted_after filters to control which values to return.

Complete Application

The completed IoT application is on Github on the master branch. With it you should be able to create devices (make sure and specify the streams up front or add an implementation of modifying a device to add a new stream!), and post datapoints to the device streams.

There are some additional files and helper routines in the repository:

  • the +JSON files provide toJSON extension functions to our objects
  • Utils.swift adds a logmsg routine

As I mentioned, there are a lot of capabilities that can be added to this API. Streams can be given datatypes and units, operations which find min and max datapoints in a stream could be added, and so on. For inspiration check out the M2X API.

What’s Next?

After spending time with the Zewo framework over the past week I’m convinced it will become one of the go to building blocks for developing web applications in Swift. This article has focused on building the server side of the REST API, and I’m looking forward to realizing the dream of developing in one language when we look at developing a client side set of classes in Swift that can be used to interact with the service. Stay tuned.

2 comments

  1. Nate HikerReply

    Can you please update this blog since Zewo has changed a couple of times since you originally published it?

    • Joe Post authorReply

      Nate, thanks for the comment. Yes, Zewo has undergone quite a transformation; I need to find the time to write up a new tutorial. In the meantime I can add a disclaimer that the information is out of date.

Leave a Reply

Your email address will not be published. Required fields are marked *