Using unix shell scripts for testing RESTful web apis

Death by a thousand cuts

What I like about in unix/linux operating systems is that much of the operating system consists of small programs, which can be piped together to make something more complex. Similar approach to test tools can also be quite beneficial as you can quickly create even complex tools just by knitting various commandline utilities together.

Here are my notes about using bash shell scripts and various small utilities for testing restful apis. I am using and demonstrating these tools under windows with cygwin, but similary you can use them on unix, linux or mac.

1. HTTP Get and Posts with curl:

Curl is a nice nifty little command-line tool for interacting various protocols (inc, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP).

When testing web-based software, I prefer to use curl for automating little things like filling and submitting a form or automatically posting and getting xml responses from SOAP-apis.

Below are commands for getting and posting of xml files to REST service.

HTTP GET:
curl http://www.my-rest-service.com/request -H Accept: application/xml -o response.xml

What happens here, curl sends HTTP GET to URL: http://www.my-rest-service.com/request
with "HTTP HEADER: Accept: Application/xml" (meaning, send me xml back please). The last "-o response.xml" means that it will write the returning response (hopefully xml formated to file response.xml). If you wish to see the HTTP codes (eg. HTTP 200 OK) which the server returns append "-i" to the curls parameters.

HTTP POST:
Similary you can also "POST" (eg. send) xml files to the server. All you need to do is to define the curl to use post-method with parameter "-X post". With "-d @MyRequest.xml" you can define that read file which has filename MyRequest.xml and add that as payload for you HTTP post request.

Example for "posting" xml-file called MyRequest.xml to my-rest-service.com/request
curl -i -X post http://www.my-rest-service.com/request -d @MyRequest.xml -H "Content-Type: application/xml"

2. Formating and parsing xml input with xmlindent and xmlstarlet.

Now when you receive xml response from the server, it might be a bit messy or the whole xml might be wrapped in to a single line. Thankfully there are utilities like xmlindent which can straighten it for you.

Example, xml often comes from server in format like this:
<?xml version="1.0" ?> <note><to>readers><to><from>samuli</from> <body>Hello!</body></note>

With xmlindent, you can format it a little bit:

$ xmlindent.exe -f note.xml
<?xml version="1.0" ?>
<note>
<to>readers
</to>
<from>samuli
</from>
<body>Hello!
</body>
</note>

Or you can do the same thing with xmlstarlet and using the option "fo" (format).

$ xmlstarlet.exe fo note.xml

<?xml version="1.0"?>

<note>

<to> readers </to>

<from>samuli</from>

<body>Hello!</body>

</note>

3. Grepping values from XML-files with xmlstarlet

"XMLStarlet is a set of command line utilities (tools) which can be used to transform, query, validate, and edit XML documents and files using simple set of shell commands in similar way it is done for plain text files using UNIX grep, sed, awk, diff, patch, join, etc commands."

In the example below I am using it for parsing values from xml file and then using these values to create a new xml file which is then posted to webservice.

Example. lets pretend we have xml response (named response.xml) we received from webservice:

<?xml version="1.0"?>

</Response>

For this we wish to grep the query id -number attribute and the amount value from the amount-element,
so we will write a bash script to grep the values and then print them out or send them again via http-get

#!/bin/bash

Var1=`xmlstarlet sel -t -v //amount`
Var2=`xmlstarlet sel -t -m "//Response/@id" -v .`
echo $Var1
curl http://example.com/statusquery?q=$Var2

The first line asks the xmlstarlet to go through all fields in the xml-file and print out values of the amount-elements. In this case I only have one amount-element but as I was too lazy to write full path to the amount-element I just ask it to select "all" amount-elements and print out the value to variable Var1.
The second line selects the value of the id-attribute inside the query element and then stores it variable Var2.

We can now use these variables as we wish. In my example, they printout the var1 to console and use var2 as parameter for http-get request.

4. Modifying xml files with xmlstarlet.

xmlstarlet can also modify existing xml-files and create a new xml-files from them like this:

xmlstarlet ed -u "//amount" -v 42 -u "//description" -v "test purchase" request.xml > newRequest.xml

This would read the request.xml file, replace all values of amount-tag with value 42, and replace all values of description with value "test purchase" and print out modified request.xml in to a new file called newRequest.xml and leave the old one as it was. When using inside shell-script you could write in in format like this: xmlstarlet ed -u "//amount" -v $1 -u "//description" -v $2 request.xml > newRequest.xml

You could then call the script with parameters like: 42 "test purchase" which would then add those to the newRequest.xml file. This is particulary useful when you want to use a value from previous requests and then send them to another service. You could also hook this up to text/number generator and create multiple testdata files.

5. Parsing json requests with resty and jsawk.

For parsing JSON request there is a nice little tool called resty.

You can grab it from following github url.
https://github.com/micha/resty
Below a short description on how to use resty for get, post, put, delete json requests.

Resty is provided as shell-scripts and aliases. To activate/setup resty you need to source the resty-file:

$ . resty

To define target host for the resty client, you need to do this:
resty http://example.com:8080/data

Then we can proceed to execute get, put, delete, post request to json services like this

$ GET /blogs.json
$ PUT /blogs/2.json '{"id" : 2, "title" : "updated post", "body" : "This is the new."}'
$ DELETE /blogs/2
$ POST -v -u user:test /blogs.json '{"title" : "new post", "body" : "This is the new new."}'

For modifying json requests there is a tool called jsawk (from https://github.com/micha/jsawk/)
jsawk uses javascript for its command language. In the example below we http-get a json-formatted response and then change the favoriteColor to blue and then http-Put it back. (copied from jsawk readme file).

GET /people/47 | jsawk 'this.favoriteColor = "blue"' | PUT /people/47

Because jsawk uses javascript you can also do tricks like these:

GET /people/47 | jsawk 'this.age++'

Thanks for reading. If you have any questions or improvement suggestions. Please add comment or send me a tweet @samuliel. Happy testing :-)