What are signed HTTP requests?

Signed HTTP requests are simply a normal HTTP request, such as a GET or a POST, that happens to include a signature as part of the request.  A signature is just a string of characters generated in a meaningful way.  This signature can be used by the receiving party (ie. a REST service) to validate the request and ensure that it hasn't been tampered with.

What are signed requests good for?

In a nutshell, validating an HTTP request where restricted access to resources are necessary. For example, a service that allows someone to update their personal information needs to ensure that people can only update their own personal information. Signed requests ensure that the service that receives the data can verify that the sender is who they claim they are (or at least possesses the proper secret key, a de-facto assumption it is the right person).

Obviously, having someone log in and create a session is another method of verifying user identity, however, this is not always efficient or possible, especially with web services. As an example, every Facebook application developer gets a secret key when they create an application on Facebook. When Facebook sends information to someone's application, it will include a signed request that is only valid with that developer's key. By checking the signature, a developer can verify that a request came from Facebook, and isn't someone trying to fool their application. They also don't need to provide a complicated session procedure or have Facebook "log in" to their application; the signed request is enough.

 What aren't signed requests good for?

Signed requests are not a form of encryption. If the data being sent is sensitive in nature, such that you wouldn't want it being sniffed in transit over a network connection, then SSL encrypted communication (HTTPS) is the way to go. For simple REST services however, often you just need to limit requests based on a user or other similar scope, and signed requests are sufficient for that.

A problem that signed requests can solve

Here is an example situation that has a security problem that can be solved by the use of signed requests. Let's say you operate a network of beer brewing web sites, and the brewers are allowed to send updates via a REST service you have provided. Each brewer has their own user id that they get when they sign up to your service. Bob's Brewery (user id 1234) wants to update their email address. The simplest of implementations might allow for a request to be sent like this: /user/update?email=newbob@example.com&userid=1234. However, if this was the only thing the REST service needed to update an email address, it would be trivial to cause serious mayhem. For example, Mr. Evil's Brewery could send in a simple request to change Bob's email address, just by finding Bob's ID somewhere, or even trying random ones:

/user/update?email=evil@example.org&userid=1
/user/update?email=evil@example.org&userid=12
/user/update?email=evil@example.org&userid=1234
etc.

Too easy! Obviously more security is needed, and this is where the concept of secret keys comes in.

Secret Keys

It is a common practice to assign each user a secret key that they use to interact with a web service, often called an API key, generally consisting of randomly generated characters. In this case, you could give Bob a secret key when he signs up with the brewery network, and it would be stored in your database along with the rest of his account information.

Here's where I often see confusion setting in. Many people think it is logical that the secret key be sent as part of the API request, because after all, it uniquely identifies the user and it was given only to that particular person. However, this is a security problem!

The thing about secret keys is that they have to remain secret, just like any password. As soon as a key is leaked, whoever finds it can impersonate that account, just as easily as Mr. Evil's Brewery did in the previous example. If you send a secret key as part of an HTTP request, it is no longer secret. You have just leaked it to whoever cared to be listening! A network sniffer could pick it up, or it could be recorded in proxy logs or web server logs that aren't secured. Perhaps your ISP is doing deep packet inspection, and a rogue employee makes off with the logs. Who knows!

The following is an example of this type of insecurity. Bob is sending his update request along with the secret key:

/user/update?email=newbob@example.com&userid=1234&secret_key=bobs-super-secret-key

Now, pretend that Bob goes to his local Tarborks coffee shop and jumps on the free open wireless connection. Mr. Evil happens to be there, and fires up his network sniffer program and starts watching all the HTTP traffic on the network. He sees Bob's request go out, since HTTP connections are not secure. Now he has Bob's secret key, and can make any kind of request he wants with it. Maybe he could even delete Bob's account with a request to /user/delete?userid=1234&secret_key=bobs-super-secret-key, or send insults to Bob's customers using a mail endpoint!

Sending the secret key as part of a network request is NOT safe. So how do we properly identify the sender if the user id can't be trusted and they can't send us their key? Well, you knew I'd get to it eventually... signed requests!

Signed HTTP Requests

To produce a signed HTTP request, the sender and the receiver must both know the rules on how to generate a signature. It can be any crazy method you care to dream up, but a common one is as follows:

1. The sender organizes the data they want to send in a logical way, such as sorting it alphabetically.
2. The data is run through a hashing algorithm using the secret key to produce a hash. Hashing algorithms produce short strings of characters that vary based on the input data, and the output is sufficiently unique such that varying the input data by even one character produces a completely different hash.
3. The hash is added to the original data and sent.
4. The receiver identifies the user and gets the secret key from their associated account information, and recreates the hash on the received data.
5. If the recreated hash matches the one included in the request, the request is valid.

To return to our Tarborks coffee shop scenario, the request going out might look like this now:

/user/update?email=newbob@example.com&userid=1234&sig=x1zz645

If Mr. Evil sniffs this request, he might be pretty pleased with himself and try to change it to /user/update?email=evil@example.org&userid=1234&sig=x1zz645. This request would be rejected by the server however, since the signature is now invalid! When Mr. Evil changed the request, he needed to change the signature to match. But since he doesn't have Bob's secret key, he can't generate a correct signature, for this request or any other he cares to make up. The only valid request he can possibly make is the exact same one he just sniffed, and that's not very malicious at all, since it's what Bob was trying to do anyway.

Get to the code already

Assuming Bob's user id is 1234, and the secret key you gave him is "bobs-super-secret-key", he might write the following PHP code.

User code

The code above outputs userid=1234&email=newbob%40example.com&sig=efffd9cc30a220f2981b5124e1caa44d91b85aa2d2181f5331f48ca719983c1d.  That's the HTTP query string that Bob would send to the /user/update endpoint to change his email address.  So how does the service verify the request?

Server-side code

There you have it. Signed requests!

Advanced Usage

The signature for a signed request can be sent in different ways. Some services, such as Amazon's S3 REST API, puts the signature in the HTTP headers. This is arguably a bit cleaner than including it in the parameters of an HTTP request, since the signature doesn't get mixed in with the data, and has implications for caching as well (browser caches and proxies). If you want to do it that way, you might have the user set a header as part of their HTTP request:

And the receiving server, instead of looking for the sig parameter in $_REQUEST['sig'] (and having to remove it before running the data through the generator function), would find it in:

Hope you found this useful!

Gearman is essentially a distributed processing framework, and seems to have community support for many programming languages.  It consists of two main components: the job server, and a Client and Worker API.  The Client and Worker API can be used in a wide variety of languages, but the job server is only available as a C library or a Perl library.  This makes it a bit tougher to get the server running on Windows, especially when you start running into some of the dependencies that it requires to build.  As well, the Client/Worker API for PHP can only be installed as a PECL extension, or a very-out-of-date PEAR extension called Net_Gearman.

Nonetheless, after yet more research I decided that I would give it a shot by using Cygwin to get the job server running (if you haven't used Cygwin before, be sure to read about it before attempting to install Gearman this way), and PEAR to use the API.  Pre-built PECL extensions aren't available for Windows anymore, and the build process for PHP extensions can be pretty painful, so it makes PEAR look good by comparison even if the code will be out of date.

I had a pretty frustrating time finally getting everything up and running due to various dependency issues, so I went back through the whole process and wrote it out step-by-step.  I used a Windows XP SP3 machine for this, but I also got it working on a Windows 7 machine as well.

Installing the Gearman job server (gearmand) on Windows with Cygwin

Installing Cygwin

1. If you don't have Cygwin already, you can get it from http://www.cygwin.com.  The setup file is located here, and the setup process is pretty straightforward; run it and follow the wizard.  Full installation instructions are available at the Cygwin site.
2. Keep the Cygwin setup.exe file handy after you've installed the default software packages, as you'll need it in the future to add packages, similar to apt-get, yum, and other Linux/UNIX package managers.
3. Cygwin installs with some basic packages, including a Cygwin Bash Shell that goes into your Start Menu.  I prefer the mintty emulator instead, as it has less of a DOS Command Prompt feel and better terminal features.  Feel free to use whatever shell you like of course.  You can get mintty by re-running the setup.exe, and at the package selection screen, type 'mintty' into the Search bar at the top left.  Expand the "Shells" category, and click on the word "Skip" under the "New" column beside the mintty package to select it before continuing the install process.

Installing Cygwin Package Dependencies needed for Gearman

If you're not already in the Cygwin setup, re-run the Cygwin setup.exe and go through to the package selection screen.  The following is a list of dependency packages you will need in order to build the Gearman job server (gearmand).  None of these packages were installed by default with Cygwin:

• gcc
• make
• libuuid1-devel
• libiconv

There's a good installation tutorial here that walks through getting gcc and make installed for people unfamiliar with Cygwin.  Finding the others is pretty straightforward, the Search bar in the package selector works well.

Installing libevent

Gearmand requires an event notification library called libevent that you cannot get as a Cygwin package, which means it has to be installed from source.  You can get the source here.

1. Download and unpack the latest libevent stable release.  At the time of this writing, I used libevent-1.4.14b-stable.
   NOTE: Download and unpack to a directory that does not contain spaces in the name, such as "C:/cygwin/home/Zombat/libevent-1.4.14b-stable".  If you unpack to something with spaces like "C:/Documents and Settings/Zombat/", the build process might not be able to install libevent correctly (libtool has a hard time with spaces)!
2. Open a Cygwin shell and cd to the unpacked libevent directory.
3. Run the following commands:

./configure
make
make install

libevent should now be installed and ready to be used when compiling the Gearman job server.

Installing the Gearman job server, gearmand.exe

1. Download and unpack the C implementation of gearmand from http://gearman.org/index.php?id=download
2. Open a cygwin shell and cd to your unpacked gearmand directory.  Same rules apply as before, make sure you've unpacked in a directory with no spaces in the path!  libtool hates that, and your build may fail.
3. Run the following commands:

./configure
make
make install

The Gearman job server should now be installed and ready to use!  Mine was installed at /usr/local/sbin/gearmand.exe, and running it with a "triple verbose" flag (-vvv) should produce the following:

That's it for the job server.  When you want to start it, simply open a Cygwin shell and run gearmand.exe.  Running it with the -d flag will cause the server to run as a daemon in the background, and running with --help will show you the full option list.

Installing the Gearman Client and Worker API (Net_Gearman)

I chose to install the PEAR Client and Worker API, as it is native PHP and doesn't involve compiling PECL extensions.  The PEAR package is called Net_Gearman, and was originally written by Joe Stump at Digg.com.  It is old and out of date now, although there appears to be a more recent fork at http://github.com/brianlmoon/net_gearman.  I stuck with the older version, as I suspect it will meet my needs, and was readily available as a PEAR package.

This also makes installation relatively painless.  Assuming you've previously set PEAR up, then all you have to do is open a command window (not a Cygwin shell) and run:

pear install Net_Gearman-alpha

The "-alpha" portion is necessary, as Net_Gearman apparently never made it to a stable release version.  That being said, it has functioned well for me so far.  Perhaps someone will pick the project up in the future.

I'll write more about getting started with the Client and Worker API in the next article, so we can actually use Gearman to get some work done.

The build server? Not so much. Immediate barfing with the cryptic message of "MSBUILD : error MSB1025: An internal failure occurred while running MSBuild". CruiseControl looked at it, sighed, and buckled to its knees under the weight of the aborting MSBuild process.

So what went wrong? Looking at the diff of the solution file, all that had happened was these two lines were removed:


Project("{D183A3D8-5FD8-494B-B014-37F57B35E655}") = "DTS", "DTS\DTS.dtproj", "{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}"
EndProject

Reading up on MSB1025 and seeing that people had fixed it by rebuilding their solution files from scratch, I figured it just had to be a badly configured solution file, and that VS2008's handling of dtproj files was sloppy.

I did a search for the project's GUID (the second guid listed there - 947C2F56-A7A8-4357-8E42-6A234C1CCCB6) and sure enough, there were some references to it left in the solution file.

One project had:

ProjectSection(ProjectDependencies) = postProject
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6} = {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}
EndProjectSection

so I removed that entire section, and of course under the GlobalSection that stores the build configuration settings there was a whole whack of:

{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Debug|Any CPU.ActiveCfg = Debug
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Debug|Mixed Platforms.ActiveCfg = Debug
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Debug|x86.ActiveCfg = Debug
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Debug)|Any CPU.ActiveCfg = Debug
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Debug)|Mixed Platforms.ActiveCfg = Debug
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Debug)|x86.ActiveCfg = Debug
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Release)|Any CPU.ActiveCfg = Release
{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Release)|Mixed Platforms.ActiveCfg = Release
...
...
...

so I removed every offending line. Saved it, handed it back to the build server, and all was good.

Obviously not.

This second part of this post gets in depth as to what's going on with the MicrosoftMvcValidation script, but to save you some time, I'll post the solution first.

To manually validate your form with MicrosoftMvcValidation.js:

In MicrosoftMvcValidation.js, change line 20 from:

return Sys.Mvc._ValidationUtil.$0($2.validate('submit'));})); return $2;

to

return Sys.Mvc._ValidationUtil.$0($2.validate('submit'));})); $0.MvcValidationFormContext = $2; return $2;

Alternately, if you're using the debug version of the script, you can achieve the same effect by adding the following to line 196 (the end of the _parseJsonOptions function):

formElement.MvcValidationFormContext = formContext;

UPDATE: As Morten Christiansen points out in the comments, there is a much cleaner way of accessing the FormContext object, without having to introduce a custom tracking property and modifying the script. Instead of a custom property, we can just use
$('form')[0]['__MVC_FormValidation']. Thanks to Morten for pointing this out!

Now we can use the following jQuery/javascript code to manually validate our form:

var $form = ("#MyForm");                            // Select our form with jQuery
var context = $form[0].MvcValidationFormContext;    // Access the new property we created
var errors;
if ($form[0]['__MVC_FormValidation']) {
    errors = $form[0]['__MVC_FormValidation'].validate("submit");        // Validate the form
}
if (!$form[0]['__MVC_FormValidation'] || errors.length == 0) {
    // No errors, do your stuff
}

I assumed (incorrectly) that there must be a client-side API to MicrosoftMvcValidation.js, the javascript responsible for dynamic client side form validation that ships with the ASP.Net MVC framework. Unfortunately, in all my digging, I could not unearth one - and it seems like a huge oversight. The amount of effort to expose an isValid property or a validate() function on the client would be pretty minimal!

Why would you need to do this? Doesn't the validation occur automatically? Well, yes it does - when you submit the form, or click around in it. However, there are lots of cases where you might want to manually trigger the error messages or perform a client-side action based on whether or not the form is valid, without running the submit handler. In my case, I wanted to make sure the form was valid before trying to submit it, as our ajax pipeline puts up the "spinner" graphic when a submit event is triggered, which meant it was flashing briefly and looked dumb.

The most common response I ran across when searching for how to manually validate was - give up and use the jQuery validate plugin. This would be my preferred solution to be honest, but I ran into at least two blockers with jQuery validate:

1. it doesn't seem to work with modal dialogs generated from jQuery UI,
2. it doesn't seem to work with forms that are loaded into the document via an ajax call.

So that was out. Since I already had most of what I needed working with the Microsoft scripts, I went back to them and started exploring the code to see if I could discover any sort of public API. This is what I came up with.

The key thing that I found was that MicrosoftMvcValidation creates a javascript object called a FormContext, which has a validate(eventName) method. This seemed promising, so I tried following the code to see how I could get at this validate method. Well - I couldn't. Not without some modifications. Here's how it works.

When you include MicrosoftMvcValidation, the main piece of script that executes is this:

Sys.Mvc.FormContext._Application_Load = function Sys_Mvc_FormContext$_Application_Load() {
    var allFormOptions = window.mvcClientValidationMetadata;
    if (allFormOptions) {
        while (allFormOptions.length > 0) {
            var thisFormOptions = allFormOptions.pop();
            Sys.Mvc.FormContext._parseJsonOptions(thisFormOptions);
        }
    }
}

Adding <% Html.EnableClientValidation(); %> to your views causes a bunch of JSON data to be output to the page, and appended to the window.mvcClientValidationMetatdata javascript object. When the page is ready, the above function runs and calls FormContext._parseJsonOptions() on all the rules and data that were output to that object.

_parseJsonOptions is what creates the FormContext object that we're interested in. In fact, the return value of _parseJsonOptions is the FormContext... but sadly, Application_Load ignores the return value, and it's just thrown to the bitbucket, never to be seen again (at least by us).

Judging from the underscores on the function names, it's pretty clear that these are meant to be internal operations, so we can (perhaps) excuse them for not storing and exposing the FormContext for us. From the framework's point of view, it doesn't need to be stored anywhere, because it still has it - _parseJsonOptions internally adds new event handlers for the form, and these handler delegates close over the FormContext object. The closures means that internally, the form handlers still have a saved reference to what they need, so the application load function doesn't need to do anything with the return value. The FormContext never surfaces - so if we want to validate the form ourselves, we need to expose dig for it.

All that my code at the top of the post is doing is modifying the _parseJsonOptions function (minified to be named $12), to start adding a new property to the form element in the DOM, which will store the created FormContext object. I named this property MvcValidationFormContext, but you could call it anything you'd like. You can subsequently access this new property to get the FormContext and call validate("submit") on it, which fires the validation as though we were trying to submit the form, and returns a collection of errors we can check.

As Morten pointed out, the reference is, of course, still there. You can access the closed over FormContext by grabbing the DOM element and digging in it's properties array. The updated code reflects this.

I started off to write a function (let's call it a caching function for simplicity) that stores some HTML from the most recently published post. Sounds easy enough. I should be able to just put a function into the functions.php file of the custom template set I'm using. That's seems to be where the "userland" custom functions go.

So I check the function reference first. Hey, wp_get_recent_posts(). Looks promising, so I give it a shot. It goes ahead and gets the most recent post just fine. Things are ok so far.

A problem appears

Now, I want to output the post exactly as it would appear in the blog, and save that output to a file on disk. Surely there's a basic function that will output a post's content? You know... take the post_content field from the database record and format it properly? Suddenly, the skies darken. Evil laughter booms out. Ha ha ha! WordPress mocks the folly of simplistic functional thinking!

The template files use functions like the_content() and the_title().  Just in case you can't tell from the excellent naming scheme, these actually produce echoed output. Checking out the_content(), we see it dutifully calls get_the_content(), then runs a couple of lines of formatting stuff on the results.  So how about using get_the_content() for my caching function? I could run the other few formatting bits manually after that. Should be ok, right? After all, the doc comment for get_the_content() says the following:

/**
* Retrieve the post content.
*

So, I can go ahead assume it simply retrieves the basic post content then? Ha ha. NO. WHY WOULD IT DO THAT? Instead, it takes a bunch of globals that get set who-the-hell-knows-where, runs through a bunch of crap seemingly unrelated to the content of a post, and does a whole lot of textual modifications to some kind of content. Reading through the function is like jabbing red-hot fire pokeys into your eyes. Here's a portion of it: