What are signed HTTP requests?

Signed HTTP requests are simply a normal HTTP request, such as a GET or a POST, that happens to include a signature as part of the request.  A signature is just a string of characters generated in a meaningful way.  This signature can be used by the receiving party (ie. a REST service) to validate the request and ensure that it hasn't been tampered with.

What are signed requests good for?

In a nutshell, validating an HTTP request where restricted access to resources are necessary. For example, a service that allows someone to update their personal information needs to ensure that people can only update their own personal information. Signed requests ensure that the service that receives the data can verify that the sender is who they claim they are (or at least possesses the proper secret key, a de-facto assumption it is the right person).

Obviously, having someone log in and create a session is another method of verifying user identity, however, this is not always efficient or possible, especially with web services. As an example, every Facebook application developer gets a secret key when they create an application on Facebook. When Facebook sends information to someone's application, it will include a signed request that is only valid with that developer's key. By checking the signature, a developer can verify that a request came from Facebook, and isn't someone trying to fool their application. They also don't need to provide a complicated session procedure or have Facebook "log in" to their application; the signed request is enough.

 What aren't signed requests good for?

Signed requests are not a form of encryption. If the data being sent is sensitive in nature, such that you wouldn't want it being sniffed in transit over a network connection, then SSL encrypted communication (HTTPS) is the way to go. For simple REST services however, often you just need to limit requests based on a user or other similar scope, and signed requests are sufficient for that.

A problem that signed requests can solve

Here is an example situation that has a security problem that can be solved by the use of signed requests. Let's say you operate a network of beer brewing web sites, and the brewers are allowed to send updates via a REST service you have provided. Each brewer has their own user id that they get when they sign up to your service. Bob's Brewery (user id 1234) wants to update their email address. The simplest of implementations might allow for a request to be sent like this: /user/update?email=newbob@example.com&userid=1234. However, if this was the only thing the REST service needed to update an email address, it would be trivial to cause serious mayhem. For example, Mr. Evil's Brewery could send in a simple request to change Bob's email address, just by finding Bob's ID somewhere, or even trying random ones:

/user/update?email=evil@example.org&userid=1
/user/update?email=evil@example.org&userid=12
/user/update?email=evil@example.org&userid=1234
etc.

Too easy! Obviously more security is needed, and this is where the concept of secret keys comes in.

Secret Keys

It is a common practice to assign each user a secret key that they use to interact with a web service, often called an API key, generally consisting of randomly generated characters. In this case, you could give Bob a secret key when he signs up with the brewery network, and it would be stored in your database along with the rest of his account information.

Here's where I often see confusion setting in. Many people think it is logical that the secret key be sent as part of the API request, because after all, it uniquely identifies the user and it was given only to that particular person. However, this is a security problem!

The thing about secret keys is that they have to remain secret, just like any password. As soon as a key is leaked, whoever finds it can impersonate that account, just as easily as Mr. Evil's Brewery did in the previous example. If you send a secret key as part of an HTTP request, it is no longer secret. You have just leaked it to whoever cared to be listening! A network sniffer could pick it up, or it could be recorded in proxy logs or web server logs that aren't secured. Perhaps your ISP is doing deep packet inspection, and a rogue employee makes off with the logs. Who knows!

The following is an example of this type of insecurity. Bob is sending his update request along with the secret key:

/user/update?email=newbob@example.com&userid=1234&secret_key=bobs-super-secret-key

Now, pretend that Bob goes to his local Tarborks coffee shop and jumps on the free open wireless connection. Mr. Evil happens to be there, and fires up his network sniffer program and starts watching all the HTTP traffic on the network. He sees Bob's request go out, since HTTP connections are not secure. Now he has Bob's secret key, and can make any kind of request he wants with it. Maybe he could even delete Bob's account with a request to /user/delete?userid=1234&secret_key=bobs-super-secret-key, or send insults to Bob's customers using a mail endpoint!

Sending the secret key as part of a network request is NOT safe. So how do we properly identify the sender if the user id can't be trusted and they can't send us their key? Well, you knew I'd get to it eventually... signed requests!

Signed HTTP Requests

To produce a signed HTTP request, the sender and the receiver must both know the rules on how to generate a signature. It can be any crazy method you care to dream up, but a common one is as follows:

1. The sender organizes the data they want to send in a logical way, such as sorting it alphabetically.
2. The data is run through a hashing algorithm using the secret key to produce a hash. Hashing algorithms produce short strings of characters that vary based on the input data, and the output is sufficiently unique such that varying the input data by even one character produces a completely different hash.
3. The hash is added to the original data and sent.
4. The receiver identifies the user and gets the secret key from their associated account information, and recreates the hash on the received data.
5. If the recreated hash matches the one included in the request, the request is valid.

To return to our Tarborks coffee shop scenario, the request going out might look like this now:

/user/update?email=newbob@example.com&userid=1234&sig=x1zz645

If Mr. Evil sniffs this request, he might be pretty pleased with himself and try to change it to /user/update?email=evil@example.org&userid=1234&sig=x1zz645. This request would be rejected by the server however, since the signature is now invalid! When Mr. Evil changed the request, he needed to change the signature to match. But since he doesn't have Bob's secret key, he can't generate a correct signature, for this request or any other he cares to make up. The only valid request he can possibly make is the exact same one he just sniffed, and that's not very malicious at all, since it's what Bob was trying to do anyway.

Get to the code already

Assuming Bob's user id is 1234, and the secret key you gave him is "bobs-super-secret-key", he might write the following PHP code.

User code

The code above outputs userid=1234&email=newbob%40example.com&sig=efffd9cc30a220f2981b5124e1caa44d91b85aa2d2181f5331f48ca719983c1d.  That's the HTTP query string that Bob would send to the /user/update endpoint to change his email address.  So how does the service verify the request?

Server-side code

There you have it. Signed requests!

Advanced Usage

The signature for a signed request can be sent in different ways. Some services, such as Amazon's S3 REST API, puts the signature in the HTTP headers. This is arguably a bit cleaner than including it in the parameters of an HTTP request, since the signature doesn't get mixed in with the data, and has implications for caching as well (browser caches and proxies). If you want to do it that way, you might have the user set a header as part of their HTTP request:

And the receiving server, instead of looking for the sig parameter in $_REQUEST['sig'] (and having to remove it before running the data through the generator function), would find it in:

Hope you found this useful!

Viewstate and lifecycle events are still very confusing for new ASP.Net programmers. There is no easy way for them to immediately "get it" - they have to spend the time in the trenches, watching their data disappear on post back, or double-bind, and flail around with building and rendering their own web controls. They have to see the DataGrid spew its html diarrhea and spend hours customizing it. They have to have that client ask them "what the hell is all this javascript, and why are the pages so big?" and figure out just what the heck all that gibberish on their pages are.

Moving up the experience chain, Viewstate and lifecycle events represent a significant amount of design and front-end time on a web application even for those that have been doing it for a while. You have to balance your state management with your html optimization, caching, and application maintainability. Often you have to build your own state-tracking structures or extend existing ones. You have to carefully consider whether to roll your own custom controls or buy third party interfaces and rely on their javascript and state programming (or maybe their support team!) And for very simple "read-only" web sites, viewstate just gets in the way.

So in summary, Viewstate and lifecycle are still be a pain in the butt, and still represent hurdles for new programmers. Years ago, it was worth the annoyances and problems, because you could write strongly typed, object oriented portable code without getting lost in folders and folders of scripted sites. So why might it be dying? Well...

The first item on the agenda is Silverlight. ASP.Net represented significant advantages for writing portable code that was closer to classic desktop programming, as it abstracted out much of the xhtml. But now I've seen web application developers swoon over Silverlight. If their reaction is any indication, and the amount of momentum that Microsoft is putting into WPF, many shops are going to give up programming their internal or non-public projects in ASP.Net and move to Silverlight instead. I'm personally still skeptical about Silverlight's market penetration, as there are many gaps it doesn't fill for content publishers. But it's not a minor consideration.

Second is the new ASP.Net MVC framework. MVC is an old model that's rapidly gaining traction in the web world, and with the introduction of the ASP.Net MVC framework, it is hard to see any clear advantage to the older webforms model. From Scott Guthrie's Introductory MVC post a couple months ago:

To help enforce testability, the MVC framework today does not support postback events directly to server controls within your Views. Instead, ASP.NET MVC applications generate hyperlink and AJAX callbacks to Controller actions - and then use Views (and any server controls within them) solely to render output. This helps ensure that your View logic stays minimal and solely focused on rendering, and that you can easily unit test your Controller classes and verify all Application and Data Logic behavior independent of your Views.

And finally, we come to REST, another model that is gaining traction. True REST and webforms are mutually exclusive since REST involves heavy reliance on mapping resource locators to known states of an application. This model has lots of advantages for web services and data-based applications, especially in the realms of testing, while ASP.Net Webform applications are often built around one URL for many states, using Viewstate and Session as your state map, which are of course lost between sessions and server restarts.

So long story short... I think we're seeing the beginning of a new paradigm. In 5 years, will anyone still be developing with true ASP.Net Webforms? It will be interesting to see!