PHP vs .Net » PHP

HTTP Signed Requests with PHP

blake — Sat, 24 Dec 2011 18:19:27 +0000

I thought I'd write a quick primer on a basic implementation of HTTP request signing with PHP. I see a lot of posts dealing with the topic, especially by people writing homebrew REST services.

What are signed HTTP requests?

Signed HTTP requests are simply a normal HTTP request, such as a GET or a POST, that happens to include a signature as part of the request. A signature is just a string of characters generated in a meaningful way. This signature can be used by the receiving party (ie. a REST service) to validate the request and ensure that it hasn't been tampered with.

What are signed requests good for?

In a nutshell, validating an HTTP request where restricted access to resources are necessary. For example, a service that allows someone to update their personal information needs to ensure that people can only update their own personal information. Signed requests ensure that the service that receives the data can verify that the sender is who they claim they are (or at least possesses the proper secret key, a de-facto assumption it is the right person).

Obviously, having someone log in and create a session is another method of verifying user identity, however, this is not always efficient or possible, especially with web services. As an example, every Facebook application developer gets a secret key when they create an application on Facebook. When Facebook sends information to someone's application, it will include a signed request that is only valid with that developer's key. By checking the signature, a developer can verify that a request came from Facebook, and isn't someone trying to fool their application. They also don't need to provide a complicated session procedure or have Facebook "log in" to their application; the signed request is enough.

What aren't signed requests good for?

Signed requests are not a form of encryption. If the data being sent is sensitive in nature, such that you wouldn't want it being sniffed in transit over a network connection, then SSL encrypted communication (HTTPS) is the way to go. For simple REST services however, often you just need to limit requests based on a user or other similar scope, and signed requests are sufficient for that.

A problem that signed requests can solve

Here is an example situation that has a security problem that can be solved by the use of signed requests. Let's say you operate a network of beer brewing web sites, and the brewers are allowed to send updates via a REST service you have provided. Each brewer has their own user id that they get when they sign up to your service. Bob's Brewery (user id 1234) wants to update their email address. The simplest of implementations might allow for a request to be sent like this: /user/update?email=newbob@example.com&userid=1234. However, if this was the only thing the REST service needed to update an email address, it would be trivial to cause serious mayhem. For example, Mr. Evil's Brewery could send in a simple request to change Bob's email address, just by finding Bob's ID somewhere, or even trying random ones:

/user/update?email=evil@example.org&userid=1 /user/update?email=evil@example.org&userid=12 /user/update?email=evil@example.org&userid=1234 etc.

Too easy! Obviously more security is needed, and this is where the concept of secret keys comes in.

Secret Keys

It is a common practice to assign each user a secret key that they use to interact with a web service, often called an API key, generally consisting of randomly generated characters. In this case, you could give Bob a secret key when he signs up with the brewery network, and it would be stored in your database along with the rest of his account information.

Here's where I often see confusion setting in. Many people think it is logical that the secret key be sent as part of the API request, because after all, it uniquely identifies the user and it was given only to that particular person. However, this is a security problem!

The thing about secret keys is that they have to remain secret, just like any password. As soon as a key is leaked, whoever finds it can impersonate that account, just as easily as Mr. Evil's Brewery did in the previous example. If you send a secret key as part of an HTTP request, it is no longer secret. You have just leaked it to whoever cared to be listening! A network sniffer could pick it up, or it could be recorded in proxy logs or web server logs that aren't secured. Perhaps your ISP is doing deep packet inspection, and a rogue employee makes off with the logs. Who knows!

The following is an example of this type of insecurity. Bob is sending his update request along with the secret key:

/user/update?email=newbob@example.com&userid=1234&secret_key=bobs-super-secret-key

Now, pretend that Bob goes to his local Tarborks coffee shop and jumps on the free open wireless connection. Mr. Evil happens to be there, and fires up his network sniffer program and starts watching all the HTTP traffic on the network. He sees Bob's request go out, since HTTP connections are not secure. Now he has Bob's secret key, and can make any kind of request he wants with it. Maybe he could even delete Bob's account with a request to /user/delete?userid=1234&secret_key=bobs-super-secret-key, or send insults to Bob's customers using a mail endpoint!

Sending the secret key as part of a network request is NOT safe. So how do we properly identify the sender if the user id can't be trusted and they can't send us their key? Well, you knew I'd get to it eventually... signed requests!

Signed HTTP Requests

To produce a signed HTTP request, the sender and the receiver must both know the rules on how to generate a signature. It can be any crazy method you care to dream up, but a common one is as follows:

The sender organizes the data they want to send in a logical way, such as sorting it alphabetically.
The data is run through a hashing algorithm using the secret key to produce a hash. Hashing algorithms produce short strings of characters that vary based on the input data, and the output is sufficiently unique such that varying the input data by even one character produces a completely different hash.
The hash is added to the original data and sent.
The receiver identifies the user and gets the secret key from their associated account information, and recreates the hash on the received data.
If the recreated hash matches the one included in the request, the request is valid.

To return to our Tarborks coffee shop scenario, the request going out might look like this now:

/user/update?email=newbob@example.com&userid=1234&sig=x1zz645

If Mr. Evil sniffs this request, he might be pretty pleased with himself and try to change it to /user/update?email=evil@example.org&userid=1234&sig=x1zz645. This request would be rejected by the server however, since the signature is now invalid! When Mr. Evil changed the request, he needed to change the signature to match. But since he doesn't have Bob's secret key, he can't generate a correct signature, for this request or any other he cares to make up. The only valid request he can possibly make is the exact same one he just sniffed, and that's not very malicious at all, since it's what Bob was trying to do anyway.

Get to the code already

Assuming Bob's user id is 1234, and the secret key you gave him is "bobs-super-secret-key", he might write the following PHP code.

User code

>> show as plain text

PHP:

$USER_ID = "1234";
$SECRET_KEY = "bobs-super-secret-key";
/**
* @param array $data Array of key/value pairs of data
* @param string $secretKey
* @return string A generated signature for the $data based on $secretKey
*/
function generateSignature($data,$secretKey)
{
//sort data array alphabetically by key
ksort($data);
//combine keys and values into one long string
$dataString = '';
foreach($data as $key => $value) {
$dataString .= $key.$value;
}
//lowercase everything
$dataString = strtolower($dataString);
//generate signature using the SHA256 hashing algorithm
return hash_hmac("sha256",$dataString,$secretKey);
}
$bobsData = array(
"userid" => $USER_ID,
"email" => "newbob@example.com"
);
$sig = generateSignature($bobsData,$SECRET_KEY);
//add signature to the outgoing data
$bobsData['sig'] = $sig;
//generate HTTP query string
$queryString = http_build_query($bobsData);
echo $queryString;

The code above outputs userid=1234&email=newbob%40example.com&sig=efffd9cc30a220f2981b5124e1caa44d91b85aa2d2181f5331f48ca719983c1d. That's the HTTP query string that Bob would send to the /user/update endpoint to change his email address. So how does the service verify the request?

Server-side code

>> show as plain text

PHP:

if (empty($_REQUEST['userid'])) {
throw new Exception("No user id was sent with the request.");
}
//look up the account associated with the value in $_REQUEST['userid']
//and get the secret key for that account - implement as necessary
$secretKey = getSecretKeyFromUserId($_REQUEST['userid']);
$data = $_REQUEST;
$receivedSignature = $data['sig'];
//generate a signature using the data sent by the user, without the 'sig'
//parameter of course. Note that the generateSignature() function is the
//SAME ONE that the users would use!
unset($data['sig']);
$generatedSignature = generateSignature($data,$secretKey);
if ($generatedSignature != $receivedSignature) {
throw new Exception("Received signature is invalid!");
}
else {
//continue on, knowing it is the right user making the request.
}

There you have it. Signed requests!

Advanced Usage

The signature for a signed request can be sent in different ways. Some services, such as Amazon's S3 REST API, puts the signature in the HTTP headers. This is arguably a bit cleaner than including it in the parameters of an HTTP request, since the signature doesn't get mixed in with the data, and has implications for caching as well (browser caches and proxies). If you want to do it that way, you might have the user set a header as part of their HTTP request:

>> show as plain text

PHP:

header("X-Brewery-Sig: ".$sig);

And the receiving server, instead of looking for the sig parameter in $_REQUEST['sig'] (and having to remove it before running the data through the generator function), would find it in:

>> show as plain text

PHP:

$_SERVER['HTTP_X_BREWERY_SIG'];

Hope you found this useful!

Installing Gearman and gearmand on Windows with Cygwin

blake — Wed, 01 Dec 2010 02:51:59 +0000

Recently I've come face-to-face with a significant processing task for a web application written in PHP. I haven't worked with process control very much, so I started researching ways of distributing the calculations to multiple processes. PHP offers several libraries for doing this (pcntl, POSIX), but it quickly became clear that if you're running Windows these libraries are not an option, and unfortunately at work I have a Windows machine. After a lot more research, I came across Gearman.

Gearman is essentially a distributed processing framework, and seems to have community support for many programming languages. It consists of two main components: the job server, and a Client and Worker API. The Client and Worker API can be used in a wide variety of languages, but the job server is only available as a C library or a Perl library. This makes it a bit tougher to get the server running on Windows, especially when you start running into some of the dependencies that it requires to build. As well, the Client/Worker API for PHP can only be installed as a PECL extension, or a very-out-of-date PEAR extension called Net_Gearman.

Nonetheless, after yet more research I decided that I would give it a shot by using Cygwin to get the job server running (if you haven't used Cygwin before, be sure to read about it before attempting to install Gearman this way), and PEAR to use the API. Pre-built PECL extensions aren't available for Windows anymore, and the build process for PHP extensions can be pretty painful, so it makes PEAR look good by comparison even if the code will be out of date.

I had a pretty frustrating time finally getting everything up and running due to various dependency issues, so I went back through the whole process and wrote it out step-by-step. I used a Windows XP SP3 machine for this, but I also got it working on a Windows 7 machine as well.

Installing the Gearman job server (gearmand) on Windows with Cygwin

Installing Cygwin

If you don't have Cygwin already, you can get it from http://www.cygwin.com. The setup file is located here, and the setup process is pretty straightforward; run it and follow the wizard. Full installation instructions are available at the Cygwin site.
Keep the Cygwin setup.exe file handy after you've installed the default software packages, as you'll need it in the future to add packages, similar to apt-get, yum, and other Linux/UNIX package managers.
Cygwin installs with some basic packages, including a Cygwin Bash Shell that goes into your Start Menu. I prefer the mintty emulator instead, as it has less of a DOS Command Prompt feel and better terminal features. Feel free to use whatever shell you like of course. You can get mintty by re-running the setup.exe, and at the package selection screen, type 'mintty' into the Search bar at the top left. Expand the "Shells" category, and click on the word "Skip" under the "New" column beside the mintty package to select it before continuing the install process.

Installing Cygwin Package Dependencies needed for Gearman

If you're not already in the Cygwin setup, re-run the Cygwin setup.exe and go through to the package selection screen. The following is a list of dependency packages you will need in order to build the Gearman job server (gearmand). None of these packages were installed by default with Cygwin:

gcc
make
libuuid1-devel
libiconv

There's a good installation tutorial here that walks through getting gcc and make installed for people unfamiliar with Cygwin. Finding the others is pretty straightforward, the Search bar in the package selector works well.

Installing libevent

Gearmand requires an event notification library called libevent that you cannot get as a Cygwin package, which means it has to be installed from source. You can get the source here.

Download and unpack the latest libevent stable release. At the time of this writing, I used libevent-1.4.14b-stable.
NOTE: Download and unpack to a directory that does not contain spaces in the name, such as "C:/cygwin/home/Zombat/libevent-1.4.14b-stable". If you unpack to something with spaces like "C:/Documents and Settings/Zombat/", the build process might not be able to install libevent correctly (libtool has a hard time with spaces)!
Open a Cygwin shell and cd to the unpacked libevent directory.
Run the following commands:

./configure make make install

libevent should now be installed and ready to be used when compiling the Gearman job server.

Installing the Gearman job server, gearmand.exe

Download and unpack the C implementation of gearmand from http://gearman.org/index.php?id=download
Open a cygwin shell and cd to your unpacked gearmand directory. Same rules apply as before, make sure you've unpacked in a directory with no spaces in the path! libtool hates that, and your build may fail.
Run the following commands:

./configure make make install

The Gearman job server should now be installed and ready to use! Mine was installed at /usr/local/sbin/gearmand.exe, and running it with a "triple verbose" flag (-vvv) should produce the following:

That's it for the job server. When you want to start it, simply open a Cygwin shell and run gearmand.exe. Running it with the -d flag will cause the server to run as a daemon in the background, and running with --help will show you the full option list.

Installing the Gearman Client and Worker API (Net_Gearman)

I chose to install the PEAR Client and Worker API, as it is native PHP and doesn't involve compiling PECL extensions. The PEAR package is called Net_Gearman, and was originally written by Joe Stump at Digg.com. It is old and out of date now, although there appears to be a more recent fork at http://github.com/brianlmoon/net_gearman. I stuck with the older version, as I suspect it will meet my needs, and was readily available as a PEAR package.

This also makes installation relatively painless. Assuming you've previously set PEAR up, then all you have to do is open a command window (not a Cygwin shell) and run:

pear install Net_Gearman-alpha

The "-alpha" portion is necessary, as Net_Gearman apparently never made it to a stable release version. That being said, it has functioned well for me so far. Perhaps someone will pick the project up in the future.

I'll write more about getting started with the Client and Worker API in the next article, so we can actually use Gearman to get some work done.

MSB1025 – Adventures in Visual Studio Project Files with MSBuild

morgan — Fri, 17 Sep 2010 23:28:28 +0000

So we had a Sql Server Integration Services project that wasn't really being used currently, so one of our guys removed the project from our VS2008 solution file and checked it in. Worked fine on his machine - heck, it worked on all our machines.

The build server? Not so much. Immediate barfing with the cryptic message of "MSBUILD : error MSB1025: An internal failure occurred while running MSBuild". CruiseControl looked at it, sighed, and buckled to its knees under the weight of the aborting MSBuild process.

So what went wrong? Looking at the diff of the solution file, all that had happened was these two lines were removed:

Project("{D183A3D8-5FD8-494B-B014-37F57B35E655}") = "DTS", "DTS\DTS.dtproj", "{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}" EndProject

Reading up on MSB1025 and seeing that people had fixed it by rebuilding their solution files from scratch, I figured it just had to be a badly configured solution file, and that VS2008's handling of dtproj files was sloppy.

I did a search for the project's GUID (the second guid listed there - 947C2F56-A7A8-4357-8E42-6A234C1CCCB6) and sure enough, there were some references to it left in the solution file.

One project had:

ProjectSection(ProjectDependencies) = postProject {947C2F56-A7A8-4357-8E42-6A234C1CCCB6} = {947C2F56-A7A8-4357-8E42-6A234C1CCCB6} EndProjectSection

so I removed that entire section, and of course under the GlobalSection that stores the build configuration settings there was a whole whack of:

{947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Debug|Any CPU.ActiveCfg = Debug {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Debug|Mixed Platforms.ActiveCfg = Debug {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Debug|x86.ActiveCfg = Debug {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Debug)|Any CPU.ActiveCfg = Debug {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Debug)|Mixed Platforms.ActiveCfg = Debug {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Debug)|x86.ActiveCfg = Debug {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Release)|Any CPU.ActiveCfg = Release {947C2F56-A7A8-4357-8E42-6A234C1CCCB6}.Development (Release)|Mixed Platforms.ActiveCfg = Release ... ... ...

so I removed every offending line. Saved it, handed it back to the build server, and all was good.

An Exercise in WordPress Integration, or Why WordPress Sucks

blake — Tue, 08 Dec 2009 08:23:50 +0000

I'd like to prefix my upcoming rant with the fact that WordPress is good at what it does: making basic blogs and publishing content. I use it, many other people use it, it works. Heck, I'm using it right now. But from a technical standpoint, WordPress sucks. I'm going to relate my experience here trying write a quick function to store post output to a file, to be used by a separate application on the same server.

I started off to write a function (let's call it a caching function for simplicity) that stores some HTML from the most recently published post. Sounds easy enough. I should be able to just put a function into the functions.php file of the custom template set I'm using. That's seems to be where the "userland" custom functions go.

So I check the function reference first. Hey, wp_get_recent_posts(). Looks promising, so I give it a shot. It goes ahead and gets the most recent post just fine. Things are ok so far.

A problem appears

Now, I want to output the post exactly as it would appear in the blog, and save that output to a file on disk. Surely there's a basic function that will output a post's content? You know... take the post_content field from the database record and format it properly? Suddenly, the skies darken. Evil laughter booms out. Ha ha ha! WordPress mocks the folly of simplistic functional thinking!

The template files use functions like the_content() and the_title(). Just in case you can't tell from the excellent naming scheme, these actually produce echoed output. Checking out the_content(), we see it dutifully calls get_the_content(), then runs a couple of lines of formatting stuff on the results. So how about using get_the_content() for my caching function? I could run the other few formatting bits manually after that. Should be ok, right? After all, the doc comment for get_the_content() says the following:

/** * Retrieve the post content. *

So, I can go ahead assume it simply retrieves the basic post content then? Ha ha. NO. WHY WOULD IT DO THAT? Instead, it takes a bunch of globals that get set who-the-hell-knows-where, runs through a bunch of crap seemingly unrelated to the content of a post, and does a whole lot of textual modifications to some kind of content. Reading through the function is like jabbing red-hot fire pokeys into your eyes. Here's a portion of it:

>> show as plain text

PHP:



$content = $pages[$page-1];


if ( preg_match('/<\!--more(.*?)?-->/', $content, $matches) ) {


$content = explode($matches[0], $content, 2);


if ( !empty($matches[1]) && !empty($more_link_text) )

$more_link_text = strip_tags(wp_kses_no_null(trim($matches[1])))
$hasTeaser = true;
}

$pages is some kind of global that doesn't seem to have any relation to a post. Then apparently we're looking for HTML comments of , and replacing them with... well, something. I'd hate to think what would happen if I ever wrote a post with an HTML comment in it that happened to hit on whatever random content markers WordPress has decided to use. (Oh wait! That just happened to me while I was trying to publish the above code fragment!) I didn't even bother to look into things like wp_kses_no_null. It probably involves dark rituals with live chicken sacrifice. Why is there so much going on in a function called get_the_content()?

In the end, it seems that get_the_content() will eventually get the content of a post, but only if you set a half-a-dozen or so globals before you call it. And what the hell post is it even getting?

"The Loop"

Digging further, it's clear that the template functions for output are all like that. They don't take any kind of parameters; they just operate on globals! There's no way to take the post data that I just retrieved with wp_get_recent_posts(), and format it using these functions. You have to be in "The Loop" in order to do that. And "The Loop" sucks. It's not a catchy, easy-to-use method of handling posts, despite WordPress's efforts to pass it off as something neat or fun. It's a mish-mash of global functions with random naming and variable schemes (incidentally, just like the rest of WordPress). You can only use "The Loop" if you're accessing WordPress in a "normal", web-requested-and-template-loaded kind of way. It doesn't work if you're outside a template file (such as in functions.php before a template gets loaded).

So back to square one. Unfortunately, it appears that if I want to have the regular blog-formatted output, I need to harness "The Loop" somehow, and clearly you can't do that on your own (ie. outside of a template file) without knowing about every global variable in the system.

After some quick googling, I came across the query_posts() function, which you can use to set up "The Loop". Reading the documentation on it, you can find this little gem:

"The query_posts function overrides and replaces the main query for the page. To save your sanity, do not use it for any other purpose."

To paraphrase: "We've created a public API function that is pretty much useless except in a very specific page-dependent situation. Please enjoy how useless it is. But don't use it."

The fact that there is a "main query" for a page is another indicator of just how global-happy WordPress is, and that in turn gives you an insight into why it has so many security holes. How do you keep track of so many globals across so many functions?

A solution... sort of.

Fortunately, the query_posts() doc page links to the WP_Query docs, which is marginally more helpful, and provides the path for a solution. Using WP_Query sets up the wacky global stuff necessary to use "The Loop", which means we can hack our way through to getting some formatted post content. While technically feasible, you have to emulate a bunch of $_REQUEST parameters to the query() method. I ended up with this:

>> show as plain text

PHP:



function cacheMostRecentPost()


{


$featuredPosts = new WP_Query();


$featuredPosts->query('showposts=1');


while ($featuredPosts->have_posts())


{


$featuredPosts->the_post();


ob_start();


//do output with stuff like the_title() and the_content()


$str = ob_get_contents();


ob_end_clean();


//write $str to cache fragment


}

}
//set up hooks for this file when a post is changed or deleted
add_action('save_post', 'cacheMostRecentPost');
add_action('deleted_post', 'cacheMostRecentPost');

So despite relying on a specific set up of incoming HTTP parameters (as a string) for the most part, at least you can pass paramters to the query if you know the right ones. In this case, "showposts=1" seems to be the total number of posts fetched, and they appear to come back ordered by posting date, most recent first. This works for what I want it to do, but guess what? It doesn't work if you try to run it anywhere that's not one of those action hooks, because "The Loop" overwrites all the globals necessary for doing output later! So I can't use that function, say, at the top of the index.php template file if I wanted to. If I do, thanks to the overwritten globals, WordPress decides that I actually want the "Archive" page instead of the index page(!), and switches templates accordingly. So while I achieved my goal of being able to cache a post to a file with this function, it's certainly not portable, and it's certainly not elegant.

Wharrgarbl

The entire code flow is mind-boggling. Basing the output functions around a bunch of globals reminds me of code someone would have written in PHP 3 a decade ago, or something a very inexperienced programmer would write. Definitely not something you would expect in an application used by what is probably now millions of people. What's wrong with having some data fetching functions, and some output functions? You could, and I know I'm talking crazy here, but you could fetch some data, and then pass it to the output functions. Then (bear with me here), you could probably fetch posts (or whatever) at any time, and get some formatted output at any time, without overwriting some important global that might be used later in the code flow. Revolutionary, I know. Sorry if I went too fast on that. I'll repeat it louder and/or slower for any WordPress core developers that happen to be reading.

So, WordPress? How about something like:

$postObjects = getRecentPostsByDate(1); $output = formatPostContent($postObject[0]);

The mere concept of having individual posts exist inside their own little encapsulated world would make the APIs a hundred times more useful (and easier to understand). You could even keep those crap the_title() and the_content() and the_something_lol_naming_scheme_lol() functions if you wanted. Just make them take parameters. Better yet, put them inside a formatting object, or even the post object itself. $post->the_content() would still work, but it would have context!

The reason this gets me worked up is not that it's so frustrating to use (although that helps). I've had to deal with a lot of frustrating code in my career. It's more the fact that it's this kind of thing that gives PHP programmers a bad name. The code is just bad. The design is random. The API functions are random. The naming schemes are random. Functions don't do what their name (or their doc comment) indicates they should do. Integrating wordpress into another application or site is next to impossible (try it, I dare you), and the other way around, integrating another application or site into wordpress is much more difficult than it should be. Global usage is rampant and ridiculous to follow.

You don't have to look any farther than a single WordPress code file to understand why there have been so many security holes over the last couple of years. And there's a lot of PHP code out there that's the quality of WordPress, or worse.

To re-iterate my opening, if you don't need to get anything special out of it, WordPress does the job. They've filled their market niche well, and it's encouraging that development is ongoing and releases occur often. I've worked with it on occasion over the last few years, and the improvements are obvious, interface-wise especially, and to some extent code-wise as well (the WP_Query object is a step forward). But working with the code is not fun. Even modifying the template files is an exercise in counter-intuitiveness.

I'm sure there are reasons the code is what it is at this point, and I'm equally as sure I don't have the full picture to go with my condemnations. I guess I should just be thankful that I don't have to maintain it.

Facebook Notifications – "An unknown error occurred (out of memory)"

blake — Sun, 19 Jul 2009 18:54:04 +0000

Over the last few months, I've worked with the Facebook Notification system many times, and there has always been a moderate-to-high level of frustration with it. It is difficult to test on a development application, because you simply don't have the same number of users as a live application, and the application settings on a dev app are different, which affects your allocation of allowed notifications.

One of the prevalent errors I was getting for a long time was that a notification sent to a large number of users (application-to-user notification) would invariably fail with an Exception thrown from the Facebook PHP client library. My notification controller code in a particular application was set up to catch Exceptions and display the error into the administration control panel. However, "An unknown error occurred (out of memory)" is about as helpful as you might imagine. Yay Facebook. The only thing going for this particular error is that it's an iota more helpful than Facebook's documentation, which you quickly learn to distrust as out-of-date or just plain wrong.

The logic

My notification function logic was fairly simple: Accept an array of Facebook user ids, run an FQL query against Facebook to make sure the users all had the application installed, remove any ids that didn't, then use the Notifications.send() method (via the PHP client library's $facebook->api_client->notifications_send() method) to fire off the message.

No problems on development. On production, it was a giant party of "An unknown error occurred (out of memory)" errors.

At first I thought that Notifications.send() couldn't handle a lot of user ids (I was giving it at least 5000), so I put a loop in to break it up into several small "chunk sends". I went all the way down to 500 users per request, and nothing seemed to change, other than I was probably wasting my allocated notifications. The only silver lining to doing this testing on a production server was that the failed notifications didn't seem to go out to anyone, although that's impossible to tell for sure.

It's not the size of the notification, it's how you verify it.

I struggled with this for a long while until I realized that there was only one other possible location for Facebook to throw this error, and that was with the FQL query. I moved my "chunking" loop to work with the FQL as well, and I finally had relief from the "unknown" error. My conclusion therefore is that FQL queries might not be able to handle a large result set.

Since the documentation and forums couldn't give me any help, I thought I'd post my working code and explanation here, in the hopes that someone might find it useful.

Note that the Facebook object has been previously set as a property of the Controller that is executing this function, so it is accessed via $this->facebook here. You can see how the original array is broken into separate requests of approximately 500 users each, with each chunk of users being verified in turn. I have no idea what a safe number of users would be, however 500 seems to be working for me (for now).

/**
 * Sends a Facebook Notification message to all given users of an application.
 *
 * @param array $facebookIds Array of Facebook IDs to send the notification to.
 * @param string $msg Notification message to send.
 * @return int Count of how many users were sent the message.
 * @link http://wiki.developers.facebook.com/index.php/Notifications.send
 */
public function notifyMultipleUsers($facebookIds, $msg)
{
	if (! is_array($facebookIds) || count($facebookIds) == 0) {
		throw new Exception('No user ids given to notifyMultipleUsers().');
	}
	$reqSize = 500;
	$numUsers = 0;
	while(count($facebookIds) > 0) {
		$tmpIds = array_splice($facebookIds,0,$reqSize);
		//Check against Facebook database to verify which users have the app installed.
		$fql = 'SELECT uid FROM user '.
		       'WHERE is_app_user=1 AND uid IN ('.implode(',',$tmpIds).')';
		$fqlRes = $this->facebook->api_client->fql_query($fql);
		$appUsers = array();
		foreach($fqlRes as $a) {
			$appUsers[] = $a['uid'];
		}
		$c = count($appUsers);
		if ($c > 0) {
			$this->facebook->api_client->notifications_send(
				implode(',',$appUsers),$msg,'app_to_user'
			);
			$numUsers += $c;
		}
	}
	return $numUsers;
}

Defend PHP

morgan — Sun, 22 Mar 2009 04:27:47 +0000

I ran across a great discussion at StackOverflow today that started with this question:

I made a tongue-in-cheek comment in another question thread calling PHP a terrible language and it got down-voted like crazy. Apparently there are lots of people here who love PHP.

So I'm genuinely curious. What am I missing? Why makes PHP a good language?

The article lists a dozen or so "flaws" with the language, and then continues:

Worst of all, PHP convinces people that designing web applications is easy. And it does indeed make much of the effort involved much easier. But the fact is, designing a web application that is both secure and efficient is a very difficult task.

By convincing so many to take up programming, PHP has taught an entire subgroup of programmers bad habits and bad design. It's given them access to capabilities that they lack the understanding to use safely. This has led to PHP's reputation as being insecure.

Are the flaws in PHP really any different from any other language?

I'm a C# programmer, and I have to disagree with what some of what the original poster asserted were flaws with PHP. Overly broad implicit type conversions? Too easy to couple presentation with logic? Heck, those are the reasons it's so popular in the first place! Making web programming easy is a flaw? Wut?

After wrestling with the back and forth in that discussion, I came to a simple conclusion. This person has fallaciously concluded that PHP is a bad language based on his perception of its compliance with ivory-tower, computer science academia concepts. Most of his criteria could basically describe Perl, Javascript, VBScript, heck, even VB6, HTML and C.

Of course PHP isn't a horrible language. It's simply a tool, and based on its wild popularity and simple learning curve, only a fool could conclude that it was a bad one. Even if it breeds more "unsafe programmers" (which I doubt highly), it's simply not the language's responsibility to restrict what you can do just because you might do it.

Every programmer should be able to see that.

PHP Variable Test reference

blake — Mon, 30 Jun 2008 02:08:33 +0000

I thought I'd post a link to this PHP Variable Tests reference page. It's a great reference that's kept up to date with the current version of PHP. I use it sometimes when I'm waffling over what function to use to validate a variable.

Something I've noticed lately with the newer versions of PHP is that ctype_digit no longer returns true when you give it an empty string (ie. ctype_digit('')). This is great, since I always thought that returning true on an empty string was counter-intuitive; by definition, it should only return true "if every character in $text is a decimal digit", so if there's no characters, it can't be true. It's also great because that means that there's lots of places in some of my code where I can change

if (ctype_digit((string) $x) && $x != '')

to just

if (ctype_digit((string) $x)),

which is cleaner and nicer.

It looks like they may have made this change back around PHP 5.1, but I never noticed it until I checked that variable reference page. Nice to see it!

Ten PHP Best Practices Tips that will get you a job

blake — Wed, 04 Jun 2008 22:06:09 +0000

The last couple of weeks have been quite the experience for me. I was part of a big layoff at my former company, which was interesting. I've never been in that position before, and it's hard not to take it personally. I started watching the job boards, and a nice-looking full-time PHP position caught my eye, so I sent out a resume and landed an interview. Before the face-to-face portion, I chatted with the owner and head programmer on a conference call, and they ended up sending me a technical assessment quiz. One particular question caught my eye on this quiz... it looked something like this:

Find the errors in the following code:

";
	}
} 

?>

So, give it a shot. How many can you find?

If you got the missing comma in the parameter list, the "new Array()" error, the colon instead of a semi-colon, the '=' instead of '=>' in the foreach statement, and the erroneous use of '+' on the echo line, then congratulations, you found all the errors! You have the basic PHP technical skills to pay the bills.

That's not how I answered the question though. I noted the errors, obviously, but I went further than that. For instance, did you notice that there were no single quotes around the array indexes ($x[sales] and $x[profit])? That won't cause a fatal PHP error, but it is a coding error! Did you also notice the use of double-quoted strings instead of single-quoted strings on the echo line? Or the usage of the opening PHP short tag? Or the usage of "
" instead of "
"?

After pointing out the actual errors, I made a point of adding comments about those things I just mentioned. It was enough to push the answer from "correct" to "impressive", and it scored me a lot of points with the programmers who were reviewing my application. Enough so that they offered me the job! (I eventually turned it down, as I have been seduced by the siren call of the contracting life, and I intend to flex my PHP skills to the benefit of my clients, and not a faceless corporate overlord who dabbles in telemarketing. I need a shower).

So, read on for my Ten PHP Best Practices Tips that will get you a job:

1. Single-quoted strings are your friend. When you surround a PHP string in double quotes, it is subsequently parsed by the PHP interpreter for variables and special characters, such as "\n". If you just want to output a basic string, use single quotes! There is a marginal performance benefit, since the string does not get parsed. If you have variables or special characters, then by all means use double-quotes, but pick single quotes when possible.

2. String output. Which line of code do you think runs faster?
print "Hi my name is $a. I am $b"; echo "Hi my name is $a. I am $b"; echo "Hi my name is ".$a.". I am ".$b; echo "Hi my name is ",$a,". I am ",$b;
This might seem weird to you, but the last one is actually the fastest operation. print is slower than echo, putting variables inline in a string is slower than concatenating them, and concatenating strings is slower than using comma-separated echo values! Not only does not-inlining your variables give you a performance boost, but it also makes your code easier to read in any editor that has syntax highlighting (your variables will show up in nice colors). The little-known use of echo as a function that takes a comma-separated list of values is the fastest of them all, since no string operations are performed, it just outputs each parameter. If you combine all this with Tip #1 and use single quotes, you're on your way to some finely-tuned strings.

3. Use single-quotes around array indexes. As you saw in the quiz question above, I pointed out that $x[sales] is technically incorrect! You should quote associative array indexes, like so: $x['sales']. This is because PHP considers the unquoted index as a "bare" string, and considers it a defined constant. When it can't find a matching symbol for this constant in the symbol table however, it converts it to a real string, which is why your code will work. Quoting the index prevents this constant-checking stuff, and makes it safer in case someone defines a future constant with the same name. I've also heard that it is up to seven times faster than referencing an unquoted index, although I haven't tested this. For more on this, see the section called "Array do's and don'ts" in the Array section of the PHP manual.

4. Don't use short open tags. Eww... are you really using these? is just bad form. It can cause conflicts with XML parsers, and if you ever distribute code, it's going to annoy the heck out of people who have to start modifying their PHP ini directives to get it to work. There's just no good reasons to use short open tags. Use the full .


5.  Don't use regular expressions if you don't need to. If you're doing basic string operations, stay away from the preg and ereg function groups whenever possible.  str_replace is much faster than preg_replace, and strtr is even faster than str_replace!  Save those crunch cycles... your enterprise applications will thank you.

6.  Don't use functions inside a loop declaration. This isn't a PHP-specific tip, but you'll see it in a lot of code.
Bad:



for ($i = 0; $i < count($array); $i++) {

//stuff

}



Good:



$count = count($array);

for($i = 0; $i < $count; $i++) {

//stuff

}



That should be pretty self-explanatory, but it's amazing how many people would rather save a line of code at the expense of performance.  If you use a function like count() inside a loop declaration, it's going to get executed at every iteration!  If your loop is large, you're using a lot of extra execution time.
7.  Never rely on register_globals or magic quotes. register_globals and magic quotes are both old features of PHP that seemed like a good idea at the time (ten years ago), but in reality turned out to be not that great.  Older installations of PHP would have these features on by default, and they cause security holes, programming errors, and all sorts of bad practices, such as relying on user input to create variables.  Both these features are now deprecated, and everyone needs to stop using them.  If you are ever working on code that relies on these features, get it out of there as soon as you can!
8.  Always initialize your variables.  PHP will automatically create a variable if it hasn't been initialized, but it's not good practice to rely on this feature.  It makes for sloppy code, and in large functions or projects can become quite confusing if you have to track down where it's being created.  In addition, incrementing an uninitialized variable is much slower than if it was initialized.  It's just a good idea.
9.  Document your code. You've heard it many times, but this can't be said enough.  I know places that won't hire people who don't document code.  I even got my previous job after an interview where the VP sat-in with the interviewer and I, where I had brought my laptop in and I was just scrolling through some code I had written for one of my sites.  He saw my documented functions and was impressed enough to ask me about my documenting habits.  A day later I had the job.
I know a lot of self-declared PHP gurus out there like to pretend that their code is so good that they don't have to spend time documenting it, and these people are full of $largeAnimal poop.  Learn docblock syntax, familiarize yourself with some PHP Documentation packages like phpDocumentor or Doxygen, and take the extra time to do it.  It's worth it.
10.  Code to a standard. This is something that you should ask potential employers about during interviews.  Ask them what kind of coding standards they use... PEAR?  Zend?  In-house?  Mention that you code to a specific standard, whether it be your own, or one of the more prevalent ones out there.  The problem with loosely-typed languages like PHP is that without a proper coding standard, code tends to start looking like huge piles of garbage.  Stinky, disgusting garbage.  A basic set of rules that includes whitespace standards, brace matching, naming conventions, etc. is a must-have, must-follow for anyone who prides themselves on their code quality.
That being said, I hate all you space-indenters.  I mean, what the hell?  4 space characters as an indent?  That's exactly four times as much whitespace to parse as a tab.  More importantly, you can set your tab-stops to any value you want if you're using any text-editor more advanced than Notepad, so every developer can having something that they like the looks of.  Set it to 4 if you want, or 0 if you're a masochist.  I don't care, but you can't do that with spaces!  You're stuck with exactly the amount that Mr. Monkey Pants in cubicle 17 decided to put in.  So why are spaces so popular?  This "4-space indent" standard everyone uses is stupid!  Stop it!  Stop doing it!
... sorry, pet peeve.
Anyway, I hope these tips are helpful.  If you want to impress at a job interview, it's the little details that will get you noticed!  Maybe don't rant about your coding standards though.



Build a CentOS LAMP server
blake — Sun, 09 Mar 2008 21:45:14 +0000
I finally finished re-building a development CentOS LAMP server.  More importantly, and the primary purpose of the whole exercise, is that I finished my guide to building a CentOS LAMP server. I am happy to now post said guide on this blog (under Articles), for everyone to read and abuse as they please!
So here you go... may I present Blake's CentOS LAMP Server Guide, single-page edition, and the perhaps friendlier multi-page edition.
The guide covers the basics of building up a CentOS box to use as a remotely-administered LAMP server, including installation and configuration of PHP, Apache and MySQL.  Other goodies are in there too, like mail server setup with Postfix/Postgrey/Dovecot.  Keep in mind this guide was written from my point of view, and contains the ways that I like to do things.  I keep my basic LAMP stuff pretty slim for maximum performance, so you'll see some installations from source in there to help keep things tight.
Let me know if the guide is useful!  I'm hoping to get Google to pick it up and get some comments going on the various ways people set up their own LAMP boxes.



PHP 5.3 – Namespaces and Late Static Binding
blake — Fri, 15 Feb 2008 00:50:44 +0000
Work on the Mono machine continues.  For reasons as yet undisclosed, I'm rebuilding the Mono test box from scratch.  I'm not expecting this to resolve the previous issues I experienced while benchmarking Mono, but it is allowing me to tie up some loose ends.  More on that soon.
In the meantime, I noticed a new article over at Sitepoint about the upcoming PHP 5.3 release, slated for later this year.  Apparently some new features which were originally scheduled for PHP 6 are now included in 5.3, including Late Static Binding, and Namespaces!
While I can't get too excited about Namespaces, I know there are people out there that will love the fact that they will finally be arriving to PHP-land.  By its very nature a lot of PHP projects are small, and you can usually get along fine without Namespaces on a small project.  But with larger PHP sites and applications like Facebook becoming more commonplace, Namespaces are bound to come in exceedingly useful for some people.
On a more personal-project level, I find Late Static Binding to be much more exciting.  The examples given in the Sitepoint article are very thorough so I won't reprint them here, but the quick summary is that if you have a function in a Parent class that uses the self:: operator, you can't call that function from the perspective of a Child class, as self:: always references the class it is in (ie. the Parent).  Late Static Binding will give PHP developers a new reference called static:: which changes that; it will resolve to the static class at runtime (ie. the Child).  This will allow for a much more intuitive extensions of parent classes in object-oriented PHP!
Good stuff, you should check it out.