Now there existed a few problems I was well aware that I wanted to address if I was going to spend any more time on a good, clean interface that promoted rapid development and readable code:

• It was dependent on a PECL extension that was nearly 4 years old
• It only worked with PHP 5.2.x (as per above)
• The actual query-building interface wasn’t pretty and hindered readability

I sat down and revised a good portion of the old code I had put together and came up with something that fixes these problems. I wrote a new collection of classes with PHP 5.3 in mind and it uses a lot of the new language features (especially late static bindings and the __callStatic magic method). Now, because of the use of these new features my code is now dependent on PHP version 5.3 (developed using 5.3.1) — but, I suppose it’s better to code looking to the future rather than being locked into an older version.Thanks to reader Robert P who introduced me to Python’s Storm ORM and how it’s querying interface looked and functioned. After digging my way through its docs, I came up with something that is quite like a hybrid morbid mix of Django and Storm for use in PHP.

Let’s get to it: here’s how my interface has changed.

Basic Querying

I wanted to get away from what I hated the most: an unreadable querying interface that prevented the creation of self-explainable queries. What before was like:

Product::get(array("make__contains" => "Sony"));

has now become:

Product::get(Field::make__contains("Sony"));

which, using the __callStatic method, allowed me to create an interface that is actually quite readable aloud: “get Products where field make contains “Sony.” This is quite a bit better than using the array() function, where in reality it is completely unnecessary to the interface. This also works when combining expressions:

$q = Product::get(Field::make__contains("Sony"), Field::price__gt(100));
print $q->create_statement();
// output: SELECT id FROM products WHERE (make LIKE '%Sony%' AND price > '100')

Complex Expressions

One of the things I love about Django is the ability to create more complex AND/OR queries using the Q interface and operator overloading (which I was using before, dependent on the PECL operator extension). Storm has this too, but it also offered up an alternative interface using AND() and OR() functions.

I wasn’t able to do this exactly in PHP (“and” and “or” are reserved keywords), but I was able to make something slightly similar:

require_once("Model.class.php");

class Product extends Model { }

$q = Product::get(_AND(Field::make__contains("Sony"), Field::price__gt(100)));
print $q->create_statement();
// output: SELECT id FROM products WHERE make LIKE '%Sony%' AND price > '100'

$q = Product::get(_OR(Field::make__contains("Panasonic"), Field::make__contains("LG")));
print $q->create_statement();
// output: SELECT id FROM products WHERE make LIKE '%Panasonic%' OR make LIKE '%LG%'

Self-Referencing Fields in Expressions

Again, another favourite feature of Django was the ability to use F objects to create expressions that references a table’s own fields. My basic research didn’t show anything like this in Storm, but using the __callStatic method and the same style interface, I was able to create a readable interface for these sorts of queries:

$q = Product::get(Field::price__gt(Field::cost__times(2)));
print $q->create_statement();
// output: SELECT id FROM products WHERE price > cost * 2

$q = Product::get(Field::price__gt(Field::cost__times("quantity_sold")));
print $q->create_statement();
// output: SELECT id FROM products WHERE price > cost * quantity_sold

Using [field]__times (or __plus, __minus, __divided_by), complex field-references can be created and is still fairly readable, as for the above example: “get Products where field price is greater than field cost times 2,” and the output should be as expected.

I’m still not entirely happy with this portion of my code at the moment. It is fairly limited in the sorts of queries you can create (for example, it’s not possible to create an expression such as “cost * 2 – price”).

I’m more than happy to offer up what code I have to date. I tried to document as much as possible, but some of the code is admittedly messy and could use a clean-up — which I’m sure I’ll get to at some point or another.

Download/view Expression.class.php — defines how Expressions are created and handled using the Field interface

Download/view ExpressionNode.class.php — slightly revised from the previous code, defines classes for Q and F objects, along with _AND() and _OR() functions for creating more complex expressions.

Download/view Model.class.php — basic class for creating models and queries.

Download/view QuerySet.class.php — defines how filter/exclude/get queries are chained together in a linked list structure.

So this is an experiment I’ve had in the making for quite some time. I’ve been working for years trying to build a reusable “generic object” querying framework that I made my PHP applications easier to maintain and faster to develop. I originally started with a small library of classes, including a ‘GenericObject’ class and a ‘Collection’, which allowed for proper iteration using the Iterator interface — all wrapped around a very messy Factory design pattern.

Up until about a year, I used hacked and modified versions of these classes for all of my projects and was getting tired of the mess. After doing some casual research I finally came across the Django Framework and found the object creation and database querying quite phenomenal as a programming interface. I didn’t want to spend years trying to get my Python up to par with my abilities with PHP, so I decided to try and replicate it. For the last year, again, I’ve been using a pretty hacked up version of what seemed to me like a “Django-like” interface, combined together with my old GenericObject and Collection classes from long before. With this, I was previously able to perform really simple database queries with an interface like so:

$users = User::get(array("fullname__contains" => "Andrew"));

foreach($users as $user) {
  // ...
}

Extending this interface, I could further create more complex queries with AND and OR’s by how I passed a series of function arguments and array items. This worked pretty great actually, definitely increased my productivity when adding to an existing application, but made maintenance somewhat difficult due to hard-to-read queries that took a lot of pausing and mental analysis.

My research went deeper into the Django documentation and I decided: what’s stopping me from making an almost (this being the key word here) identical interface/framework for PHP, complete with complex querying using Q() and F() objects? Well, ultimately: a lot. But I’ve started, and I’m proud of my work that I want to share.

The Problems

First and foremost, the biggest hurdle in my goal was the lack of operator overloading in PHP. This is what I would need in order to successfully achieve complex querying with Q() and F() objects, such as listed in the Django examples:

Q(question__startswith='Who') | Q(question__startswith='What')

I also discovered this topic has been brought up many times before, with the final word being an unquestionable “never.” This was obviously disappointing for two reason: 1) it meant my goal was out of reach; and 2) PHP is a mature language which I have based my career around up to this point, and now I’ve come across what I believe to be a major limitation for the future.

Some more research turned up the PECL operator Package, which does override PHP’s functionality and allow for some basic operator overloading. Perfect! But another roadblock: it took me long enough to dig up a precompiled version of the package (since PECL doesn’t compile on Windows systems) and I only had a version that was compatible with PHP 5.2.x. Tough news. This meant I didn’t have access to beautiful things in PHP 5.3 such as late static bindings and other new additions to the language. I was able to overcome this by writing my own hacked-up, squirrely version of the get_called_class function and gratuitous cringe-worthy use of the eval function.

My final problem was Python’s beautiful ability to pass arguments to functions/methods in a format such as:

Entry.objects.filter(headline__startswith="What")

As much research as I could possibly stand to do, there really is no way around this in PHP. there is no way of passing a variable name a value as a function argument. My work around, despite being what I originally wanted to get away from, was to do something as follows:

Entry::get(array("headline__startswith" => "What"));

Obviously this creates some unfortunate coding overhead and reduces fast readability, it’s my only solution thus far.

Despite these trade-offs, I now had just about everything I needed to create a respectable PHP equivalent of Django querying.

The Solution

I went through the Django document on making queries up and down, left and right, and tried to get the best grasp the concepts of the interface and how they could be applied in PHP. Before even getting to code, I spent a couple nights just thinking about how this could be done in the back-end code of my framework/interface, and finally began coding backwards: first I started writing out some of the Django examples in PHP, then writing backwards through all the classes and objects that I would require in order for these examples to output what I wanted.

Before I get into the nitty gritty of how all the code works together, I’ll provide a few examples of my interface looks like so far and the queries it’s able to generate. I’m not yet at the point where I have QuerySets and usable objects, but I’ve done the hard part by creating the MySQL statement query building.

First I start by creating my initial database Model:

require_once("Model.class.php");

class Product extends Model {
// optional: table name is automatically generated
// using the class name and pluralizing it
//const table_name = "products";
}

From here, I can use my static get() method to create queries:

$r = Product::get();
print $r->create_statement();
// output: SELECT id FROM products

$r = Product::get(array("make__contains" => "Sony"));
print $r->create_statement();
// output: SELECT id FROM products WHERE make LIKE '%Sony%'

$r = Product::get(array("make__contains" => "Sony"), array("category__exact" => "TV"));
print $r->create_statement();
// output: SELECT id FROM products WHERE (make LIKE '%Sony%' AND category = 'TV')

I can then filter() and exclude() these Query objects like with Django:

$r = Product::get(array("make__contains" => "Sony"));
$r = $r->exclude(array("model__contains" => "RD3"));
print $r->create_statement();
// output: SELECT id FROM products WHERE make LIKE '%Sony%' AND NOT model LIKE '%RD3%'

$r = Product::get(array("category__contains" => "Stereo"));
$r = $r->filter(array("make__exact" => "Panasonic"));
print $r->create_statement();
// output: SELECT id FROM products WHERE category LIKE '%Stereo%' AND make = 'Panasonic'

And I can also use the critical Q() and F() functions, along with operator overloading to create complex queries:

$r = Product::get(Q(array("make__contains" => "Sony")) | ~Q(array("category__exact" => "TV")));
print $r->create_statement();
// output: SELECT id FROM products WHERE (make LIKE '%Sony%' OR  NOT category = 'TV')

$r = Product::get(array("selling_price__gte" => F("cost") * 2));
print $r->create_statement();
// output: SELECT id FROM products WHERE selling_price >= cost * 2

Of course, this isn’t as perfectly pretty as the Django interface, but it is quite readable and more maintainable than any other existing PHP solution I am aware of.

Additionally, I also added basic functionality for delete() and update() methods:

$r = Product::get(array("selling_price__gte" => F("cost") * 2));

print $r->delete();
// output: DELETE FROM products WHERE selling_price >= cost * 2

print $r->update(array("make" => "Sony", "category" => "TV"));
// output: UPDATE products SET make = 'Sony', category = 'TV' WHERE selling_price >= cost * 2

Use of limiting (through the ArrayAccess interface) and order_by() methods is also available:

$r = Product::get(array("make__exact" => "Sony"))->order_by("-model");
print $r->create_statement();
// output: SELECT id FROM products WHERE make = 'Sony' ORDER BY model DESC

$r = Product::get(array("make__exact" => "Sony"));
print $r["5:10"]->create_statement();
// output: SELECT id FROM products WHERE make = 'Sony' LIMIT 5 OFFSET 5

print $r[":10"]->create_statement();
// output: SELECT id FROM products WHERE make = 'Sony' LIMIT 10

At the moment, as mentioned previously, this interface is only good for creating MySQL queries. But now that my base functionality exists, applying this to full iterable query sets and database objects should be easy in comparison.

I still have a lot of work to go to round this all together, but I’m quite happy with my progress so far with a couple days of on-and-off investment of time.

I’m more than happy to offer up my code so far, but before there is any severe judgment in the quality of it, I should point out that I made little to no reference to the core Django code (except for one small piece that originally lead me to object overloading). If any readers show interest in my code, then I’ll do a small explanatory write-up of the classes and design patterns in another blog at a later date.

View/download the Model abstract class.

View/download my hacked up get_called_class function.

View/download the Query class.

View/download the S class (used for Q and F objects, shows operator overloading).

View/download the examples used in this blog entry.

Of course, please let me know if any of this code piques any interest in you or if you use any of the code in your applications.

Update – August 11, 2010 – So I caved and took a good hard look at the Django source. It’s not nearly as complicated as I imagined and my suspicions about how the back-end works just from looking at the docs is pretty close. I’m now either considering a full line-for-line port of Django to PHP or continuing with what I currently have and making it my own.

Map of Canada chart with province value colour coding.

This is PHP class I developed for work when I needed a Canada-specific map chart in some reporting tools I was creating. Obviously Google’s Map Charts API would’ve been my first choice, but it didn’t support Canada as it’s zoom level for reporting province-based data sets.

This is where I sat down, thought about a quick and dirty way to create this chart and went to work. I was able to create a chart as show on the right, with full control of output dimensions and colours.

Before I jump right into the full code, I want give a brief explanation of some of the techniques I used to accomplish this task.

Map of Canada chart with each province uniquely coloured

The idea I had was to have a pre-coloured map of Canada such that each province has it’s own very distinct colour (as shown on left). From this starting point, I could replace each unique province colour with a degree of color based on a primary foreground color (e.g. red, #FF0000, as in the example above). To do this properly, I set up an array based on province short-forms (for data entry) and their unique colour on the base map:

$provs = array(
  "AB" => "00DF6A",
  "BC" => "8AC707",
  "MB" => "3D5DFE",
  "NB" => "FF6BAA",
  "NL" => "C0FF00",
  "NT" => "FF8029",
  "NS" => "FFE26B",
  "NU" => "FFE328",
  "ON" => "D43CFF",
  "PE" => "6BFF7C",
  "QC" => "FF3DC0",
  "SK" => "01D7DF",
  "YT" => "FE293D"
);

and I was able to use an excellent function that I found which can take a HTML/hex color code and make it lighter by a percentage factor:

function hexLighter($hex,$factor = 30) {
  $new_hex = '';

  $base['R'] = hexdec($hex{0}.$hex{1});
  $base['G'] = hexdec($hex{2}.$hex{3});
  $base['B'] = hexdec($hex{4}.$hex{5});

  foreach ($base as $k => $v)
      {
      $amount = 255 - $v;
      $amount = $amount / 100;
      $amount = round($amount * $factor);
      $new_decimal = $v + $amount;

      $new_hex_component = dechex($new_decimal);
      if(strlen($new_hex_component) < 2)
          { $new_hex_component = "0".$new_hex_component; }
      $new_hex .= $new_hex_component;
      }

  return $new_hex;
}

From here, replacing one colour with another is quite a simple task:

function colorReplace($image, $oldcolor, $newcolor) {
  // find old color
  $oldcolor = $this->html2rgb($oldcolor);
  $old = imagecolorexact($image, $oldcolor['r'], $oldcolor['g'], $oldcolor['b']);
  // replace with new color
  $newcolor = $this->html2rgb($newcolor);
  imagecolorset($image, $old, $newcolor['r'], $newcolor['g'], $newcolor['b']);
}

…the rest is simply basic PHP image manipulation.

View/download the complete Canada Chart PHP class.
Click here to download the full-size coloured map of Canada required by the class.

Now since I was already using Google’s Charts API, I wanted to make my own URL API to be relatively similar in fashion. So you could use this class like so, click to see result:

canada.chart.php?chld=ON,QC,AB,NL,BC,NB,MB,PE,NS,SK&chd=1000,450,132,56,6,775,320,950,100,645&chf=ff0000&w=450&h=450

This should make using this class in a project in which you’re already using Google Charts API fairly consistent.

I did add basic support for data labeling, but this wasn’t something I specifically needed so I didn’t pretty it up too much, but you can see an example here:

canada.chart.php?chld=ON,QC,AB,NL,BC,NB,MB,PE,NS,SK&chd=1000,450,132,56,6,775,320,950,100,645&chf=ff0000&w=450&h=450&clb=1

This requires a GDF conversion of the Arial font for use with PHP’s GD library. Feel free to never use this functionality, take it out completely, or make it better.

If you successfully use this code, take it apart, or make it better — please let me know! I love to be constantly improving reusable classes like this, and it’s great to share good code with everyone.

This first function will return the coordinates of specific address (I find pretty much anything works, whether it’s a full address with street number, or just a postal code, or simply a city name) as an array of coordinates:

function getCoordinates($address) {
  $url = "http://maps.google.com/maps/geo?q=" . urlencode($address)
         . "&output=json";
  $result = file_get_contents($url);
  $result = json_decode($result, 1);
  if (isset($result['Placemark'])) {
    list($lat, $long) = $result['Placemark'][0]['Point']['coordinates'];
  } else {
    $lat = $long = false;
  }
  return array($lat, $long);
}

and the following function will accept the lat/long coordinates of two separate points and return their distance (not road/traveling distance, but straight-shot distance) in KM:

function calculateDistance($lat1, $long1, $lat2, $long2) {
  $R = 6371; // earth's mean radius in km
  $dLat = deg2rad($lat2-$lat1);
  $dLon = deg2rad($long2-$long1);
  $lat1 = deg2rad($lat1);
  $lat2 = deg2rad($lat2);

  $a = sin($dLat/2) * sin($dLat/2) +
      cos($lat1) * cos($lat2) *
      sin($dLon/2) * sin($dLon/2);
  $c = 2 * atan2(sqrt($a), sqrt(1-$a));
  $d = $R * $c;

  // return distance in KM
  return $d;
}

Now it certainly may be worthwhile to cache coordinates for a given location, depending on your usage, because retrieving the coordinates from Google for a large quantity of addresses at once can take some time.

Waveform generated using Dancing With Paris' "(Boardwalk)" MP3.

A few days of Googling, I did come up with a couple solutions but none seemed to really produce the results I wanted. All I was looking to generate was a simple waveform like that in Cubase or other recording software. I don’t need axis labels or left/right audio splits, just a simple mono waveform from an MP3. I also didn’t want to go through hundreds of audio files and convert them into WAV files just to do this — if I was going to do this as an experiment, I wanted it on the fly. I did come across some interesting links though:

The Sound eXchange (SoX) audio tool kit has a lot of great features that might come in handy later on, including audio effects, but support for MP3 wasn’t built-in and seemed a bit too much work to bother compiling just for a side experiment. SoX can generate spectrograms, but this wasn’t particularly what I was looking for.

I did find this classAudioFile.php, which does support generating waveforms from WAV files, but this seemed a bit more hefty than I really needed — and not as accomplishing as doing it myself. That being said, and though I never bothered testing it out, it does seem (based on the code) to create a simple waveform image that I was aiming to create.

Ultimately, I wanted something flexible: upload MP3s files without pre-converting them, adjust size of the image easily, and change foreground and background colours. The key that makes my script different, and most flexible, is converting the files to WAV on the fly. Though this is time consuming, it’s ultimately a lot easier than batch processing them on my computer first.

Click here to see the source code for the script.

The key to making this convenient has been using the LAME MP3 encoder/decoder through the command line (on Windows for my script, I haven’t yet tested with any flavour of Linux). I simply downloaded a windows build of LAME, threw the executable into my script directory and ran the following command:

lame original_file.mp3 -f -m m -b 16 --resample 8 resampled.mp3 && lame --decode resampled.mp3 final_file_for_processing.wav

Because you don’t really need a high quality audio file for generating a waveform, this set of commands will resample the original MP3 to a 16 bit, 8 KHz, mono MP3 file and then turn that into a WAV file for processing the image from. Processing does take some time, but it’s significantly faster than processing a huge 40+ MB WAV file from the start. Also, no amount of research showed a way to directly downsample an MP3 straight to a WAV file, hence the intermediary step.

After that, I used an algorithm I found on the DevShed forums for processing 16 bit WAV files, and used the data it generates to create the waveform (which, ultimately, is pretty simple).

If this is something you have a use for, feel free to use it as you wish and give credit where it’s due (see the source comments for where I pulled some code snippets from to make this work). You may want to fix it up a bit for better error checking and so forth though.

A note of warning: it’s not fast. Far from it. A couple basic tests showed anywhere between 10-30+ seconds to generate the waveforms for an average sized MP3 (decent quality, 2-4 minutes in length). If you will be generating and re-generating waveforms multiple times for the same audio file, you may want to cache the WAV data in a database for future use, as using that data to create the image itself is comparatively quite quick.

I may use this, along with the SoX library, later on to create — simply for fun and discovery — an online audio mixer/editor similar to Cubase, which I’ve been working with quite a lot lately. This would be a great way to create a new web-based interface with a lot of widgets and controls not commonly used, as well as using SoX to add audio effects, mix, combine files, in a multi-layered audio interface. I’ll see if I ever get around to this however.

Environment tested in, your results may vary:

• Windows Server 2003
• PHP 5.2.10
• LAME 3.98.4