An experiment: Django Framework-like querying/model interface in PHP

I have continued with this experiment and created a new querying interface compatible with PHP 5.3 and has no dependent PECL extensions: <a href="http://andrewfreiday.com/2010/08/23/experiment-update-revised-django-like-php-5-3-compatible-querying-interface/" target="_blank"><em><strong>Experiment update: revised Django-like PHP (5.3 compatible) querying interface</strong></em></a>.

So this is an experiment I’ve had in the making for quite some time. I’ve been working for years trying to build a reusable “generic object” querying framework that I made my PHP applications easier to maintain and faster to develop. I originally started with a small library of classes, including a ‘GenericObject’ class and a ‘Collection’, which allowed for proper iteration using the Iterator interface — all wrapped around a very messy Factory design pattern.

Up until about a year, I used hacked and modified versions of these classes for all of my projects and was getting tired of the mess. After doing some casual research I finally came across the Django Framework and found the object creation and database querying quite phenomenal as a programming interface. I didn’t want to spend years trying to get my Python up to par with my abilities with PHP, so I decided to try and replicate it. For the last year, again, I’ve been using a pretty hacked up version of what seemed to me like a “Django-like” interface, combined together with my old GenericObject and Collection classes from long before. With this, I was previously able to perform really simple database queries with an interface like so:

$users = User::get(array("fullname__contains" => "Andrew"));

foreach($users as $user) {
  // ...
}

Extending this interface, I could further create more complex queries with AND and OR’s by how I passed a series of function arguments and array items. This worked pretty great actually, definitely increased my productivity when adding to an existing application, but made maintenance somewhat difficult due to hard-to-read queries that took a lot of pausing and mental analysis.

My research went deeper into the Django documentation and I decided: what’s stopping me from making an almost (this being the key word here) identical interface/framework for PHP, complete with complex querying using Q() and F() objects? Well, ultimately: a lot. But I’ve started, and I’m proud of my work that I want to share.

The Problems

First and foremost, the biggest hurdle in my goal was the lack of operator overloading in PHP. This is what I would need in order to successfully achieve complex querying with Q() and F() objects, such as listed in the Django examples:

Q(question__startswith='Who') | Q(question__startswith='What')

I also discovered this topic has been brought up many times before, with the final word being an unquestionable “never.” This was obviously disappointing for two reason: 1) it meant my goal was out of reach; and 2) PHP is a mature language which I have based my career around up to this point, and now I’ve come across what I believe to be a major limitation for the future.

Some more research turned up the PECL operator Package, which does override PHP’s functionality and allow for some basic operator overloading. Perfect! But another roadblock: it took me long enough to dig up a precompiled version of the package (since PECL doesn’t compile on Windows systems) and I only had a version that was compatible with PHP 5.2.x. Tough news. This meant I didn’t have access to beautiful things in PHP 5.3 such as late static bindings and other new additions to the language. I was able to overcome this by writing my own hacked-up, squirrely version of the get_called_class function and gratuitous cringe-worthy use of the eval function.

My final problem was Python’s beautiful ability to pass arguments to functions/methods in a format such as:

Entry.objects.filter(headline__startswith="What")

As much research as I could possibly stand to do, there really is no way around this in PHP. there is no way of passing a variable name a value as a function argument. My work around, despite being what I originally wanted to get away from, was to do something as follows:

Entry::get(array("headline__startswith" => "What"));

Obviously this creates some unfortunate coding overhead and reduces fast readability, it’s my only solution thus far.

Despite these trade-offs, I now had just about everything I needed to create a respectable PHP equivalent of Django querying.

The Solution

I went through the Django document on making queries up and down, left and right, and tried to get the best grasp the concepts of the interface and how they could be applied in PHP. Before even getting to code, I spent a couple nights just thinking about how this could be done in the back-end code of my framework/interface, and finally began coding backwards: first I started writing out some of the Django examples in PHP, then writing backwards through all the classes and objects that I would require in order for these examples to output what I wanted.

Before I get into the nitty gritty of how all the code works together, I’ll provide a few examples of my interface looks like so far and the queries it’s able to generate. I’m not yet at the point where I have QuerySets and usable objects, but I’ve done the hard part by creating the MySQL statement query building.

First I start by creating my initial database Model:

require_once("<a href="http://andrewfreiday.com/source.php?f=django/Model.class.php" target="_blank">Model.class.php</a>");

class Product extends Model {
// optional: table name is automatically generated
// using the class name and pluralizing it
//const table_name = "products";
}

From here, I can use my static get() method to create queries:

$r = Product::get();
print $r->create_statement();
// output: SELECT id FROM products

$r = Product::get(array("make__contains" => "Sony"));
print $r->create_statement();
// output: SELECT id FROM products WHERE make LIKE '%Sony%'

$r = Product::get(array("make__contains" => "Sony"), array("category__exact" => "TV"));
print $r->create_statement();
// output: SELECT id FROM products WHERE (make LIKE '%Sony%' AND category = 'TV')

I can then filter() and exclude() these Query objects like with Django:

$r = Product::get(array("make__contains" => "Sony"));
$r = $r->exclude(array("model__contains" => "RD3"));
print $r->create_statement();
// output: SELECT id FROM products WHERE make LIKE '%Sony%' AND NOT model LIKE '%RD3%'

$r = Product::get(array("category__contains" => "Stereo"));
$r = $r->filter(array("make__exact" => "Panasonic"));
print $r->create_statement();
// output: SELECT id FROM products WHERE category LIKE '%Stereo%' AND make = 'Panasonic'

And I can also use the critical Q() and F() functions, along with operator overloading to create complex queries:

$r = Product::get(Q(array("make__contains" => "Sony")) | ~Q(array("category__exact" => "TV")));
print $r->create_statement();
// output: SELECT id FROM products WHERE (make LIKE '%Sony%' OR  NOT category = 'TV')

$r = Product::get(array("selling_price__gte" => F("cost") * 2));
print $r->create_statement();
// output: SELECT id FROM products WHERE selling_price >= cost * 2

Of course, this isn’t as perfectly pretty as the Django interface, but it is quite readable and more maintainable than any other existing PHP solution I am aware of.

Additionally, I also added basic functionality for delete() and update() methods:

$r = Product::get(array("selling_price__gte" => F("cost") * 2));

print $r->delete();
// output: DELETE FROM products WHERE selling_price >= cost * 2

print $r->update(array("make" => "Sony", "category" => "TV"));
// output: UPDATE products SET make = 'Sony', category = 'TV' WHERE selling_price >= cost * 2

Use of limiting (through the ArrayAccess interface) and order_by() methods is also available:

$r = Product::get(array("make__exact" => "Sony"))->order_by("-model");
print $r->create_statement();
// output: SELECT id FROM products WHERE make = 'Sony' ORDER BY model DESC

$r = Product::get(array("make__exact" => "Sony"));
print $r["5:10"]->create_statement();
// output: SELECT id FROM products WHERE make = 'Sony' LIMIT 5 OFFSET 5

print $r[":10"]->create_statement();
// output: SELECT id FROM products WHERE make = 'Sony' LIMIT 10

At the moment, as mentioned previously, this interface is only good for creating MySQL queries. But now that my base functionality exists, applying this to full iterable query sets and database objects should be easy in comparison.

I still have a lot of work to go to round this all together, but I’m quite happy with my progress so far with a couple days of on-and-off investment of time.

I’m more than happy to offer up my code so far, but before there is any severe judgment in the quality of it, I should point out that I made little to no reference to the core Django code (except for one small piece that originally lead me to object overloading). If any readers show interest in my code, then I’ll do a small explanatory write-up of the classes and design patterns in another blog at a later date.

View/download the Model abstract class.

View/download my hacked up get_called_class function.

View/download the Query class.

View/download the S class (used for Q and F objects, shows operator overloading).

View/download the examples used in this blog entry.

Of course, please let me know if any of this code piques any interest in you or if you use any of the code in your applications.

Update – August 11, 2010 – So I caved and took a good hard look at the Django source. It’s not nearly as complicated as I imagined and my suspicions about how the back-end works just from looking at the docs is pretty close. I’m now either considering a full line-for-line port of Django to PHP or continuing with what I currently have and making it my own.

7 thoughts on “An experiment: Django Framework-like querying/model interface in PHP

  1. Robert P

    “full line-for-line port of Django to PHP or continuing with what I currently have and making it my own”

    Whichever you choose, keep it up! A little more of the Python mentality in the PHP world is a good thing. πŸ™‚

    That this relies on an out-of-date PECL package is a real turn-off though. I really like these kinds of features from duck-type languages. When PHP implemented namespaces (and even type hinting) I felt like it was showcasing what it couldn’t do.

    Have you looked at other Python ORMs, like for example Storm (http://storm.canonical.com/)? Its architecture may be more suited for PHP than Djangos, but that’s just a guess…

    • Andrew

      Hi Robert, thanks for your comment and suggestion.

      To be quite honest, I couldn’t possibly agree anymore about the dependency on a PECL package whose last update was February of ’06. This is definitely not reliable for any production environment, but for me it was simply experiment. That being said though, a full port of the Django back-end would be much more organized and efficient than anything I could ever write myself. It’d be nice to do a port of the core Django source but find a better way to handle the querying interface — primarily removing the dependency on any PECL package like this, but finding a clean way to have the same functionality with the Q and F query modifiers. This is something I will start to think about.

      I took a very brief look at Storm, thank you for the alternative reference. I’m not sure how I feel about the ‘store’ interface, but I’ll look into a bit deeper over the next few days and consider this as another option.

      Edit: I was just looking at Expressions under the Storm manual and I do quite like the use of the And() as a replacement for non-existent operator overloading in my case. And as it explains,

      Using an explicit β€˜And’ is particularly useful if you have a loop or generator function which build lists of expressions. Since this is all serializable, you can easily build up a python list of expressions, and then And() it together

      Also, I do quite like

      Foo.name.like(foo.name)

      style expressions in place of my hacked-up

      array(“name__contains” => $name)

      expressions.

      Maybe my querying interface will be a hybrid mix up of Django and Storm!

      • Robert P

        As Isaac Newton once said: “Experimenting and stuff is probably good.”

        I have a feeling that Django is too strongly tied to Python to suffer a conversion to a not-less-but-certainly-not-greater language like PHP. It’s organisational structure could be loosely adapted to PHP, but the fine details are nigh-on impossible. I still haven’t had metaclasses explained to me in a paragraph…

        I mentioned Storm because at a quick glance it seemed to be more achievable that Djangos ORM.

        Person.objects.filter(person__firstname=’Robert’) # Django
        or
        store.find(Person, Person.firstname==’Robert’) # Storm

        And as for the interfaces, I generally find that if you can comfortably read it aloud as a sentence then you can repeatedly code it with no issues. There’s not too much difference between “filtering the Person objects” and “finding Persons in the store” when you know you’re working with a database.

        For a Django-PHP port the emphasis should be on the philosophy and design patterns of Django, with serious concessions for Python-specific features.

        I even made a couple of super-basic attempt myself. The first used the classes Model, IdentityMap, Context and Constraint and would have looked something like this:
        User::objects() // returns the instance of IdentityMap tied to the User singleton
        ->filter(array(‘username’ => ‘Robert));

        The second went off into model fields, and isn’t even worth a mention.

        Django-PHP may be a pipe-dream. For small projects what I really want is a “so cool I can’t believe it’s PHP” micro-framework inspired by Django. Something like Sinatra-Django but better. And for anything bigger why not use Python?

        • Andrew

          You are absolutely right in terms of the language differences, that anything even vaguely considered a true “port” of Django from Python to PHP is impossible. As an experiment, this was definitely more of a learning experience about language features and design patterns, as you mentioned, than simply copying from one language to another.

          I know there are about a million different, all very tight and refined PHP frameworks already out there, but there’s always something to be said for doing it myself so that it is exactly what I want to be coding (the framework’s interface) all while making it a learning process.

          My obsession with Django, despite my almost complete lack of experience with Python, was that it always just had some indefinable quality about it that made me want to use it. With PHP, it’s just become my language of choice and I’m working on my expertise with it while still early in my web development career. I really had no intention of developing yet another bloated PHP framework with command-line driven application directory building and database table creation (i.e. Rails). I just wanted a framework, for myself, that I could swap into an existing project with a pre-made database table structure, without defining a handful of models/classes and redefining all of the fields/properties I already previously laid out in the database. It would also be nice to have a framework that works very tightly with Smarty templates (which I already use, and at times despise), also a bit more like Django’s own templates.

          I think I might continue on with this little side project, with more focus on ultimately being used for “rapid development”, and just see where it goes. If nothing else, I bridge a huge gap in my knowledge of using a variety of design patterns.

          Your comments and suggestions have been incredibly insightful, and I very much thank you for stimulating some conversation and providing me with some new direction.

  2. […] After some research, discussion and a bit of hacking together some poorly organized code, I’ve come up with a revised interface from my previous post An experiment: Django Framework-like querying/model interface in PHP. […]

  3. Hi, came across the same whish. My way works more like a parser, the modelclass holds private properties needed for any model and methods for parsing querys e.g. news->filter(). Subclasses extend it with their public properties.
    subclasses properties work like:
    public $link = array(“admin”=>”true”,
    “type”=>”link”,
    “relation”=>array(“class”=>”news_links”,
    “type”=>”foreignkey”,
    “render”=>”inline”));
    queries are defined such as:
    public function article_news_overview(){
    parent::reset();
    $news->filter(“dtime__lte=ts_today|type__exact=1”)->load()->execute();
    /loading&executing is not strictly needed. Simply use render_to_response() to fill a view/

    for parsing i simply use regex like:
    private function checktrigger($statement){
    GLOBAL $_USER;
    $classfind = preg_split(‘/___/’,$statement,2,PREG_SPLIT_NO_EMPTY| PREG_SPLIT_DELIM_CAPTURE);
    if (count($classfind)>1){
    $statement = preg_split(‘/__|=/’,$classfind[1],3,PREG_SPLIT_NO_EMPTY| PREG_SPLIT_DELIM_CAPTURE);
    } else {
    if (preg_match(‘/exists__/’,$statement)==0){
    $statement = preg_split(‘/__|=/’,$statement,3,PREG_SPLIT_NO_EMPTY| PREG_SPLIT_DELIM_CAPTURE);
    } ELSE {
    $statement = preg_split(‘/__/’,$statement,2,PREG_SPLIT_NO_EMPTY| PREG_SPLIT_DELIM_CAPTURE);
    }
    }
    switch ($statement[0]){
    case “exists”:
    return ” EXISTS (“.$statement[1].”)”;
    }
    switch ($statement[1]){
    case “gte”:
    $statement[1] = “>=”;
    break;
    (…)
    switch ($statement[2]){
    case “me”:
    $statement[2] = $_USER->id;
    break;
    case “ts_today”:
    $statement[2] = time();
    break;
    default:
    if (count($classfind)>1){
    $return = $classfind[0].”.”.$statement[0].$statement[1].$statement[2].” “;
    } ELSE {
    $return = $this->classname.”.”.$statement[0].$statement[1].$statement[2].” “;
    }
    return $return;
    break;
    }

    At least i’m no professional coder but my solution is highly flexible to me. I’m doing some weird stuff like loading subclasses from an imported directory structure array:

    protected function import(){
    $value = false;
    foreach ($this->obj as $key=>$value){
    try {
    if (!file_exists(“path/active/class.”.$value.”.php”)){
    throw new Exception(“Die Datei mit dem Namen “.$value.” wurde nicht gefunden. Traceback:”);
    unset($key);
    } ELSE {
    require_once(“path/active/class.”.$value.”.php”);
    $this->objs[$value] = new $value;
    }
    } catch (Exception $e) {
    (…)
    I guess for professional that’s a no go…

  4. […] added the ability to create more complex expressions with ANDs and ORs using predefined functions. Last time I was experimenting with the idea of operator overloading using the PECL operator package. It […]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>