fíam

I'm starting to notice a problematic pattern in Django: some components are not extensible at all. There's no problem when you are using it in the same way the framework developers intended, but when you want to do something which deviates from the Django standard, you're screwed.

That's what happened to me yesterday. I wanted to add a RSS feed for public notes at byNotes, featuring GeoRSS information, but as far as I'm concerned Django Syndication Framework doesn't allow the framework user to define new fields. So I've choosen to write a simple but extensible RSS Framework.

Each new field is a class, which can be plugged inside a channel or inside an item. In addition, I've also used a view-as-class approach without any templates, requiring the application feed class to return the field contents. However, every field also defines its own formatter, so the application feed class does not need to return the text, only the object the formatter expects (e.g, the pubDate field expects a datetime object, while the link field expects an URL).

There are some things I like in the Django Syndication Framework, so I've kept them. For example, feeds can receive parameters via get_object() and the bits argument. I'm also using a method-based approach for retrieving the data, however I've changed the method names, since they depend on the field name. For example, the channel title is returned by get_channel_title(), while get_item_description() takes care of item descriptions. The general rule is get_(channel|item)_xmlns_tagname(), but standard fields lack the xmlns part (as get_channel_title and get_item_description).

I'm not very good at explaining frameworks, so let's see the byNotes feed class as an example:

from django.utils.text import capfirst
from django.utils.html import urlize
from django.shortcuts import get_object_or_404

from notes import rss 
from notes.utils import escape_note
from notes.models import Note, UserProfile

class PublicFeed(rss.Feed):
    additional_item_fields = [rss.FeedFieldGeoRSSPoint, rss.FeedFieldDCCreator]
    def get_object(self, bits):
        if len(bits) == 0:
            return None

        if len(bits) == 1:
            try:
                return get_object_or_404(UserProfile, user__username=bits[0])
            except UserProfile.DoesNotExist:
                pass

    def get_channel_title(self, obj):
        if obj:
            return _('Public notes by %(author)s') % \ 
                {   
                    'author': obj.display_name,
                }   

        return _('byNotes public timeline')

    def get_channel_link(self, obj):
        if obj:
            return '/%s/' % obj.user.username

        return '/_timeline/'

    def get_channel_description(self, obj):
        if obj:
            return _('Latest public notes by %s') % obj.display_name

        return _('Latest public notes posted at byNotes')

    def get_items(self, obj):
        if obj:
            return Note.objects.filter(submitter=obj.user, public=True).order_by('-id')[:50]

        return Note.objects.filter(public=True).order_by('-id')[:50]

    def get_item_title(self, obj):
        return _('%(type)s by %(author)s') % \
            {
                'type': capfirst(obj.get_note_type_display()),
                'author': obj.submitter.get_profile().display_name,
            }

    def get_item_pubdate(self, obj):
        return obj.submitted

    def get_item_description(self, obj):
        val = escape_note(obj.message)
        if obj.note_type == 'L':
            val += '<b>&raquo; %s:</b> %s' % (_('Link'), urlize(obj.get_attachment.uri))
        elif obj.note_type == 'P':
            val += '&raquo; <a href="http://maps.google.com/?ll=%s,%s&z=8">%s</a>' % \
                (obj.public_position.latitude, obj.public_position.longitude,
                    _('View on Google Maps'))


        if obj.note_type != 'E' and obj.position:
            val += '<br><b>%s:</b> %s' % (_('From'), capfirst(obj.public_position.display))

        return val

    def get_item_guid(self, obj):
        return 'http://bynotes.com%s' % obj.get_absolute_url()

    def get_item_georss__point(self, obj):
        try:
            return (obj.public_position.latitude, obj.public_position.longitude)
        except AttributeError:
            return None

        def get_item_dc__author(self, obj):
            return obj.submitter.username

I have no intentions to extend this framework with Atom support for now, since RSS is enough for me. If you are interested in the code, I'm releasing it under a MIT License.

UPDATE This framework uses the class-as-view approach, meaning urls.py should look like:

from django.conf.urls.defaults import *
from notes.feeds import PublicFeed  # Change to match your feed class name    

urlpatterns = patterns('bynotes.notes.views',
...
...
(r'^_feeds/public/((?P<params>.*)/)?$', PublicFeed()),
...

I've just launched the new geomicroblogging site I mentioned a few days ago. It still isn't as powerful as FireEagle, since the web API is still not ready, but I expect to finish it by next week. However, there are also additional features not present in FireEagle (according to what I've read, since I can't try FireEagle):

• You can send messages.
• You can send events, and when people reply to them they can specify if they're attending.
• You can also send links, with oEmbed support.
• You can build a network of friends and send them private notes.
• You can become a fan of someone and you'll receive his public notes (like following someone on twitter, but you can also know where he is).
• You can update your position only for some of your friends, the rest of them will see your previous position.
• Registration with OpenID is supported.

Registration is currently open to everyone, but I'm going to close it as soon as there are users enough for doing some real-world testing. Wanna try it? Go ahead and start geomicroblogging today.

As for the source code, I'm not sure if I'll release all of it. However, there are some portions almost ready for release. For example, the django-geonames application has been upgraded to work with PostgreSQL and GIS support (proximity searches) has been added. In addition, I've also developed an application for geocoding and reverse geocoding using the Google Maps API and a oEmbed client (model-based instead of the tag based implementation which has been released by someone a few days ago). All of these applications are going to be released next week.

UPDATE Thanks to all who started geomicroblogging today!

I think I've finally found the perfect syntax for defining new serializations in your application code. You no longer will need to return the element name. Instead, it'll be taken from the function name, but you can override it via decorator. However, there's an exception to this rule. The default function will always return the object class name into lowercase. In addition, there's a new decorator for extending a serilization. Let's see how it looks:

from wapi import serializers

class UserProfileSerializer(serializers.BaseSerializer):
    @serializers.objname(name='my_friend')
        def friend(self, obj, *args, **kwargs):
        return {
            'username': obj.user.username,
            'avatar': 'http://bynotes.com/_m/avatars/%s' % obj.avatar,
        }   

    @serializers.objname('my_friend_location')
    @serializers.extends('friend')
    def friend_position(self, obj, *args, **kwargs):
         return {   
            'position': {
                'latitude': 5,
                'longitude': 9,
            }   
        }   

from notes.models import UserProfile
serializers.register(UserProfile, UserProfileSerializer)

As you can see, now serializations can be extended in the same way they are created and you won't need to write the object name again if it matches the function name.

On the other hand, I've finally decided to bundle this serializers as part of the (soon-to-be-released) wapi application, which provides a foundation for exposing web apis in an easy way.

I've made some progress on the serialization framework and I think it's ready for an alpha release. Here is the code, but I suggest you to finish reading this entry, since there are a few changes since the last post :).

In first place, I decided to remove backwards compatibility, because these serializers are totally different than the ones in Django. Django serializers are clearly designed for exchanging full objects between applications or dumping the object database into a file. They happen to be usable for exposing a web API, but that's not their main purpose.

On the other hand, these serializers are designed with a web API framework in mind. They are not intented for exchanging full objects, instead they intend to help you exposing data for your web API clients. That's a totally different purpose and I think both things don't mix well. In addition, I like the UNIX design philosophy, do one thing and do it right.

As for importing this code into Django, I think there's no problem keeping both serialization frameworks. In my humble opinion, these newserializers should replace the current serializers, because the former have a potentially wider audience. I don't find myself needing exchanging full objects between applications, however some people may need it (I could be totally wrong here, perhaps I'm the corner case). The current serializers could be renamed to something like exchangeserializers (I know I'm really bad at naming APIs).

Well, let's move to the code. There's a slight change in the user-defined serializers syntax. Following the suggestion by Ken, I've decided to return a tuple in the form of (object_name, object_data), because it seems the best solution for me. The decorator syntax could be nice, but there was a problem: the BaseSerializer class defines a default method, which works on any object you pass to serialize() and if the object name was in the method name every serialization using the default "preset" would wrap the object inside a container named default (not very nice in my opinion). So, a "preset" could be something like this:

import newserializers

class UserProfileSerializer(newserializers.BaseSerializer):
    def friend(self, obj, *args, **kwargs):
        return ('friend',
                {   
                    'username': obj.user.username,
                    'avatar': 'http://bynotes.com/_m/avatars/%s' % obj.avatar,
                }   
            )   

    def friend_position(self, obj, *args, **kwargs):
        data = self.friend(obj, *args, **kwargs)
        data[1].update({
            'position': {
                'latitude': 5,
                'longitude': 9,
            }   
        })  

        return data

from notes.models import UserProfile
newserializers.register(UserProfile, UserProfileSerializer)

Well, there are really two presets in this block, because I also wanted to demostrate how you can extend an existing preset. Note you could also subclass UserProfileSerializer, override any method (or define new ones, with or without calling the superclass) and register your new serializer class. Now let's see how the xml formatter would format this data:

serialize('xml', UserProfile.objects.all(), method='friend')

This uses the friend preset:

<?xml version='1.0' encoding='utf8'?>\n<objects><friend><username>manolo</username><avatar>http://bynotes.com/_m/avatars/None</avatar></friend><friend><username>fiam</username><avatar>http://bynotes.com/_m/avatars/None</avatar></friend></objects>

This adds the position field, extending friend using a new method:

serialize('xml', UserProfile.objects.all(), method='friend_position')

Which results in:

<?xml version='1.0' encoding='utf8'?>\n<objects><friend><username>manolo</username><position><latitude>5</latitude><longitude>9</longitude></position><avatar>http://bynotes.com/_m/avatars/None</avatar></friend><friend><username>fiam</username><position><latitude>5</latitude><longitude>9</longitude></position><avatar>http://bynotes.com/_m/avatars/None</avatar></friend></objects>

Let's see another example using the User class, which hasn't any registered serializers, so it gets the default serialization:

<?xml version='1.0' encoding='utf8'?>\n<objects><user><username>fiam</username><first_name /><last_name /><is_active>True</is_active><email>fiam@foo.bar</email><is_superuser>False</is_superuser><is_staff>False</is_staff><last_login>2008-06-13 16:53:13.076327</last_login><password>sha1$c262d$04067075864b8417d9719c6f2eb5fbb4ebe4fc8f</password><id>8</id><date_joined>2008-06-11 20:11:32.159342</date_joined></user><user><username>manolo</username><first_name /><last_name /><is_active>True</is_active><email>manolo@django.foo</email><is_superuser>False</is_superuser><is_staff>False</is_staff><last_login>2008-06-12 08:52:50.598065</last_login><password>sha1$d43c5$125e0e7f01971320b469a2f5b40ea99cca65c20c</password><id>7</id><date_joined>2008-06-11 02:42:07.366955</date_joined></user></objects>

Let's move to the formatters. As I mentioned in the previous entry, they're also pluggable and you can add your new formatters from your application code. I've already coded formatters for JSON, XML and Yaml, but any other one could be added easily. Here is the JsonFormatter as an example:

from newserializers import formatters
from django.utils import simplejson

from datetime import datetime

class JsonEncoder(simplejson.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return str(obj)
    return simplejson.JSONEncoder.default(self, obj)

class JsonFormatter(formatters.BaseFormatter):
    def start(self):
        self.buffer = ['[']

    def end(self):
        self.buffer.pop() # remove last comma
        self.buffer.append(']')
        self.data = ''.join(self.buffer)

    def format(self, data):
        self.buffer.append(simplejson.dumps(data[1], cls=JsonEncoder))
        self.buffer.append(',') 

    formatters.register('json', JsonFormatter)

The BaseFormatter class has a get() method which deals with either returning the string or writing to the file object passed to serialize(), so the formatters only need to write their output to self.data when the user calls end().

And finally, I've also added some subclasses of HttpResponse which get a QuerySet and take care of serializing and setting the mimetype. For example:

from newserializers.responses import JsonResponse

...
return JsonResponse(User.objects.all())

Please, keep posting comments, your feedback is helping me a lot.

The current Django serialization framework is not good enough for me. My current annoyances are:

• You can only serialize object fields. If you have defined some properties over the elements, you can't serialize them.
• You can not properly nest fields (e.g. serialize the user profile and include the username, you'll only get the foreign key)
• There's no way to configure some "presets". If I always serialize a class with the same fields, I need to write fields=('foo', 'bar') multiple times (where's DRY when you need it?).
• You cannot pass arguments to serializations. This could seem like something strange, but I find it usefull and others might do. E.g.: I'm currently developing a geomicroblogging site (yes, I've coined that term - 0 hits on Google right now -, think of it like Twitter+Pownce+FireEagle). One of the API endpoints returns where your friends are, but their locations depend on the viewer because you can send your position privately to just one of your friends (the rest will see the previous one).

Let me say this serialization framework is useful for passing raw objects between multiple applications understanding the format, since you can do deserialization on the data and get the exact object you serialized. But it doesn't work very well when you're trying to build a web API and you want more fine-grained control over the exposed data. I've been patching the serializers framework for a long time, but there are some limitations which can't be solved without a full redesign. So, I've came up with newserializers, a new serialization framework for Django.

I'm not a Django developer, so I don't currently have plans to import this in the SVN. However, if people find this framework useful, I'll consider enhancing it and talking to the Django devs about merging it.

Let's start from the design principles I've followed:

• Serialization should be done based on "presets".
• "Presets" should be easy to create, extendable and decoupled from the models and the views.
• Proper nesting
• Separation between the serialization (specifying which information is going to be serialized) and the formatting (converting the data to the output format).
• Adding new formatters from application code should be possible
• Backwards compatible.

I've hacked a 156 lines implementation of this in one hour. It currently works for both JSON and XML but it's not backwards compatible. However, as well as YAML support, it should be trivial to implement. In fact, I think both things should take no more than 50 lines if I reuse the code from the Django serializers, because the formatters are really short. (JSON is 18 lines, XML, 27).

Let's see an example of application code using this framework. In first place, you need to define the serialization "presets" in some place and execute that code on startup. For example, I've added a serializers.py to my application and I've put "from serializers import *" inside the app __init__.py. I like this layout, but you're free to define the "presets" in models.py or views.py if you prefer doing so. Here is a "preset" definition:

import newserializers

class UserProfileSerializer(newserializers.BaseSerializer):
    def friend(self, obj, *args, **kwargs):
        position = obj.position_for(kwargs.get('other'))
        return {
            'friend': {
                'username': obj.user.username,
                'avatar': 'http://bynotes.com/_m/avatars/%s' % obj.avatar,
                'position': {
                        'latitude': position.latitude,
                        'longitude': position.longitude,
                }   
            }   
        }   

from notes.models import UserProfile
newserializers.register(UserProfile, UserProfileSerializer)

Every method inside the UserProfileSerializer class defines a preset and every one of them returns a dictionary containing only dictionaries, lists, tuples and basic types (anything that can be transformed to string by smart_unicode). The only restriction is the first item in the dictionary must be another dictionary, because when serializing into XML you'll need a name for the full object (in this case it matches the method name, but they don't need to be equal). However, since JSON does not impose this restriction I'm thinking of setting the full object name using a decorator, so the JSON formatter can ignore it. For example:

import newserializers

class UserProfileSerializer(newserializers.BaseSerializer):
    @newserializers.object(name='close_friend') # if unspecified, defaults to func.__name__
    def friend(self, obj, *args, **kwargs):
        position = obj.position_for(kwargs.get('other'))
        return {
            'username': obj.user.username,
            'avatar': 'http://bynotes.com/_m/avatars/%s' % obj.avatar,
            'position': {
                'latitude': position.latitude,
                 'longitude': position.longitude,
            }   
        }   

from notes.models import UserProfile
newserializers.register(UserProfile, UserProfileSerializer)

The nice thing about this syntax is it lets you extends the presets easily:

class UserProfileAnotherSerializer(UserProfileSerializer):
    def other_method(self, obj, *args, **kwargs):
         return {
               'username': obj.user.username
         }

newserializers.register(UserProfile, UserProfileAnotherSerializer)

UserProfileAnotherSerializer will override UserProfileSerializer into the serializers registry, so you can add new methods or modify the existing ones.

And, finally, the syntax for calling this serializers is almost the same as for the ones in Django. However, the getserializer idiom is still not supported:

serialize('xml', UserProfile.objects.all(), method='other_method')
serialize('json', UserProfile.objects.all(), method='friend', other=other_profile)
serialize('json', UserProfile.objects.all()) #uses the method called 'default', which should call the old serializers

I'm still not posting the code for the framework, since it needs a bit of cleaning before I can publish it. But getting some feedback about the functionality and the syntax would help me to finish and publish it soon, so I'm waiting for your (always very appreciated) comments ;).