Simple datastore batch writes in Google AppEngine

Google AppEngine has just came out of "Preview" today,  & changed the pricing model & quotas accordingly. A major change in the pricing model is that instead of charging for the total CPU time of DB operations, they are now charged by the number of DB operations. This means that instead of tunning the DB operations themselves, a much better approach will be to just batch them together (when possible).


The Pull Task Queue API of AppEngine is very useful for batching DB operations, which can be written to the queue & pulled reliably in batches for execution.


When working with Django over AppEngine (using the Django Nonrel framework), you employ Django's ORM, which doesn't yet support batch operations. How will you therefore implement this highly needed optimization?


A simple approach is to convert the Django model instances you want to write in a batch, back into AppEngine datastore instances, & use the datastore Put command to save them in 1 DB operation. I assembled code written by Vladimir Mihailenco into a small package which you can add to your project, & from it invoke the method: convert_to_gae_entity, providing it the Django instance you want to convert. A parameter allows you to allocate an ID for this instance in advance, in order to save related entities. Once you have the AppEngine entity, you can add it to a list, & write the list to the datastore using datastore Put command.



Hopefully, this will be integrated in Django Nonrel & Django soon.