Django integration with Google App Engine
Project description
# Djangae
The best way to run Django on Google App Engine.
Djangae (djan-gee) is a Django app that allows you to run Django applications on Google App Engine, including (if you
want to) using Django's models with the App Engine Datastore as the underlying database.
https://potatolondon.github.io/djangae/
https://github.com/potatolondon/djangae
**Note: Djangae is under heavy development, stability is not guaranteed. A 1.0 release will happen when it's ready**
## Features
* A WSGI middleware that provides a clean way via which your Django app is plugged into App Engine.
* A hook to allow App Engine's deferred tasks and mapreduce handlers to run through the same environment.
* The ability to use use the Datastore as the database for Django's models. See **The Database Backend** for details.
You can also use App Engine's NDB, or you can use Google Cloud SQL (via the standard django MySQL backend) instead of
or along side the Datastore. Or use all 3!
* `djangae.contrib.auth` which provides a custom user model, auth backend and middleware that makes django.contrib.auth
work on the datastore (i.e. without Many-To-Many relationships).
* A `runserver` command which fires up the App Engine SDK to serve your app (while still using Django's code reloading).
* The ability to run management commands locally or on the remote App Engine Datastore.
* A `shell` command that correctly sets up the environment/database. (Note, we should support this set up for any
custom commands as well, see TODO.md).
## Supported Django Versions
The intention is always to support the last two versions of Django, although older versions may work. Currently only
Django 1.6 is supported, but 1.7 support is in the pipeline.
# Installation
* Create a Django project, add app.yaml to the root. Make sure Django 1.6+ is in your project and importable
* Install Djangae into your project, make sure it's importable (you'll likely need to manipulate the path in manage.py and wsgi.py)
* Add djangae to `INSTALLED_APPS`.
* At the top of your settings, insert the following line: `from djangae.settings_base import *` - this sets up some
default settings.
* In app.yaml add the following handlers:
```yml
- url: /_ah/(mapreduce|queue|warmup).*
script: YOUR_DJANGO_APP.wsgi.application
login: admin
- url: /.*
script: YOUR_DJANGO_APP.wsgi.application
```
* Make your manage.py look something like this:
```python
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
from djangae.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
```
* Use the Djangae WSGI handler in your wsgi.py, something like
```python
from django.core.wsgi import get_wsgi_application
from djangae.wsgi import DjangaeApplication
application = DjangaeApplication(get_wsgi_application())
```
* Add the following to your URL handler: `url(r'^_ah/', include('djangae.urls'))`
* It is recommended that for improved security you add `djangae.contrib.security.middleware.AppEngineSecurityMiddleware` as the first
of your middleware classes. This middleware patches a number of insecure parts of the Python and App Engine libraries and warns if your
Django settings aren't as secure as they could be.
## The Database Backend
Previously, in order to use Django's ORM with the App Engine Datastore, django-nonrel was required, along with
djangoappengine. That's now changed. With Djangae you can use vanilla Django with the Datastore. Heavily inspired by
djangoappengine (thanks Waldemar!) Djangae provides an intelligent database backend that allows vanilla Django to be
used, and makes use of many of the Datastore's speed and efficiency features such as projection queries.
Here's the full list of magic:
* Database-level enforcement of `unique` and `unique_together` constraints.
* A transparent caching layer for queries which return a single result (`.get` or any query filtering on a unique field
or unique-together fields). This helps to avoid Datastore
[consistency issues](https://developers.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency_).
* Automatic creation of additional index fields containing pre-manipulated values, so that queries such as `__iexact`
work out of the box. These index fields are created automatically when you use the queries. Use
`settings.GENERATE_SPECIAL_INDEXES_DURING_TESTING` to control whether that automatic creation happens during tests.
* Support for queries which weren't possible with djangoappengine, such as OR queries using `Q` objects.
* A `ListField` which provides a "normal" django model field for storing lists (a feature of the Datastore).
## Roadmap
1.0-alpha
- Basic ancestor query support
- All tests either passing or throwing NotSupportedError in the testapp
- Configurable constraints
1.0-beta
- Support for 'infinite' descendents in the ancestor support. Lots of tests
- Memcache backed caching by PK and unique constraints
- Mapreduce handlers and utilities
- All NotSupportedError tests being skipped, everything passes in the testapp
### What Can't It Do?
Due to the limitations of the App Engine Datastore (it being a non-relational database for a start), there are some
things which you still can't do with the Django ORM when using the djangae backend. The easiest way to find these out
is to just build your app and look out for the `NotSupportedError` exceptions. But if you don't like surprises, here's
a quick list:
* `ManyToManyField` - a non-relational database simply can't do these (or not efficiently). However, you can probably
solve these kind of problems using djangae's `ListField`. We may even create a many-to-many replacement based on
that in the future.
* `__in` queries with more than 30 values. This is a limitation of the Datastore. You can filter for up to 500 values
on the primary key field though.
* More than one inequality filter, i.e. you can't do `.exclude(a=1, b=2)`. This is a limitation of the Datastore.
* Transactions. The Datastore has transactions, but they are not "normal" transactions in the SQL sense. Transactions
should be done using `google.appengine.api.datastore.RunInTransaction`.
### Other Considerations
When using the Datastore you should bear in mind its capabilities and limitations. While Djangae allows you to run
Django on the Datastore, it doesn't turn the Datastore into a relational database. There are things which the
datastore is good at (e.g. handling huge bandwidth of reads and writes) and things which it isn't good at
(e.g. counting). Djangae is not a substitute for knowing how to use the
[Datastore](https://developers.google.com/appengine/docs/python/datastore/).
## Local/remote management commands
If you set your manage.py up as described above, djangae will allow you to run management commands locally or
remotely, by specifying a `--sandbox`. Eg.
```
./manage.py --sandbox=local shell # Starts a shell locally (the default)
./manage.py --sandbox=remote shell # Starts a shell using the remote datastore
```
With no arguments, management commands are run locally.
## djangae.contrib.auth
This includes a custom user model, auth backend and middleware that makes django.contrib.auth work on the datastore.
To use, do the following:
- At the bottom of your settings.py add: from djangae.contrib.auth.settings import * (this sets up the auth backend,
login url and custom user model)
- Replace 'django.contrib.auth.middleware.AuthenticationMiddleware' with
'djangae.contrib.auth.middleware.AuthenticationMiddleware'
- Add 'djangae.contrib.auth' to INSTALLED_APPS probably after 'django.contrib.auth'
## Using other databases
You can use Google Cloud SQL or sqlite (locally) instead of or along side the Datastore.
Note that the Database backend and settings for the Datastore remain the same whether you're in local development on on
App Engine Production, djanagae switches between the SDK and the production datastore appropriately. However, with
Cloud SQL you will need to switch the settings yourself, otherwise you could find yourself developing on your
live database!
Here's an example of how your `DATABASES` might look in settings.py if you're using both Cloud SQL and the Datastore.
```python
from djangae.utils import on_production
DATABASES = {
'default': {
'ENGINE': 'djangae.db.backends.appengine'
}
}
if on_production():
DATABASES['sql'] = {
'ENGINE': 'django.db.backends.mysql',
'HOST': '/cloudsql/YOUR_GOOGLE_CLOUD_PROJECT:YOUR_INSTANCE_NAME',
'NAME': 'YOUR_DATABASE_NAME',
'USER': 'root',
}
else:
DATABASES['sql'] = {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'development.sqlite3'
}
```
See the Google documentation for more information on connecting to Cloud SQL via the
[MySQL client](https://developers.google.com/cloud-sql/docs/mysql-client) and from
[external applications](https://developers.google.com/cloud-sql/docs/external).
## Contributing
Contributions are accepted via pull request and will be reviewed as soon as possible. If you have access to master, please do not commit directly! Pull requests only!
The best way to run Django on Google App Engine.
Djangae (djan-gee) is a Django app that allows you to run Django applications on Google App Engine, including (if you
want to) using Django's models with the App Engine Datastore as the underlying database.
https://potatolondon.github.io/djangae/
https://github.com/potatolondon/djangae
**Note: Djangae is under heavy development, stability is not guaranteed. A 1.0 release will happen when it's ready**
## Features
* A WSGI middleware that provides a clean way via which your Django app is plugged into App Engine.
* A hook to allow App Engine's deferred tasks and mapreduce handlers to run through the same environment.
* The ability to use use the Datastore as the database for Django's models. See **The Database Backend** for details.
You can also use App Engine's NDB, or you can use Google Cloud SQL (via the standard django MySQL backend) instead of
or along side the Datastore. Or use all 3!
* `djangae.contrib.auth` which provides a custom user model, auth backend and middleware that makes django.contrib.auth
work on the datastore (i.e. without Many-To-Many relationships).
* A `runserver` command which fires up the App Engine SDK to serve your app (while still using Django's code reloading).
* The ability to run management commands locally or on the remote App Engine Datastore.
* A `shell` command that correctly sets up the environment/database. (Note, we should support this set up for any
custom commands as well, see TODO.md).
## Supported Django Versions
The intention is always to support the last two versions of Django, although older versions may work. Currently only
Django 1.6 is supported, but 1.7 support is in the pipeline.
# Installation
* Create a Django project, add app.yaml to the root. Make sure Django 1.6+ is in your project and importable
* Install Djangae into your project, make sure it's importable (you'll likely need to manipulate the path in manage.py and wsgi.py)
* Add djangae to `INSTALLED_APPS`.
* At the top of your settings, insert the following line: `from djangae.settings_base import *` - this sets up some
default settings.
* In app.yaml add the following handlers:
```yml
- url: /_ah/(mapreduce|queue|warmup).*
script: YOUR_DJANGO_APP.wsgi.application
login: admin
- url: /.*
script: YOUR_DJANGO_APP.wsgi.application
```
* Make your manage.py look something like this:
```python
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
from djangae.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
```
* Use the Djangae WSGI handler in your wsgi.py, something like
```python
from django.core.wsgi import get_wsgi_application
from djangae.wsgi import DjangaeApplication
application = DjangaeApplication(get_wsgi_application())
```
* Add the following to your URL handler: `url(r'^_ah/', include('djangae.urls'))`
* It is recommended that for improved security you add `djangae.contrib.security.middleware.AppEngineSecurityMiddleware` as the first
of your middleware classes. This middleware patches a number of insecure parts of the Python and App Engine libraries and warns if your
Django settings aren't as secure as they could be.
## The Database Backend
Previously, in order to use Django's ORM with the App Engine Datastore, django-nonrel was required, along with
djangoappengine. That's now changed. With Djangae you can use vanilla Django with the Datastore. Heavily inspired by
djangoappengine (thanks Waldemar!) Djangae provides an intelligent database backend that allows vanilla Django to be
used, and makes use of many of the Datastore's speed and efficiency features such as projection queries.
Here's the full list of magic:
* Database-level enforcement of `unique` and `unique_together` constraints.
* A transparent caching layer for queries which return a single result (`.get` or any query filtering on a unique field
or unique-together fields). This helps to avoid Datastore
[consistency issues](https://developers.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency_).
* Automatic creation of additional index fields containing pre-manipulated values, so that queries such as `__iexact`
work out of the box. These index fields are created automatically when you use the queries. Use
`settings.GENERATE_SPECIAL_INDEXES_DURING_TESTING` to control whether that automatic creation happens during tests.
* Support for queries which weren't possible with djangoappengine, such as OR queries using `Q` objects.
* A `ListField` which provides a "normal" django model field for storing lists (a feature of the Datastore).
## Roadmap
1.0-alpha
- Basic ancestor query support
- All tests either passing or throwing NotSupportedError in the testapp
- Configurable constraints
1.0-beta
- Support for 'infinite' descendents in the ancestor support. Lots of tests
- Memcache backed caching by PK and unique constraints
- Mapreduce handlers and utilities
- All NotSupportedError tests being skipped, everything passes in the testapp
### What Can't It Do?
Due to the limitations of the App Engine Datastore (it being a non-relational database for a start), there are some
things which you still can't do with the Django ORM when using the djangae backend. The easiest way to find these out
is to just build your app and look out for the `NotSupportedError` exceptions. But if you don't like surprises, here's
a quick list:
* `ManyToManyField` - a non-relational database simply can't do these (or not efficiently). However, you can probably
solve these kind of problems using djangae's `ListField`. We may even create a many-to-many replacement based on
that in the future.
* `__in` queries with more than 30 values. This is a limitation of the Datastore. You can filter for up to 500 values
on the primary key field though.
* More than one inequality filter, i.e. you can't do `.exclude(a=1, b=2)`. This is a limitation of the Datastore.
* Transactions. The Datastore has transactions, but they are not "normal" transactions in the SQL sense. Transactions
should be done using `google.appengine.api.datastore.RunInTransaction`.
### Other Considerations
When using the Datastore you should bear in mind its capabilities and limitations. While Djangae allows you to run
Django on the Datastore, it doesn't turn the Datastore into a relational database. There are things which the
datastore is good at (e.g. handling huge bandwidth of reads and writes) and things which it isn't good at
(e.g. counting). Djangae is not a substitute for knowing how to use the
[Datastore](https://developers.google.com/appengine/docs/python/datastore/).
## Local/remote management commands
If you set your manage.py up as described above, djangae will allow you to run management commands locally or
remotely, by specifying a `--sandbox`. Eg.
```
./manage.py --sandbox=local shell # Starts a shell locally (the default)
./manage.py --sandbox=remote shell # Starts a shell using the remote datastore
```
With no arguments, management commands are run locally.
## djangae.contrib.auth
This includes a custom user model, auth backend and middleware that makes django.contrib.auth work on the datastore.
To use, do the following:
- At the bottom of your settings.py add: from djangae.contrib.auth.settings import * (this sets up the auth backend,
login url and custom user model)
- Replace 'django.contrib.auth.middleware.AuthenticationMiddleware' with
'djangae.contrib.auth.middleware.AuthenticationMiddleware'
- Add 'djangae.contrib.auth' to INSTALLED_APPS probably after 'django.contrib.auth'
## Using other databases
You can use Google Cloud SQL or sqlite (locally) instead of or along side the Datastore.
Note that the Database backend and settings for the Datastore remain the same whether you're in local development on on
App Engine Production, djanagae switches between the SDK and the production datastore appropriately. However, with
Cloud SQL you will need to switch the settings yourself, otherwise you could find yourself developing on your
live database!
Here's an example of how your `DATABASES` might look in settings.py if you're using both Cloud SQL and the Datastore.
```python
from djangae.utils import on_production
DATABASES = {
'default': {
'ENGINE': 'djangae.db.backends.appengine'
}
}
if on_production():
DATABASES['sql'] = {
'ENGINE': 'django.db.backends.mysql',
'HOST': '/cloudsql/YOUR_GOOGLE_CLOUD_PROJECT:YOUR_INSTANCE_NAME',
'NAME': 'YOUR_DATABASE_NAME',
'USER': 'root',
}
else:
DATABASES['sql'] = {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'development.sqlite3'
}
```
See the Google documentation for more information on connecting to Cloud SQL via the
[MySQL client](https://developers.google.com/cloud-sql/docs/mysql-client) and from
[external applications](https://developers.google.com/cloud-sql/docs/external).
## Contributing
Contributions are accepted via pull request and will be reviewed as soon as possible. If you have access to master, please do not commit directly! Pull requests only!
