Fabric tasks for Django and Amazon Web Services
***********************************************
Required arrangements
---------------------
.. note:: `django_fabfile` package should be listed in your path. It's already
in place if your are installed it using `pip install django-fabfile`. But if
you are working within repository, then::
export PYTHONPATH=<path-to-folder-with-django_fabfile-package>
in your shell.
Pre-run configuration
---------------------
Django settings will be checked first if environment variable
`DJANGO_SETTINGS_MODULE` configured properly. If option was not configured
within Django settings, then its value will be taken from ``./fabfile.cfg``
file - copy-paste there rows that should be overriden from defaults
:download:`django_fabfile/fabfile.cfg.def <../django_fabfile/fabfile.cfg.def>`.
Django settings should be configured in following form (with preserving
sections case)::
FABFILE = {
'DEFAULT': {
'LOGGING_FOLDER': '.',
},
'Credentials': {
'AWS_ACCESS_KEY_ID': 'AKIAABCDEFGHIJKLMNOP',
'AWS_SECRET_ACCESS_KEY': 'ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLM',
},
}
.. note::
Boto config files http://code.google.com/p/boto/wiki/BotoConfig will be
examined at least - if ``[Credentials]`` or some other options was filled
neither in Django settings nor in ``./fabfile.cfg`` or :download:`django_fabfile/fabfile.cfg.def <../django_fabfile/fabfile.cfg.def>`.
Example of required AWS IAM Policy
----------------------------------
For using Fabric tasks you need AWS account with following permissions:
* create-snapshot::
{
"Statement": [
{
"Sid": "Stmt1304003463574",
"Action": [
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
* delete-snapshot::
{
"Statement": [
{
"Sid": "Stmt1306410750989",
"Action": [
"ec2:DescribeRegions",
"ec2:DescribeSnapshots",
"ec2:DeleteSnapshot"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
* reboot-instance::
{
"Statement": [
{
"Sid": "Stmt1312204628195",
"Action": [
"ec2:RebootInstances"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "Stmt1312276311113",
"Action": [
"ec2:DetachVolume"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
* rsync-snapshot::
{
"Statement": [
{
"Effect": "Allow",
"Action": [
"EC2:AttachVolume",
"EC2:CreateKeyPair",
"EC2:CreateVolume",
"EC2:DeleteKeyPair",
"EC2:DeleteVolume",
"EC2:DescribeAvailabilityZones",
"EC2:DescribeImages",
"EC2:DescribeKeyPairs",
"EC2:RunInstances",
"EC2:TerminateInstances"
],
"Resource": "*"
}
]
}
Backup script
-------------
.. note:: Instances and snaphots are filtered by tag "Earmarking" value
"production". This can be configured in settings - see "Pre-run
configuration" section above.
Following backup commands may be configured with crontab to run daily::
#!/bin/sh
VE=/home/backup-agent/.virtualenvs/backup
. $VE/bin/activate && \
fab --keepalive=60 -f $VE/lib/python2.6/site-packages/django_fabfile/backup.py \
backup_instances_by_tag \
trim_snapshots \
rsync_region:src_region_name=us-east-1,dst_region_name=eu-west-1 \
rsync_region:src_region_name=us-west-1,dst_region_name=eu-west-1 \
rsync_region:src_region_name=eu-west-1,dst_region_name=us-east-1
With 30 production instances it tooks around 6 hours to replicate
snapshots updates. First replication tooks much more time obviously.
Running tests
-------------
The package has test cases to make sure, that all methods, being tested,
keep on the entire program logic. The test cases are using packages ``fudge``
and ``unittest``. The test files (``'test_*.py'``) are located in folder
``'/django_fabfile/tests/'``. To run tests from command line, implement::
python -m unittest discover django_fabfile
Recent changes
--------------
See :doc:`changelog` for recent updates.
Licensing
---------
The product is licensed by oDesk http://www.odesk.com/ under
:download:`GNU GENERAL PUBLIC LICENSE Version 3<license.txt>` except
portions with attached inline license information like
:func:`django_fabfile.backup._trim_snapshots`.
