Skip to main content

export pleanrio data sets to s3

Project description


This is both a simple application and a rather complicated application. The actions it performs are straight forward - create a job to export a data set, zip up the generated CSVs, upload them to Amazon S3, and email the requestor.

The application uses django-channels: it’s a new asynchronous back end for Django, thus relieving us of the hassle of running separate instances for celery and flower. It has its own baggage though - in production it has its own server environment and relies on Redis for message passing.

Overall, it’s pretty snappy and I’m confident that channels is the future of Django, and not celery.

To wire this up, it’s a relatively simple install:

# your site/
    'whatever django and local stuff',
    'channels',  # necessary to make exporter go
    'plenario_exporter_s3',  # this app

    'default': {
        'BACKEND': 'asgi_redis.RedisChannelLayer',
        'CONFIG': {
            'hosts': [os.environ.get('REDIS_URL', 'redis://localhost:6379')],
        'ROUTING': 'plenario_exporter_s3.routing.channel_routing',

You’re also going to need an file alongside your wsgi file:

import os
import channels.asgi

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your app.settings')
channel_layer = channels.asgi.get_channel_layer()

And in your views, wire up the provided export service function as the async_handler argument to the plenario-core export view:

from plenario_core.views.export import GenericMetaExportView
from import create_models_and_start_job

from .models import EtlEventMeta

export_meta = GenericMetaExportView.as_view(


Fire up a virtualenv and install the dev requirements:

$ python3.6 -m venv .env
$ source .env/bin/activate
$ pip install -r dev-requirements.txt

To run the tests, in a separate terminal pull in the PostGIS docker image and create the database:

$ docker pull mdillon/postgis
$ docker run -d -p 5432:5432 mdillon/postgis
$ docker ps
$ docker exec -it {container hash} /bin/bash
# su postgres -c psql
> create database plenario;

You’re also going to need a local Redis server running:

$ docker pull redis
$ docker run -d -p 6379:6379 redis

Then all you have to do is run the tests normally:

$ coverage run test
$ coverage report
$ flake8

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for plenario-exporter-s3, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size plenario_exporter_s3-0.0.4-py3-none-any.whl (21.2 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page