This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Helpers for executing Django app code concurrently within Django tests.

Tested against the same versions of Python that Django supports:

x Py2.6 Py2.7 Py3.2 Py3.3 Py3.4 Py3.5
Django 1.4
       
Django 1.5
   
Django 1.6
   
Django 1.7  
 
Django 1.8  
Django 1.9  
   

Getting started

pip install django-concurrent-test-helper

Goes well with https://github.com/box/flaky (pip install flaky), as you may want to run a test several times while trying to trigger a rare race condition.

You need to add it to your Django project settings too:

INSTALLED_APPS = (
    # ...
    'django_concurrent_tests',
)

Two helpers are provided:

from django_concurrent_tests.helpers import call_concurrently

def is_success(result):
    return result is True and not isinstance(result, Exception)

def test_concurrent_code():
    results = call_concurrently(5, racey_function, first_arg=1)
    # results contains the return value from each call
    successes = list(filter(is_success, results))
    assert len(successes) == 1

and:

from django_concurrent_tests.helpers import make_concurrent_calls

def is_success(result):
    return result is True and not isinstance(result, Exception)

def test_concurrent_code():
    calls = [
        (first_func, {'first_arg': 1}),
        (second_func, {'other_arg': 'wtf'}),
    ] * 3
    results = make_concurrent_calls(*calls)
    # results contains the return value from each call
    successes = list(filter(is_success, results))
    assert len(successes) == 1

Note that if your called function raises an exception, the exception object will be returned by the concurrent test helper.

from django_concurrent_tests.helpers import make_concurrent_calls

def test_concurrent_code():
    calls = [
        (first_func, {'first_arg': 1}),
        (raises_error, {'other_arg': 'wtf'}),
    ] * 3
    results = make_concurrent_calls(*calls)
    # results contains the return value from each call
    errors = list(filter(lambda r: isinstance(r, Exception), results))
    assert len(errors) == 3

Another thing to remember is if you are using the override_settings decorator in your test. You need to also decorate your called functions (since the subprocesses won’t see the overridden settings from your main test process):

from django_concurrent_tests.helpers import make_concurrent_calls

@override_settings(SPECIAL_SETTING=False)
def test_concurrent_code():
    calls = [
        (first_func, {'first_arg': 1}),
        (raises_error, {'other_arg': 'wtf'}),
    ] * 3
    results = make_concurrent_calls(*calls)

@override_settings(SPECIAL_SETTING=False)
def first_func(first_arg):
    return first_arg * 2

def raises_error(other_arg):
    # can also be used as a context manager
    with override_settings(SPECIAL_SETTING=False):
        raise SomeError(other_arg)

Lastly, you can pass a string import path to a function rather than the function itself. The format is: 'dotted module.path.to:function' (NOTE colon separates the name to import, after the dotted module path).

This can be nice when you don’t want to import the function itself in your test to pass it. But more importantly it is essential in some cases, such as when f is a decorated function whose decorator returns a new object (and functools.wraps was not used). In that situation we will not be able to introspect the import path from the function object’s __module__ (which will point to the decorator’s module instead), so for those cases calling by string is mandatory.

from django_concurrent_tests.helpers import call_concurrently

@bad_decorator
def myfunc():
    return True

def test_concurrent_code():
    results = call_concurrently('mymodule.module:myfunc', 3)
    # results contains the return value from each call
    results = list(filter(None, results))
    assert len(results) == 3

NOTES

Why subprocesses?

We originally wanted to implement this purely using multiprocessing.Pool to call the function you want to test. If that had worked then this module would hardly be necessary.

Unfortunately we hit a problem with this approach: multiprocessing works by forking the parent process. The forked processes inherit the parent’s sockets, so in a Django project this will include things like the socket opened by psycopg2 to your Postgres database. However the inherited sockets are in a broken state. There’s a bunch of questions about this on SO and no solutions presented, it seems basically you can’t fork a Django process and do anything with the db afterwards.

(Note in Python 3 you may be able to use the ‘spawn’ start method of multiprocessing to avoid the fork problems - have not tried this)

So in order to make this work we have to use subprocess.Popen to run with un-forked ‘virgin’ processes. To be able to test an arbitrary function in this way we do an ugly/clever hack and provide a manage.py concurrent_call_wrapper command (which is why you have to add this module to your INSTALLED_APPS) which handles the serialization of kwargs and return values.

This does mean that your kwargs and return value must be pickleable.

Another potential gotcha is if you are using SQLite db when running your tests. By default Django will use :memory: for the test-db in this case. But that means the concurrent processes would each have their own in-memory db and wouldn’t be able to see data created by the parent test run.

For these tests to work you need to be sure to set TEST_NAME for the SQLite db to a real filename in your DATABASES settings (in Django 1.9 this is a dict, i.e. {'TEST': {'NAME': 'test.db'}}).

Finally you need to be careful with Django’s implicit transactions, otherwise data you create in the parent test has not yet been committed and is therefore not visible to the subprocesses.

Ensure that you use Django’s TransactionTestCase or a derivative (to prevent all the code in your test from being inside an uncommitted transaction).
Release History

Release History

0.2.8

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.7

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.6

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.4

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.10

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
django-concurrent-test-helper-0.2.8.tar.gz (10.8 kB) Copy SHA256 Checksum SHA256 Source Sep 5, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting