surveydata 0.1.13

Flexible access to survey data

The surveydata Python package offers flexible access to survey data and support for multiple local and cloud storage options.

Installation

Installing the latest version with pip:

pip install surveydata

Overview

To use the surveydata package, you access data from specific survey platforms via an appropriate SurveyPlatform object:

• SurveyCTOPlatform provides support for SurveyCTO data, including methods to process text audits and submit submission updates via the review and correction workflow (in support of SurveyCTO’s machine learning roadmap, with the ml4qc project)

• ODKPlatform provides support for Open Data Kit data via an ODK Central server

All survey data must be stored somewhere, and storage is handled via an appropriate StorageSystem object:

• FileStorage provides support for local file storage

• S3Storage provides support for AWS S3 storage

• DynamoDBStorage provides support for AWS DynamoDB storage

• GoogleCloudStorage provides support for Google Cloud Storage

• AzureBlobStorage provides support for Azure Blob Storage

• SurveyCTOExportStorage provides support for local data exported with SurveyCTO Desktop (in wide format)

• ODKExportStorage provides support for local data downloaded and unzipped from an ODK Central All data and Attachments export

In general, the workflow goes like this:

1. Initialize the survey platform

2. Initialize one or more storage systems

3. Synchronize data between the survey platform and the storage system(s) to ensure that data in storage is fully up-to-date (except for static export storage, via SurveyCTOExportStorage or ODKExportStorage, which doesn’t require synchronization)

4. Load data and/or attachments via the survey platform and storage API’s

5. Optionally: Save processed data and then, later, load it back again, for cases where ingestion and processing tasks are separated from actual analysis or use

Examples

See these notebooks for detailed usage examples:

• SurveyCTO usage examples

• ODK usage examples

Documentation

See the full reference documentation here:

https://surveydata.readthedocs.io/

Development

To develop locally:

1. git clone https://github.com/orangechairlabs/py-surveydata.git

2. cd py-surveydata

3. python -m venv venv

4. source venv/bin/activate

5. pip install -r requirements.txt

For convenience, the repo includes .idea project files for PyCharm.

To rebuild the documentation:

1. Update version number in /docs/source/conf.py

2. Update layout or options as needed in /docs/source/index.rst

3. In a terminal window, from the project directory:

   1. cd docs

   2. SPHINX_APIDOC_OPTIONS=members,show-inheritance sphinx-apidoc -o source ../src/surveydata --separate --force

   3. make clean html

To rebuild the distribution packages:

1. For the PyPI package:

   1. Update version number (and any build options) in /setup.py

   2. Confirm credentials and settings in ~/.pypirc

   3. Run /setup.py for the bdist_wheel and sdist build types (Tools… Run setup.py task… in PyCharm)

   4. Delete old builds from /dist

   5. In a terminal window:

      1. twine upload dist/* --verbose

2. For GitHub:

   1. Commit everything to GitHub and merge to main branch

   2. Add new release, linking to new tag like v#.#.# in main branch

3. For readthedocs.io:

   1. Go to https://readthedocs.org/projects/surveydata/, log in, and click to rebuild from GitHub (only if it doesn’t automatically trigger)