Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

DataSure

DataSure is IPA's Data Management System Dashboard - a comprehensive tool for survey data quality monitoring and high-frequency checks (HFCs) in research projects.

Built for data managers, survey coordinators, and research teams, DataSure provides real-time monitoring of survey data quality with interactive dashboards, automated checks, and flexible reporting capabilities.

Key Features

  • 📊 Data Quality Monitoring: Real-time dashboards for comprehensive survey data analysis
  • 🔍 Automated Checks: 10 specialized quality check modules including duplicates, outliers, GPS validation, and missing data analysis
  • 📈 Interactive Visualizations: Charts and maps for data exploration and quality assessment
  • 🔗 Multi-Source Integration: Direct SurveyCTO API connection plus CSV/Excel file support
  • ⚙️ Flexible Configuration: Project-based settings with customizable check parameters
  • 📋 Comprehensive Reporting: Export capabilities for different audiences and formats
  • 🎯 Enumerator Performance: Monitor data collection team productivity and quality metrics

Installation

Step 1: Install uv from terminal

# WINDOWS
winget install astral-sh.uv

# MACOS/LINUX
brew install uv

Step 2: Install datasure with uv

# install
uv tool install datasure

# ON WINDOWS: update windows path after installation
uv tool update-shell 

Step 3: verify installation

datasure --version

Getting the latest release

# if datasure is already install, get latest version with
uv tool upgrade datasure

Quick Start

  1. Launch the application:

    datasure
    
  2. Create your first project and configure data quality checks

  3. Import survey data:

    • Connect directly to your SurveyCTO server
    • Upload CSV or Excel files from local storage
  4. Monitor data quality with interactive dashboards organized into specialized check modules

  5. Generate reports and export results for your research team

System Requirements

  • Python: Version 3.11 or higher
  • Operating System: Windows, macOS, or Linux
  • Memory: Minimum 4GB RAM (8GB recommended for large datasets)
  • Storage: 1GB free space for application and data cache
  • Internet: Required for SurveyCTO integration and updates

Data Quality Check Modules

DataSure includes 10 specialized modules for comprehensive survey data quality monitoring:

Module Purpose
Summary Overall project progress and completion tracking
Missing Data Identify patterns in incomplete responses
Duplicates Find and manage duplicate survey entries
GPS Validation Verify location data accuracy with interactive maps
Outliers Identify unusual responses requiring review
Enumerator Performance Monitor data collection team productivity
Progress Tracking Real-time survey completion monitoring
Descriptive Statistics Data distribution analysis and summaries
Back-checks Verification workflow support
Custom Checks Configure additional quality checks per project

Core Capabilities

Data Import and Management

  • SurveyCTO Integration: Direct API connection with form metadata and authentication
  • Local File Support: CSV and Excel upload with automatic type detection
  • Multi-Project Organization: Manage multiple surveys simultaneously
  • Data Preparation: Cleaning and transformation workflows

Interactive Dashboards

  • Real-time Monitoring: Live updates as new data arrives
  • Customizable Views: Configure dashboards per project requirements
  • Export Options: Generate reports in PDF, Excel, and other formats
  • Automated Alerts: Notifications for quality issues requiring attention

Performance and Scalability

  • High-Performance Processing: DuckDB backend for fast analytical queries
  • Large Dataset Support: Optimized for datasets with hundreds of thousands of records
  • Intelligent Caching: Reduces processing time and API calls
  • Cross-Platform Compatibility: Works on Windows, macOS, and Linux

Getting Started - Application Usage

Using DataSure

Once DataSure is installed, you can begin monitoring your survey data quality:

1. Launch the Application

datasure

The web interface will open in your default browser (typically at http://localhost:8501).

2. Import Data

  • Import Data Page: Start here to connect your data sources
  • SurveyCTO Integration: Connect directly to your SurveyCTO server with authentication
  • Local Files: Upload CSV or Excel files from your computer
  • Multiple Datasets: Import and manage up to 10 datasets per project

3. Prepare and Configure

  • Prepare Data Page: Preview your imported datasets in separate tabs
  • Configure Checks Page: Set up High-Frequency Checks (HFCs)
    • Enter a page name for your quality monitoring dashboard
    • Select the dataset to analyze
    • Configure check parameters and thresholds
    • Save settings to create your HFC page

4. Monitor Data Quality

  • HFC Dashboard: Access your configured quality check page
  • Interactive Tabs: Each check type has its own tab (Summary, Missing Data, Duplicates, etc.)
  • Settings Expanders: Configure specific parameters for each check
  • Real-time Updates: Dashboard refreshes as new data becomes available

5. Export and Share

  • Generate reports for different audiences
  • Export findings in various formats
  • Monitor trends over time

Command Line Options

# Show version information
datasure --version

# Launch with custom host/port  
datasure --host 0.0.0.0 --port 8080

# View all available options
datasure --help

Data Storage and Cache

DataSure automatically manages data storage and caching for optimal performance:

Cache Directory Locations

  • Development Mode: ./cache/ (in project root)
  • Production Mode:
    • Windows: %APPDATA%/datasure/cache/
    • Linux/macOS: ~/.local/share/datasure/cache/

What's Stored

  • Project configurations: HFC page settings and form configurations
  • Database files: DuckDB databases for processed survey data
  • SurveyCTO cache: Cached form metadata and server connections
  • User settings: Check configurations and preferences

Cache directories are created automatically - no manual setup required.

Support and Resources

Getting Help

Version Information

  • Current Version: See RELEASENOTES.md for the latest release information
  • Version History: Track all changes and improvements
  • Upgrade Instructions: Follow installation commands above to get the latest version

Contributing

We welcome contributions from the research community! DataSure is developed by Innovations for Poverty Action (IPA) with input from data managers and survey coordinators worldwide.

Ways to Contribute

  • Report Issues: Found a bug or have a feature request? Open an issue
  • Suggest Features: Share ideas for new data quality checks or workflow improvements
  • Share Use Cases: Help us understand how DataSure fits into different research workflows
  • Code Contributions: Developers can contribute code improvements and new features

For Developers

If you're interested in contributing code or setting up a development environment, see our comprehensive CONTRIBUTING.md guide which includes:

  • Development environment setup
  • Code quality standards and testing requirements
  • Package building and distribution workflows
  • Release process and documentation guidelines
  • Technical architecture and development patterns

Community Standards

  • Use clear, descriptive language when reporting issues
  • Follow our code of conduct and treat all contributors with respect
  • Help create a welcoming environment for researchers and developers from all backgrounds

Authors and Acknowledgments

DataSure is developed and maintained by the Global Research & Data Science (GRDS) team at Innovations for Poverty Action (IPA). Contact GRDS at researchsupport@poverty-action.org.

Core Development Team

License and Contact


DataSure - Ensuring data quality for better research outcomes.

Metadata

Release files for DataSure 0.6.2rc1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for DataSure 0.6.2rc1
File Size Uploaded Provenance Actions
datasure-0.6.2rc1.tar.gz 1.1 MB Without provenance Details for datasure-0.6.2rc1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for DataSure 0.6.2rc1
File Interpreter ABI Platform Actions
datasure-0.6.2rc1-py3-none-any.whl Python 3 none any Details for datasure-0.6.2rc1-py3-none-any.whl

Total release size: 2.2 MB

Release files / datasure-0.6.2rc1.tar.gz

Download URL datasure-0.6.2rc1.tar.gz
Size 1.1 MB
Tags Source
SHA-256 checksum
How to use checksums
5184e48fc0bbdb1d305fbfcba1877d8b01892b3f43afb75e2d69dcb433f5fa31
BLAKE2b-256 checksum
How to use checksums
f3f6b7ccc45923c6a9cad1325e091bf12f60283d32101dad0a41099991158850
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.25 {"installer":{"name":"uv","version":"0.9.25","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / datasure-0.6.2rc1-py3-none-any.whl

Download URL datasure-0.6.2rc1-py3-none-any.whl
Size 1.1 MB
Tags Python 3
SHA-256 checksum
How to use checksums
a933d83b668a0bcea60281770db5c018e79c95af5553c26987b703e5660a6865
BLAKE2b-256 checksum
How to use checksums
424f0f370c246b3b0a251395743cebf27d4706b0aaa13dbe9a1b095773939fa1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.25 {"installer":{"name":"uv","version":"0.9.25","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for DataSure 0.6.2rc1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page