easy-dataset-share 0.4.0

CLI tool to responsibly share datasets by gzipping, canarying, and tracking provenance.

Easy Dataset Share

easy-dataset-share helps AI researchers share datasets responsibly. It prevents evaluation contamination by making datasets easy for researchers to use but hard for automated scrapers to ingest.

The easy-dataset-share CLI tool provides basic protection against scraping by making the dataset text itself less scrapeable.

However, sophisticated actors will still be able to scrape your content. Rather than providing unsophisticated further defenses which inconvenience real users, we think you should outsource that defense to a provider like CloudFlare. We wrote an easy tutorial on signing up with CloudFlare Turnstile, which is like CAPTCHA but a. actually effective and b. doesn't inconvenience your real users. See WEB_HOSTING_GUIDE.md

Features

In easy-data-share we include features for:

• Canary markers: Unique identifiers to detect if your dataset was used for training.
• Hash verification: Ensuring the process of adding + removing canaries does not alter the dataset through hashing before and after the protection process.
• Protection layers: Zipping the data to make it not readable in plaintext by basic crawlers. Optional encryption as well.
• Default Best Practices: We generate a robots.txt and Terms of Service which prohibits use for AI training.
• Clean removal: Removing all protection while preserving original data (use hash verification to confirm data-integrity).
• Web hosting (optional): Deploying a protected download site with Cloudflare-Turnstile (a CAPTCHA-replacement)- see WEB_HOSTING_GUIDE.md

Installation

pip install easy-dataset-share

Quick Start

Protect a dataset

easy-dataset-share protect-dir /path/to/dataset

Unprotect and clean

easy-dataset-share unprotect-dir dataset.zip --remove-canaries

Verify integrity

easy-dataset-share hash /path/to/dataset

Options

Run easy-dataset-share --help or for a subcommand easy-dataset-share protect-dir --help. To get a description of the options available.

How it Works

1. Hash original dataset for integrity baseline
2. Add canary markers throughout the dataset
3. Package with robots.txt and optional encryption
4. Verify integrity when unprotecting (canaries removed, data unchanged)

Example Workflow

# Protect with a password
easy-dataset-share protect-dir my_dataset

# Share dataset.zip publicly

# Recipients unprotect and remove canaries
easy-dataset-share unprotect-dir dataset.zip --remove-canaries
# Output shows: "📊 Dataset hash: abc123..." (matches original)

Use -p your-password to add a password when you zip (and for others to use when they unzip - will now be .zip.enc) Use -v for verbose output to see hashing details and canary operations.

Notes about licensing for Robots.txt (Text & Data Mining (TDM) opt-out)

The robots.txt generated by this helper add TDM protections, see EDRlab for more information.

Hosting with Anti-Scraper Protection

For datasets hosted outside of Hugging Face, we strongly recommend using Cloudflare Turnstile to add an additional layer of protection against automated AI scrapers. See WEB_HOSTING_GUIDE.md for a guide on how to do this.

This layered approach (dataset protection + hosting protection) provides comprehensive defense against automated data harvesting while maintaining accessibility for legitimate researchers.

Maintainence + Development

This is meant to be a collaborative and community project. Please feel encouraged to make PRs to update this repo!

For development:

git clone https://github.com/Responsible-Dataset-Sharing/easy-dataset-share.git
cd easy-dataset-share
pip install -e .
git config core.hooksPath .githooks

Current Maintainers

• Roy Rinberg [e-mail]
• Dipika Khullar [e-mail]
• Edward Turner [e-mail]

Acknowledgements

This project was kickstarted by Alex Turner and then funded by the following supporters:

• Alex Turner ($500) - https://turntrout.com/
• Anna Wang ($500) - https://www.linkedin.com/in/annawang01/
• James Aung ($500) - https://jamesaung.com/
• Girish Sastry ($1000) - https://www.linkedin.com/in/girish-sastry-2a39348/