datasluice 0.2.2

One Python interface for open-data discovery, extraction, format normalization, and pipeline integration

⚠️ Unstable — under active development. DataSluice is pre-1.0 and evolving fast. Breaking changes may occur at any time without notice. Use it at your own risk.

• GitHub | PyPI | Documentation
• Created by Nitish Raj | GitHub @nitish-raj | PyPI @nitish-raj
• MIT License

Installation

pip install datasluice

Optional extras for format and integration support:

pip install "datasluice[pandas,polars,parquet,xlsx]"
pip install "datasluice[all]"          # everything

Apache Airflow

Airflow integration is a separate distribution that imports from the airflow.providers.datasluice namespace:

pip install apache-airflow-providers-datasluice

Quick Start

from datasluice import DataSluice

# Point at any supported portal — the portal type is auto-detected
ds = DataSluice("https://www.data.gouv.fr")

# Search for datasets
results = ds.search("climate")
for dataset in results:
    print(dataset.title, len(dataset.resources))

# Inspect a dataset and list its resources
dataset = ds.get_dataset("some-dataset-id")
for resource in dataset.resources:
    print(resource.format, resource.url)

# Download a resource
path = ds.download(dataset.resources[0], "data/")
print(path)

CLI:

datasluice search "climate" --portal https://www.data.gouv.fr
datasluice inspect -p https://www.data.gouv.fr <dataset-id>
datasluice detect https://demo.ckan.org
datasluice download -p https://www.data.gouv.fr <dataset-id> --format csv

Features

• Unified API — one interface for CKAN, data.gouv.fr, Socrata, and custom portals
• Auto-detection — point at a URL and DataSluice figures out the portal type
• Format normalization — CSV, JSON, XLSX, Parquet, and GeoJSON readers
• Integrations — pandas, Polars, dlt, DuckDB, and Apache Airflow (separate provider)
• CLI — search, inspect, download, and detect from the command line
• Pipeline-ready — retry, rate-limiting, caching, and checksum verification built in

Documentation

Documentation is built with Zensical and deployed to GitHub Pages.

• Live site: https://nitish-raj.github.io/datasluice/
• Preview locally: just docs-serve (serves at http://localhost:8000)
• Build: just docs-build

API documentation is auto-generated from docstrings using mkdocstrings.

Docs deploy automatically on push to main via GitHub Actions. To enable this, go to your repo's Settings > Pages and set the source to GitHub Actions.

Development

To set up for local development:

# Clone your fork
git clone git@github.com:your_username/datasluice.git
cd datasluice

# Install dependencies (including all optional deps for dev)
uv sync --all-extras

# Install just (task runner) — one-time setup
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to .venv/bin

# Install in editable mode with live updates
uv tool install --editable .

This installs the CLI globally but with live updates - any changes you make to the source code are immediately available when you run datasluice.

Install pre-commit hooks:

uv run pre-commit install

Run tests:

uv run pytest

Run quality checks (format, lint, type check, test):

just qa

Release Process

Releases are automated with Release Please. There is no manual version bumping or tagging.

1. Use Conventional Commits (feat:, fix:, docs:, …) — see CONTRIBUTING.md for the full list.
2. Release Please maintains a release PR on main that bumps the version and updates the changelog.
3. Merge the release PR → Release Please creates a Git tag and a GitHub Release.
4. The GitHub Release auto-triggers publishing to TestPyPI, then waits for approval before publishing to PyPI.

Contributing

Contributions are welcome! See CONTRIBUTING.md for setup, conventions, and the release workflow. Please follow the Code of Conduct.