TL;DR

"""
Module mypackage/datasets.py
"""
import pooch

# Get the version string from your project. You have one of these, right?
from . import version


# Create a new friend to manage your sample data storage
GOODBOY = pooch.create(
    # Folder where the data will be stored. For a sensible default, use the default
    # cache folder for your OS.
    path=pooch.os_cache("mypackage"),
    # Base URL of the remote data store. Will call .format on this string to insert
    # the version (see below).
    base_url="https://github.com/myproject/mypackage/raw/{version}/data/",
    # Pooches are versioned so that you can use multiple versions of a package
    # simultaneously. Use PEP440 compliant version number. The version will be
    # appended to the path.
    version=version,
    # If a version as a "+XX.XXXXX" suffix, we'll assume that this is a dev version
    # and replace the version with this string.
    version_dev="master",
    # An environment variable that overwrites the path.
    env="MYPACKAGE_DATA_DIR",
    # The cache file registry. A dictionary with all files managed by this pooch.
    # Keys are the file names (relative to *base_url*) and values are their
    # respective SHA256 hashes. Files will be downloaded automatically when needed
    # (see fetch_gravity_data).
    registry={"gravity-data.csv": "89y10phsdwhs09whljwc09whcowsdhcwodcy0dcuhw"}
)
# You can also load the registry from a file. Each line contains a file name and
# it's sha256 hash separated by a space. This makes it easier to manage large
# numbers of data files. The registry file should be in the same directory as this
# module.
GOODBOY.load_registry("registry.txt")


# Define functions that your users can call to get back some sample data in memory
def fetch_gravity_data():
    """
    Load some sample gravity data to use in your docs.
    """
    # Fetch the path to a file in the local storae. If it's not there, we'll
    # download it.
    fname = GOODBOY.fetch("gravity-data.csv")
    # Load it with numpy/pandas/etc
    data = ...
    return data

About

Does your Python package include sample datasets? Are you shipping them with the code? Are they getting too big?

Pooch will manage downloading your sample data files over HTTP from a server and storing them in a local directory:

• Download a file only if it’s not in the local storage.

• Check the SHA256 hash to make sure the file is not corrupted or needs updating.

• If the hash is different from the registry, Pooch will download a new version of the file.

• If the hash still doesn’t match, Pooch will raise an exception warning of possible data corruption.

Projects using Pooch

• MetPy

• Verde

If you’re using Pooch, send us a pull request adding your project to the list.

Contacting Us

• Most discussion happens on Github. Feel free to open an issue or comment on any open issue or pull request.

• We have chat room on Gitter where you can ask questions and leave comments.

Contributing

License

This is free software: you can redistribute it and/or modify it under the terms of the BSD 3-clause License. A copy of this license is provided in LICENSE.txt.

Documentation for other versions

• Development (reflects the master branch on Github)

• Latest release

• v0.2.0

• v0.1.1

• v0.1