Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Download remote data (HTTP, FTP, SFTP) and store locally for data pipeline

Project description

fetch-data

Build Status Documentation Status codecov pypi License:MIT DOI

Download remote data (HTTP, FTP, SFTP) and store locally for data pipeline.

This package was created out of the frustration that it is very difficult to download data easily with intake. fetch-data is a mash-up of fsspec and pooch making it easy to download multiple files and store all the info, making it good for data pipeline applications.

Installation

Currently, this package is pip install git+https://github.com/lukegre/fetch-data.git

Basic usage

Use the download function directly:

flist = fd.download(url)

The file will be downloaded to the current directory and will be populated with a readme file, cached file list, and logging information.

Using with YAML catalogs

Use the catalog YAML entry

import fetch_data as fd
cat = fd.read_catalog(cat_fname)
flist = fd.download(**cat['entry_name'])

The catalog should be structured as shown below:

entry_name:
    url: remote path to file/s. Can contain *
    dest: where the file/s will be stored - can have optional {} placeholders that will be replaced
    meta:  # this will be written to the README file
        doi: url to the data source
        description: info about the data
        citation: how to cite this dataset
    placeholder: value  # optional will replace values in dest

<small>Project based on the cookiecutter science project template.</small>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fetch-data, version 0.2.4
Filename, size File type Python version Upload date Hashes
Filename, size fetch_data-0.2.4-py3-none-any.whl (14.4 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size fetch_data-0.2.4.tar.gz (26.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page