Skip to main content

Library for downloading and preprocessing data

Project description

# BAS Download Toolbox

![GitHub issues](https://img.shields.io/github/issues/antarctica/download-toolbox?style=plastic) ![GitHub closed issues](https://img.shields.io/github/issues-closed/antarctica/download-toolbox?style=plastic) ![GitHub](https://img.shields.io/github/license/antarctica/download-toolbox) ![GitHub forks](https://img.shields.io/github/forks/antarctica/download-toolbox?style=social) ![GitHub forks](https://img.shields.io/github/stars/antarctica/download-toolbox?style=social)

This is the core python library for the download toolbox utilities used by BAS data pipelines.

This is only just getting started, more info will appear soon.

Contact jambyr <at> bas <dot> ac <dot> uk if you want further information.

## Table of contents

  • [Overview](#overview)

  • [Installation](#installation)

  • [Implementation](#implementation)

  • [Contributing](#contributing)

  • [Credits](#credits)

  • [License](#license)

## Installation

Not currently released to pip.

Please refer to [the contribution guidelines for more information.](CONTRIBUTING.rst)

## Implementation

When installed, the library will provide a series of CLI commands. Please use the –help switch for more initial information, or the documentation.

### Basic principles

The library sets up downloaders that will go through the following steps, for a variety of different data sources:

  1. Set up a data store or if it exists, read the provenance config

  2. Naively optimise the requested download

  3. Download from the source in parallel

  4. Transform the dataset into convenient to use files, ready for processing

That last step is important, as it might result in a different dataset to that which comes from source. The tool is intended to record this in the provenenace configuration, which is why it might exist in step (1), so that new data downloaded is consistent with what’s there - as well as the differences from the source data recorded for consistency (you should not be able to screw up existing datasets), posterity and reproducibility.

## Limitations

There are some major limitations to this as a general purpose tool, these will hopefully be dealt with in time! They likely don’t have issues related, yet.

  • Works only for hemisphere level downloading - north or south. The overhaul for this intends to ensure that identifiers are used so that someone can specify “north” or “south” but equally specify “Norway” or “The Shops” and then provide a geolocation that would identify the dataset within the filesystem.

This is currently very heavy development functionality, but the following downloaders should work:

  • download_amsr2

  • download_cmip

  • download_era5

  • download_osisaf

Other stubs probably don’t work, unless I forgot to update these docs!

## Contributing

Please refer to [the contribution guidelines for more information.](CONTRIBUTING.rst)

## Credits

<a href=”https://github.com/antarctica/download-toolbox/graphs/contributors”><img src=”https://contrib.rocks/image?repo=antarctica/download-toolbox” /></a>

## License

This is licensed using the [MIT License](LICENSE)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

download_toolbox-0.0.2.tar.gz (11.3 kB view details)

Uploaded Source

Built Distribution

download_toolbox-0.0.2-py3-none-any.whl (5.0 kB view details)

Uploaded Python 3

File details

Details for the file download_toolbox-0.0.2.tar.gz.

File metadata

  • Download URL: download_toolbox-0.0.2.tar.gz
  • Upload date:
  • Size: 11.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.19

File hashes

Hashes for download_toolbox-0.0.2.tar.gz
Algorithm Hash digest
SHA256 40707755e1257a3452896ac1fd8b09bddd5d94c5398b5563fd1d7d3b8bd3f111
MD5 a4253e7cc7f5ac3278f7bcfef265a4d6
BLAKE2b-256 61e9c6aab5f07b559f25c798e943f9ed3008321ec737339aed43363c891de1e6

See more details on using hashes here.

File details

Details for the file download_toolbox-0.0.2-py3-none-any.whl.

File metadata

File hashes

Hashes for download_toolbox-0.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 7a9250b4856768b50fe68675c7995583e9ca6e19794e0b1f94a999e4eaf6171f
MD5 b4c5d753e243de87b19ccf1d29e62020
BLAKE2b-256 0b097b4a3c117fcf56d526c8105e323794d9e9e62251bc53eec9187a62bed110

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page