Library for downloading and preprocessing data
Project description
# BAS Download Toolbox
![GitHub issues](https://img.shields.io/github/issues/antarctica/download-toolbox?style=plastic) ![GitHub closed issues](https://img.shields.io/github/issues-closed/antarctica/download-toolbox?style=plastic) ![GitHub](https://img.shields.io/github/license/antarctica/download-toolbox) ![GitHub forks](https://img.shields.io/github/forks/antarctica/download-toolbox?style=social) ![GitHub forks](https://img.shields.io/github/stars/antarctica/download-toolbox?style=social)
This is the core python library for the download toolbox utilities used by BAS data pipelines.
This is only just getting started, more info will appear soon.
Contact jambyr <at> bas <dot> ac <dot> uk if you want further information.
## Table of contents
[Overview](#overview)
[Installation](#installation)
[Implementation](#implementation)
[Contributing](#contributing)
[Credits](#credits)
[License](#license)
## Installation
Not currently released to pip.
Please refer to [the contribution guidelines for more information.](CONTRIBUTING.rst)
## Implementation
When installed, the library will provide a series of CLI commands. Please use the –help switch for more initial information, or the documentation.
### Basic principles
The library sets up downloaders that will go through the following steps, for a variety of different data sources:
Set up a data store or if it exists, read the provenance config
Naively optimise the requested download
Download from the source in parallel
Transform the dataset into convenient to use files, ready for processing
That last step is important, as it might result in a different dataset to that which comes from source. The tool is intended to record this in the provenenace configuration, which is why it might exist in step (1), so that new data downloaded is consistent with what’s there - as well as the differences from the source data recorded for consistency (you should not be able to screw up existing datasets), posterity and reproducibility.
## Limitations
There are some major limitations to this as a general purpose tool, these will hopefully be dealt with in time! They likely don’t have issues related, yet.
Works only for hemisphere level downloading - north or south. The overhaul for this intends to ensure that identifiers are used so that someone can specify “north” or “south” but equally specify “Norway” or “The Shops” and then provide a geolocation that would identify the dataset within the filesystem.
This is currently very heavy development functionality, but the following downloaders should work:
download_amsr2
download_cmip
download_era5
download_osisaf
Other stubs probably don’t work, unless I forgot to update these docs!
## Contributing
Please refer to [the contribution guidelines for more information.](CONTRIBUTING.rst)
## Credits
<a href=”https://github.com/antarctica/download-toolbox/graphs/contributors”><img src=”https://contrib.rocks/image?repo=antarctica/download-toolbox” /></a>
## License
This is licensed using the [MIT License](LICENSE)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file download_toolbox-0.0.2.tar.gz
.
File metadata
- Download URL: download_toolbox-0.0.2.tar.gz
- Upload date:
- Size: 11.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40707755e1257a3452896ac1fd8b09bddd5d94c5398b5563fd1d7d3b8bd3f111 |
|
MD5 | a4253e7cc7f5ac3278f7bcfef265a4d6 |
|
BLAKE2b-256 | 61e9c6aab5f07b559f25c798e943f9ed3008321ec737339aed43363c891de1e6 |
File details
Details for the file download_toolbox-0.0.2-py3-none-any.whl
.
File metadata
- Download URL: download_toolbox-0.0.2-py3-none-any.whl
- Upload date:
- Size: 5.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a9250b4856768b50fe68675c7995583e9ca6e19794e0b1f94a999e4eaf6171f |
|
MD5 | b4c5d753e243de87b19ccf1d29e62020 |
|
BLAKE2b-256 | 0b097b4a3c117fcf56d526c8105e323794d9e9e62251bc53eec9187a62bed110 |