Utility scripts and tools for tsdat.
Project description
Tsdat Tools
This repository contains helpful scripts and notes for several tsdat-related tools.
Some tools are available as jupyter notebooks, and others are available as a command-line utility.
To get access to the command-line utilities, just run:
pip install tsdat-tools
To use all the other tools, we recommend cloning this repository.
Data to Yaml
The goal of this tool is to reduce the tediousness of writing tsdat configuration files for data that you can already
read and convert into an xr.Dataset
object in tsdat. It generates two output files: dataset.yaml
and
retriever.yaml
, which are used by tsdat
to define metadata and how the input variables should be mapped to output
variables.
If your file is in one of the following formats, this tool can already do this for you. Formats supported out-of-box:
netCDF
: Files ending with.nc
or.cdf
will use thetsdat.NetCDFReader
classcsv
: Files ending with.csv
will use thetsdat.CSVReader
classparquet
: Files ending with.parquet
or.pq
or.pqt
will use thetsdat.ParquetReader
classzarr
: Files/folders ending with.zarr
will use thetsdat.ZarrReader
class
Usage
Then you can run the tool with:
tsdat-tools data2yaml path/to/data/file --input-config path/to/current/dataset.yaml
Full usage instructions can be obtained using the --help
flag:
>>> tsdat-tools data2yaml --help
Usage: tsdat-tools data2yaml [OPTIONS] DATAPATH
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────╮
│ * datapath PATH Path to the input data file that should be used to generate tsdat configurations. │
│ [default: None] │
│ [required] │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────╮
│ --outdir DIRECTORY The path to the directory where │
│ the 'dataset.yaml' and │
│ 'retriever.yaml' files should be │
│ written. │
│ [default: .] │
│ --input-config PATH Path to a dataset.yaml file to be │
│ used in addition to │
│ configurations derived from the │
│ input data file. Configurations │
│ defined here take priority over │
│ auto-detected properties in the │
│ input file. │
│ [default: None] │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
This tool is designed to be run in the following workflow:
- Generate new ingest/pipeline from cookiecutter template (e.g.,
make cookies
command) - Put an example data file for your pipeline in the
test/data/input
folder - Clean up the autogenerated
dataset.yaml
file.- Add metadata and remove any unused variables
- Don't add additional variables yet; just make sure that the info in the current file is accurate
- Commit your changes in
git
or back up your changes so you can compare before & after the script runs. - Run this script, passing it the path to your input data file and using the
--input-config
option to tell it where your cleaneddataset.yaml
file is. By default this will generate a newdataset.yaml
file in the current working directory (location ofpwd
on the command line), but you can also use the--outdir
option to specify the path where it should write to. - Review the changes the script made to each file. Note that it is not capable of standardizing units or other metadata, so you will still need to clean those up manually.
- Continue with the rest of the ingest/pipeline development steps
Excel to Yaml
Please consult the documentation in the excel2yaml/README.md file for more information about this tool.
NetCDF to CSV
Please consult the documentation in the netcdf2csv/README.md file for more information about this tool.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tsdat_tools-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00bde37ecb7cffa92aa9cf28cb427298656c2002a4551c3324b890b374cca48a |
|
MD5 | 9501c834b56af643711f38a465be6af9 |
|
BLAKE2b-256 | 7aa6e6dae1ae536653a4787bd270eeed355f7718c6c6760125e1af41514b5033 |