Skip to main content

Utility scripts and tools for tsdat.

Project description

Tsdat Tools

This repository contains helpful scripts and notes for several tsdat-related tools.

Some tools are available as jupyter notebooks, and others are available as a command-line utility.

To get access to the command-line utilities, just run:

pip install tsdat-tools

To use all the other tools, we recommend cloning this repository.

Data to Yaml

The goal of this tool is to reduce the tediousness of writing tsdat configuration files for data that you can already read and convert into an xr.Dataset object in tsdat. It generates two output files: dataset.yaml and retriever.yaml, which are used by tsdat to define metadata and how the input variables should be mapped to output variables.

If your file is in one of the following formats, this tool can already do this for you. Formats supported out-of-box:

  • netCDF: Files ending with .nc or .cdf will use the tsdat.NetCDFReader class
  • csv: Files ending with .csv will use the tsdat.CSVReader class
  • parquet: Files ending with .parquet or .pq or .pqt will use the tsdat.ParquetReader class
  • zarr: Files/folders ending with .zarr will use the tsdat.ZarrReader class

Usage

Then you can run the tool with:

tsdat-tools data2yaml path/to/data/file --input-config path/to/current/dataset.yaml

Full usage instructions can be obtained using the --help flag:

>>> tsdat-tools data2yaml --help

Usage: tsdat-tools data2yaml [OPTIONS] DATAPATH

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    datapath   PATH  Path to the input data file that should be used to generate tsdat configurations. │
│                       [default: None]                                                                   │
│                       [required]                                                                        │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────╮
│ --outdir                               DIRECTORY                      The path to the directory where   │
│                                                                       the 'dataset.yaml' and            │
│                                                                       'retriever.yaml' files should be  │
│                                                                       written.                          │
│                                                                       [default: .]                      │
│ --input-config                         PATH                           Path to a dataset.yaml file to be │
│                                                                       used in addition to               │
│                                                                       configurations derived from the   │
│                                                                       input data file. Configurations   │
│                                                                       defined here take priority over   │
│                                                                       auto-detected properties in the   │
│                                                                       input file.                       │
│                                                                       [default: None]                   │
│ --help                                                                Show this message and exit.       │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯

This tool is designed to be run in the following workflow:

  1. Generate new ingest/pipeline from cookiecutter template (e.g., make cookies command)
  2. Put an example data file for your pipeline in the test/data/input folder
  3. Clean up the autogenerated dataset.yaml file.
    • Add metadata and remove any unused variables
    • Don't add additional variables yet; just make sure that the info in the current file is accurate
  4. Commit your changes in git or back up your changes so you can compare before & after the script runs.
  5. Run this script, passing it the path to your input data file and using the --input-config option to tell it where your cleaned dataset.yaml file is. By default this will generate a new dataset.yaml file in the current working directory (location of pwd on the command line), but you can also use the --outdir option to specify the path where it should write to.
  6. Review the changes the script made to each file. Note that it is not capable of standardizing units or other metadata, so you will still need to clean those up manually.
  7. Continue with the rest of the ingest/pipeline development steps

Excel to Yaml

Please consult the documentation in the excel2yaml/README.md file for more information about this tool.

NetCDF to CSV

Please consult the documentation in the netcdf2csv/README.md file for more information about this tool.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tsdat_tools-0.3.0.tar.gz (50.6 kB view details)

Uploaded Source

Built Distribution

tsdat_tools-0.3.0-py3-none-any.whl (31.6 kB view details)

Uploaded Python 3

File details

Details for the file tsdat_tools-0.3.0.tar.gz.

File metadata

  • Download URL: tsdat_tools-0.3.0.tar.gz
  • Upload date:
  • Size: 50.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.5

File hashes

Hashes for tsdat_tools-0.3.0.tar.gz
Algorithm Hash digest
SHA256 7731b9a978e7711db2959fdbb16c46ef4f785176fd84a940f5dcf1cbb8bfd2a5
MD5 120bca0872608408b822bf12624104c4
BLAKE2b-256 1b7ff3522f1b32612dc9cfb85967d5f538a2e06d7f786580121c297b4e2289f3

See more details on using hashes here.

File details

Details for the file tsdat_tools-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: tsdat_tools-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 31.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.5

File hashes

Hashes for tsdat_tools-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 00bde37ecb7cffa92aa9cf28cb427298656c2002a4551c3324b890b374cca48a
MD5 9501c834b56af643711f38a465be6af9
BLAKE2b-256 7aa6e6dae1ae536653a4787bd270eeed355f7718c6c6760125e1af41514b5033

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page