Skip to main content

A library for common scientific model transforms

Project description

mixmasta

Python Tests

A library for common scientific model transforms. This library enables fast and intuitive transforms including:

  • Converting a geotiff to a csv
  • Converting a NetCDF to a csv
  • Geocoding csv, xls, and xlsx data that contains latitude and longitude

Setup

See docs/docker.md for instructions on running Mixmasta in Docker (easiest!).

Ensure you have a working installation of GDAL

You also need to ensure that numpy is installed prior to mixmasta installation. This is an artifact of GDAL, which will build incorrectly if numpy is not already configured:

pip install numpy==1.20.1
pip install mixmasta

Note: if you had a prior installation of GDAL you may need to run pip install mixmasta --no-cache-dir in a clean environment.

You must install the GADM2 and GADM3 data with:

mixmasta download

Usage

Examples can be found in the input directory.

Convert a geotiff to a dataframe with:

from mixmasta import mixmasta as mix
df = mix.raster2df('chirps-v2.0.2021.01.3.tif', feature_name='rainfall', band=1)

Note that you should specify the data band of the geotiff to process if it is multi-band. You may also specify the name of the feature column to produce. You may optionally specify a date if the geotiff has an associated date. For example:

Convert a NetCDF to a dataframe with:

from mixmasta import mixmasta as mix
df = mix.netcdf2df('tos_O1_2001-2002.nc')

Geocode a dataframe:

from mixmasta import mixmasta as mix

# First, load in the geotiff as a dataframe
df = mix.raster2df('chirps-v2.0.2021.01.3.tif', feature_name='rainfall', band=1)

# next, we can geocode the dataframe to the admin-level desired (`admin2` or `admin3`)
# by specifying the names of the x and y columns
# in this case, we will geocode to admin2 where x,y are are 'longitude' and 'latitude', respectively.
df_g = mix.geocode("admin2", df, x='longitude', y='latitude')

Running with CLI

After cloning the repository and changing to the mixmasta directory, you can run mixmasta via the command line.

Set-up:

While you can point mixmasta to any file you would like to transform, the examples below assume your file is in the inputs folder; the transformed .csv file will be written to the outputs folder.

  • Transform geotiff to geocoded csv:
mixmasta mix --xform=geotiff --input_file=chirps-v2.0.2021.01.3.tif --output_file=geotiffTEST.csv --geo=admin2 --feature_name=rainfall --band=1 --date='5/4/2010' --x=longitude --y=latitude
  • Transform geotiff to csv:
mixmasta mix --xform=geotiff --input_file=maxhop1.tif --output_file=maxhopOUT.csv --geo=admin2 --feature_name=probabilty --band=1 --x=longitude --y=latitude
  • Transform netcdf to geocoded csv:
mixmasta mix --xform=netcdf --input_file=tos_O1_2001-2002.nc --output_file=netcdf.csv --geo=admin2 --x=lon --y=lat
  • Transform netcdf to csv:
mixmasta mix --xform=netcdf --input_file=tos_O1_2001-2002.nc --output_file=netcdf.csv

-geocode an existing csv file:

mixmasta mix --xform=geocode --input_file=no_geo.csv --geo=admin3 --output_file=geoed_no_geo.csv --x=longitude --y=latitude

World Modelers Specific Normalization

For the World Modelers program, it is necessary to convert arbitrary csv, geotiff, and netcdf files into a CauseMos compliant format. This can be accomplished by leveraging a mapping annotation file and the causemosify command. The output is a gzipped parquet file. This may be invoked with:

mixmasta causemosify --input_file=chirps-v2.0.2021.01.3.tif --mapper=mapper.json --geo=admin3 --output_file=causemosified_example

This will produce a file called causemosified_example.parquet.gzip which can be read using Pandas with:

pd.read_parquet('causemosified_example.parquet.gzip')

Other Documents

  • Docker Instructions: docs/docker.md
  • Geo Entity Resolution Description: docs/geo-tentity-resolution.md
  • Package Testing in SpaceTag Env: docs/spacetag-test.md

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.0 (2021-02-24)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mixmasta-0.6.9.tar.gz (2.9 MB view details)

Uploaded Source

Built Distribution

mixmasta-0.6.9-py2.py3-none-any.whl (32.0 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file mixmasta-0.6.9.tar.gz.

File metadata

  • Download URL: mixmasta-0.6.9.tar.gz
  • Upload date:
  • Size: 2.9 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.9.2 requests/2.28.1 setuptools/57.5.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.8.10

File hashes

Hashes for mixmasta-0.6.9.tar.gz
Algorithm Hash digest
SHA256 20d4f27c46732b1b606e18aad255612b91030968067bbdb8fccf776a3f2e81c3
MD5 39c698c50ad9db243e3b0e7216ffdedf
BLAKE2b-256 4221cf29f591d0a0fa76e0f4ad46febc7e7a63dcd84251054513b67d2531c5cc

See more details on using hashes here.

File details

Details for the file mixmasta-0.6.9-py2.py3-none-any.whl.

File metadata

  • Download URL: mixmasta-0.6.9-py2.py3-none-any.whl
  • Upload date:
  • Size: 32.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.9.2 requests/2.28.1 setuptools/57.5.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.8.10

File hashes

Hashes for mixmasta-0.6.9-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 2b327cdddef20f8f92ec06865281e13834a7b5bf5f02a5aebbd88013a75c3b23
MD5 9d404092075973ceabc3de47965135d6
BLAKE2b-256 066964f8fd00f0d8869753c5811ad805f10da936c211ab386e8a5ed7c678ad71

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page