A library for common scientific model transforms
Project description
mixmasta
A library for common scientific model transforms. This library enables fast and intuitive transforms including:
- Converting a
geotiff
to acsv
- Converting a
NetCDF
to acsv
- Geocoding
csv
,xls
, andxlsx
data that contains latitude and longitude
Setup
See docs/docker.md
for instructions on running Mixmasta in Docker (easiest!).
Ensure you have a working installation of GDAL
You also need to ensure that numpy
is installed prior to mixmasta
installation. This is an artifact of GDAL, which will build incorrectly if numpy
is not already configured:
pip install numpy==1.20.1
pip install mixmasta
Note: if you had a prior installation of GDAL you may need to run
pip install mixmasta --no-cache-dir
in a clean environment.
You must install the GADM2 and GADM3 data with:
mixmasta download
Usage
Examples can be found in the input
directory.
Convert a geotiff to a dataframe with:
from mixmasta import mixmasta as mix
df = mix.raster2df('chirps-v2.0.2021.01.3.tif', feature_name='rainfall', band=1)
Note that you should specify the data band of the geotiff to process if it is multi-band. You may also specify the name of the feature column to produce. You may optionally specify a date
if the geotiff has an associated date. For example:
Convert a NetCDF to a dataframe with:
from mixmasta import mixmasta as mix
df = mix.netcdf2df('tos_O1_2001-2002.nc')
Geocode a dataframe:
from mixmasta import mixmasta as mix
# First, load in the geotiff as a dataframe
df = mix.raster2df('chirps-v2.0.2021.01.3.tif', feature_name='rainfall', band=1)
# next, we can geocode the dataframe to the admin-level desired (`admin2` or `admin3`)
# by specifying the names of the x and y columns
# in this case, we will geocode to admin2 where x,y are are 'longitude' and 'latitude', respectively.
df_g = mix.geocode("admin2", df, x='longitude', y='latitude')
Running with CLI
After cloning the repository and changing to the mixmasta
directory, you can run mixmasta via the command line.
Set-up:
While you can point mixmasta
to any file you would like to transform, the examples below assume your file is in the inputs
folder; the transformed .csv
file will be written to the outputs
folder.
- Transform geotiff to geocoded csv:
mixmasta mix --xform=geotiff --input_file=chirps-v2.0.2021.01.3.tif --output_file=geotiffTEST.csv --geo=admin2 --feature_name=rainfall --band=1 --date='5/4/2010' --x=longitude --y=latitude
- Transform geotiff to csv:
mixmasta mix --xform=geotiff --input_file=maxhop1.tif --output_file=maxhopOUT.csv --geo=admin2 --feature_name=probabilty --band=1 --x=longitude --y=latitude
- Transform netcdf to geocoded csv:
mixmasta mix --xform=netcdf --input_file=tos_O1_2001-2002.nc --output_file=netcdf.csv --geo=admin2 --x=lon --y=lat
- Transform netcdf to csv:
mixmasta mix --xform=netcdf --input_file=tos_O1_2001-2002.nc --output_file=netcdf.csv
-geocode an existing csv file:
mixmasta mix --xform=geocode --input_file=no_geo.csv --geo=admin3 --output_file=geoed_no_geo.csv --x=longitude --y=latitude
World Modelers Specific Normalization
For the World Modelers program, it is necessary to convert arbitrary csv
, geotiff
, and netcdf
files into a CauseMos compliant format. This can be accomplished by leveraging a mapping
annotation file and the causemosify
command. The output is a gzipped
parquet
file. This may be invoked with:
mixmasta causemosify --input_file=chirps-v2.0.2021.01.3.tif --mapper=mapper.json --geo=admin3 --output_file=causemosified_example
This will produce a file called causemosified_example.parquet.gzip
which can be read using Pandas with:
pd.read_parquet('causemosified_example.parquet.gzip')
Other Documents
- Docker Instructions:
docs/docker.md
- Geo Entity Resolution Description:
docs/geo-tentity-resolution.md
- Package Testing in SpaceTag Env:
docs/spacetag-test.md
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
History
0.1.0 (2021-02-24)
- First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file mixmasta-0.6.9.tar.gz
.
File metadata
- Download URL: mixmasta-0.6.9.tar.gz
- Upload date:
- Size: 2.9 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.14.0 pkginfo/1.9.2 requests/2.28.1 setuptools/57.5.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.8.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20d4f27c46732b1b606e18aad255612b91030968067bbdb8fccf776a3f2e81c3 |
|
MD5 | 39c698c50ad9db243e3b0e7216ffdedf |
|
BLAKE2b-256 | 4221cf29f591d0a0fa76e0f4ad46febc7e7a63dcd84251054513b67d2531c5cc |
File details
Details for the file mixmasta-0.6.9-py2.py3-none-any.whl
.
File metadata
- Download URL: mixmasta-0.6.9-py2.py3-none-any.whl
- Upload date:
- Size: 32.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.14.0 pkginfo/1.9.2 requests/2.28.1 setuptools/57.5.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.8.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b327cdddef20f8f92ec06865281e13834a7b5bf5f02a5aebbd88013a75c3b23 |
|
MD5 | 9d404092075973ceabc3de47965135d6 |
|
BLAKE2b-256 | 066964f8fd00f0d8869753c5811ad805f10da936c211ab386e8a5ed7c678ad71 |