Skip to main content

A python package for obtaining and cleaning Tb files

Project description

Disclaimer

  • Currently this script is not supported in Windows due to pynco only supporting Mac OS or Unix

  • Web scrapping currently only working on the northern hemisphere, fix coming soon

  • Requires python 3.6 and Anaconda 3

Setup:

Setup Earthdata Login

Create an Earthdata account to be able to download data: https://urs.earthdata.nasa.gov/

Optional (.netrc file vs passing Username and Password):

Setup your username and password in a .netrc file
Run this command in the directory you will be working in

echo "machine urs.earthdata.nasa.gov login <uid> password <password>" >> ~/.netrc
chmod 0600 ~/.netrc

uid is your Earthdata username. Do not include the brackets <>.

https://nsidc.org/support/faq/what-options-are-available-bulk-downloading-data-https-earthdata-login-enabled

2. Setup conda environment from yaml (Optional):

Using the yaml file (.yml) create a new conda environment

conda env create -f swepy_env.yml

3. Install ipykernel (if using jupyter)

source activate swepy_env
python -m ipykernel install --user --name swepy_env --display-name "Python (swepy_env)"

Main Dependencies:

  • gdal
  • affine
  • requests
  • scikit-image
  • nco (pynco)
  • netCDF4
  • datetime
  • tqdm
  • mapboxgl
  • pandas

Using SWEpy for analyzing SWE:

  1. Import the Library:
from swepy.swepy import swepy
  1. Instantiate the class with working directory, date range, bounding coordinates, and earthdata username and password
upper_left = [lat_upleft, lon_upleft]
lower_right = [lat_lowright, lon_lowright]

start = datetime.date(startY, startM, startD)
end = datetime.date(endY, endM, endD)

path = os.getcwd()

swepy = swepy(path, start, end, upper_left, lower_right, username, password)
  1. Use desired functionality, either separate or individually:
swepy.scrape()
swepy.subset()
swepy.concatenate()

swepy.concatenate(swepy.subset(swepy.scrape()))
  1. Or, use scrape_all to avoid massive file sizes:
swepy.scrape_all()

This limits the number of full-size images on your disk at one time.

Using SWEpy's Web Scraper Alone:

  • Note: Web scraper is enabled automatically in the scrape_all workflow, however it can also be used as a stanalone function!
from swepy.nsidcDownloader import nsidcDownloader

## Ways to instantiate nsidcDownloader
nD = nsidcDownloader.nsidcDownloader(username="user", password="pass", folder=".") ## user/pass combo and folder

nD = nsidcDownloader(sensor="SSMIS") ## user/pass combo from .netrc and default folder, setting the default key of sensor

## Download a file:

file = {
    "resolution": "3.125km",
    "platform": "F17",
    "sensor": "SSMIS",
    "date": datetime(2015,10,10),
    "channel": "37H"
}

nD.download_file(**file)

nD.download_range(sensor="SSMIS", date=[datetime(2014,01,01), datetime(2015,01,01)])
  • Authentication will work if the user/pass combo is saved in ~/.netrc, or if it is passed in the nsidcDownloader instance

  • The class formats the following string:

 "{protocol}://{server}/{datapool}/{dataset}.{version}/{date:%Y.%m.%d}" \
                    "/{dataset}-{projection}_{grid}{resolution}-{platform}_{sensor}" \
                    "-{date:%Y%j}-{channel}-{pass}-{algorithm}-{input}-{dataversion}.nc"

Function Summaries

Descriptions of included functions

get_xy(ll_ul, ll_lr)
  • Parameters: lists of latitude/longitude upper left, latitude/longitude lower right
  • Uses NSIDC scripts to convert user inputted lat/lon into Ease grid 2.0 coordinates
  • Returns: Ease grid 2.0 coordinates of inputted lat/longs
subset(list6, path)
  • Parameters: coordinates of area of interest, current working directory
  • Subset will get the files from wget directory and subset them geographically
  • Returns: subsetted file
concatenate(path, outfile_19, outfile_37, final=False)
  • Parameters: current working directory, output file for 19Ghz, output file for 37Ghz
  • The concatenate function merges all netCDF files into one large file
  • Returns: concatenated netCDF file
file_setup(path)
  • Parameters: current working directory
  • setup files needed for other functions
  • Returns: create correct folders for use by other functions
scrape_all(start, end, list3, path=None)
  • Parameters: start date, end date, list, current working directory(optional)
  • Complete function that downloads, concatenates, and subsets data
  • Returns: file names of concatenated 19/37 time cubes
plot_a_day(file1, file2, path, token)
  • Parameters: 19Ghz files, 37Ghz files, current working directory, mapbox token
  • Plots a day of data using Mapbox Jupyter
  • Returns: interactive map of inputted data

Troubleshooting

  1. ‘image not found’ errors If encountering ‘image not found’ errors then one possible fix is to add theconda-forge channel on top of the defaults in your .condarc file. This is a hidden file, show hidden files and then edit the .condarc file and make your file look like this:

    $ cat .condarc channels:

    • conda-forge
    • defaults

After saving this file, update conda:

conda update all

https://conda-forge.org/docs/conda-forge_gotchas.html#using-multiple-channels

  1. HDF5 errors: If getting HDF5 errors, try deleting all the netCDF files in your directories.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

swepy-0.0.17.tar.gz (11.2 kB view hashes)

Uploaded Source

Built Distribution

swepy-0.0.17-py3-none-any.whl (22.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page