Skip to main content

EMIT L2A Estimated Surface Reflectance and Uncertainty and Masks 60 m Search and Download Utility

This Python package is a tool for searching, downloading, and reading hypspectral surface reflectance from the Earth Surface Mineral Dust Source Investigation (EMIT) L2A RFL data product.

Installation

Standard Installation (pip)

pip install EMITL2ARFL

HPC Systems with HDF5 Issues

If you encounter HDF5 errors (like HDF Error -101) on HPC systems, the issue is typically caused by conflicts between system HDF5 libraries and pip-installed packages. The solution is to use conda-forge packages which bundle compatible HDF5 libraries.

Using the makefile (recommended):

git clone https://github.com/STARS-Data-Fusion/EMIT-L2A-RFL.git
cd EMIT-L2A-RFL
make environment  # Creates mamba environment with conda-forge HDF5
mamba activate EMITL2ARFL
make install

Manual setup:

# Create environment with HDF5 from conda-forge
mamba create -n EMITL2ARFL -c conda-forge python=3.10 hdf5 h5py netcdf4
mamba activate EMITL2ARFL

# Install package
pip install EMITL2ARFL
# Or for development: pip install -e .

Set environment variable (HPC systems only):

# For fish shell
set -Ux HDF5_USE_FILE_LOCKING FALSE

# For bash shell (add to ~/.bashrc)
export HDF5_USE_FILE_LOCKING=FALSE

See diagnostics/Install HPC.md for detailed HPC installation instructions and troubleshooting.

Caveats

Full Granule Download Requirement

The current version of this package requires downloading entire EMIT L2A RFL granules in order to obtain spatial subsets. While the package provides functions to extract data for a specific geometry or region of interest, the spatial subsetting occurs after the complete granule files (reflectance, mask, and uncertainty) have been downloaded to the local filesystem.

This means:

  • Full files are downloaded: Each granule (~300-500 MB) is downloaded in its entirety, regardless of subset size
  • Local processing: Spatial subsetting is performed locally after download using the geolocation information
  • Bandwidth impact: For small areas of interest, this results in downloading significantly more data than necessary

Workaround considerations:

  • Cache full granules locally if processing multiple overlapping geometries, as the same granule can be reused
  • Consider the download bandwidth constraints when planning time series processing for large date ranges
  • Granules that do not intersect your geometry are skipped automatically through the search and filtering process

Future versions of this package may support server-side subsetting via OGC Web Coverage Service (WCS) endpoints or cloud-native data formats (Zarr, Cloud-Optimized GeoTIFF) if such capabilities become available through NASA LPDAAC.

Future Work: Server-Side Subsetting

Several promising approaches could enable efficient remote subsetting without downloading full granules:

1. HTTP Range Requests with Kerchunk (Most Feasible)

Use the kerchunk library to create virtual Zarr indices of remote NetCDF files, allowing byte-range requests for specific data chunks. This approach:

  • Requires no infrastructure changes at NASA LPDAAC
  • Leverages existing earthaccess authentication
  • Downloads only the bytes needed for the requested subset
  • Works immediately if EMIT data servers support HTTP Range requests (standard for most NASA data systems)

Implementation: Add optional use_remote=True parameter to retrieval functions that bypasses full download in favor of remote chunked access.

2. OGC Web Coverage Service (WCS)

If NASA LPDAAC exposes WCS endpoints for EMIT data, the package could:

  • Submit server-side subsetting requests with geometry parameters
  • Receive pre-subset data directly from the server
  • Reduce bandwidth by orders of magnitude for small AOIs

Implementation: Create retrieve_via_wcs.py module using owslib library to construct and execute WCS GetCoverage requests.

3. Cloud-Native Data Formats

If EMIT data becomes available in cloud-optimized formats:

  • Zarr: Native chunked format with efficient remote access via xarray and fsspec
  • Cloud-Optimized GeoTIFF (COG): GDAL can read remote tiles via HTTP Range requests
  • Both formats enable subsetting at the storage level without custom protocols

Implementation: Add format detection and route requests to appropriate readers based on available data sources.

4. OpenDAP Protocol

If EMIT granules are served via OpenDAP (OPeNDAP) servers:

  • Use xarray.open_dataset() with constraint expressions
  • Request subsets by coordinate ranges directly in the URL
  • Standard protocol supported across NASA data centers

Implementation: Modify retrieve_EMIT_L2A_RFL_granule() to accept OpenDAP URLs and use xarray remote access instead of download.

5. Enhanced Earthaccess Integration

Monitor and contribute to the earthaccess library for:

  • Native subsetting support in earthaccess.download()
  • Automatic detection and utilization of OGC services
  • Chunked download capabilities for large files

Community involvement: File feature requests and collaborate with earthaccess maintainers on subsetting capabilities.

Development Priority

The HTTP Range + Kerchunk approach (Option 1) is recommended as the first implementation target because:

  • It has the fewest external dependencies
  • No waiting for NASA infrastructure changes
  • Can be implemented and tested immediately
  • Provides graceful fallback to full download if Range requests are unavailable

Contributions and pull requests implementing any of these approaches are welcome!

Diagnostics

Diagnostic tools and troubleshooting guides are available in the diagnostics/ folder:

  • Install HPC.md - Complete HPC installation guide
  • HPC Troubleshooting.md - Common HPC issues and solutions
  • verify_installation.py - Check if HDF5/NetCDF libraries are properly configured
  • check_hdf5_version.py - Display library versions and test file opening
  • debug_hpc_environment.py - Comprehensive environment diagnostic tool

References

  • Green, R. O., et al. (2023). Earth Surface Mineral Dust Source Investigation (EMIT) L2A Estimated Surface Reflectance and Uncertainty and Masks, Version 1. [Data set]. NASA EOSDIS Land Processes DAAC. doi:10.5067/EMIT/EMITL2ARFL.001

  • Green, R. O., et al. (2024). The Earth Surface Mineral Dust Source Investigation (EMIT) on the International Space Station: In-flight instrument performance and first results. Remote Sensing of Environment, 282, 113277. doi:10.1016/j.rse.2023.113277

Metadata

Release files for EMITL2ARFL 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for EMITL2ARFL 1.2.0
File Size Uploaded Provenance Actions
emitl2arfl-1.2.0.tar.gz 327.7 kB Without provenance Details for emitl2arfl-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for EMITL2ARFL 1.2.0
File Interpreter ABI Platform Actions
emitl2arfl-1.2.0-py3-none-any.whl Python 3 none any Details for emitl2arfl-1.2.0-py3-none-any.whl

Total release size: 387.2 kB

Release files / emitl2arfl-1.2.0.tar.gz

Download URL emitl2arfl-1.2.0.tar.gz
Size 327.7 kB
Tags Source
SHA-256 checksum
How to use checksums
f22d7591312dfa455589c7f7dca18288fb24f9f62a7196515a1637582c88222f
BLAKE2b-256 checksum
How to use checksums
6a8811bb5cdecd052b444c7a3f3cfc49685060abdbac38d8cde9b7400f7c7f74
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Release files / emitl2arfl-1.2.0-py3-none-any.whl

Download URL emitl2arfl-1.2.0-py3-none-any.whl
Size 59.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7f3b3fb2cc508f74081df2d47c23e177a0f4151f6733fc98adefdc02977a93c5
BLAKE2b-256 checksum
How to use checksums
ed794beb3f0b34feb701860ce7cf1e3cc46f7e003c45d7525cb2a44a7c59fff3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Security for EMITL2ARFL 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page