EMIT L2A Estimated Surface Reflectance and Uncertainty and Masks 60 m Search and Download Utility
This Python package is a tool for searching, downloading, and reading hypspectral surface reflectance from the Earth Surface Mineral Dust Source Investigation (EMIT) L2A RFL data product.
Installation
Standard Installation (pip)
pip install EMITL2ARFL
HPC Systems with HDF5 Issues
If you encounter HDF5 errors (like HDF Error -101) on HPC systems, the issue is typically caused by conflicts between system HDF5 libraries and pip-installed packages. The solution is to use conda-forge packages which bundle compatible HDF5 libraries.
Using the makefile (recommended):
git clone https://github.com/STARS-Data-Fusion/EMIT-L2A-RFL.git
cd EMIT-L2A-RFL
make environment # Creates mamba environment with conda-forge HDF5
mamba activate EMITL2ARFL
make install
Manual setup:
# Create environment with HDF5 from conda-forge
mamba create -n EMITL2ARFL -c conda-forge python=3.10 hdf5 h5py netcdf4
mamba activate EMITL2ARFL
# Install package
pip install EMITL2ARFL
# Or for development: pip install -e .
Set environment variable (HPC systems only):
# For fish shell
set -Ux HDF5_USE_FILE_LOCKING FALSE
# For bash shell (add to ~/.bashrc)
export HDF5_USE_FILE_LOCKING=FALSE
See diagnostics/Install HPC.md for detailed HPC installation instructions and troubleshooting.
Caveats
Full Granule Download Requirement
The current version of this package requires downloading entire EMIT L2A RFL granules in order to obtain spatial subsets. While the package provides functions to extract data for a specific geometry or region of interest, the spatial subsetting occurs after the complete granule files (reflectance, mask, and uncertainty) have been downloaded to the local filesystem.
This means:
- Full files are downloaded: Each granule (~300-500 MB) is downloaded in its entirety, regardless of subset size
- Local processing: Spatial subsetting is performed locally after download using the geolocation information
- Bandwidth impact: For small areas of interest, this results in downloading significantly more data than necessary
Workaround considerations:
- Cache full granules locally if processing multiple overlapping geometries, as the same granule can be reused
- Consider the download bandwidth constraints when planning time series processing for large date ranges
- Granules that do not intersect your geometry are skipped automatically through the search and filtering process
Future versions of this package may support server-side subsetting via OGC Web Coverage Service (WCS) endpoints or cloud-native data formats (Zarr, Cloud-Optimized GeoTIFF) if such capabilities become available through NASA LPDAAC.
Future Work: Server-Side Subsetting
Several promising approaches could enable efficient remote subsetting without downloading full granules:
1. HTTP Range Requests with Kerchunk (Most Feasible)
Use the kerchunk library to create virtual Zarr indices of remote NetCDF files, allowing byte-range requests for specific data chunks. This approach:
- Requires no infrastructure changes at NASA LPDAAC
- Leverages existing
earthaccessauthentication - Downloads only the bytes needed for the requested subset
- Works immediately if EMIT data servers support HTTP Range requests (standard for most NASA data systems)
Implementation: Add optional use_remote=True parameter to retrieval functions that bypasses full download in favor of remote chunked access.
2. OGC Web Coverage Service (WCS)
If NASA LPDAAC exposes WCS endpoints for EMIT data, the package could:
- Submit server-side subsetting requests with geometry parameters
- Receive pre-subset data directly from the server
- Reduce bandwidth by orders of magnitude for small AOIs
Implementation: Create retrieve_via_wcs.py module using owslib library to construct and execute WCS GetCoverage requests.
3. Cloud-Native Data Formats
If EMIT data becomes available in cloud-optimized formats:
- Zarr: Native chunked format with efficient remote access via
xarrayandfsspec - Cloud-Optimized GeoTIFF (COG): GDAL can read remote tiles via HTTP Range requests
- Both formats enable subsetting at the storage level without custom protocols
Implementation: Add format detection and route requests to appropriate readers based on available data sources.
4. OpenDAP Protocol
If EMIT granules are served via OpenDAP (OPeNDAP) servers:
- Use
xarray.open_dataset()with constraint expressions - Request subsets by coordinate ranges directly in the URL
- Standard protocol supported across NASA data centers
Implementation: Modify retrieve_EMIT_L2A_RFL_granule() to accept OpenDAP URLs and use xarray remote access instead of download.
5. Enhanced Earthaccess Integration
Monitor and contribute to the earthaccess library for:
- Native subsetting support in
earthaccess.download() - Automatic detection and utilization of OGC services
- Chunked download capabilities for large files
Community involvement: File feature requests and collaborate with earthaccess maintainers on subsetting capabilities.
Development Priority
The HTTP Range + Kerchunk approach (Option 1) is recommended as the first implementation target because:
- It has the fewest external dependencies
- No waiting for NASA infrastructure changes
- Can be implemented and tested immediately
- Provides graceful fallback to full download if Range requests are unavailable
Contributions and pull requests implementing any of these approaches are welcome!
Diagnostics
Diagnostic tools and troubleshooting guides are available in the diagnostics/ folder:
- Install HPC.md - Complete HPC installation guide
- HPC Troubleshooting.md - Common HPC issues and solutions
- verify_installation.py - Check if HDF5/NetCDF libraries are properly configured
- check_hdf5_version.py - Display library versions and test file opening
- debug_hpc_environment.py - Comprehensive environment diagnostic tool
References
-
Green, R. O., et al. (2023). Earth Surface Mineral Dust Source Investigation (EMIT) L2A Estimated Surface Reflectance and Uncertainty and Masks, Version 1. [Data set]. NASA EOSDIS Land Processes DAAC. doi:10.5067/EMIT/EMITL2ARFL.001
-
Green, R. O., et al. (2024). The Earth Surface Mineral Dust Source Investigation (EMIT) on the International Space Station: In-flight instrument performance and first results. Remote Sensing of Environment, 282, 113277. doi:10.1016/j.rse.2023.113277
Metadata
Release files for EMITL2ARFL 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| emitl2arfl-1.2.0.tar.gz | 327.7 kB | Without provenance | Details for emitl2arfl-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| emitl2arfl-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for emitl2arfl-1.2.0-py3-none-any.whl |
Total release size: 387.2 kB
Release files / emitl2arfl-1.2.0.tar.gz
| Download URL | emitl2arfl-1.2.0.tar.gz |
|---|---|
| Size | 327.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f22d7591312dfa455589c7f7dca18288fb24f9f62a7196515a1637582c88222f
|
|
BLAKE2b-256 checksum How to use checksums |
6a8811bb5cdecd052b444c7a3f3cfc49685060abdbac38d8cde9b7400f7c7f74
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Release files / emitl2arfl-1.2.0-py3-none-any.whl
| Download URL | emitl2arfl-1.2.0-py3-none-any.whl |
|---|---|
| Size | 59.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7f3b3fb2cc508f74081df2d47c23e177a0f4151f6733fc98adefdc02977a93c5
|
|
BLAKE2b-256 checksum How to use checksums |
ed794beb3f0b34feb701860ce7cf1e3cc46f7e003c45d7525cb2a44a7c59fff3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Security for EMITL2ARFL 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.