lpjg2nc2 1.0.0

Convert LPJ-GUESS output files (.out) to NetCDF format

LPJ-GUESS to NetCDF Converter (lpjg2nc2)

lpjg2nc2 is a powerful tool for converting LPJ-GUESS output files (.out) to NetCDF format. It efficiently searches for run* folders in a given path, finds .out files inside their output directories, and converts them to the widely-used NetCDF format for easier analysis and visualization.

🚀 Key Features

• Efficient Processing: Convert LPJ-GUESS .out files to NetCDF with proper coordinates and metadata
• High Performance: Vectorized implementation delivers up to 20× speedup compared to point-by-point processing
• Parallel Processing: Two-level parallelization optimizes conversion on multi-core systems
• Flexible Configuration: Control various aspects of the conversion process
• Grid Handling: Process both 2D and 3D data (with depth levels), preserving spatial coordinates
• Time Dimensions: Handle time series data (years, months, and days, if present)
• Grid Remapping: Option to remap to regular global grids using CDO
• Data Analysis: Built-in analysis of NaN values to understand data sparsity in global datasets
• Test Mode: Option to process specific files for testing

📦 Installation

From PyPI (recommended)

pip install lpjg2nc2

From Source

git clone https://github.com/JanStreffing/lpjg2nc2.git
cd lpjg2nc2
pip install -e .

📋 Requirements

• Python 3.8+
• Dependencies (automatically installed):
  • numpy>=1.20.0
  • pandas>=1.3.0
  • xarray>=0.19.0
  • netCDF4>=1.5.7
  • tqdm>=4.61.0
• Optional: Climate Data Operators (CDO) for remapping capabilities

🔍 Usage

Basic Conversion

lpjg2nc -p /path/to/lpj_guess_runs/

This will:

1. Search for all run* folders in the specified path
2. Find .out files in each run's output directory
3. Convert them to NetCDF format
4. Save the output to "../../outdata/lpj_guess" relative to the input path

Verbose Output with Custom Output Directory

lpjg2nc -p /path/to/lpj_guess_runs/ -v -o /path/to/output/

Remapping to Regular Grid

lpjg2nc -p /path/to/lpj_guess_runs/ --remap 1
# or specify grid dimensions
lpjg2nc -p /path/to/lpj_guess_runs/ --remap 360x180

Testing with a Specific File

lpjg2nc -p /path/to/lpj_guess_runs/ -f /path/to/runs/run1/output/somefile.out -v

🛠️ Command Line Arguments

Required Arguments

• -p PATH, --path PATH: Path to the directory containing run* folders

Optional Arguments

• -f FILE, --file FILE: Process a specific file (for testing)
• -o OUTPUT, --output OUTPUT: Output directory for NetCDF files
• -v, --verbose: Increase output verbosity
• --remap RES: Remap output to a regular global grid using CDO (specify resolution in degrees or XxY dimensions)
• --test {ifs_input}: Test with specific file pattern (e.g., ifs_input.out)
• -j JOBS, --jobs JOBS: Number of parallel jobs for outer parallelization (patterns), default: 8
• --inner-jobs INNER_JOBS: Number of parallel jobs for inner parallelization, default: 16
• --chunk-size CHUNK_SIZE: Chunk size for processing arrays, default: 50000
• --pattern PATTERN: Specific pattern to process (internal use for parallelization)

Recommended Settings

Environment	--jobs (-j)	--inner-jobs	--chunk-size
Desktop (16GB RAM)	4	8	25000
Workstation (32GB RAM)	8	16	50000
HPC Node (128GB+ RAM)	8	64	75000

📚 Documentation

For complete documentation, visit lpjg2nc2.readthedocs.io.

📜 License

This project is licensed under the GNU General Public License v3 (GPLv3) - see the license file for details.