Pre-release
This release is a pre-release and may not be stable for production use.
ACCESS-MOPPy (Model Output Post-processor in Python)
ACCESS-MOPPy is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets.
Key Features
- Python API for integration into notebooks and scripts
- Batch processing system for HPC environments with PBS
- Real-time monitoring with web-based dashboard
- Flexible CMORisation of individual variables
- Dask-enabled for scalable parallel processing
- Cross-platform compatibility (not limited to NCI Gadi)
- CMIP6 and CMIP7 FastTrack support
Installation
ACCESS-MOPPy requires Python >= 3.11. Install with:
pip install numpy pandas xarray netCDF4 cftime dask pyyaml tqdm requests streamlit
pip install .
Quick Start
Interactive Usage (Python API)
import glob
from access_moppy import ACCESS_ESM_CMORiser
# Select input files
files = glob.glob("/path/to/model/output/*mon.nc")
# Create CMORiser instance
cmoriser = ACCESS_ESM_CMORiser(
input_data=files,
compound_name="Amon.pr", # table.variable format
experiment_id="historical",
source_id="ACCESS-ESM1-5",
variant_label="r1i1p1f1",
grid_label="gn",
activity_id="CMIP",
output_path="/path/to/output"
)
# Run CMORisation
cmoriser.run()
cmoriser.write()
Batch Processing (HPC/PBS)
For large-scale processing on HPC systems:
- Create a configuration file (
batch_config.yml):
variables:
- Amon.pr
- Omon.tos
- Amon.ts
experiment_id: piControl
source_id: ACCESS-ESM1-5
variant_label: r1i1p1f1
grid_label: gn
input_folder: "/g/data/project/model/output"
output_folder: "/scratch/project/cmor_output"
file_patterns:
Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
Omon.tos: "output[0-4][0-9][0-9]/ocean/*temp*.nc"
Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
# PBS configuration
queue: normal
cpus_per_node: 16
mem: 32GB
walltime: "02:00:00"
scheduler_options: "#PBS -P your_project"
storage: "gdata/project+scratch/project"
worker_init: |
module load conda
conda activate your_environment
- Submit batch job:
moppy-cmorise batch_config.yml
- Monitor progress at http://localhost:8501
Batch Processing Features
The batch processing system provides:
- Parallel execution: Each variable processed as a separate PBS job
- Real-time monitoring: Web dashboard showing job status and progress
- Automatic tracking: SQLite database maintains job history and status
- Error handling: Failed jobs can be easily identified and resubmitted
- Resource optimization: Configurable CPU, memory, and storage requirements
- Environment management: Automatic setup of conda/module environments
Monitoring Tools
- Streamlit Dashboard: Real-time web interface at http://localhost:8501
- Command line: Use standard PBS commands (
qstat,qdel) - Database: SQLite tracking at
{output_folder}/cmor_tasks.db - Log files: Individual stdout/stderr for each job
File Organization
work_directory/
├── batch_config.yml # Your configuration
├── cmor_job_scripts/ # Generated PBS scripts and logs
│ ├── cmor_Amon_pr.sh # PBS script
│ ├── cmor_Amon_pr.py # Python processing script
│ ├── cmor_Amon_pr.out # Job output
│ └── cmor_Amon_pr.err # Job errors
└── output_folder/
├── cmor_tasks.db # Progress tracking
└── [CMORised files] # Final output
Documentation
- Getting Started:
docs/source/getting_started.rst - Example Configuration:
src/access_moppy/examples/batch_config.yml - API Reference: [Coming soon]
Test Data Override
Integration and end-to-end tests require an external test-data tree set via
the ACCESS_MOPPY_DATA_ROOT environment variable.
- Covered tests: full CMOR integration and end-to-end real-file tests
- No fallback: test-data fixtures in
tests/data/are not used by these tests - Requirement:
ACCESS_MOPPY_DATA_ROOTmust point to a valid dataset root containingoutput*/atmosphere/netCDF,output*/ocean, andoutput*/ice
Example:
export ACCESS_MOPPY_DATA_ROOT=/path/to/CMIP7_Test_data/esm-historical
pixi run -e dev python -m pytest tests/integration/test_full_cmorisation.py
pixi run -e dev python -m pytest tests/integration/test_cmip7_baseline_cmorisation.py
pixi run -e dev python -m pytest tests/e2e/test_end_to_end.py
CMIP7 baseline test note:
tests/integration/test_cmip7_baseline_cmorisation.pyruns one case per CMIP7 baseline variable listed insrc/access_moppy/examples/batch_config_esm1-6_cmip7_baseline.yml- By default, this suite checks end-to-end CMORisation success (run/write/output)
- To additionally enforce WCRP compliance-checker validation for this suite,
set
ACCESS_MOPPY_BASELINE_VALIDATE_WCRP=1
Example with strict WCRP validation enabled:
export ACCESS_MOPPY_DATA_ROOT=/path/to/CMIP7_Test_data/esm-historical
export ACCESS_MOPPY_BASELINE_VALIDATE_WCRP=1
pixi run -e dev python -m pytest tests/integration/test_cmip7_baseline_cmorisation.py --validation-tool=wcrp
Current Limitations
- Alpha version: Intended for evaluation only
- Ocean variables: Limited support in current release
- Variable mapping: Under review for CMIP6/CMIP7 compliance
Support
- Issues: Submit via GitHub Issues
- Questions: Contact ACCESS-NRI support
- Contributions: Welcome via Pull Requests
License
ACCESS-MOPPy is licensed under the Apache-2.0 License.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
access_moppy-1.7.5b0.tar.gz
(12.8 MB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file access_moppy-1.7.5b0.tar.gz.
File metadata
- Download URL: access_moppy-1.7.5b0.tar.gz
- Upload date:
- Size: 12.8 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
643ee5a0f6c44bfc62ca62fa014b11b029d37b47db2d5e0bfb7e62c8a72af25d
|
|
| MD5 |
07e240f5505a570265e860c1406e4ca7
|
|
| BLAKE2b-256 |
b45e6b8f4880fd176806108e944fd78f4307eef350820a7bde4e3ff3972529c0
|
Provenance
The following attestation bundles were made for access_moppy-1.7.5b0.tar.gz:
Publisher:
cd.yml on ACCESS-NRI/ACCESS-MOPPy
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
access_moppy-1.7.5b0.tar.gz -
Subject digest:
643ee5a0f6c44bfc62ca62fa014b11b029d37b47db2d5e0bfb7e62c8a72af25d - Sigstore transparency entry: 2339609128
- Sigstore integration time:
-
Permalink:
ACCESS-NRI/ACCESS-MOPPy@7d16a74e0810b0bf8e1847c9ac3ba5b919cf1b9e -
Branch / Tag:
refs/tags/moppy-v1.7.5b - Owner: https://github.com/ACCESS-NRI
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
cd.yml@7d16a74e0810b0bf8e1847c9ac3ba5b919cf1b9e -
Trigger Event:
push
-
Statement type:
File details
Details for the file access_moppy-1.7.5b0-py3-none-any.whl.
File metadata
- Download URL: access_moppy-1.7.5b0-py3-none-any.whl
- Upload date:
- Size: 13.2 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
88a6398332e44c0756d4e4f70104cbb62819f3ed5adf8c2319bd96c97e35a6ea
|
|
| MD5 |
1b17a7a7cff1ebafce8325a23454dc8e
|
|
| BLAKE2b-256 |
66e4041d0808c6c7dc002be9d47820137de72ad14fccfdcfe0a93f3f6561d8e4
|
Provenance
The following attestation bundles were made for access_moppy-1.7.5b0-py3-none-any.whl:
Publisher:
cd.yml on ACCESS-NRI/ACCESS-MOPPy
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
access_moppy-1.7.5b0-py3-none-any.whl -
Subject digest:
88a6398332e44c0756d4e4f70104cbb62819f3ed5adf8c2319bd96c97e35a6ea - Sigstore transparency entry: 2339609153
- Sigstore integration time:
-
Permalink:
ACCESS-NRI/ACCESS-MOPPy@7d16a74e0810b0bf8e1847c9ac3ba5b919cf1b9e -
Branch / Tag:
refs/tags/moppy-v1.7.5b - Owner: https://github.com/ACCESS-NRI
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
cd.yml@7d16a74e0810b0bf8e1847c9ac3ba5b919cf1b9e -
Trigger Event:
push
-
Statement type:
