This project has been archived by its maintainers, and is no longer receiving any updates.
ACCESS Model Output Post-Processor (ACCESS-MOPPeR) v2
ACCESS-MOPPeR is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets. This version represents a significant rewrite focusing on usability, flexibility, and integration with modern Python workflows.
Key Features
- Python API for integration into notebooks and scripts
- Batch processing system for HPC environments with PBS
- Real-time monitoring with web-based dashboard
- Flexible CMORisation of individual variables
- Dask-enabled for scalable parallel processing
- Cross-platform compatibility (not limited to NCI Gadi)
- CMIP6 and CMIP7 FastTrack support
Installation
ACCESS-MOPPeR requires Python >= 3.11. Install with:
pip install numpy pandas xarray netCDF4 cftime dask pyyaml tqdm requests streamlit
pip install .
Quick Start
Interactive Usage (Python API)
import glob
from access_mopper import ACCESS_ESM_CMORiser
# Select input files
files = glob.glob("/path/to/model/output/*mon.nc")
# Create CMORiser instance
cmoriser = ACCESS_ESM_CMORiser(
input_paths=files,
compound_name="Amon.pr", # table.variable format
experiment_id="historical",
source_id="ACCESS-ESM1-5",
variant_label="r1i1p1f1",
grid_label="gn",
activity_id="CMIP",
output_path="/path/to/output"
)
# Run CMORisation
cmoriser.run()
cmoriser.write()
Batch Processing (HPC/PBS)
For large-scale processing on HPC systems:
- Create a configuration file (
batch_config.yml):
variables:
- Amon.pr
- Omon.tos
- Amon.ts
experiment_id: piControl
source_id: ACCESS-ESM1-5
variant_label: r1i1p1f1
grid_label: gn
input_folder: "/g/data/project/model/output"
output_folder: "/scratch/project/cmor_output"
file_patterns:
Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
Omon.tos: "output[0-4][0-9][0-9]/ocean/*temp*.nc"
Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
# PBS configuration
queue: normal
cpus_per_node: 16
mem: 32GB
walltime: "02:00:00"
scheduler_options: "#PBS -P your_project"
storage: "gdata/project+scratch/project"
worker_init: |
module load conda
conda activate your_environment
- Submit batch job:
mopper-cmorise batch_config.yml
- Monitor progress at http://localhost:8501
Batch Processing Features
The batch processing system provides:
- Parallel execution: Each variable processed as a separate PBS job
- Real-time monitoring: Web dashboard showing job status and progress
- Automatic tracking: SQLite database maintains job history and status
- Error handling: Failed jobs can be easily identified and resubmitted
- Resource optimization: Configurable CPU, memory, and storage requirements
- Environment management: Automatic setup of conda/module environments
Monitoring Tools
- Streamlit Dashboard: Real-time web interface at http://localhost:8501
- Command line: Use standard PBS commands (
qstat,qdel) - Database: SQLite tracking at
{output_folder}/cmor_tasks.db - Log files: Individual stdout/stderr for each job
File Organization
work_directory/
├── batch_config.yml # Your configuration
├── cmor_job_scripts/ # Generated PBS scripts and logs
│ ├── cmor_Amon_pr.sh # PBS script
│ ├── cmor_Amon_pr.py # Python processing script
│ ├── cmor_Amon_pr.out # Job output
│ └── cmor_Amon_pr.err # Job errors
└── output_folder/
├── cmor_tasks.db # Progress tracking
└── [CMORised files] # Final output
Documentation
- Getting Started:
docs/source/getting_started.rst - Example Configuration:
src/access_mopper/examples/batch_config.yml - API Reference: [Coming soon]
Current Limitations
- Alpha version: Intended for evaluation only
- Ocean variables: Limited support in current release
- Variable mapping: Under review for CMIP6/CMIP7 compliance
Support
- Issues: Submit via GitHub Issues
- Questions: Contact ACCESS-NRI support
- Contributions: Welcome via Pull Requests
License
ACCESS-MOPPeR is licensed under the Apache-2.0 License.
Background: ACCESS-MOPPeR v2 is a complete rewrite using modern Python libraries (xarray, dask) instead of CMOR, providing improved flexibility and integration with contemporary data science workflows.
1 maintainer
Metadata
1 maintainer
Release files for access-mopper 2.3.0a24
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| access_mopper-2.3.0a24.tar.gz | 310.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| access_mopper-2.3.0a24-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 658.9 kB
Release files / access_mopper-2.3.0a24.tar.gz
| Download URL | access_mopper-2.3.0a24.tar.gz |
|---|---|
| Size | 310.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
62b0b80f46e5999da2f24d5f52766bd962488f526420319484eac5339fdcfc88
|
|
BLAKE2b-256 checksum How to use checksums |
20a5aac0fa72ea474732d2178c1836dad17b87118861e09d66ef20a672b87bf9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 28, 2025.
Transparency logRelease files / access_mopper-2.3.0a24-py3-none-any.whl
| Download URL | access_mopper-2.3.0a24-py3-none-any.whl |
|---|---|
| Size | 348.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7e5387dd0c906b3d7fe3786bc01a741bc3d5cd8c29a736c78e332e46cac8d55c
|
|
BLAKE2b-256 checksum How to use checksums |
a076b26a5aa758219ddf51a90f241b8497adfb841fcbef2605a9850baec0d93e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 28, 2025.
Transparency log
