absconda 0.1.0

Generate Dockerfiles and container build workflows from Conda environments

Absconda

Turn conda environments into optimized container images for development and HPC deployment.

Absconda bridges the gap between conda's reproducible environments and container-based workflows. Define your scientific computing environment once with conda, then deploy it anywhere—Docker, Singularity/Apptainer, HPC clusters—with production-ready optimizations.

Key Features

🚀 Multi-Stage Builds - Automatic optimization that reduces image sizes by 40-60%
🔐 Policy Validation - Enforce security and compliance rules organization-wide
🏗️ Remote Builders - Offload builds to cloud instances with automatic provisioning
🧪 HPC Integration - First-class Singularity support with module files and wrappers
📦 R + renv Support - Combine conda environments with R package management
🎯 Flexible Deployment - Multiple modes: full-env, tarball, requirements, export-explicit
🔧 Custom Templates - Jinja2-based system for advanced customization

Quick Start

Installation

# Install from PyPI
pip install absconda

# Or with pipx (recommended)
pipx install absconda

# Verify installation
absconda --version

Basic Usage

1. Create a conda environment file:

# environment.yaml
name: my-analysis
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.11
  - numpy=1.26
  - pandas=2.1
  - scikit-learn=1.3

2. Build a Docker image:

absconda build \
  --file environment.yaml \
  --repository ghcr.io/myorg/my-analysis \
  --tag latest \
  --push

3. Use the image:

docker run --rm ghcr.io/myorg/my-analysis:latest python -c "import numpy; print(numpy.__version__)"

For HPC with Singularity:

# Build Docker image and convert to Singularity
absconda build \
  --file environment.yaml \
  --repository ghcr.io/myorg/my-analysis \
  --tag latest \
  --push

# Convert to Singularity
absconda singularity \
  --image ghcr.io/myorg/my-analysis:latest \
  --output my-analysis.sif

# Generate HPC module
absconda module \
  --image my-analysis.sif \
  --module-path /apps/modules/my-analysis/1.0

See the Quick Start Guide for a complete walkthrough.

Documentation

For New Users

• Installation - Get up and running
• Quick Start - 5-minute tutorial
• Core Concepts - Understanding absconda

Guides

• Basic Usage - Essential workflows
• Building Images - Docker build process
• HPC Deployment - Singularity on HPC clusters
• Remote Builders - Cloud-based builds
• R + renv Integration - Combining conda and renv
• Requirements Mode - Deployment mode comparison
• Advanced Templating - Custom Dockerfiles

How-To Guides

• Multi-Stage Builds - Optimize image size
• Custom Base Images - GPU and specialized bases
• Secrets Management - Handle credentials safely
• CI/CD Integration - Automate with GitHub Actions

Examples

Complete working examples with explanations:

• Minimal Python - Simple Python environment
• Data Science Stack - NumPy/pandas/scikit-learn
• R + Bioconductor - RNA-seq analysis
• GPU + PyTorch - Deep learning with CUDA
• HPC Workflow - Complete HPC deployment

Reference

• CLI Reference - Complete command documentation
• Environment Files - YAML specification
• Configuration - System configuration
• Policies - Policy validation system

For Contributors

• Contributing - How to contribute
• Testing - Running and writing tests
• Architecture - Technical design documentation

Why Absconda?

The Problem

Scientific computing has conflicting requirements:

• Reproducibility: Need exact package versions
• Portability: Must run on laptops, HPC clusters, cloud
• Performance: Large conda environments create huge containers
• HPC Reality: Singularity/Apptainer, not Docker; module systems, not Docker Compose

The Solution

Absconda solves this by:

1. Starting with conda - Use the ecosystem you already know
2. Optimizing automatically - Multi-stage builds reduce image sizes by 40-60%
3. Targeting HPC - Native Singularity support with modules and wrappers
4. Enforcing policies - Organization-wide security and compliance
5. Enabling remote builds - Build on powerful cloud instances, not your laptop

Compared to Alternatives

Feature	Absconda	repo2docker	docker-conda	Manual Dockerfile
Multi-stage optimization	✅ Automatic	❌ No	❌ No	⚠️ Manual
Singularity integration	✅ First-class	❌ No	❌ No	⚠️ Manual
HPC modules	✅ Built-in	❌ No	❌ No	⚠️ Manual
Policy enforcement	✅ Yes	❌ No	❌ No	❌ No
Remote builders	✅ Yes	❌ No	❌ No	❌ No
R + renv support	✅ Yes	⚠️ Limited	❌ No	⚠️ Manual
Custom templates	✅ Jinja2	❌ No	❌ No	✅ Full control

Real-World Example

Complete workflow for deploying to NCI Gadi HPC:

# 1. Build optimized Docker image (locally or on GCP)
absconda build \
  --file rnaseq-env.yaml \
  --repository ghcr.io/lab/rnaseq \
  --tag v1.0 \
  --remote-builder gcp-builder \
  --push

# 2. Convert to Singularity on HPC
ssh gadi.nci.org.au
singularity pull rnaseq.sif docker://ghcr.io/lab/rnaseq:v1.0

# 3. Generate module file
absconda module \
  --image /apps/rnaseq/v1.0/rnaseq.sif \
  --module-path /apps/Modules/modulefiles/rnaseq/1.0 \
  --wrapper-dir /apps/rnaseq/v1.0/wrappers

# 4. Use in PBS job
module load rnaseq/1.0
python analysis.py  # Uses containerized environment transparently

The result: reproducible, optimized, compliant environments deployed consistently from development through production.

Use Cases

🔬 Research Computing

• Reproducible analysis pipelines
• Sharing environments with collaborators
• Publishing with computational papers
• Archive environments for long-term reproducibility

🏢 Multi-User HPC

• Centralized environment management
• Policy enforcement across teams
• Module system integration
• Singularity deployment at scale

☁️ Cloud + HPC Hybrid

• Build in cloud (fast, powerful VMs)
• Deploy to HPC (Singularity)
• Consistent environments across platforms
• Cost-optimized with remote builders

🧬 Bioinformatics

• Bioconda + R/Bioconductor workflows
• GPU-accelerated analysis
• Large-scale genomics pipelines
• Compliance with data policies

Project Status

Absconda is production-ready and actively maintained. It powers scientific computing workflows for research teams at the Garvan Institute and beyond.

Current version: 0.1.0
Python support: 3.10, 3.11, 3.12
License: MIT

Getting Help

• Documentation: docs/
• Issues: GitHub Issues
• Discussions: GitHub Discussions

Contributing

Contributions welcome! See Contributing Guide.

Areas needing help:

• Documentation improvements
• Example workflows
• Testing on different platforms
• Feature requests and feedback

Acknowledgments

Developed at the Garvan Institute of Medical Research by the Swarbrick Lab.

Built with:

• micromamba - Fast conda package manager
• Jinja2 - Template engine
• Singularity/Apptainer - HPC containers
• Terraform - Infrastructure as code

License

MIT License - see LICENSE for details.

---

Ready to get started? → Installation Guide

Have questions? → GitHub Discussions