Skip to main content

Unified Conda and Pip requirements management.

Project description

:rocket: conda-join - Unified Conda and Pip Requirements Management :rocket:

PyPI Build Status CodeCov

conda-join simplifies Python project dependency management by enabling a single requirements.yaml file to handle both Conda and Pip dependencies. This streamlined approach allows for creating a unified Conda environment.yaml, while also seamlessly integrating with setup.py or pyproject.toml. In addition, it can be used as a CLI to combine multiple requirements.yaml files into a single environment.yaml file. Simplify your setup and maintain all your dependencies in one place with conda-join.

:books: Table of Contents

:package: Installation

To install conda-join, run the following command:

pip install -U conda-join

Or just copy the script to your computer:

wget https://raw.githubusercontent.com/basnijholt/conda-join/main/conda_join.py

:page_facing_up: requirements.yaml structure

conda-join processes requirements.yaml files with a specific format:

  • name (Optional): For documentation, not used in the output.
  • channels: List of sources for packages, such as conda-forge.
  • dependencies: Mix of Conda and Pip packages.

Example

Example of a requirements.yaml file:

name: example_environment
channels:
  - conda-forge
dependencies:
  - numpy  # same name on conda and pip
  - conda: python-graphviz  # When names differ between Conda and Pip
    pip: graphviz
  - pip: slurm-usage  # pip-only
  - conda: mumps  # conda-only

⚠️ conda-join can process this file in pyproject.toml or setup.py and create a environment.yaml file.

Key Points

  • Standard names (e.g., - numpy) are assumed to be the same for Conda and Pip.
  • Use conda: <package> and pip: <package> to specify different names across platforms.
  • Use pip: to specify packages that are only available through Pip.
  • Use conda: to specify packages that are only available through Conda.

Using the CLI conda-join will combine these dependencies into a single environment.yaml file, structured as follows:

name: some_name
channels:
  - conda-forge
dependencies:
  - numpy
  - python-graphviz
  - mumps
  pip:
    - slurm-usage

Platform Selectors

This tool supports a range of platform selectors that allow for specific handling of dependencies based on the user's operating system and architecture. This feature is particularly useful for managing conditional dependencies in diverse environments.

Supported Selectors

The following selectors are supported:

  • linux: For all Linux-based systems.
  • linux64: Specifically for 64-bit Linux systems.
  • aarch64: For Linux systems on ARM64 architectures.
  • ppc64le: For Linux on PowerPC 64-bit Little Endian architectures.
  • osx: For all macOS systems.
  • osx64: Specifically for 64-bit macOS systems.
  • arm64: For macOS systems on ARM64 architectures (Apple Silicon).
  • macos: An alternative to osx for macOS systems.
  • unix: A general selector for all UNIX-like systems (includes Linux and macOS).
  • win: For all Windows systems.
  • win64: Specifically for 64-bit Windows systems.

Usage

Selectors are used in requirements.yaml files to conditionally include dependencies based on the platform:

dependencies:
  - some-package  # [unix]
  - another-package  # [win]
  - special-package  # [osx64]
  - pip: cirq  # [macos]
    conda: cirq  # [linux]

In this example:

  • some-package is included only in UNIX-like environments (Linux and macOS).
  • another-package is specific to Windows.
  • special-package is included only for 64-bit macOS systems.
  • cirq is managed by pip on macOS and by conda on Linux. This demonstrates how you can specify different package managers for the same package based on the platform.

Implementation

The tool parses these selectors and filters dependencies according to the platform where it's being run. This is particularly useful for creating environment files that are portable across different platforms, ensuring that each environment has the appropriate dependencies installed.

:memo: Usage

With pyproject.toml or setup.py

To use conda-join in your project, you can configure it in pyproject.toml. This setup works alongside a requirements.yaml file located in the same directory. The behavior depends on your project's setup:

  • When using only pyproject.toml: The dependencies field in pyproject.toml will be automatically populated based on the contents of requirements.yaml.
  • When using setup.py: The install_requires field in setup.py will be automatically populated, reflecting the dependencies defined in requirements.yaml.

Here's an example pyproject.toml configuration:

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel", "conda-join"]

[project]
dynamic = ["dependencies"]

In this configuration, conda-join is included as a build requirement, allowing it to process the Python dependencies in the requirements.yaml file and update the project's dependencies accordingly.

:memo: As a CLI

Use conda-join to scan directories for requirements.yaml file(s) and combine them into an environment.yaml file. See example for more information or check the output of conda-join -h:

usage: conda-join [-h] [-d DIRECTORY] [-o OUTPUT] [-n NAME] [--depth DEPTH]
                  [--stdout] [-v]

Unified Conda and Pip requirements management.

options:
  -h, --help            show this help message and exit
  -d DIRECTORY, --directory DIRECTORY
                        Base directory to scan for requirements.yaml files, by
                        default `.`
  -o OUTPUT, --output OUTPUT
                        Output file for the conda environment, by default
                        `environment.yaml`
  -n NAME, --name NAME  Name of the conda environment, by default `myenv`
  --depth DEPTH         Depth to scan for requirements.yaml files, by default
                        1
  --stdout              Output to stdout instead of a file
  -v, --verbose         Print verbose output

Limitations

  • No Conflict Resolution: Doesn't resolve version conflicts between different requirements.yaml files.
  • Conda-Focused: Best suited for Conda environments.

Try conda-join today for a streamlined approach to managing your Conda environment dependencies across multiple projects! 🎉👏

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

conda-join-0.13.0.tar.gz (17.0 kB view details)

Uploaded Source

Built Distribution

conda_join-0.13.0-py3-none-any.whl (12.8 kB view details)

Uploaded Python 3

File details

Details for the file conda-join-0.13.0.tar.gz.

File metadata

  • Download URL: conda-join-0.13.0.tar.gz
  • Upload date:
  • Size: 17.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.0

File hashes

Hashes for conda-join-0.13.0.tar.gz
Algorithm Hash digest
SHA256 90d95539e4332834e75176c1e167ffe302f63f39de55fec554374f25a666b438
MD5 c06c75c332bbc02d61dd3089317110ce
BLAKE2b-256 c3944b332ba20840b384d85a44426d43a05454e4b5de41a6af64738a601d8fdd

See more details on using hashes here.

File details

Details for the file conda_join-0.13.0-py3-none-any.whl.

File metadata

  • Download URL: conda_join-0.13.0-py3-none-any.whl
  • Upload date:
  • Size: 12.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.0

File hashes

Hashes for conda_join-0.13.0-py3-none-any.whl
Algorithm Hash digest
SHA256 dc140a73d23d41b9a081a7a915bd79ef92383be7214e0aaed0d45c5cf6de750f
MD5 59e40c56428126296786b3d153590c28
BLAKE2b-256 e7c418ba1f8e289295f4b60eb60cde400c94171cc54a37b40723549ed0e8c64b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page