Unified Conda and Pip requirements management.
Project description
:rocket: UniDep - Unified Conda and Pip Dependency Management :rocket:
unidep
simplifies Python project dependency management by enabling a single requirements.yaml
file to handle both Conda and Pip dependencies.
This approach allows for creating a unified Conda environment.yaml
, while also integrating with setup.py
or pyproject.toml
.
In addition, it can be used as a CLI to combine multiple requirements.yaml
files into a single environment.yaml
file.
Simplify your setup and maintain all your dependencies in one place with unidep
.
:rocket: Features
- 🔗 Unified Management: Single-file handling of Conda and Pip dependencies.
- ⚙️ Project Tool Integration: Easily works with
pyproject.toml
andsetup.py
, sorequirements.yaml
is used duringpip install
. - 🏢 Monorepo Support: Merge multiple
requirements.yaml
into one Conda environmentenvironment.yaml
using the CLI tool. - 🌍 Platform-Specific Support: Specify dependencies for different operating systems or architectures.
- 🛠️ Conflict Resolution: Simplifies complex dependency management by resolving version conflicts.
- 🔄
unidep install
CLI: Automates installation of Conda, Pip, and local package dependencies.
:books: Table of Contents
:package: Installation
To install unidep
, run the following command:
pip install -U unidep
Or just copy the script to your project:
wget https://raw.githubusercontent.com/basnijholt/unidep/main/unidep.py
:page_facing_up: requirements.yaml
structure
unidep
processes requirements.yaml
files with a specific format (similar but not the same as a Conda environment.yaml
file):
- name (Optional): For documentation, not used in the output.
- channels: List of sources for packages, such as
conda-forge
. - dependencies: Mix of Conda and Pip packages.
Example
Example of a requirements.yaml
file:
name: example_environment
channels:
- conda-forge
dependencies:
- numpy # same name on conda and pip
- conda: python-graphviz # When names differ between Conda and Pip
pip: graphviz
- pip: slurm-usage # pip-only
- conda: mumps # conda-only
# Use platform selectors; below only on linux64
- conda: cuda-toolkit # [linux64]
platforms: # (Optional) specify platforms that are supported (like conda-lock)
- linux-64
- osx-arm64
⚠️ unidep
can process this file in pyproject.toml
or setup.py
and create a environment.yaml
file.
For a more in-depth example, see the example
directory.
Key Points
- Standard names (e.g.,
- numpy
) are assumed to be the same for Conda and Pip. - Use
conda: <package>
andpip: <package>
to specify different names across platforms. - Use
pip:
to specify packages that are only available through Pip. - Use
conda:
to specify packages that are only available through Conda.
Using the CLI unidep
will combine these dependencies into a single conda installable environment.yaml
file.
Platform Selectors
This tool supports a range of platform selectors that allow for specific handling of dependencies based on the user's operating system and architecture. This feature is particularly useful for managing conditional dependencies in diverse environments.
Supported Selectors
The following selectors are supported:
linux
: For all Linux-based systems.linux64
: Specifically for 64-bit Linux systems.aarch64
: For Linux systems on ARM64 architectures.ppc64le
: For Linux on PowerPC 64-bit Little Endian architectures.osx
: For all macOS systems.osx64
: Specifically for 64-bit macOS systems.arm64
: For macOS systems on ARM64 architectures (Apple Silicon).macos
: An alternative toosx
for macOS systems.unix
: A general selector for all UNIX-like systems (includes Linux and macOS).win
: For all Windows systems.win64
: Specifically for 64-bit Windows systems.
Usage
Selectors are used in requirements.yaml
files to conditionally include dependencies based on the platform:
dependencies:
- some-package # [unix]
- another-package # [win]
- special-package # [osx64]
- pip: cirq # [macos]
conda: cirq # [linux]
In this example:
some-package
is included only in UNIX-like environments (Linux and macOS).another-package
is specific to Windows.special-package
is included only for 64-bit macOS systems.cirq
is managed bypip
on macOS and byconda
on Linux. This demonstrates how you can specify different package managers for the same package based on the platform.
Implementation
The tool parses these selectors and filters dependencies according to the platform where it's being run. This is particularly useful for creating environment files that are portable across different platforms, ensuring that each environment has the appropriate dependencies installed.
Conflict Resolution
unidep
features a conflict resolution mechanism to manage version conflicts and platform-specific dependencies in requirements.yaml
files. This functionality ensures optimal package version selection based on specified requirements.
How It Works
-
Version Pinning Priority:
unidep
gives priority to version-pinned packages when multiple versions of the same package are specified. For instance, if bothfoo
andfoo <1
are listed,foo <1
is selected due to its specific version pin. -
Minimal Scope Selection:
unidep
resolves platform-specific dependency conflicts by preferring the version with the most limited platform scope. For instance, givenfoo <1 # [linux64]
andfoo >1
, it installsfoo <1
exclusively on Linux-64 andfoo >1
on all other platforms. This approach ensures platform-specific requirements are precisely met. -
Resolving Intractable Conflicts: When conflicts are irreconcilable (e.g.,
foo >1
vs.foo <1
),unidep
issues a warning and defaults to the first encountered specification.
:memo: Usage
With pyproject.toml
or setup.py
To use unidep
in your project, you can configure it in pyproject.toml
. This setup works alongside a requirements.yaml
file located in the same directory. The behavior depends on your project's setup:
- When using only
pyproject.toml
: Thedependencies
field inpyproject.toml
will be automatically populated based on the contents ofrequirements.yaml
. - When using
setup.py
: Theinstall_requires
field insetup.py
will be automatically populated, reflecting the dependencies defined inrequirements.yaml
.
Here's an example pyproject.toml
configuration:
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "unidep"]
[project]
dynamic = ["dependencies"]
In this configuration, unidep
is included as a build requirement, allowing it to process the Python dependencies in the requirements.yaml
file and update the project's dependencies accordingly.
:memo: As a CLI
See example for more information or check the output of unidep -h
for the available sub commands:
usage: unidep [-h] {merge,pip,conda,install,conda-lock} ...
Unified Conda and Pip requirements management.
positional arguments:
{merge,pip,conda,install,conda-lock}
Subcommands
merge Merge requirements to conda installable
environment.yaml
pip Get the pip requirements for the current platform
only.
conda Get the conda requirements for the current platform
only.
install Install the dependencies of a single
`requirements.yaml` file in the currently activated
conda environment with conda, then install the
remaining dependencies with pip, and finally install
the current package with `pip install [-e] .`.
conda-lock Generate a global conda-lock file of a collection of
`requirements.yaml` files. Additionally, generate a
conda-lock file for each separate `requirements.yaml`
file based on the global lock file.
options:
-h, --help show this help message and exit
unidep merge
Use unidep merge
to scan directories for requirements.yaml
file(s) and combine them into an environment.yaml
file.
See unidep merge -h
for more information:
usage: unidep merge [-h] [-o OUTPUT] [-n NAME] [--stdout]
[--selector {sel,comment}] [-d DIRECTORY] [-v]
[--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}]
[--depth DEPTH]
options:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file for the conda environment, by default
`environment.yaml`
-n NAME, --name NAME Name of the conda environment, by default `myenv`
--stdout Output to stdout instead of a file
--selector {sel,comment}
The selector to use for the environment markers, if
`sel` then `- numpy # [linux]` becomes `sel(linux):
numpy`, if `comment` then it remains `- numpy #
[linux]`, by default `sel`
-d DIRECTORY, --directory DIRECTORY
Base directory to scan for requirements.yaml file(s),
by default `.`
-v, --verbose Print verbose output
--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}, -p {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}
The platform(s) to get the requirements for. Multiple
platforms can be specified. By default, the current
platform (`linux-64`) is used.
--depth DEPTH Maximum depth to scan for requirements.yaml files, by
default 1
unidep install
Use unidep install
on a requirements.yaml
file and install the dependencies on the current platform using conda, then install the remaining dependencies with pip, and finally install the current package with pip install [-e] .
.
See unidep install -h
for more information:
usage: unidep install [-h] [-v] [-e]
[--conda-executable {conda,mamba,micromamba}]
[--dry-run]
file
positional arguments:
file The requirements.yaml file to parse or folder that
contains that file, by default `.`
options:
-h, --help show this help message and exit
-v, --verbose Print verbose output
-e, --editable Install the project in editable mode
--conda-executable {conda,mamba,micromamba}
The conda executable to use
--dry-run Only print the commands that would be run
unidep pip
Use unidep pip
on a requirements.yaml
file and output the pip installable dependencies on the current platform (default).
See unidep pip -h
for more information:
usage: unidep pip [-h] [-f FILE] [-v]
[--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}]
[--separator SEPARATOR]
options:
-h, --help show this help message and exit
-f FILE, --file FILE The requirements.yaml file to parse or folder that
contains that file, by default `requirements.yaml`
-v, --verbose Print verbose output
--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}, -p {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}
The platform(s) to get the requirements for. Multiple
platforms can be specified. By default, the current
platform (`linux-64`) is used.
--separator SEPARATOR
The separator between the dependencies, by default ` `
unidep conda
Use unidep conda
on a requirements.yaml
file and output the conda installable dependencies on the current platform (default).
See unidep conda -h
for more information:
usage: unidep conda [-h] [-f FILE] [-v]
[--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}]
[--separator SEPARATOR]
options:
-h, --help show this help message and exit
-f FILE, --file FILE The requirements.yaml file to parse or folder that
contains that file, by default `requirements.yaml`
-v, --verbose Print verbose output
--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}, -p {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}
The platform(s) to get the requirements for. Multiple
platforms can be specified. By default, the current
platform (`linux-64`) is used.
--separator SEPARATOR
The separator between the dependencies, by default ` `
unidep conda-lock
Use unidep conda-lock
on one or multiple requirements.yaml
files and output the conda-lock file.
Optionally, when using a monorepo with multiple subpackages (with their own requirements.yaml
files), generate a lock file for each subpackage.
See unidep conda -h
for more information:
usage: unidep conda-lock [-h] [--only-global] [-d DIRECTORY] [-v]
[--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}]
[--depth DEPTH]
options:
-h, --help show this help message and exit
--only-global Only generate the global lock file
-d DIRECTORY, --directory DIRECTORY
Base directory to scan for requirements.yaml file(s),
by default `.`
-v, --verbose Print verbose output
--platform {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}, -p {linux-64,linux-aarch64,linux-ppc64le,osx-64,osx-arm64,win-64}
The platform(s) to get the requirements for. Multiple
platforms can be specified. By default, the current
platform (`linux-64`) is used.
--depth DEPTH Maximum depth to scan for requirements.yaml files, by
default 1
Limitations
- Conda-Focused: Best suited for Conda environments.
Try unidep
today for a streamlined approach to managing your Conda environment dependencies across multiple projects! 🎉👏
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.