pyproject2conda 0.3.2

A script to convert a Python project declared on a pyproject.toml to a conda environment.

pyproject2conda

A script to convert pyproject.toml dependecies to environemnt.yaml files.

Overview

The main goal of pyproject2conda is to provide a means to keep all basic dependency information, for both pip based and conda based environments, in pyproject.toml. I often use a mix of pip and conda when developing packages, and in my everyday workflow. Some packages just aren't available on both. If you use poetry, I'd highly recommend poetry2conda.

Features

• Simple comment based syntax to add information to dependencies when creating environment.yaml

Status

This package is actively used by the author, but is still very much a work in progress. Please feel free to create a pull request for wanted features and suggestions!

Quick start

Use one of the following

pip install pyproject2conda

or

conda install -c wpk-nist pyproject2conda

Example usage

Basic usage

Consider the toml file test-pyproject.toml.

[project]
name = "hello"
requires-python = ">=3.8,<3.11"
dependencies = [
    "athing", # p2c: -p # a comment
    "bthing", # p2c: -s "bthing-conda"
    "cthing; python_version < '3.10'", # p2c: -c conda-forge
]
# ...

Note the comment lines # p2c:.... These are special tokens that pyproject2conda will analyze. The basic options are:

usage: -c [-h] [-c CHANNEL] [-p] [-s] [package ...]

Parser searches for comments '# p2c: [OPTIONS]

positional arguments:
  package

options:
  -h, --help            show this help message and exit
  -c CHANNEL, --channel CHANNEL
                        Channel to add to the pyproject requirement
  -p, --pip             If specified, install dependency on pyproject
                        dependency (on this line) with pip
  -s, --skip            If specified skip pyproject dependency on this line

So, if we run the following, we get:

$ pyproject2conda yaml -f tests/test-pyproject.toml
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Note that other comments can be mixed in.

By default, the python version is not included in the resulting conda output. To include the specification from pyproject.toml, use -p/--python option:

$ pyproject2conda yaml -f tests/test-pyproject.toml --python-include
channels:
  - conda-forge
dependencies:
  - python>=3.8,<3.11
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

To specify a specific value of python in the output, pass a value with:

$ pyproject2conda yaml -f tests/test-pyproject.toml --python-include python=3.9
channels:
  - conda-forge
dependencies:
  - python=3.9
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Note that this is for including python in the resulting environment file.

You can also constrain packages by the python version using the standard pyproject.toml syntax "...; python_version < 'some-version-number'". For is parsed for for both the pip packages and conda packages:

$ pyproject2conda yaml -f tests/test-pyproject.toml --python-version 3.10
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - pip
  - pip:
      - athing

Installing extras

Given the extra dependency:

# ...
[project.optional-dependencies]
test = [
    "pandas",
    "pytest", # p2c: -c conda-forge

]
dev-extras = [
    # p2c: -s "additional-thing; python_version < '3.9'" # this is an additional conda package
    ## p2c: -s "another-thing" # this will be skipped because of ## before p2c.
    "matplotlib", # p2c: -s conda-matplotlib

]
dev = ["hello[test]", "hello[dev-extras]"]
# ...

and running the the following gives:

$ pyproject2conda yaml -f tests/test-pyproject.toml -e test
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pandas
  - conda-forge::pytest
  - pip
  - pip:
      - athing

pyproject2conda also works with self referenced dependencies:

$ pyproject2conda yaml -f tests/test-pyproject.toml -e dev
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pandas
  - conda-forge::pytest
  - additional-thing
  - conda-matplotlib
  - pip
  - pip:
      - athing

This also shows that p2c comments without dependencies are also parsed. To comment out such lines, make sure p2c is preceded by ##.

Usage within python

pyproject2conda can also be used within python:

>>> from pyproject2conda import PyProject2Conda
>>> p = PyProject2Conda.from_path("./tests/test-pyproject.toml")

# Basic environment
>>> print(p.to_conda_yaml(python_include="get").strip())
channels:
  - conda-forge
dependencies:
  - python>=3.8,<3.11
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

# Environment with extras
>>> print(p.to_conda_yaml(extras="test").strip())
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pandas
  - conda-forge::pytest
  - pip
  - pip:
      - athing

Configuration

pyproject2conda can be configured with a [tool.pyproject2conda] section in pyproject.toml. To specify conda channels use:

# ...
[tool.pyproject2conda]
channels = ['conda-forge']
# ...

Note that specifying channels at the comand line overrides tool.pyproject2conda.channels.

You can also specify environments without the base dependencies (those under project.dependencies) by passing the --no-base flag. This is useful for defining environments for build, etc, that do not require the package be installed. For example:

!-- markdownlint-disable-next-line MD013 -->

# ...
dist-pypi = [ # this is intended to be parsed with --no-base option
    "setuptools",
    "build", # p2c: -p
]
# ...

These can be accessed using either of the following:

$ pyproject2conda yaml -f tests/test-pyproject.toml -e dist-pypi --no-base
channels:
  - conda-forge
dependencies:
  - setuptools
  - pip
  - pip:
      - build

or

>>> from pyproject2conda import PyProject2Conda
>>> p = PyProject2Conda.from_path("./tests/test-pyproject.toml")

# Basic environment
>>> print(p.to_conda_yaml(extras='dist-pypi', include_base_dependencies=False).strip())
channels:
  - conda-forge
dependencies:
  - setuptools
  - pip
  - pip:
      - build

CLI options

$ pyproject2conda --help
Usage: pyproject2conda [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  conda-requirements  Create requirement files for conda and pip.
  json                Create json representation.
  list                List available extras
  requirements        Create requirements.txt for pip depedencies.
  yaml                Create yaml file from dependencies and...

$ pyproject2conda list --help
Usage: pyproject2conda list [OPTIONS]

  List available extras

Options:
  -f, --file PATH  input pyproject.toml file
  -v, --verbose
  --help           Show this message and exit.

$ pyproject2conda yaml --help
Usage: pyproject2conda yaml [OPTIONS]

  Create yaml file from dependencies and optional-dependencies.

Options:
  -e, --extra TEXT       Extra depenedencies. Can specify multiple times for
                         multiple extras.
  -c, --channel TEXT     conda channel.  Can specify. Overrides
                         [tool.pyproject2conda.channels]
  -f, --file PATH        input pyproject.toml file
  -n, --name TEXT        Name of conda env
  -o, --output PATH      File to output results
  --python-include TEXT  If flag passed without options, include python spec
                         from pyproject.toml in yaml output.  If value passed,
                         use this value (exactly) in the output. So, for
                         example, pass `--python-include "python=3.8"`
  --python-version TEXT  Python version to check `python_verion <=>
                         {python_version}` lines against.  That is, this
                         version is used to limit packages in resulting
                         output. For example, if have a line like
                         `a-package; python_version < '3.9'`, Using `--python-
                         version 3.10` will not include `a-package`, while
                         `--python-version 3.8` will include `a-package`.
  --no-base              Default is to include base (project.dependencies)
                         with extras. However, passing `--no-base` will
                         exclude base dependencies. This is useful to define
                         environments that should exclude base dependencies
                         (like build, etc) in pyproject.toml.
  --help                 Show this message and exit.

$ pyproject2conda requirements --help
Usage: pyproject2conda requirements [OPTIONS]

  Create requirements.txt for pip depedencies.

Options:
  -e, --extra TEXT   Extra depenedencies. Can specify multiple times for
                     multiple extras.
  -f, --file PATH    input pyproject.toml file
  -o, --output PATH  File to output results
  --no-base          Default is to include base (project.dependencies) with
                     extras. However, passing `--no-base` will exclude base
                     dependencies. This is useful to define environments that
                     should exclude base dependencies (like build, etc) in
                     pyproject.toml.
  --help             Show this message and exit.

$ pyproject2conda conda-requirements --help
Usage: pyproject2conda conda-requirements [OPTIONS] [PATH_CONDA] [PATH_PIP]

  Create requirement files for conda and pip.

  These can be install with, for example,

  conda install --file {path_conda} pip install -r {path_pip}

Options:
  -e, --extra TEXT       Extra depenedencies. Can specify multiple times for
                         multiple extras.
  --python-include TEXT  If flag passed without options, include python spec
                         from pyproject.toml in yaml output.  If value passed,
                         use this value (exactly) in the output. So, for
                         example, pass `--python-include "python=3.8"`
  --python-version TEXT  Python version to check `python_verion <=>
                         {python_version}` lines against.  That is, this
                         version is used to limit packages in resulting
                         output. For example, if have a line like
                         `a-package; python_version < '3.9'`, Using `--python-
                         version 3.10` will not include `a-package`, while
                         `--python-version 3.8` will include `a-package`.
  -c, --channel TEXT     conda channel.  Can specify. Overrides
                         [tool.pyproject2conda.channels]
  -f, --file PATH        input pyproject.toml file
  --no-base              Default is to include base (project.dependencies)
                         with extras. However, passing `--no-base` will
                         exclude base dependencies. This is useful to define
                         environments that should exclude base dependencies
                         (like build, etc) in pyproject.toml.
  --prefix TEXT          set conda-output=prefix + 'conda.txt', pip-
                         output=prefix + 'pip.txt'
  --prepend-channel
  --help                 Show this message and exit.

$ pyproject2conda json --help
Usage: pyproject2conda json [OPTIONS]

  Create json representation.

  Keys are: "dependencies": conda dependencies. "pip": pip dependencies.
  "channels": conda channels.

Options:
  -e, --extra TEXT       Extra depenedencies. Can specify multiple times for
                         multiple extras.
  --python-include TEXT  If flag passed without options, include python spec
                         from pyproject.toml in yaml output.  If value passed,
                         use this value (exactly) in the output. So, for
                         example, pass `--python-include "python=3.8"`
  --python-version TEXT  Python version to check `python_verion <=>
                         {python_version}` lines against.  That is, this
                         version is used to limit packages in resulting
                         output. For example, if have a line like
                         `a-package; python_version < '3.9'`, Using `--python-
                         version 3.10` will not include `a-package`, while
                         `--python-version 3.8` will include `a-package`.
  -c, --channel TEXT     conda channel.  Can specify. Overrides
                         [tool.pyproject2conda.channels]
  -f, --file PATH        input pyproject.toml file
  -o, --output PATH      File to output results
  --no-base              Default is to include base (project.dependencies)
                         with extras. However, passing `--no-base` will
                         exclude base dependencies. This is useful to define
                         environments that should exclude base dependencies
                         (like build, etc) in pyproject.toml.
  --help                 Show this message and exit.

License

This is free software. See LICENSE.

Related work

TBD

Contact

The author can be reached at [e-mail].

Credits

This package was created with Cookiecutter and the wpk-nist-gov/cookiecutter-pypackage Project template forked from audreyr/cookiecutter-pypackage.

Changelog

Changelog for pyproject2conda

Unreleased

See the fragment files in changelog.d

This software was developed by employees of the National Institute of Standards and Technology (NIST), an agency of the Federal Government. Pursuant to title 17 United States Code Section 105, works of NIST employees are not subject to copyright protection in the United States and are considered to be in the public domain. Permission to freely use, copy, modify, and distribute this software and its documentation without fee is hereby granted, provided that this notice and disclaimer of warranty appears in all copies.

THE SOFTWARE IS PROVIDED 'AS IS' WITHOUT ANY WARRANTY OF ANY KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND FREEDOM FROM INFRINGEMENT, AND ANY WARRANTY THAT THE DOCUMENTATION WILL CONFORM TO THE SOFTWARE, OR ANY WARRANTY THAT THE SOFTWARE WILL BE ERROR FREE. IN NO EVENT SHALL NIST BE LIABLE FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO, DIRECT, INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF, RESULTING FROM, OR IN ANY WAY CONNECTED WITH THIS SOFTWARE, WHETHER OR NOT BASED UPON WARRANTY, CONTRACT, TORT, OR OTHERWISE, WHETHER OR NOT INJURY WAS SUSTAINED BY PERSONS OR PROPERTY OR OTHERWISE, AND WHETHER OR NOT LOSS WAS SUSTAINED FROM, OR AROSE OUT OF THE RESULTS OF, OR USE OF, THE SOFTWARE OR SERVICES PROVIDED HEREUNDER.

Distributions of NIST software should also include copyright and licensing statements of any third-party software that are legally bundled with the code in compliance with the conditions of those licenses.