pyproject2conda 0.4.0

A script to convert a Python project declared on a pyproject.toml to a conda environment.

pyproject2conda

A script to convert pyproject.toml dependecies to environemnt.yaml files.

Overview

The main goal of pyproject2conda is to provide a means to keep all basic dependency information, for both pip based and conda based environments, in pyproject.toml. I often use a mix of pip and conda when developing packages, and in my everyday workflow. Some packages just aren't available on both. If you use poetry, I'd highly recommend poetry2conda.

Features

• Simple comment based syntax to add information to dependencies when creating environment.yaml

Status

This package is actively used by the author, but is still very much a work in progress. Please feel free to create a pull request for wanted features and suggestions!

Quick start

Use one of the following

pip install pyproject2conda

or

conda install -c wpk-nist pyproject2conda

Example usage

Basic usage

Consider the toml file test-pyproject.toml.

[project]
name = "hello"
requires-python = ">=3.8,<3.11"
dependencies = [
    "athing", # p2c: -p # a comment
    "bthing", # p2c: -s "bthing-conda"
    "cthing; python_version < '3.10'", # p2c: -c conda-forge

# ...

Note the comment lines # p2c:.... These are special tokens that pyproject2conda will analyze. The basic options are:

usage: -c [-h] [-c CHANNEL] [-p] [-s] [package ...]

Parser searches for comments '# p2c: [OPTIONS]

positional arguments:
  package

options:
  -h, --help            show this help message and exit
  -c CHANNEL, --channel CHANNEL
                        Channel to add to the pyproject requirement
  -p, --pip             If specified, install dependency on pyproject
                        dependency (on this line) with pip
  -s, --skip            If specified skip pyproject dependency on this line

So, if we run the following, we get:

$ pyproject2conda yaml -f tests/test-pyproject.toml
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Note that other comments can be mixed in.

By default, the python version is not included in the resulting conda output. To include the specification from pyproject.toml, use -p/--python option:

$ pyproject2conda yaml -f tests/test-pyproject.toml --python-include
channels:
  - conda-forge
dependencies:
  - python>=3.8,<3.11
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

To specify a specific value of python in the output, pass a value with:

$ pyproject2conda yaml -f tests/test-pyproject.toml --python-include python=3.9
channels:
  - conda-forge
dependencies:
  - python=3.9
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Note that this is for including python in the resulting environment file.

You can also constrain packages by the python version using the standard pyproject.toml syntax "...; python_version < 'some-version-number'". For is parsed for for both the pip packages and conda packages:

$ pyproject2conda yaml -f tests/test-pyproject.toml --python-version 3.10
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - pip
  - pip:
      - athing

Installing extras

Given the extra dependency:

# ...

[project.optional-dependencies]
test = [
    "pandas",
    "pytest", # p2c: -c conda-forge

]
dev-extras = [
    # p2c: -s "additional-thing; python_version < '3.9'" # this is an additional conda package
    ## p2c: -s "another-thing" # this will be skipped because of ## before p2c.
    "matplotlib", # p2c: -s conda-matplotlib

]
# ...

and running the the following gives:

$ pyproject2conda yaml -f tests/test-pyproject.toml -e test
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pandas
  - conda-forge::pytest
  - pip
  - pip:
      - athing

pyproject2conda also works with self referenced dependencies:

$ pyproject2conda yaml -f tests/test-pyproject.toml -e dev
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pandas
  - conda-forge::pytest
  - additional-thing
  - conda-matplotlib
  - pip
  - pip:
      - athing

This also shows that p2c comments without dependencies are also parsed. To comment out such lines, make sure p2c is preceded by ##.

Header in output

By default, pyproject2conda includes a header in most output files to note that the files are auto generated. No header is included by default when writing to standard output. To override this behavior, pass --header/--noheader:

$ pyproject2conda yaml -f tests/test-pyproject.toml --header
#
# This file is autogenerated by pyrpoject2conda.
# You should not manually edit this file.
# Instead edit the corresponding pyproject.toml file.
#
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Usage within python

pyproject2conda can also be used within python:

>>> from pyproject2conda import PyProject2Conda
>>> p = PyProject2Conda.from_path("./tests/test-pyproject.toml")

# Basic environment
>>> print(p.to_conda_yaml(python_include="get").strip())
channels:
  - conda-forge
dependencies:
  - python>=3.8,<3.11
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

# Environment with extras
>>> print(p.to_conda_yaml(extras="test").strip())
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pandas
  - conda-forge::pytest
  - pip
  - pip:
      - athing

Configuration

pyproject2conda can be configured with a [tool.pyproject2conda] section in pyproject.toml. To specify conda channels use:

# ...
]

[tool.pyproject2conda]
channels = ['conda-forge']
# ...

Note that specifying channels at the comand line overrides tool.pyproject2conda.channels.

You can also specify environments without the base dependencies (those under project.dependencies) by passing the --no-base flag. This is useful for defining environments for build, etc, that do not require the package be installed. For example:

!-- markdownlint-disable-next-line MD013 -->

# ...
dev = ["hello[test]", "hello[dev-extras]"]
dist-pypi = [
    # this is intended to be parsed with --no-base option
    "setuptools",
# ...

These can be accessed using either of the following:

$ pyproject2conda yaml -f tests/test-pyproject.toml -e dist-pypi --no-base
channels:
  - conda-forge
dependencies:
  - setuptools
  - pip
  - pip:
      - build

or

>>> from pyproject2conda import PyProject2Conda
>>> p = PyProject2Conda.from_path("./tests/test-pyproject.toml")

# Basic environment
>>> print(p.to_conda_yaml(extras='dist-pypi', include_base_dependencies=False).strip())
channels:
  - conda-forge
dependencies:
  - setuptools
  - pip
  - pip:
      - build

CLI options

$ pyproject2conda --help
Usage: pyproject2conda [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  conda-requirements  Create requirement files for conda and pip.
  json                Create json representation.
  list                List available extras
  requirements        Create requirements.txt for pip depedencies.
  yaml                Create yaml file from dependencies and...

$ pyproject2conda list --help
Usage: pyproject2conda list [OPTIONS]

  List available extras

Options:
  -f, --file PATH  input pyproject.toml file
  -v, --verbose
  --help           Show this message and exit.

$ pyproject2conda yaml --help
Usage: pyproject2conda yaml [OPTIONS]

  Create yaml file from dependencies and optional-dependencies.

Options:
  -e, --extra TEXT        Extra depenedencies. Can specify multiple times for
                          multiple extras.
  -c, --channel TEXT      conda channel.  Can specify. Overrides
                          [tool.pyproject2conda.channels]
  -f, --file PATH         input pyproject.toml file
  -n, --name TEXT         Name of conda env
  -o, --output PATH       File to output results
  --python-include TEXT   If flag passed without options, include python spec
                          from pyproject.toml in yaml output.  If value
                          passed, use this value (exactly) in the output. So,
                          for example, pass `--python-include "python=3.8"`
  --python-version TEXT   Python version to check `python_verion <=>
                          {python_version}` lines against.  That is, this
                          version is used to limit packages in resulting
                          output. For example, if have a line like
                          `a-package; python_version < '3.9'`, Using
                          `--python-version 3.10` will not include
                          `a-package`, while `--python-version 3.8` will
                          include `a-package`.
  --base / --no-base      Default is to include base (project.dependencies)
                          with extras. However, passing `--no-base` will
                          exclude base dependencies. This is useful to define
                          environments that should exclude base dependencies
                          (like build, etc) in pyproject.toml.
  --header / --no-header  If True (--header) include header line in output.
                          Default is to include the header for output to a
                          file, and not to include header when writing to
                          stdout.
  --help                  Show this message and exit.

$ pyproject2conda requirements --help
Usage: pyproject2conda requirements [OPTIONS]

  Create requirements.txt for pip depedencies.

Options:
  -e, --extra TEXT        Extra depenedencies. Can specify multiple times for
                          multiple extras.
  -f, --file PATH         input pyproject.toml file
  -o, --output PATH       File to output results
  --base / --no-base      Default is to include base (project.dependencies)
                          with extras. However, passing `--no-base` will
                          exclude base dependencies. This is useful to define
                          environments that should exclude base dependencies
                          (like build, etc) in pyproject.toml.
  --header / --no-header  If True (--header) include header line in output.
                          Default is to include the header for output to a
                          file, and not to include header when writing to
                          stdout.
  --help                  Show this message and exit.

$ pyproject2conda conda-requirements --help
Usage: pyproject2conda conda-requirements [OPTIONS] [PATH_CONDA] [PATH_PIP]

  Create requirement files for conda and pip.

  These can be install with, for example,

  conda install --file {path_conda} pip install -r {path_pip}

Options:
  -e, --extra TEXT        Extra depenedencies. Can specify multiple times for
                          multiple extras.
  --python-include TEXT   If flag passed without options, include python spec
                          from pyproject.toml in yaml output.  If value
                          passed, use this value (exactly) in the output. So,
                          for example, pass `--python-include "python=3.8"`
  --python-version TEXT   Python version to check `python_verion <=>
                          {python_version}` lines against.  That is, this
                          version is used to limit packages in resulting
                          output. For example, if have a line like
                          `a-package; python_version < '3.9'`, Using
                          `--python-version 3.10` will not include
                          `a-package`, while `--python-version 3.8` will
                          include `a-package`.
  -c, --channel TEXT      conda channel.  Can specify. Overrides
                          [tool.pyproject2conda.channels]
  -f, --file PATH         input pyproject.toml file
  --base / --no-base      Default is to include base (project.dependencies)
                          with extras. However, passing `--no-base` will
                          exclude base dependencies. This is useful to define
                          environments that should exclude base dependencies
                          (like build, etc) in pyproject.toml.
  --header / --no-header  If True (--header) include header line in output.
                          Default is to include the header for output to a
                          file, and not to include header when writing to
                          stdout.
  --prefix TEXT           set conda-output=prefix + 'conda.txt', pip-
                          output=prefix + 'pip.txt'
  --prepend-channel
  --help                  Show this message and exit.

$ pyproject2conda json --help
Usage: pyproject2conda json [OPTIONS]

  Create json representation.

  Keys are: "dependencies": conda dependencies. "pip": pip dependencies.
  "channels": conda channels.

Options:
  -e, --extra TEXT       Extra depenedencies. Can specify multiple times for
                         multiple extras.
  --python-include TEXT  If flag passed without options, include python spec
                         from pyproject.toml in yaml output.  If value passed,
                         use this value (exactly) in the output. So, for
                         example, pass `--python-include "python=3.8"`
  --python-version TEXT  Python version to check `python_verion <=>
                         {python_version}` lines against.  That is, this
                         version is used to limit packages in resulting
                         output. For example, if have a line like
                         `a-package; python_version < '3.9'`, Using `--python-
                         version 3.10` will not include `a-package`, while
                         `--python-version 3.8` will include `a-package`.
  -c, --channel TEXT     conda channel.  Can specify. Overrides
                         [tool.pyproject2conda.channels]
  -f, --file PATH        input pyproject.toml file
  -o, --output PATH      File to output results
  --base / --no-base     Default is to include base (project.dependencies)
                         with extras. However, passing `--no-base` will
                         exclude base dependencies. This is useful to define
                         environments that should exclude base dependencies
                         (like build, etc) in pyproject.toml.
  --help                 Show this message and exit.

License

This is free software. See LICENSE.

Related work

TBD

Contact

The author can be reached at [e-mail].

Credits

This package was created with Cookiecutter and the wpk-nist-gov/cookiecutter-pypackage Project template forked from audreyr/cookiecutter-pypackage.

Changelog

Changelog for pyproject2conda

Unreleased

See the fragment files in changelog.d

This software was developed by employees of the National Institute of Standards and Technology (NIST), an agency of the Federal Government. Pursuant to title 17 United States Code Section 105, works of NIST employees are not subject to copyright protection in the United States and are considered to be in the public domain. Permission to freely use, copy, modify, and distribute this software and its documentation without fee is hereby granted, provided that this notice and disclaimer of warranty appears in all copies.

THE SOFTWARE IS PROVIDED 'AS IS' WITHOUT ANY WARRANTY OF ANY KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND FREEDOM FROM INFRINGEMENT, AND ANY WARRANTY THAT THE DOCUMENTATION WILL CONFORM TO THE SOFTWARE, OR ANY WARRANTY THAT THE SOFTWARE WILL BE ERROR FREE. IN NO EVENT SHALL NIST BE LIABLE FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO, DIRECT, INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF, RESULTING FROM, OR IN ANY WAY CONNECTED WITH THIS SOFTWARE, WHETHER OR NOT BASED UPON WARRANTY, CONTRACT, TORT, OR OTHERWISE, WHETHER OR NOT INJURY WAS SUSTAINED BY PERSONS OR PROPERTY OR OTHERWISE, AND WHETHER OR NOT LOSS WAS SUSTAINED FROM, OR AROSE OUT OF THE RESULTS OF, OR USE OF, THE SOFTWARE OR SERVICES PROVIDED HEREUNDER.

Distributions of NIST software should also include copyright and licensing statements of any third-party software that are legally bundled with the code in compliance with the conditions of those licenses.