Abode: Friendly Python Packaging

Most experienced Python users know that Python packaging is rough. Abode is an attempt to make things nicer by extending Conda. It uses Conda under the hood, but makes things a bit easier and robust.

Conda is by far my preferred packaging solution. It works as an environment manager (better than virtualenv in my opinion) as well as a package manager. The default repository is focused on data science software. However, Conda also installs packages from PyPI with pip. Also, unlike pip, Conda installs non-Python libraries such as MKL and CUDA that often increase operation speeds by 10-100x.

An aside: I recommend installing Miniconda instead of the full Anaconda distribution. The vast majority of people won't need every single package in the Anaconda distribution. So save yourself some time, bandwidth, and storage. Install Miniconda, then create environments and install packages as needed.

The Main Attraction

The biggest issue with Conda is saving an environment's dependencies and sharing it across platforms. You can get the dependencies using conda env export > environment.yml. Say I have an environment where I have installed Flask, Numpy, and PyTorch. Even though those are the only packages I intentionally installed, Conda installs all the dependencies as well. Conda exports the environment to a YAML file that looks like:

name: flask
channels:
  - pytorch
  - defaults
dependencies:
  - blas=1.0=mkl
  - ca-certificates=2019.5.15=1
  - certifi=2019.6.16=py37_1
  - cffi=1.12.3=py37hb5b8e2f_0
  - intel-openmp=2019.4=233
  - libcxx=4.0.1=hcfea43d_1
  - libcxxabi=4.0.1=hcfea43d_1
  - libedit=3.1.20181209=hb402a30_0
  - libffi=3.2.1=h475c297_4
  - libgfortran=3.0.1=h93005f0_2
  - mkl=2019.4=233
  - mkl-service=2.0.2=py37h1de35cc_0
  - mkl_fft=1.0.14=py37h5e564d8_0
  - mkl_random=1.0.2=py37h27c97d8_0
  - ncurses=6.1=h0a44026_1
  - ninja=1.9.0=py37h04f5b5a_0
  - numpy=1.16.4=py37hacdab7b_0
  - numpy-base=1.16.4=py37h6575580_0
  - openssl=1.1.1c=h1de35cc_1
  - pip=19.1.1=py37_0
  - pycparser=2.19=py37_0
  - python=3.7.4=h359304d_1
  - pytorch=1.2.0=py3.7_0
  - readline=7.0=h1de35cc_5
  - setuptools=41.0.1=py37_0
  - six=1.12.0=py37_0
  - sqlite=3.29.0=ha441bb4_0
  - tk=8.6.8=ha441bb4_0
  - wheel=0.33.4=py37_0
  - xz=5.2.4=h1de35cc_4
  - zlib=1.2.11=h1de35cc_3
  - pip:
    - click==7.0
    - flask==1.1.1
    - itsdangerous==1.1.0
    - jinja2==2.10.1
    - markupsafe==1.1.1
    - werkzeug==0.15.5
prefix: /Users/mat/miniconda3/envs/flask

The issue here is that defining the versions potentially breaks the environment on platforms other than the original one. I created this environment on my MacBook. It is not guaranteed that all these dependencies are available for Linux or Windows, likely breaking the environment on these platforms.

Also, a lot of the time we're not concerned with the exact versions of our packages. Instead we are okay with the newest versions, or any version greater than some release with a specific feature. In these cases, locking to specific versions is overly strict.

You can create essentially the same environment with this file:

name: flask
channels:
  - pytorch
  - defaults
dependencies:
  - numpy
  - pip
  - python=3
  - pytorch
  - pip:
    - flask
prefix: /Users/mat/miniconda3/envs/flask

Conda will take this file and solve all the necessary dependencies on whatever platform you're using. Of course if there are specific versions you want (like python=3 here) you can define those.

Abode manages Conda environments by creating and editing minimal environment files like these. Hopefully this will allow users to take advantage of the great things Conda is doing, while making the environments portable.

Abode Dependencies

• Python 3.6+ because I like f-strings
• PyYAML
• Conda, as noted above, I suggest installing Miniconda instead of the full Anaconda distribution

Installation

Abode is available from PyPI:

pip install abode

Usage

Warning: This is very early. Use at your own risk.

So far this is what I have implemented.

Create an environment

To create a new environment with Python 3 installed:

abode create -n env_name python=3

Behind the scenes this creates an environment file and uses Conda to create an environment from the file.

Create from environment file

To create an environment from an environment file (a YAML file created by Abode or Conda):

abode create -f FILE

Enter an environment

I haven't figure out how to do this without using conda in the shell, so for now:

conda activate env_name

This is pretty high priority for me. It's awkward to switch between abode and conda commands.

Install packages

Installing packages is the same as Conda:

abode install numpy matplotlib

Behind the scenes, Abode is adding these dependencies to the environment file, then updating the environment from the file.

Install with pip

Use the --pip flag to install a package with pip.

abode install flask --pip

Install from a non-default channel

The channel option -c adds the channel to the environment file.

abode install pytorch -c pytorch

Update packages

This updates all the packages in the environment without locked versions.

abode update

Export the environment

abode export > environment.yml

This just copies the active environment's dependency file to the new file.

List packages in the current environment

To print out the list of all installed packages, equivalent to conda list:

abode list

If you want to see which packages have been installed with Abode:

abode list -f

or

abode list --file

List environments managed with Abode

abode env list

Contributions

Happy to work with you on this. Create an issue or a pull request. Say hi.